Comments on Creative code and cuisine: SnTT: tracking down space hogs in Notes

Kerr - We never did any performance monitoring or ...

2009-12-08T08:51:49.886-05:00

Kerr - We never did any performance monitoring or methodical statistics gathering so I can only make some assumptions and guesses. I think that the NotesDatabase.Open call forces a behind-the-scenes load of the index for the default view or folder. In the case of a mail database that is the ($Inbox) folder, and when the user has 10,000 messages in his Inbox that index is very large.

Calling NSFDBOpen instead only creates a handle but doesn't force the default view to load.

Charles, Those are certainly impressive numbers. ...

2009-12-08T07:57:57.184-05:00

Charles, Those are certainly impressive numbers. I'd love to know what was going on under the hood that would make that much difference. Obviously the LS is wasting a huge amount of time somewhere. Just one of those corner cases I suppose.

Wow, I leave for vacation and the discussion conti...

2009-11-23T08:38:46.210-05:00

Wow, I leave for vacation and the discussion continues! Thanks for keeping up, Craig. :-)

@Kerr - This was something I helped a friend come up with because opening every database was taking far too long. He was killing it after an hour. This technique completes in under 10 minutes.

@Grant - You are correct about this only working with databases up to 2GB. You can use Currency instead of Long to get precision up to 838 terabytes. I read about NSFDbSpaceUsageScaled but it seemed like a lot of indirection to me for no real gain.

Another tip: If we expect that the database size ...

2009-11-09T20:02:56.268-05:00

Another tip:

If we expect that the database size may grow beyond 4GB (quite likely when dealing with space hogs,) a better function to use would be NSFDbSpaceUsageScaled( ByVal hDB As Long, retAllocatedBytes As Long, retFreeBytes As Long, retGranularity as Long ).

The return results (retAllocatedBytes and retFreeBytes) are multiplied by the size of the chunks (retGranularity) to get the actual values.

But, I haven't tested any of this and could be way off base.

@Mike - this example is WIN32 only, though you can...

2009-11-08T17:47:28.374-05:00

@Mike - this example is WIN32 only, though you can easily change the Declares to support what ever platform you need. For example, AIX would be libnotes.a, LINUX/Solaris would be libnotes.so, etc.

Presumably, this approach ties you, hand and foot,...

2009-11-08T16:53:17.654-05:00

Presumably, this approach ties you, hand and foot, to Windows? (You have declared DLLs here).

Is it even possible to use the C API on other platforms?

@craig, Interesting. I thought the LS classes wh...

2009-11-06T10:33:00.611-05:00

@craig, Interesting. I thought the LS classes where pretty lightweight wrappers that delegated most of the work down to the capi as required. Certainly if they are doing a huge chunk of extra work then that would explain it. Still, I'd love to see soem numbers. I might have to try a run some benchmarks.

@kerr - The performance difference isn't really t...

2009-11-06T08:41:42.020-05:00

@kerr - The performance difference isn't really to do with the 'interpreted' aspect of LS. It has everything to do with the way the LS classes are implemented. When you open a document for example in LS, the LS class needs to load a lot more information in order to get the item list, parent database, etc members of the NotesDocument object setup. When you open a document via the API, none of this happens and you only get a handle to the document. If you need the rest of the info the LS class provides, you make other API calls to set that up. Because of this, in my experience, an API program can run circles around a LS version that does the exact same thing.

Charles, it surprises me that there would be much ...

2009-11-06T08:33:01.053-05:00

Charles, it surprises me that there would be much performance improvement. If I were to guess I'd have said that IO performance would swamp the LS interpretation overhead. Do you have any comparison figures on running the C api version versus a pure LS version?

There definitely could be a problem with accuracy,...

2009-11-05T11:48:05.123-05:00

There definitely could be a problem with accuracy, do doubt. There are too many shops that have turned off the catalog for one reason or another and end up missing out on an extremely valuable resource. The other common misunderstanding is surrounding the 'Do not list in catalog' database property. These databases are still in the catalog, the documents just get excluded from the view via the selection formula. A script like yours would also need to take that into account and use 'AllDocuments' or something similar. A FT search could also be used and just report dbs over a threashold. Heck you could even just create the view in the catalog itself and not worry about the agent... but I'm getting carried away.
Just to backtrack a minute, any script should double check the last updated date for the database and the individual documents and post warnings when they are out of date, or just a statement on the document 'Valid as of this date'.

All that being said, I still stand by my original comment that this is a good example of using the API to do something that otherwise would take longer than it needs to. The more tools you have in your tool box the better off you are.

I completely forgot that was in the catalog becaus...

2009-11-05T11:30:53.445-05:00

I completely forgot that was in the catalog because I've enver had any luck getting useful information out of it. The numbers were never accurate when I looked at them. Maybe it has improved over the years.

The catalog will also have the information about d...

2009-11-05T10:38:27.439-05:00

The catalog will also have the information about database size (DbSize), free space(DbPercentUsed), number of documents (DbNumDocuments), etc. Seems to me that the cost of opening the catalog and walking the view reading the summary info would be a lot faster than opening every database individually. NSFDbOpen will still take a hit, performance wise, if it encountered a large database, not not as severe as the LS version.

Oh, I just remembered the Catalog just has documen...

2009-11-05T09:30:56.809-05:00

Oh, I just remembered the Catalog just has documents describing the databases. You would still have to open them, which is what I was trying to avoid. NotesDBDirectory will give you a not-full-populated database object already.

By using the NotesDatabase from NotesDBDirectory in conjunction with the database handle from NSFDbOpen you can sort of dance around actually opening the database, which is why this runs so quickly.

Craig, you're right. I have been working on this p...

2009-11-05T08:42:52.798-05:00

Craig, you're right. I have been working on this post for a while and didn't notice I didn't post the final version of the code. I'll update that shortly.

On this server the catalog is gigantic. Opening that was a significant hit itself and a NotesDBDirectory was a much faster option.

Hi Charles, This is a great example of using the A...

2009-11-05T07:06:18.067-05:00

Hi Charles, This is a great example of using the API to make a simple task run even faster! However, you need one more API function: NSFDbClose(). Without this you are leaking a database handle and on a large server will eventually start throwing 'out of handles' type errors.

Also, wouldn't it be easier to just loop through the documents in the catalog?