Marc's Musings Reality from Marc's point of view!

I recently had an issue where the mail boxes on one of my Domino servers were not compacting at all during their normal window.

My logs showed

09/03/2009 04:00:01 AM  Router: Beginning mailbox file compaction of mail1.box
09/03/2009 04:00:01 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\mail1.box due to unique fields names table size
09/03/2009 04:00:02 AM  Router: Shutdown is in progress
09/03/2009 04:01:49 AM  Router: Unable to compact mailbox file mail1.box: Database is currently in use by you or another user
09/03/2009 04:01:49 AM  Router: Completed mailbox file compaction of mail1.box
09/03/2009 04:01:49 AM  Router: Beginning mailbox file compaction of mail2.box
09/03/2009 04:02:02 AM  Router: Completed mailbox file compaction of mail2.box
09/03/2009 04:02:02 AM  Router: Beginning mailbox file compaction of mail3.box
09/03/2009 04:02:02 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\mail3.box due to unique fields names table size
09/03/2009 04:03:49 AM  Router: Unable to compact mailbox file mail3.box: Database is currently in use by you or another user
09/03/2009 04:03:49 AM  Router: Completed mailbox file compaction of mail3.box
09/03/2009 04:03:50 AM  Router operation resumed.

Since I haven’t had the need to monitor for failure of mailbox compaction, I was not aware of this issue until I received complaints of Non Delivery Errors. Specifically “554 Error writing message to safe storage; message could not be stored to disk“.

I was unable to find anything useful either from Lotus’ technotes online, their forums or using Google in general so I thought I’d post something to help out others who travel this path.

I contacted Lotus and opened a PMR but they were clueless as well. I really wanted to know what the “unique fields names table size” error was all about but they really didn’t know and tossed me a technote pertaining to “Database has too many unique field names” which really didn’t apply. They also focused on database corruption caused by  anti-virus programs. I figure that, if this was the issue, this should  have come up in the past 5 years on some of the 30+ other servers I’m working with so I was pretty sure this was also not a cause.

Long and short, I still don’t have an effective means to generically monitor for mail box compact failure. I *could* set up a monitor for the “unique fields names table size” string but it seems that seems like overkill.

To resolve the issue is simple enough:

• shut down the server,
• rename all the mail boxes,
• restart the server (allowing new mail boxes to be created),
• copy over any not-dead messages to one of the newly created mail boxes and
• delete the old mail boxes

A side-effect of the above is that my name becomes the “from” address for all the copied messages.  Anybody have a simple solution for that? I haven’t actually researched it yet since it comes up so infrequently but it would be handy to know.

Posted under Lotus Domino, Work Portfolio

*** Update May 8, 2009 ***

Verizon finally did see fit to release the OS 4.5 update and, so far, it appears that it is addressing the issue for us. We set a date of May 15 (approximately 3 weeks after upgrading) as our “OK Date”. This means that folks in our pilot group would have gone 100% longer than before without a recurrence of their messages vanishing.

So, while the memory does still dip significantly in the devices I’ve seen, it appears that the updated OS is addressing the worst of the memory symptoms. Kudos! From the below you could see I was entirely not expecting this.

*** End Update May 8, 2009 ***

We have been having a problem for a while know that has surged to the forefront of our corporate consciousness at the beginning of spring break (April 4, 2009).

Folks on that day began reporting in droves (basically about 5% of my total blackberry user base) that some or all of their email had disappeared, many on that Saturday, but many others on the days after that.

In looking into the issue I discovered that our Blackberry help desk has been fielding similar issues on the order of (anecdotally) 5-10 such incidents per week.

The problem has been largely experienced by our 8830 users. Made more awful by the fact that only this past summer we upgraded most of our handhelds to the new 8830 model.

Facts that I have at my disposal:

• RIM claims that the memory manager in the 4.5 OS addresses this issue
• I have a BOLD (OS 4.6.x) that shows massive memory consumption that is corrected by removing and replacing the battery or by simply performing a reset (Alt – Right Shift – Del).
• People are complaining about this issue whether their devices communicate with a corporate BES or not (in the various forums that I have been reading)
• The vanishing messages are a normal function of the low memory manager on the Blackberry OS. The priority is to allow new incoming messages and (supposedly) the last resort is to delete the oldest or least accessed existing messages to make that room

This is what I am thinking:

• The problem is device related and is a memory handling issue
• The problem has not been resolved with OS 4.5 but rather folks now receive a warning at about 400K of memory so they can take action before messages begin to vanish
• The problem manifests in devices other than the 8830 but newer devices have so much memory that the issue is masked and wreaks less havoc. My bold can easily get to 6 Megabytes of file free out of 37 that are normally available. That’s 31 Megabytes of space that’s being wasted. That’s also more than double the maximum File Free that I used to have on my old 8830. Except for those apps supplied with the bold, my 3rd party apps are the same on this device as I had on my 8830.
• The sudden surge in reports stems more from the fact that so many people were going to be out of the office and needed their email accessible on their handheld devices than any sudden change in the environment (indeed, I had been out of the office for the 2 days prior to this issue and I’m the only one making changes to our BES). Unless Verizon is monkeying around with stuff on their network.

RIM is insistent that the issue is resolved in the 4.5 OS. Since Verizon is our primary service provider and they’ve been unable to certify the 4.5 OS for their devices for about a year now, this is not a reasonable solution for us.

Plus, it seems that there are a fair number of other Blackberry users out there who are encountering this issue regardless of their handheld version or OS.

What needs to happen:

• Short term: I need to create an application that would reset the device on a schedule. One of my team members has found one called “QuickPull” that *almost* fits the bill but needs some tweaks (see below) and can’t be distributed via BES.
• Long Term:  RIM needs to get off its behind and acknowledge this memory management issue and actually address it. Pull back on the whiz-bang (crappy storm for example) and make sure that the basics are rock solid. Folks needing to reset their devices on a daily basis is just silly.
• Long Term as well: What’s with this limited memory in these devices? We should be able to upgrade or add memory as we need to. The advice from RIM to “remove applications” and “minimize what you store” belongs back in the late-80’s along with a 640K memory limit and QEMM software!

Specs for reset application:

Since it is going to be necessary to create a workaround to allow folks to use their devices with the security of a corporate email system (i.e. messages not disappearing) I need to find or create a reset application that -

• Resets the device (reclaims all that wasted memory)
• Is distributable by BES (ALX and COD files)
• Operates on a schedule that is user configurable but has a default time set by policy. That time is localized (i.e. 3 am is 3 am your timezone).
• Prompts user to allow delay or skip of reset
• Recognizes phone in use and delays reset
• Can be set to only reset if device is locked
• Inexpensive

That’s what I have. So far I’ve gotten to the point where I can produce a “Hello World!” app on my handheld. Let’s see how long it takes to address this workaround.

We’ve been working unproductively with RIM – they keep wanting us to delete applications / themes etc.

We’ve been working with Verizon, they’re promising the 4.5 OS RSN (Real Soon Now).

Posted under Blackberry, Opinions, Tech Stuff, Work Portfolio

In Lotus Notes, a user has recurring meetings scheduled. When the reminder comes up, they acknowledge the dialog normally.

The next day, the recurring meeting will show up again along with any others that have accumulated.

In the forums and Lotus’ Technotes they indicate that you should get a handle to the user’s “CalendarSettings” profile (some refer to the “CalendarProfile” profile but you really need “CalendarSettings”) and reset the “lastAlarmDate” field to “Now”.

Using NotesPeek I could see that the user that I was dealing with had 3 CalendarSettings profiles. One for their current name, one for an early variation on their name with which they had begun their employ with us and one with the name of one of our Junior Administrators in it.

The correct profile (user’s current name) also matched the owner’s field name in the CalendarProfile but the “lastAlarmDate” field was already current. So this wasn’t going to help me. Figuring that the CalendarSettings document was likely damaged in some way, I created the following agent to blow away ALL of the calendarsettings profiles.

Get the user to exit Lotus Notes completely, then add the following agent to their mail file and run it. Don’t forget to remove it afterwards.

Then have them open Lotus Notes. They will see ALL their old reminders which they should just acknowledge. Also, some settings will need to be redone: they will go back to the week view of the calendar if they had changed it to some other view, etc.

But, compared with the annoyance of more and more accumulating meeting reminders, the reset is a minor issue.

Name:    Getcalendarsettings
Last Modification:    02/19/2009 04:05:54 PM
Comment:    [Not Assigned]
Shared Agent:    Yes
Type:    LotusScript
State:    Enabled
Trigger:    Manually From Actions Menu
Acts On:    None
LotusScript Code:
Option Public
Option Declare
Sub Initialize
    Dim sess As New notessession
    Dim coll As notesdocumentcollection
    Dim doc As NotesDocument
    Dim docdel As NotesDocument
    Set coll = sess.CurrentDatabase.GetProfileDocCollection(“Calendarsettings”)
    Set doc = coll.GetFirstDocument
    While Not doc Is Nothing
        Set docDel = doc   
        Set doc = coll.GetNextDocument(doc)
        If Not docDel Is Nothing Then
            Call docDel.Remove(True)
        End If
    Wend
End Sub

Posted under Lotus Domino, Work Portfolio

Teamstudio recently had a contest where they wanted you to showcase your applications so that others can see what folks are doing out there and so you could toot your own horn a bit.

My entry was probably a lot less snazzy than the ultimate winners (I haven’t seen them) as I tend to focus on non-public facing utilities and infrastructure improvements in my development work.

But I thought it would be worthwhile posting my entry here (now that they’ve let me know I *didn’t* win… *sniff*) so those of you who know me might get some idea what I spend part of my days doing.

I just call my application the “Admin Agent Repository”. It was originally created to house a single major utility and to serve as a common repository for the myriad agents that are useful for administrators on a day-to-day basis. But I have refined it to allow non-developer administrators access to its functionality.
The most visible application was to buffer us from the rather absolute and immediate nature of Domino’s user termination where the need to reinstate caused a lot of headaches.
A document is created for each terminated employee – this can be created by anybody granted the correct role so security or helpdesk folks can input terms directly – and, on the appointed date the user’s name is submitted to the deny access group.
The app also verifies whether or not the user has a Blackberry device and notifies our Blackberry group to remove their account.

Designed when background agents could not access databases on other servers, the application will then run an agent on the user’s home server and, if necessary, update their database ACL with the name of a person responsible for cleaning up the mail file and send a note, with a link to that responsible individual explaining their responsibilities.
When the purge date arrives (the default is 4 days, 30 if a responsible person is defined,  all controlled via profile), the actual signed adminp request is submitted and the process proceeds as per Domino normal.
Reinstate is often as simple as putting the termination document on hold and removing the user from the deny group. No ACL updates, no repopulating in groups.
The repository is replete with utilities that I have found useful over the years, most are profile driven:

For instance the below profile drives our dynamically created groups. Each morning a subset of our groups are destroyed and recreated based on person document information input by either the users or our identity management system. The groups are used both to control application access based on certain criteria (i.e. all reports to a person can have access to a calendar app used by that team) or for communications (i.e. message to be sent to all folks within a particular state or management structure).

This one inspects all mail files and ensures that they not only have a template, but that it’s one of the standard ones that we mandate:

One ad-hoc utility that is quite popular is one that runs, inspects all of the calendar documents in a mail file and ensures that the $Busyname matches the Owner name from the calendar profile. Very handy for botched renames and transfers of mail files from other domains.
I also have a couple of “Rebuild Busytime” agents for clustered and non-clustered servers that run weekly to keep things there tip-top as well.
These utilities save us literally hours each week, any job that’s wasting my time I invest the effort into automating it and add it to this growing application.

Posted under Lotus Domino, Tech Stuff, Utils / Tools, Work Portfolio

From the timestamps I see that this gem has been available since June of 2006 but I certainly had not heard about it.

Microsoft (M$), always there with useful products, also always manages to make things *just* different enough from standard that you can’t really seem to call them on it but they make it impossible to cleanly interface with their products as a competitor.

Case in point is Outlook and Exchange. If you create a message using M$’s email client and send it out the door using their Rich Text Format which seems to be of a proprietary nature, the message that arrives at the other end will be wrapped up in a file called “Winmail.Dat”.

For Lotus Notes this has always been a problem since the Domino server does not have the ability to unravel this proprietary formatting and so the recipient ends up with a useless (winmail.dat )attachment.

I know that there have been some fairly sophisticated attempts at providing solutions but the recipients of these messages are often business users with better things to do than try to master the arcane world of command line utilities with multistep processes.

Until now all we could really tell our users was to contact the sender and have them re-send the message but ensuring that they send with “Plain Text” instead of “RTF”. This is understandably awkward for a business user trying to look professional and it wasn’t doing my ego any wonders either.

The only really practical solution, besides Lotus actually dealing with the issue and including it in their SMTP router is this one presented to me by a co-worker (thanks Anton!). Created by Julian Robichaux (as far as I can tell as his name is all over the database) it is a simple mail-in database that you set up in your Domino environment. Your internal (and external if you wish to help out the rest of the world) Users then forward these nuisance messages to it and it replies back with the “unraveled” files.

So again the link is to Project WinmailExtractor. I had it set up and running in about 15 minutes including inspecting the code for any weirdness.

Posted under Lotus Domino, Utils / Tools, Work Portfolio

“If you haven’t got the time to do it right,
when will you find the time to do it over?”

This is actually the title of a book that I keep on my desk at work.

The book actually only proved moderately useful in a time management sense (at least for me), but the title is killer. Whenever I get involved in a discussion with someone who wants to do something “quick and dirty and we can circle around and do it properly later” I point to the book and ask the obvious question that is practically bursting from the cover.

I’ve had many arguments around this topic, won most, lost some (when an initiative is being driven by the wrong motivations/people) but this philosophy helps put some perspective on the effort that is about to be expended and helps to guide the thinking and planning process into more strategic pathways.

Posted under Opinions, Work Portfolio

Fast, Cheap or Good

Pick any two of the above.

I keep the above on my office wall at all times. I wish I could find a proper attribution for it, a quick Google search reveals that this is a fairly widely known designer’s creed but I’m not able to nail down who originally coined it.

But I love the sentiments represented.

Posted under Opinions, Work Portfolio