Sametime High Availability problem using Round Robin DNS where at least one server is down for an extended period of time

There is an issue where people will begin having issues accessing Sametime if one server in a Sametime HA environment is down beyond a certain length of time when the Server Community DNS is set up as round robin.

Essentially either the clients or the servers “forget” about the downed server. You are OK for up to some number of days where they recognize that the server is a ST server but no longer available. After that some cache is cleared and this problem arises.

This is why normal HA works just fine (when a server goes down for just a few minutes or hours) the servers and / or the clients are able to ignore the proffered IP address since they are aware that the server is not longer responding.

If you are taking a server out of service for longer than a week (or the period within which you will be recycling all of the other servers in the community), remove the downed server from the round robin DNS until you are ready to put it back into service.

Save Window State in Lotus Notes 8.5.x

In Lotus Notes 7.x the ability to “Save Window State” (File > Save Window State) was introduced to allow users to have certain databases available to them on startup.

This ability is available in Lotus Notes 8.5.x but is set up a little differently. Now you must bookmark the databases you wish to appear automatically at startup in a special “Startup” folder.

To do this, right click on the open tab and select “Create Bookmark”

Then select the “Startup” Bookmark folder located under “More Bookmarks”

If you want ONLY the bookmarks in the startup folder to show up when the Lotus Notes Client starts, then go to Files – Preferences – Windows and Themes and DESELECT the “On restart, reopen any tabs that were open when I closed the client”.

If you leave this checked, all tabs that were open will show up on restart PLUS the items in your startup folder. So, even if you had closed any of the tabs represented in the startup folder they will be available when the client starts as expected.

Create a local replica of your mail file using LotusScript

I recently had the need to create local mail file replicas for a bunch of folks. This is something that *had* to be in place for a certain time but we wanted people to be able to create the replicas at a time of their choosing up until the drop dead date.

So I have two versions of this code. The first, which I’m not posting here because it’s a part of a much larger housekeeping module invoked from the PostOpen event of the user’s mail file, gives no choice to put it off. The end user will receive a dialog indicating what is about to happen and then the replica creation will kick off.

The second, which is sent as a button in an email explaining to the user why they would want to do this and that this allows them to run it at a time of their choosing, I include below. It has some friendlier dialogs for the end user.

When run, the code will pop up a message box explaining what is about to happen and give them an opportunity to stop before they get started.

Then I check to make sure they are the owner of the database (so an admin opening the message on their manager’s behalf doesn’t inadvertently create a local replica of the managers file on their own workstation).

Then – and this is probably overkill – I make sure that the user is explicitly in the ACL (not in there via a group) and is a Manager. In our environment this will always be the case so it doesn’t hurt to leave this check in.  You can probably delete it. This was put in out of a concern that folks would open this in a misconfigured mail-in database (where the owner in the preferences had been allowed to default to the users name instead of that of the mailbox). We manage access to our mail-in databases via groups so it would be unusual for the user to be explicitly listed and HIGHLY unusual for them to have manager access under any circumstances.

I then validate whether or not there already exists a file with the same name and along the same path as the one I wish to create. I’m lifting the path and file directly from the server-side database. I’ve actually come across a few cases where folks have local mail file replicas with completely different names and/or in surprisingly different folders than you would expect. Ah… the creativity of the end user!  For our purposes we need to have the file path and name consistent between the local and server copies.

Note that for the LotusScript Dir$ command, Notes will only return an empty string if the file doesn’t exist. If the containing folder doesn’t exist you will receive an error “76” . But if you create a handler that contains simply “Resume Next” Notes will gleefully pick up INSIDE the conditional statement. I was prepared to create a Boolean variable that would be set if either the file was not there or would be set in the error handler but I was able to take this little bizarre shortcut.

The tricky part for me was to get the new replica to show up automatically on the replication tab. For this I need to thank Notes 411 for a little bit of code that I really should have figured out for myself. This is what the UIWorkspace is doing in the code. Basically you just open both replicas in the UIWorkspace and they will be added to the replication tab. Easy as pie. I did not have any issues getting the replica appear on the workspace as the Notes 411 logic also purports to do.

Once done, I let the user know and exit the script.

This is designed to work on Lotus Notes 6.5.x and 7.

Sorry about the lack of formatting, WordPress seems to have taken to stripping it out.

Sub Click(Source As Button)
‘Check if there is a local mail file replica and create one if it doesn’t already exist.
‘Created by Marc Bourassa – July 16, 2010

Dim Sess As New NotesSession
Dim ws As New NotesUIWorkspace
Dim dbUI As NotesUIDatabase
Dim db As NotesDatabase
Dim ACL As NotesACL
Dim ACLEntry As NotesACLEntry
Dim dbLocal As NotesDatabase
Dim docCalProf As NotesDocument
Dim namName As NotesName
Dim sDataDir As String
Dim iRC As Integer
Dim sMsg As String
Dim sPrintText As String

On Error Goto ErrorGeneral
On Error 4005 Goto ErrorDBExists
On Error 76 Goto ErrorPathNotFound

Set db = sess.Currentdatabase

sMsg = “This button will create a local replica of your mail file (” & db.Title & “) if one does not already exist with the correct name.” & Chr$(13)& Chr$(13)&_
“Please press OK to continue or CANCEL to Abort”

iRC = Messagebox(sMsg,49,”Create Mail File Replica”)

If iRC = 1 Then
Set docCalProf = db.GetProfileDocument(“CalendarProfile”)
Set namName = New NotesName(docCalProf.Owner(0))
If namName.common = sess.Commonusername Then ‘User is the owner of this mail database
Set ACL = db.Acl
Set ACLEntry = ACL.Getentry(namName.Abbreviated )
If Not ACLEntry Is Nothing Then
If ACLEntry.Level=ACLLEVEL_MANAGER Then ‘User is explicitly in ACL AND is a manager
sDataDir = sess.getenvironmentstring(“Directory”,True)
If sDataDir = “” Then ‘something probably wrong with ini file. Don’t expect system to work correctly
Messagebox “There is a problem retrieveing your data directory information from the notes.ini file. Cannot proceed. This issue needs to be resolved by the helpdesk. Aborting Replica Creation.”,0,”Create Mail Database Replica”
Goto LeaveSub
End If
If Dir$( sDataDir & “\” & db.Filepath, 0 ) = “” Then ‘No replica by this name exists. Will fail to next line from error handling if path is bad (i.e. missing mail folder)
Call db.Createreplica(“”, db.Filepath)
‘Now add to the Replication tab
Call ws.opendatabase(db.Server,db.FilePath)
Set dbUI = ws.CurrentDatabase
Call ws.opendatabase(“”,db.FilePath)
Set dbUI=ws.CurrentDatabase
Messagebox “Local mail file replica has been successfully created (” & db.Title & “).”,0,”Create Mail Database Replica”
Messagebox “Replica already exists for your mail file (” & db.Title & “). Nothing to do here.”,0,”Create Mail Database Replica”
End If
End If
Messagebox “You are not a manager of this mail database (” & db.Title & “). Aborting Replica Creation.”,0,”Create Mail Database Replica”
End If
Messagebox “You are not the owner of this mail database (” & db.Title & “). Aborting Replica Creation.”,0,”Create Mail Database Replica”
End If
Messagebox “Replica Creation cancelled by you.”,0,”Create Mail Database Replica”
End If

Exit Sub

Resume LeaveSub

Resume Next

sPrintText = “Error: ” + Cstr(Err) + ” defn: ” + Error$ + “. ”
Messagebox sPrintText & Chr$(13) & Chr$(13) & “Could not complete this task you will need to create this replica manually. Please contact the help desk for assistance.” ,0,”Unexpected Error”
Resume LeaveSub

End Sub

On Access Scanners Killing App Performance

At work I’m wrestling with McAfee’s on access scanner absolutely crippling Lotus Notes latest client R8.5.1.  By killing the mcshield process you can realize a dramatic improvement in the startup time and overall performance of the Lotus client. A side benefit is that the omnipresent crashes I was facing have virtually disappeared.

Also, if you use the designer client you will see at least one order of magnitude increase in performance. I went from being amazed that such a terrible designer client had been released to being horrified that a relatively capable designer client had been so crippled by on access anti malware software as to render it useless.

Obviously our security folks are not going to be satisfied with simply disabling the scanner so the challenge is now to find some way to make these crappy security products ignore the right files so that the business applications can get on with being useful.

Today I was working at home with Quicken 2010 and I was miffed by how sluggishly it was behaving. I’ve got an i7-920 computer chip overclocked to 3 GHz, 12 Gigabytes of RAM and 64-Bit Windows 7 Professional pushing Quicken along. In theory it should be screaming.  Pulling up the task manager, what do I see but for pretty much every mouse click the MsMpEng executable (Microsoft Security Essentials Anti Malware scanner) is sucking up 10-30 percent of my CPU.

I’ve tried configuring the scanner to ignore my executable (C:\Program Files (x86)\Quicken\qw.exe), the Quicken QDF data file type (QDF extension) and even the entire dedicated drive on which the Quicken data files reside. But nothing has any effect. I’m pretty sure that those options are provided for the same reason as “Close” buttons on elevators exist: to give you something to do while things proceed at their normal pace.

Now killing the MsMpEng process absolutely fixes the issue. Quicken takes off like there is no tomorrow. Again, this is not an appropriate answer. You need to have rather sophisticated protections in place to work in today’s sophisticated cyber world. But it does point a finger directly at the resource hogging culprit.

So tell me, why does our protection software have to be so crude and bloated that this is even an issue? Does anybody have any suggestions or alternatives that you are happy with? Until this issue I thought I was VERY happy with Microsoft Security Essentials. But I’ve never needed to tweak it before. Now I want something effective. Symantec’s products have proven themselves to be monstrous resource hogs in the past. My McAfee experience at work is leading me away from that direction. Suggestions are very welcome and even more so if you’ve worked with Quicken in a Windows 7 64 bit environment!

“unique fields names table size” error in Lotus Domino

I recently had an issue where the mail boxes on one of my Domino servers were not compacting at all during their normal window.

My logs showed

09/03/2009 04:00:01 AM  Router: Beginning mailbox file compaction of
09/03/2009 04:00:01 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\ due to unique fields names table size
09/03/2009 04:00:02 AM  Router: Shutdown is in progress
09/03/2009 04:01:49 AM  Router: Unable to compact mailbox file Database is currently in use by you or another user
09/03/2009 04:01:49 AM  Router: Completed mailbox file compaction of
09/03/2009 04:01:49 AM  Router: Beginning mailbox file compaction of
09/03/2009 04:02:02 AM  Router: Completed mailbox file compaction of
09/03/2009 04:02:02 AM  Router: Beginning mailbox file compaction of
09/03/2009 04:02:02 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\ due to unique fields names table size
09/03/2009 04:03:49 AM  Router: Unable to compact mailbox file Database is currently in use by you or another user
09/03/2009 04:03:49 AM  Router: Completed mailbox file compaction of
09/03/2009 04:03:50 AM  Router operation resumed.

Since I haven’t had the need to monitor for failure of mailbox compaction, I was not aware of this issue until I received complaints of Non Delivery Errors. Specifically “554 Error writing message to safe storage; message could not be stored to disk“.

I was unable to find anything useful either from Lotus’ technotes online, their forums or using Google in general so I thought I’d post something to help out others who travel this path.

I contacted Lotus and opened a PMR but they were clueless as well. I really wanted to know what the “unique fields names table size” error was all about but they really didn’t know and tossed me a technote pertaining to “Database has too many unique field names” which really didn’t apply. They also focused on database corruption caused by  anti-virus programs. I figure that, if this was the issue, this should  have come up in the past 5 years on some of the 30+ other servers I’m working with so I was pretty sure this was also not a cause.

Long and short, I still don’t have an effective means to generically monitor for mail box compact failure. I *could* set up a monitor for the “unique fields names table size” string but it seems that seems like overkill.

To resolve the issue is simple enough:

  • shut down the server,
  • rename all the mail boxes,
  • restart the server (allowing new mail boxes to be created),
  • copy over any not-dead messages to one of the newly created mail boxes and
  • delete the old mail boxes

A side-effect of the above is that my name becomes the “from” address for all the copied messages.  Anybody have a simple solution for that? I haven’t actually researched it yet since it comes up so infrequently but it would be handy to know.

Memory leak issue for RIM Blackberry Devices – Especially 8830

*** Update May 8, 2009 ***

Verizon finally did see fit to release the OS 4.5 update and, so far, it appears that it is addressing the issue for us. We set a date of May 15 (approximately 3 weeks after upgrading) as our “OK Date”. This means that folks in our pilot group would have gone 100% longer than before without a recurrence of their messages vanishing.

So, while the memory does still dip significantly in the devices I’ve seen, it appears that the updated OS is addressing the worst of the memory symptoms. Kudos! From the below you could see I was entirely not expecting this.

*** End Update May 8, 2009 ***

We have been having a problem for a while know that has surged to the forefront of our corporate consciousness at the beginning of spring break (April 4, 2009).

Folks on that day began reporting in droves (basically about 5% of my total blackberry user base) that some or all of their email had disappeared, many on that Saturday, but many others on the days after that.

In looking into the issue I discovered that our Blackberry help desk has been fielding similar issues on the order of (anecdotally) 5-10 such incidents per week.

The problem has been largely experienced by our 8830 users. Made more awful by the fact that only this past summer we upgraded most of our handhelds to the new 8830 model.

Facts that I have at my disposal:

  • RIM claims that the memory manager in the 4.5 OS addresses this issue
  • I have a BOLD (OS 4.6.x) that shows massive memory consumption that is corrected by removing and replacing the battery or by simply performing a reset (Alt – Right Shift – Del).
  • People are complaining about this issue whether their devices communicate with a corporate BES or not (in the various forums that I have been reading)
  • The vanishing messages are a normal function of the low memory manager on the Blackberry OS. The priority is to allow new incoming messages and (supposedly) the last resort is to delete the oldest or least accessed existing messages to make that room

This is what I am thinking:

  • The problem is device related and is a memory handling issue
  • The problem has not been resolved with OS 4.5 but rather folks now receive a warning at about 400K of memory so they can take action before messages begin to vanish
  • The problem manifests in devices other than the 8830 but newer devices have so much memory that the issue is masked and wreaks less havoc. My bold can easily get to 6 Megabytes of file free out of 37 that are normally available. That’s 31 Megabytes of space that’s being wasted. That’s also more than double the maximum File Free that I used to have on my old 8830. Except for those apps supplied with the bold, my 3rd party apps are the same on this device as I had on my 8830.
  • The sudden surge in reports stems more from the fact that so many people were going to be out of the office and needed their email accessible on their handheld devices than any sudden change in the environment (indeed, I had been out of the office for the 2 days prior to this issue and I’m the only one making changes to our BES). Unless Verizon is monkeying around with stuff on their network.

RIM is insistent that the issue is resolved in the 4.5 OS. Since Verizon is our primary service provider and they’ve been unable to certify the 4.5 OS for their devices for about a year now, this is not a reasonable solution for us.

Plus, it seems that there are a fair number of other Blackberry users out there who are encountering this issue regardless of their handheld version or OS.

What needs to happen:

  • Short term: I need to create an application that would reset the device on a schedule. One of my team members has found one called “QuickPull” that *almost* fits the bill but needs some tweaks (see below) and can’t be distributed via BES.
  • Long Term:  RIM needs to get off its behind and acknowledge this memory management issue and actually address it. Pull back on the whiz-bang (crappy storm for example) and make sure that the basics are rock solid. Folks needing to reset their devices on a daily basis is just silly.
  • Long Term as well: What’s with this limited memory in these devices? We should be able to upgrade or add memory as we need to. The advice from RIM to “remove applications” and “minimize what you store” belongs back in the late-80’s along with a 640K memory limit and QEMM software!

Specs for reset application:

Since it is going to be necessary to create a workaround to allow folks to use their devices with the security of a corporate email system (i.e. messages not disappearing) I need to find or create a reset application that –

  • Resets the device (reclaims all that wasted memory)
  • Is distributable by BES (ALX and COD files)
  • Operates on a schedule that is user configurable but has a default time set by policy. That time is localized (i.e. 3 am is 3 am your timezone).
  • Prompts user to allow delay or skip of reset
  • Recognizes phone in use and delays reset
  • Can be set to only reset if device is locked
  • Inexpensive

That’s what I have. So far I’ve gotten to the point where I can produce a “Hello World!” app on my handheld. Let’s see how long it takes to address this workaround.

We’ve been working unproductively with RIM – they keep wanting us to delete applications / themes etc.

We’ve been working with Verizon, they’re promising the 4.5 OS RSN (Real Soon Now).

Old Reminders Keep Appearing for Recurring Meetings

In Lotus Notes, a user has recurring meetings scheduled. When the reminder comes up, they acknowledge the dialog normally.

The next day, the recurring meeting will show up again along with any others that have accumulated.

In the forums and Lotus’ Technotes they indicate that you should get a handle to the user’s “CalendarSettings” profile (some refer to the “CalendarProfile” profile but you really need “CalendarSettings”) and reset the “lastAlarmDate” field to “Now”.

Using NotesPeek I could see that the user that I was dealing with had 3 CalendarSettings profiles. One for their current name, one for an early variation on their name with which they had begun their employ with us and one with the name of one of our Junior Administrators in it.

The correct profile (user’s current name) also matched the owner’s field name in the CalendarProfile but the “lastAlarmDate” field was already current. So this wasn’t going to help me. Figuring that the CalendarSettings document was likely damaged in some way, I created the following agent to blow away ALL of the calendarsettings profiles.

Get the user to exit Lotus Notes completely, then add the following agent to their mail file and run it. Don’t forget to remove it afterwards.

Then have them open Lotus Notes. They will see ALL their old reminders which they should just acknowledge. Also, some settings will need to be redone: they will go back to the week view of the calendar if they had changed it to some other view, etc.

But, compared with the annoyance of more and more accumulating meeting reminders, the reset is a minor issue.

Name:    Getcalendarsettings
Last Modification:    02/19/2009 04:05:54 PM
Comment:    [Not Assigned]
Shared Agent:    Yes
Type:    LotusScript
State:    Enabled
Trigger:    Manually From Actions Menu
Acts On:    None
LotusScript Code:
Option Public
Option Declare
Sub Initialize
    Dim sess As New notessession
    Dim coll As notesdocumentcollection
    Dim doc As NotesDocument
    Dim docdel As NotesDocument
    Set coll = sess.CurrentDatabase.GetProfileDocCollection(“Calendarsettings”)
    Set doc = coll.GetFirstDocument
    While Not doc Is Nothing
        Set docDel = doc   
        Set doc = coll.GetNextDocument(doc)
        If Not docDel Is Nothing Then
            Call docDel.Remove(True)
        End If
End Sub

My Entry for Teamstudio’s Spotlight Award

Teamstudio recently had a contest where they wanted you to showcase your applications so that others can see what folks are doing out there and so you could toot your own horn a bit.

My entry was probably a lot less snazzy than the ultimate winners (I haven’t seen them) as I tend to focus on non-public facing utilities and infrastructure improvements in my development work.

But I thought it would be worthwhile posting my entry here (now that they’ve let me know I *didn’t* win… *sniff*) so those of you who know me might get some idea what I spend part of my days doing.

I just call my application the “Admin Agent Repository”. It was originally created to house a single major utility and to serve as a common repository for the myriad agents that are useful for administrators on a day-to-day basis. But I have refined it to allow non-developer administrators access to its functionality.
The most visible application was to buffer us from the rather absolute and immediate nature of Domino’s user termination where the need to reinstate caused a lot of headaches.
A document is created for each terminated employee – this can be created by anybody granted the correct role so security or helpdesk folks can input terms directly – and, on the appointed date the user’s name is submitted to the deny access group.
The app also verifies whether or not the user has a Blackberry device and notifies our Blackberry group to remove their account.

Designed when background agents could not access databases on other servers, the application will then run an agent on the user’s home server and, if necessary, update their database ACL with the name of a person responsible for cleaning up the mail file and send a note, with a link to that responsible individual explaining their responsibilities.
When the purge date arrives (the default is 4 days, 30 if a responsible person is defined,  all controlled via profile), the actual signed adminp request is submitted and the process proceeds as per Domino normal.
Reinstate is often as simple as putting the termination document on hold and removing the user from the deny group. No ACL updates, no repopulating in groups.
The repository is replete with utilities that I have found useful over the years, most are profile driven:

For instance the below profile drives our dynamically created groups. Each morning a subset of our groups are destroyed and recreated based on person document information input by either the users or our identity management system. The groups are used both to control application access based on certain criteria (i.e. all reports to a person can have access to a calendar app used by that team) or for communications (i.e. message to be sent to all folks within a particular state or management structure).

This one inspects all mail files and ensures that they not only have a template, but that it’s one of the standard ones that we mandate:

One ad-hoc utility that is quite popular is one that runs, inspects all of the calendar documents in a mail file and ensures that the $Busyname matches the Owner name from the calendar profile. Very handy for botched renames and transfers of mail files from other domains.
I also have a couple of “Rebuild Busytime” agents for clustered and non-clustered servers that run weekly to keep things there tip-top as well.
These utilities save us literally hours each week, any job that’s wasting my time I invest the effort into automating it and add it to this growing application.

Winmail.dat file issue in Lotus Notes workaround (finally)

Lotus Notes LogoFrom the timestamps I see that this gem has been available since June of 2006 but I certainly had not heard about it.

Microsoft (M$), always there with useful products, also always manages to make things *just* different enough from standard that you can’t really seem to call them on it but they make it impossible to cleanly interface with their products as a competitor.

Case in point is Outlook and Exchange. If you create a message using M$’s email client and send it out the door using their Rich Text Format which seems to be of a proprietary nature, the message that arrives at the other end will be wrapped up in a file called “Winmail.Dat”.

For Lotus Notes this has always been a problem since the Domino server does not have the ability to unravel this proprietary formatting and so the recipient ends up with a useless (winmail.dat )attachment.

I know that there have been some fairly sophisticated attempts at providing solutions but the recipients of these messages are often business users with better things to do than try to master the arcane world of command line utilities with multistep processes.

Until now all we could really tell our users was to contact the sender and have them re-send the message but ensuring that they send with “Plain Text” instead of “RTF”. This is understandably awkward for a business user trying to look professional and it wasn’t doing my ego any wonders either.

The only really practical solution, besides Lotus actually dealing with the issue and including it in their SMTP router is this one presented to me by a co-worker (thanks Anton!). Created by Julian Robichaux (as far as I can tell as his name is all over the database) it is a simple mail-in database that you set up in your Domino environment. Your internal (and external if you wish to help out the rest of the world) Users then forward these nuisance messages to it and it replies back with the “unraveled” files.

So again the link is to Project WinmailExtractor. I had it set up and running in about 15 minutes including inspecting the code for any weirdness.

Marc’s Philosophies – 02

“If you haven’t got the time to do it right,
when will you find the time to do it over?”

This is actually the title of a book that I keep on my desk at work.

The book actually only proved moderately useful in a time management sense (at least for me), but the title is killer. Whenever I get involved in a discussion with someone who wants to do something “quick and dirty and we can circle around and do it properly later” I point to the book and ask the obvious question that is practically bursting from the cover.

I’ve had many arguments around this topic, won most, lost some (when an initiative is being driven by the wrong motivations/people) but this philosophy helps put some perspective on the effort that is about to be expended and helps to guide the thinking and planning process into more strategic pathways.