Sametime High Availability problem using Round Robin DNS where at least one server is down for an extended period of time

There is an issue where people will begin having issues accessing Sametime if one server in a Sametime HA environment is down beyond a certain length of time when the Server Community DNS is set up as round robin.

Essentially either the clients or the servers “forget” about the downed server. You are OK for up to some number of days where they recognize that the server is a ST server but no longer available. After that some cache is cleared and this problem arises.

This is why normal HA works just fine (when a server goes down for just a few minutes or hours) the servers and / or the clients are able to ignore the proffered IP address since they are aware that the server is not longer responding.

If you are taking a server out of service for longer than a week (or the period within which you will be recycling all of the other servers in the community), remove the downed server from the round robin DNS until you are ready to put it back into service.

Save Window State in Lotus Notes 8.5.x

In Lotus Notes 7.x the ability to “Save Window State” (File > Save Window State) was introduced to allow users to have certain databases available to them on startup.

This ability is available in Lotus Notes 8.5.x but is set up a little differently. Now you must bookmark the databases you wish to appear automatically at startup in a special “Startup” folder.

To do this, right click on the open tab and select “Create Bookmark”

Then select the “Startup” Bookmark folder located under “More Bookmarks”

If you want ONLY the bookmarks in the startup folder to show up when the Lotus Notes Client starts, then go to Files – Preferences – Windows and Themes and DESELECT the “On restart, reopen any tabs that were open when I closed the client”.

If you leave this checked, all tabs that were open will show up on restart PLUS the items in your startup folder. So, even if you had closed any of the tabs represented in the startup folder they will be available when the client starts as expected.

Create a local replica of your mail file using LotusScript

I recently had the need to create local mail file replicas for a bunch of folks. This is something that *had* to be in place for a certain time but we wanted people to be able to create the replicas at a time of their choosing up until the drop dead date.

So I have two versions of this code. The first, which I’m not posting here because it’s a part of a much larger housekeeping module invoked from the PostOpen event of the user’s mail file, gives no choice to put it off. The end user will receive a dialog indicating what is about to happen and then the replica creation will kick off.

The second, which is sent as a button in an email explaining to the user why they would want to do this and that this allows them to run it at a time of their choosing, I include below. It has some friendlier dialogs for the end user.

When run, the code will pop up a message box explaining what is about to happen and give them an opportunity to stop before they get started.

Then I check to make sure they are the owner of the database (so an admin opening the message on their manager’s behalf doesn’t inadvertently create a local replica of the managers file on their own workstation).

Then – and this is probably overkill – I make sure that the user is explicitly in the ACL (not in there via a group) and is a Manager. In our environment this will always be the case so it doesn’t hurt to leave this check in.  You can probably delete it. This was put in out of a concern that folks would open this in a misconfigured mail-in database (where the owner in the preferences had been allowed to default to the users name instead of that of the mailbox). We manage access to our mail-in databases via groups so it would be unusual for the user to be explicitly listed and HIGHLY unusual for them to have manager access under any circumstances.

I then validate whether or not there already exists a file with the same name and along the same path as the one I wish to create. I’m lifting the path and file directly from the server-side database. I’ve actually come across a few cases where folks have local mail file replicas with completely different names and/or in surprisingly different folders than you would expect. Ah… the creativity of the end user!  For our purposes we need to have the file path and name consistent between the local and server copies.

Note that for the LotusScript Dir$ command, Notes will only return an empty string if the file doesn’t exist. If the containing folder doesn’t exist you will receive an error “76” . But if you create a handler that contains simply “Resume Next” Notes will gleefully pick up INSIDE the conditional statement. I was prepared to create a Boolean variable that would be set if either the file was not there or would be set in the error handler but I was able to take this little bizarre shortcut.

The tricky part for me was to get the new replica to show up automatically on the replication tab. For this I need to thank Notes 411 for a little bit of code that I really should have figured out for myself. This is what the UIWorkspace is doing in the code. Basically you just open both replicas in the UIWorkspace and they will be added to the replication tab. Easy as pie. I did not have any issues getting the replica appear on the workspace as the Notes 411 logic also purports to do.

Once done, I let the user know and exit the script.

This is designed to work on Lotus Notes 6.5.x and 7.

Sorry about the lack of formatting, WordPress seems to have taken to stripping it out.

Sub Click(Source As Button)
‘Check if there is a local mail file replica and create one if it doesn’t already exist.
‘Created by Marc Bourassa – July 16, 2010

Dim Sess As New NotesSession
Dim ws As New NotesUIWorkspace
Dim dbUI As NotesUIDatabase
Dim db As NotesDatabase
Dim ACL As NotesACL
Dim ACLEntry As NotesACLEntry
Dim dbLocal As NotesDatabase
Dim docCalProf As NotesDocument
Dim namName As NotesName
Dim sDataDir As String
Dim iRC As Integer
Dim sMsg As String
Dim sPrintText As String

On Error Goto ErrorGeneral
On Error 4005 Goto ErrorDBExists
On Error 76 Goto ErrorPathNotFound

Set db = sess.Currentdatabase

sMsg = “This button will create a local replica of your mail file (” & db.Title & “) if one does not already exist with the correct name.” & Chr$(13)& Chr$(13)&_
“Please press OK to continue or CANCEL to Abort”

iRC = Messagebox(sMsg,49,”Create Mail File Replica”)

If iRC = 1 Then
Set docCalProf = db.GetProfileDocument(“CalendarProfile”)
Set namName = New NotesName(docCalProf.Owner(0))
If namName.common = sess.Commonusername Then ‘User is the owner of this mail database
Set ACL = db.Acl
Set ACLEntry = ACL.Getentry(namName.Abbreviated )
If Not ACLEntry Is Nothing Then
If ACLEntry.Level=ACLLEVEL_MANAGER Then ‘User is explicitly in ACL AND is a manager
sDataDir = sess.getenvironmentstring(“Directory”,True)
If sDataDir = “” Then ‘something probably wrong with ini file. Don’t expect system to work correctly
Messagebox “There is a problem retrieveing your data directory information from the notes.ini file. Cannot proceed. This issue needs to be resolved by the helpdesk. Aborting Replica Creation.”,0,”Create Mail Database Replica”
Goto LeaveSub
End If
If Dir$( sDataDir & “\” & db.Filepath, 0 ) = “” Then ‘No replica by this name exists. Will fail to next line from error handling if path is bad (i.e. missing mail folder)
Call db.Createreplica(“”, db.Filepath)
‘Now add to the Replication tab
Call ws.opendatabase(db.Server,db.FilePath)
Set dbUI = ws.CurrentDatabase
Call ws.opendatabase(“”,db.FilePath)
Set dbUI=ws.CurrentDatabase
Messagebox “Local mail file replica has been successfully created (” & db.Title & “).”,0,”Create Mail Database Replica”
Messagebox “Replica already exists for your mail file (” & db.Title & “). Nothing to do here.”,0,”Create Mail Database Replica”
End If
End If
Messagebox “You are not a manager of this mail database (” & db.Title & “). Aborting Replica Creation.”,0,”Create Mail Database Replica”
End If
Messagebox “You are not the owner of this mail database (” & db.Title & “). Aborting Replica Creation.”,0,”Create Mail Database Replica”
End If
Messagebox “Replica Creation cancelled by you.”,0,”Create Mail Database Replica”
End If

Exit Sub

Resume LeaveSub

Resume Next

sPrintText = “Error: ” + Cstr(Err) + ” defn: ” + Error$ + “. ”
Messagebox sPrintText & Chr$(13) & Chr$(13) & “Could not complete this task you will need to create this replica manually. Please contact the help desk for assistance.” ,0,”Unexpected Error”
Resume LeaveSub

End Sub

On Access Scanners Killing App Performance

At work I’m wrestling with McAfee’s on access scanner absolutely crippling Lotus Notes latest client R8.5.1.  By killing the mcshield process you can realize a dramatic improvement in the startup time and overall performance of the Lotus client. A side benefit is that the omnipresent crashes I was facing have virtually disappeared.

Also, if you use the designer client you will see at least one order of magnitude increase in performance. I went from being amazed that such a terrible designer client had been released to being horrified that a relatively capable designer client had been so crippled by on access anti malware software as to render it useless.

Obviously our security folks are not going to be satisfied with simply disabling the scanner so the challenge is now to find some way to make these crappy security products ignore the right files so that the business applications can get on with being useful.

Today I was working at home with Quicken 2010 and I was miffed by how sluggishly it was behaving. I’ve got an i7-920 computer chip overclocked to 3 GHz, 12 Gigabytes of RAM and 64-Bit Windows 7 Professional pushing Quicken along. In theory it should be screaming.  Pulling up the task manager, what do I see but for pretty much every mouse click the MsMpEng executable (Microsoft Security Essentials Anti Malware scanner) is sucking up 10-30 percent of my CPU.

I’ve tried configuring the scanner to ignore my executable (C:\Program Files (x86)\Quicken\qw.exe), the Quicken QDF data file type (QDF extension) and even the entire dedicated drive on which the Quicken data files reside. But nothing has any effect. I’m pretty sure that those options are provided for the same reason as “Close” buttons on elevators exist: to give you something to do while things proceed at their normal pace.

Now killing the MsMpEng process absolutely fixes the issue. Quicken takes off like there is no tomorrow. Again, this is not an appropriate answer. You need to have rather sophisticated protections in place to work in today’s sophisticated cyber world. But it does point a finger directly at the resource hogging culprit.

So tell me, why does our protection software have to be so crude and bloated that this is even an issue? Does anybody have any suggestions or alternatives that you are happy with? Until this issue I thought I was VERY happy with Microsoft Security Essentials. But I’ve never needed to tweak it before. Now I want something effective. Symantec’s products have proven themselves to be monstrous resource hogs in the past. My McAfee experience at work is leading me away from that direction. Suggestions are very welcome and even more so if you’ve worked with Quicken in a Windows 7 64 bit environment!

“unique fields names table size” error in Lotus Domino

I recently had an issue where the mail boxes on one of my Domino servers were not compacting at all during their normal window.

My logs showed

09/03/2009 04:00:01 AM  Router: Beginning mailbox file compaction of
09/03/2009 04:00:01 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\ due to unique fields names table size
09/03/2009 04:00:02 AM  Router: Shutdown is in progress
09/03/2009 04:01:49 AM  Router: Unable to compact mailbox file Database is currently in use by you or another user
09/03/2009 04:01:49 AM  Router: Completed mailbox file compaction of
09/03/2009 04:01:49 AM  Router: Beginning mailbox file compaction of
09/03/2009 04:02:02 AM  Router: Completed mailbox file compaction of
09/03/2009 04:02:02 AM  Router: Beginning mailbox file compaction of
09/03/2009 04:02:02 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\ due to unique fields names table size
09/03/2009 04:03:49 AM  Router: Unable to compact mailbox file Database is currently in use by you or another user
09/03/2009 04:03:49 AM  Router: Completed mailbox file compaction of
09/03/2009 04:03:50 AM  Router operation resumed.

Since I haven’t had the need to monitor for failure of mailbox compaction, I was not aware of this issue until I received complaints of Non Delivery Errors. Specifically “554 Error writing message to safe storage; message could not be stored to disk“.

I was unable to find anything useful either from Lotus’ technotes online, their forums or using Google in general so I thought I’d post something to help out others who travel this path.

I contacted Lotus and opened a PMR but they were clueless as well. I really wanted to know what the “unique fields names table size” error was all about but they really didn’t know and tossed me a technote pertaining to “Database has too many unique field names” which really didn’t apply. They also focused on database corruption caused by  anti-virus programs. I figure that, if this was the issue, this should  have come up in the past 5 years on some of the 30+ other servers I’m working with so I was pretty sure this was also not a cause.

Long and short, I still don’t have an effective means to generically monitor for mail box compact failure. I *could* set up a monitor for the “unique fields names table size” string but it seems that seems like overkill.

To resolve the issue is simple enough:

  • shut down the server,
  • rename all the mail boxes,
  • restart the server (allowing new mail boxes to be created),
  • copy over any not-dead messages to one of the newly created mail boxes and
  • delete the old mail boxes

A side-effect of the above is that my name becomes the “from” address for all the copied messages.  Anybody have a simple solution for that? I haven’t actually researched it yet since it comes up so infrequently but it would be handy to know.

Memory leak issue for RIM Blackberry Devices – Especially 8830

*** Update May 8, 2009 ***

Verizon finally did see fit to release the OS 4.5 update and, so far, it appears that it is addressing the issue for us. We set a date of May 15 (approximately 3 weeks after upgrading) as our “OK Date”. This means that folks in our pilot group would have gone 100% longer than before without a recurrence of their messages vanishing.

So, while the memory does still dip significantly in the devices I’ve seen, it appears that the updated OS is addressing the worst of the memory symptoms. Kudos! From the below you could see I was entirely not expecting this.

*** End Update May 8, 2009 ***

We have been having a problem for a while know that has surged to the forefront of our corporate consciousness at the beginning of spring break (April 4, 2009).

Folks on that day began reporting in droves (basically about 5% of my total blackberry user base) that some or all of their email had disappeared, many on that Saturday, but many others on the days after that.

In looking into the issue I discovered that our Blackberry help desk has been fielding similar issues on the order of (anecdotally) 5-10 such incidents per week.

The problem has been largely experienced by our 8830 users. Made more awful by the fact that only this past summer we upgraded most of our handhelds to the new 8830 model.

Facts that I have at my disposal:

  • RIM claims that the memory manager in the 4.5 OS addresses this issue
  • I have a BOLD (OS 4.6.x) that shows massive memory consumption that is corrected by removing and replacing the battery or by simply performing a reset (Alt – Right Shift – Del).
  • People are complaining about this issue whether their devices communicate with a corporate BES or not (in the various forums that I have been reading)
  • The vanishing messages are a normal function of the low memory manager on the Blackberry OS. The priority is to allow new incoming messages and (supposedly) the last resort is to delete the oldest or least accessed existing messages to make that room

This is what I am thinking:

  • The problem is device related and is a memory handling issue
  • The problem has not been resolved with OS 4.5 but rather folks now receive a warning at about 400K of memory so they can take action before messages begin to vanish
  • The problem manifests in devices other than the 8830 but newer devices have so much memory that the issue is masked and wreaks less havoc. My bold can easily get to 6 Megabytes of file free out of 37 that are normally available. That’s 31 Megabytes of space that’s being wasted. That’s also more than double the maximum File Free that I used to have on my old 8830. Except for those apps supplied with the bold, my 3rd party apps are the same on this device as I had on my 8830.
  • The sudden surge in reports stems more from the fact that so many people were going to be out of the office and needed their email accessible on their handheld devices than any sudden change in the environment (indeed, I had been out of the office for the 2 days prior to this issue and I’m the only one making changes to our BES). Unless Verizon is monkeying around with stuff on their network.

RIM is insistent that the issue is resolved in the 4.5 OS. Since Verizon is our primary service provider and they’ve been unable to certify the 4.5 OS for their devices for about a year now, this is not a reasonable solution for us.

Plus, it seems that there are a fair number of other Blackberry users out there who are encountering this issue regardless of their handheld version or OS.

What needs to happen:

  • Short term: I need to create an application that would reset the device on a schedule. One of my team members has found one called “QuickPull” that *almost* fits the bill but needs some tweaks (see below) and can’t be distributed via BES.
  • Long Term:  RIM needs to get off its behind and acknowledge this memory management issue and actually address it. Pull back on the whiz-bang (crappy storm for example) and make sure that the basics are rock solid. Folks needing to reset their devices on a daily basis is just silly.
  • Long Term as well: What’s with this limited memory in these devices? We should be able to upgrade or add memory as we need to. The advice from RIM to “remove applications” and “minimize what you store” belongs back in the late-80’s along with a 640K memory limit and QEMM software!

Specs for reset application:

Since it is going to be necessary to create a workaround to allow folks to use their devices with the security of a corporate email system (i.e. messages not disappearing) I need to find or create a reset application that –

  • Resets the device (reclaims all that wasted memory)
  • Is distributable by BES (ALX and COD files)
  • Operates on a schedule that is user configurable but has a default time set by policy. That time is localized (i.e. 3 am is 3 am your timezone).
  • Prompts user to allow delay or skip of reset
  • Recognizes phone in use and delays reset
  • Can be set to only reset if device is locked
  • Inexpensive

That’s what I have. So far I’ve gotten to the point where I can produce a “Hello World!” app on my handheld. Let’s see how long it takes to address this workaround.

We’ve been working unproductively with RIM – they keep wanting us to delete applications / themes etc.

We’ve been working with Verizon, they’re promising the 4.5 OS RSN (Real Soon Now).

Old Reminders Keep Appearing for Recurring Meetings

In Lotus Notes, a user has recurring meetings scheduled. When the reminder comes up, they acknowledge the dialog normally.

The next day, the recurring meeting will show up again along with any others that have accumulated.

In the forums and Lotus’ Technotes they indicate that you should get a handle to the user’s “CalendarSettings” profile (some refer to the “CalendarProfile” profile but you really need “CalendarSettings”) and reset the “lastAlarmDate” field to “Now”.

Using NotesPeek I could see that the user that I was dealing with had 3 CalendarSettings profiles. One for their current name, one for an early variation on their name with which they had begun their employ with us and one with the name of one of our Junior Administrators in it.

The correct profile (user’s current name) also matched the owner’s field name in the CalendarProfile but the “lastAlarmDate” field was already current. So this wasn’t going to help me. Figuring that the CalendarSettings document was likely damaged in some way, I created the following agent to blow away ALL of the calendarsettings profiles.

Get the user to exit Lotus Notes completely, then add the following agent to their mail file and run it. Don’t forget to remove it afterwards.

Then have them open Lotus Notes. They will see ALL their old reminders which they should just acknowledge. Also, some settings will need to be redone: they will go back to the week view of the calendar if they had changed it to some other view, etc.

But, compared with the annoyance of more and more accumulating meeting reminders, the reset is a minor issue.

Name:    Getcalendarsettings
Last Modification:    02/19/2009 04:05:54 PM
Comment:    [Not Assigned]
Shared Agent:    Yes
Type:    LotusScript
State:    Enabled
Trigger:    Manually From Actions Menu
Acts On:    None
LotusScript Code:
Option Public
Option Declare
Sub Initialize
    Dim sess As New notessession
    Dim coll As notesdocumentcollection
    Dim doc As NotesDocument
    Dim docdel As NotesDocument
    Set coll = sess.CurrentDatabase.GetProfileDocCollection(“Calendarsettings”)
    Set doc = coll.GetFirstDocument
    While Not doc Is Nothing
        Set docDel = doc   
        Set doc = coll.GetNextDocument(doc)
        If Not docDel Is Nothing Then
            Call docDel.Remove(True)
        End If
End Sub