Sametime High Availability problem using Round Robin DNS where at least one server is down for an extended period of time

There is an issue where people will begin having issues accessing Sametime if one server in a Sametime HA environment is down beyond a certain length of time when the Server Community DNS is set up as round robin.

Essentially either the clients or the servers “forget” about the downed server. You are OK for up to some number of days where they recognize that the server is a ST server but no longer available. After that some cache is cleared and this problem arises.

This is why normal HA works just fine (when a server goes down for just a few minutes or hours) the servers and / or the clients are able to ignore the proffered IP address since they are aware that the server is not longer responding.

If you are taking a server out of service for longer than a week (or the period within which you will be recycling all of the other servers in the community), remove the downed server from the round robin DNS until you are ready to put it back into service.

Save Window State in Lotus Notes 8.5.x

In Lotus Notes 7.x the ability to “Save Window State” (File > Save Window State) was introduced to allow users to have certain databases available to them on startup.

This ability is available in Lotus Notes 8.5.x but is set up a little differently. Now you must bookmark the databases you wish to appear automatically at startup in a special “Startup” folder.

To do this, right click on the open tab and select “Create Bookmark”

Then select the “Startup” Bookmark folder located under “More Bookmarks”

If you want ONLY the bookmarks in the startup folder to show up when the Lotus Notes Client starts, then go to Files – Preferences – Windows and Themes and DESELECT the “On restart, reopen any tabs that were open when I closed the client”.

If you leave this checked, all tabs that were open will show up on restart PLUS the items in your startup folder. So, even if you had closed any of the tabs represented in the startup folder they will be available when the client starts as expected.

Create a local replica of your mail file using LotusScript

I recently had the need to create local mail file replicas for a bunch of folks. This is something that *had* to be in place for a certain time but we wanted people to be able to create the replicas at a time of their choosing up until the drop dead date.

So I have two versions of this code. The first, which I’m not posting here because it’s a part of a much larger housekeeping module invoked from the PostOpen event of the user’s mail file, gives no choice to put it off. The end user will receive a dialog indicating what is about to happen and then the replica creation will kick off.

The second, which is sent as a button in an email explaining to the user why they would want to do this and that this allows them to run it at a time of their choosing, I include below. It has some friendlier dialogs for the end user.

When run, the code will pop up a message box explaining what is about to happen and give them an opportunity to stop before they get started.

Then I check to make sure they are the owner of the database (so an admin opening the message on their manager’s behalf doesn’t inadvertently create a local replica of the managers file on their own workstation).

Then – and this is probably overkill – I make sure that the user is explicitly in the ACL (not in there via a group) and is a Manager. In our environment this will always be the case so it doesn’t hurt to leave this check in.  You can probably delete it. This was put in out of a concern that folks would open this in a misconfigured mail-in database (where the owner in the preferences had been allowed to default to the users name instead of that of the mailbox). We manage access to our mail-in databases via groups so it would be unusual for the user to be explicitly listed and HIGHLY unusual for them to have manager access under any circumstances.

I then validate whether or not there already exists a file with the same name and along the same path as the one I wish to create. I’m lifting the path and file directly from the server-side database. I’ve actually come across a few cases where folks have local mail file replicas with completely different names and/or in surprisingly different folders than you would expect. Ah… the creativity of the end user!  For our purposes we need to have the file path and name consistent between the local and server copies.

Note that for the LotusScript Dir$ command, Notes will only return an empty string if the file doesn’t exist. If the containing folder doesn’t exist you will receive an error “76” . But if you create a handler that contains simply “Resume Next” Notes will gleefully pick up INSIDE the conditional statement. I was prepared to create a Boolean variable that would be set if either the file was not there or would be set in the error handler but I was able to take this little bizarre shortcut.

The tricky part for me was to get the new replica to show up automatically on the replication tab. For this I need to thank Notes 411 for a little bit of code that I really should have figured out for myself. This is what the UIWorkspace is doing in the code. Basically you just open both replicas in the UIWorkspace and they will be added to the replication tab. Easy as pie. I did not have any issues getting the replica appear on the workspace as the Notes 411 logic also purports to do.

Once done, I let the user know and exit the script.

This is designed to work on Lotus Notes 6.5.x and 7.

Sorry about the lack of formatting, WordPress seems to have taken to stripping it out.

Sub Click(Source As Button)
‘Check if there is a local mail file replica and create one if it doesn’t already exist.
‘Created by Marc Bourassa – July 16, 2010

Dim Sess As New NotesSession
Dim ws As New NotesUIWorkspace
Dim dbUI As NotesUIDatabase
Dim db As NotesDatabase
Dim ACL As NotesACL
Dim ACLEntry As NotesACLEntry
Dim dbLocal As NotesDatabase
Dim docCalProf As NotesDocument
Dim namName As NotesName
Dim sDataDir As String
Dim iRC As Integer
Dim sMsg As String
Dim sPrintText As String

On Error Goto ErrorGeneral
On Error 4005 Goto ErrorDBExists
On Error 76 Goto ErrorPathNotFound

Set db = sess.Currentdatabase

sMsg = “This button will create a local replica of your mail file (” & db.Title & “) if one does not already exist with the correct name.” & Chr$(13)& Chr$(13)&_
“Please press OK to continue or CANCEL to Abort”

iRC = Messagebox(sMsg,49,”Create Mail File Replica”)

If iRC = 1 Then
Set docCalProf = db.GetProfileDocument(“CalendarProfile”)
Set namName = New NotesName(docCalProf.Owner(0))
If namName.common = sess.Commonusername Then ‘User is the owner of this mail database
Set ACL = db.Acl
Set ACLEntry = ACL.Getentry(namName.Abbreviated )
If Not ACLEntry Is Nothing Then
If ACLEntry.Level=ACLLEVEL_MANAGER Then ‘User is explicitly in ACL AND is a manager
sDataDir = sess.getenvironmentstring(“Directory”,True)
If sDataDir = “” Then ‘something probably wrong with ini file. Don’t expect system to work correctly
Messagebox “There is a problem retrieveing your data directory information from the notes.ini file. Cannot proceed. This issue needs to be resolved by the helpdesk. Aborting Replica Creation.”,0,”Create Mail Database Replica”
Goto LeaveSub
End If
If Dir$( sDataDir & “\” & db.Filepath, 0 ) = “” Then ‘No replica by this name exists. Will fail to next line from error handling if path is bad (i.e. missing mail folder)
Call db.Createreplica(“”, db.Filepath)
‘Now add to the Replication tab
Call ws.opendatabase(db.Server,db.FilePath)
Set dbUI = ws.CurrentDatabase
Call ws.opendatabase(“”,db.FilePath)
Set dbUI=ws.CurrentDatabase
Messagebox “Local mail file replica has been successfully created (” & db.Title & “).”,0,”Create Mail Database Replica”
Else
Messagebox “Replica already exists for your mail file (” & db.Title & “). Nothing to do here.”,0,”Create Mail Database Replica”
End If
End If
Else
Messagebox “You are not a manager of this mail database (” & db.Title & “). Aborting Replica Creation.”,0,”Create Mail Database Replica”
End If
Else
Messagebox “You are not the owner of this mail database (” & db.Title & “). Aborting Replica Creation.”,0,”Create Mail Database Replica”
End If
Else
Messagebox “Replica Creation cancelled by you.”,0,”Create Mail Database Replica”
End If

LeaveSub:
Exit Sub

ErrorDBExists:
Resume LeaveSub

ErrorPathNotFound:
Resume Next

ErrorGeneral:
sPrintText = “Error: ” + Cstr(Err) + ” defn: ” + Error$ + “. ”
Messagebox sPrintText & Chr$(13) & Chr$(13) & “Could not complete this task you will need to create this replica manually. Please contact the help desk for assistance.” ,0,”Unexpected Error”
Resume LeaveSub

End Sub

Searching in Lotus Notes

There are 3 ways you can search for items in Lotus Notes:

  • Make sure that the view you are in is sorted by the column you wish to search by (usually the leftmost column, otherwise the one with the highlighted arrow at the top)
    • Begin typing – i.e. if you are in your inbox and you click on the “Who” column, then start typing the name of someone in that view. A box will appear to capture the letters and when you hit enter you will jump to the first line beginning with those letters.
  • Hit Ctrl-F or press the binocular icon a the top of the screen (). This will bring up a dialog that lets you search anything you can SEE IN THAT VIEW. It will search on subjects, names, whatever you can see it can search.
  • Finally, clicking the Magnifying glass icon or clicking View > Search This View depending on your version of Lotus Notes, will allow you to search by any criteria within the documents (body of email, attachments, to, from, subject, etc.) This is called a “Full text search”.

Can’t see update to a meeting in your sent folder

When you send a meeting invitation you will see it in your “Sent” folder.

If you make subsequent updates to the invitation, you do NOT see them as separate notes in the sent folder, however if you find the original note you will see that it is updated with your latest changes.

The Gory Details:

This is because the original sent message, your calendar entry and the update you just made are all one and the same document. When you open your sent message you will see it’s exactly the same document as if you opened the meeting from the calendar view.

There is no obvious way to “confirm” that the update has been sent except that, if you did not encounter any issues when you saved the update and if your calendar entry reflects the update, then it has been sent.

Lotusphere 2010 – Knowledge from Concentrate!

I just got back from this year’s Lotusphere held in Orlando, Florida. As usual, it is a great opportunity to immerse myself in the world of Lotus Notes / Domino.

There is always so much available to do that you have to pick and choose what’s most important to you and some stuff you just won’t get to see. In my case, I focus on the presentations rather than the product showcase primarily since I’m not in a position to get any of those products into my organization and secondarily I develop most of my own utilities and applications anyway. I spread my attention between the developer and the Admin tracks.

Hits and Misses

As with any event there are good and bad things. In my opinion:

The Hits:

– an unexpected highlight of the conference was having Dr. Brian Cox speak at the closing session. Absolutely brilliant!

– The JumpStart sessions held on Sunday (January 17) every one of these 2 hour sessions that I attended was great. Well prepared presenters disseminating highly relevant information.

– The Show-‘n-tell sessions. A deep dive involving some end-to-end process that was like one long (1 hr 45 min) demo. I loved experiencing all the gory details of installs that I’ll implementing myself later this year first hand like this.

– The Swan and Dolphin hotel. These folks know how to manage people flow. Meals had legions of staff directing people and ensuring that food stations were kept stocked. Cleanup of tables was unobtrusive and thorough and staff were always pleasant.

– The Audio visual folks. Not sure if they are Disney or were brought in by Lotusphere but there were no glitches at any of the sessions that I could detect. When there was a question or a presenter made a comment about something that they couldn’t figure out, someone would unobtrusively show up and do what was needed.

– On Time. Sessions started on time and ended on time. Period. This is a consistent theme and one that respects both attendees and presenter’s commitments alike.

The Misses:

– The Information and assistance folks were particularly impotent. They had no idea how to escalate questions and seemed intent on just getting rid of you. My biggest peeve was that my registration never came off pending status and so my Lotus Online account was never activated. Three trips to those yahoos failed to resolve the issue.

– The BOF (Birds of a Feather) sessions were a big disappointment for me this year. The first one I tried to attend, the facilitator was detained (but sent word so we could perhaps try to find another session of interest) and the session was rescheduled for some time in the future. The second one the facilitator didn’t bother to show or send word. This was vexing as it was an early morning session that I’d dragged my not-a-morning-person butt out of bed for at 5:30 am to be there on time.  Also, the sessions exactly overlapped with breakfast – well if you were serious about attending the following normal morning sessions. I think it’s time these ceased being an afterthought and perhaps gained some more prominence in both the schedule and maybe in the minds of the hosts? Last time I was here I attended one or two of these and found them pretty valuable.

– No use of mobile device access to session evaluation. Having attended WES 2009 and seen what can be done in terms of integrating the nearly ubiquitous smartphones into the scheduling and evaluation process, it was kind of a let down to see that these technologies were not being exploited (indeed *showcased*) as part of this event.

My Sessions

Sunday:

BOOT103 Running with Scissors: Sharpen Your Skills for a Pain-free IBM Lotus Domino 8.5 Upgrade

Boot 101 8.5 dom admin

BOOT104 IBM Lotus Sametime 8.5 Deployment Workshop

SHOW102 Using IBM Lotus Domino 8.5 Policies to Manage Your Clients

Monday:

BP108 Worst Practices

AD206 Filthy Rich User Interfaces

SHOW113 Integrating IBM Lotus Domino Data and Applications on Smartphones, Blackberry

BP306 How to Sell IBM Lotus Notes

BOF316 Managing your IBM Lotus Domino Environment with Lotus Domino (Facilitator no show)

Tuesday:

BOF314 How to Tune Your IBM Lotus Domino Server for Maximum Performance (Facilitator no show)

AD102 Extreme Makeover — LotusScript and Java Editor Edition

AD107 Enhance Your Existing Applications with XPages

BP106 The Top 11 Tips for Keeping Your Servers Healthy

ID102 Enterprise IBM Lotus Notes Client Deployments

BP212 Delivering IBM Lotus Domino to Mobile Devices: Top 10 Mobile Browser Dev Tricks – and More!

ID615 Best Practices for Upgrading to IBM Lotus Notes and Domino 8.5x

Wednesday:

BP107 Real World Examples: Upgrading to IBM Lotus Notes and Domino 8.5

SHOW201 Installation and Setup of IBM Lotus Sametime 8.5: From “Zero to Hero” in Just Two Hours

BP206 Let’s Give Your LotusScript a Tune-Up

ID604 IBM Lotus Notes and Domino in the Cloud – IBM LotusLive Notes

Thursday:

SHOW103 Roaming in IBM Lotus Domino 8.5: Configure the Best Choice for You

ID611 The Best-Laid Plans: Networking That Can Hurt Your Applications

GURU101 GURUpalooza!

ASK101 Ask the Developers

“unique fields names table size” error in Lotus Domino

I recently had an issue where the mail boxes on one of my Domino servers were not compacting at all during their normal window.

My logs showed

09/03/2009 04:00:01 AM  Router: Beginning mailbox file compaction of mail1.box
09/03/2009 04:00:01 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\mail1.box due to unique fields names table size
09/03/2009 04:00:02 AM  Router: Shutdown is in progress
09/03/2009 04:01:49 AM  Router: Unable to compact mailbox file mail1.box: Database is currently in use by you or another user
09/03/2009 04:01:49 AM  Router: Completed mailbox file compaction of mail1.box
09/03/2009 04:01:49 AM  Router: Beginning mailbox file compaction of mail2.box
09/03/2009 04:02:02 AM  Router: Completed mailbox file compaction of mail2.box
09/03/2009 04:02:02 AM  Router: Beginning mailbox file compaction of mail3.box
09/03/2009 04:02:02 AM  Switching to copy-style compaction for E:\Lotus\Domino\Data\mail3.box due to unique fields names table size
09/03/2009 04:03:49 AM  Router: Unable to compact mailbox file mail3.box: Database is currently in use by you or another user
09/03/2009 04:03:49 AM  Router: Completed mailbox file compaction of mail3.box
09/03/2009 04:03:50 AM  Router operation resumed.

Since I haven’t had the need to monitor for failure of mailbox compaction, I was not aware of this issue until I received complaints of Non Delivery Errors. Specifically “554 Error writing message to safe storage; message could not be stored to disk“.

I was unable to find anything useful either from Lotus’ technotes online, their forums or using Google in general so I thought I’d post something to help out others who travel this path.

I contacted Lotus and opened a PMR but they were clueless as well. I really wanted to know what the “unique fields names table size” error was all about but they really didn’t know and tossed me a technote pertaining to “Database has too many unique field names” which really didn’t apply. They also focused on database corruption caused by  anti-virus programs. I figure that, if this was the issue, this should  have come up in the past 5 years on some of the 30+ other servers I’m working with so I was pretty sure this was also not a cause.

Long and short, I still don’t have an effective means to generically monitor for mail box compact failure. I *could* set up a monitor for the “unique fields names table size” string but it seems that seems like overkill.

To resolve the issue is simple enough:

  • shut down the server,
  • rename all the mail boxes,
  • restart the server (allowing new mail boxes to be created),
  • copy over any not-dead messages to one of the newly created mail boxes and
  • delete the old mail boxes

A side-effect of the above is that my name becomes the “from” address for all the copied messages.  Anybody have a simple solution for that? I haven’t actually researched it yet since it comes up so infrequently but it would be handy to know.

Old Reminders Keep Appearing for Recurring Meetings

In Lotus Notes, a user has recurring meetings scheduled. When the reminder comes up, they acknowledge the dialog normally.

The next day, the recurring meeting will show up again along with any others that have accumulated.

In the forums and Lotus’ Technotes they indicate that you should get a handle to the user’s “CalendarSettings” profile (some refer to the “CalendarProfile” profile but you really need “CalendarSettings”) and reset the “lastAlarmDate” field to “Now”.

Using NotesPeek I could see that the user that I was dealing with had 3 CalendarSettings profiles. One for their current name, one for an early variation on their name with which they had begun their employ with us and one with the name of one of our Junior Administrators in it.

The correct profile (user’s current name) also matched the owner’s field name in the CalendarProfile but the “lastAlarmDate” field was already current. So this wasn’t going to help me. Figuring that the CalendarSettings document was likely damaged in some way, I created the following agent to blow away ALL of the calendarsettings profiles.

Get the user to exit Lotus Notes completely, then add the following agent to their mail file and run it. Don’t forget to remove it afterwards.

Then have them open Lotus Notes. They will see ALL their old reminders which they should just acknowledge. Also, some settings will need to be redone: they will go back to the week view of the calendar if they had changed it to some other view, etc.

But, compared with the annoyance of more and more accumulating meeting reminders, the reset is a minor issue.

Name:    Getcalendarsettings
Last Modification:    02/19/2009 04:05:54 PM
Comment:    [Not Assigned]
Shared Agent:    Yes
Type:    LotusScript
State:    Enabled
Trigger:    Manually From Actions Menu
Acts On:    None
LotusScript Code:
Option Public
Option Declare
Sub Initialize
    Dim sess As New notessession
    Dim coll As notesdocumentcollection
    Dim doc As NotesDocument
    Dim docdel As NotesDocument
    Set coll = sess.CurrentDatabase.GetProfileDocCollection(“Calendarsettings”)
    Set doc = coll.GetFirstDocument
    While Not doc Is Nothing
        Set docDel = doc   
        Set doc = coll.GetNextDocument(doc)
        If Not docDel Is Nothing Then
            Call docDel.Remove(True)
        End If
    Wend
End Sub

My Entry for Teamstudio’s Spotlight Award

Teamstudio recently had a contest where they wanted you to showcase your applications so that others can see what folks are doing out there and so you could toot your own horn a bit.

My entry was probably a lot less snazzy than the ultimate winners (I haven’t seen them) as I tend to focus on non-public facing utilities and infrastructure improvements in my development work.

But I thought it would be worthwhile posting my entry here (now that they’ve let me know I *didn’t* win… *sniff*) so those of you who know me might get some idea what I spend part of my days doing.

I just call my application the “Admin Agent Repository”. It was originally created to house a single major utility and to serve as a common repository for the myriad agents that are useful for administrators on a day-to-day basis. But I have refined it to allow non-developer administrators access to its functionality.
The most visible application was to buffer us from the rather absolute and immediate nature of Domino’s user termination where the need to reinstate caused a lot of headaches.
A document is created for each terminated employee – this can be created by anybody granted the correct role so security or helpdesk folks can input terms directly – and, on the appointed date the user’s name is submitted to the deny access group.
The app also verifies whether or not the user has a Blackberry device and notifies our Blackberry group to remove their account.

Designed when background agents could not access databases on other servers, the application will then run an agent on the user’s home server and, if necessary, update their database ACL with the name of a person responsible for cleaning up the mail file and send a note, with a link to that responsible individual explaining their responsibilities.
When the purge date arrives (the default is 4 days, 30 if a responsible person is defined,  all controlled via profile), the actual signed adminp request is submitted and the process proceeds as per Domino normal.
Reinstate is often as simple as putting the termination document on hold and removing the user from the deny group. No ACL updates, no repopulating in groups.
The repository is replete with utilities that I have found useful over the years, most are profile driven:

For instance the below profile drives our dynamically created groups. Each morning a subset of our groups are destroyed and recreated based on person document information input by either the users or our identity management system. The groups are used both to control application access based on certain criteria (i.e. all reports to a person can have access to a calendar app used by that team) or for communications (i.e. message to be sent to all folks within a particular state or management structure).

This one inspects all mail files and ensures that they not only have a template, but that it’s one of the standard ones that we mandate:

One ad-hoc utility that is quite popular is one that runs, inspects all of the calendar documents in a mail file and ensures that the $Busyname matches the Owner name from the calendar profile. Very handy for botched renames and transfers of mail files from other domains.
I also have a couple of “Rebuild Busytime” agents for clustered and non-clustered servers that run weekly to keep things there tip-top as well.
These utilities save us literally hours each week, any job that’s wasting my time I invest the effort into automating it and add it to this growing application.

Winmail.dat file issue in Lotus Notes workaround (finally)

Lotus Notes LogoFrom the timestamps I see that this gem has been available since June of 2006 but I certainly had not heard about it.

Microsoft (M$), always there with useful products, also always manages to make things *just* different enough from standard that you can’t really seem to call them on it but they make it impossible to cleanly interface with their products as a competitor.

Case in point is Outlook and Exchange. If you create a message using M$’s email client and send it out the door using their Rich Text Format which seems to be of a proprietary nature, the message that arrives at the other end will be wrapped up in a file called “Winmail.Dat”.

For Lotus Notes this has always been a problem since the Domino server does not have the ability to unravel this proprietary formatting and so the recipient ends up with a useless (winmail.dat )attachment.

I know that there have been some fairly sophisticated attempts at providing solutions but the recipients of these messages are often business users with better things to do than try to master the arcane world of command line utilities with multistep processes.

Until now all we could really tell our users was to contact the sender and have them re-send the message but ensuring that they send with “Plain Text” instead of “RTF”. This is understandably awkward for a business user trying to look professional and it wasn’t doing my ego any wonders either.

The only really practical solution, besides Lotus actually dealing with the issue and including it in their SMTP router is this one presented to me by a co-worker (thanks Anton!). Created by Julian Robichaux (as far as I can tell as his name is all over the database) it is a simple mail-in database that you set up in your Domino environment. Your internal (and external if you wish to help out the rest of the world) Users then forward these nuisance messages to it and it replies back with the “unraveled” files.

So again the link is to Project WinmailExtractor. I had it set up and running in about 15 minutes including inspecting the code for any weirdness.