E-mail Storage Maintenance
Monday, May 29th, 2006Tonight we will be upgrading our hosting service to a new e-mail storage format. The new format will give us better performance and will allow us to put some improved virus and spam checking tools in place.
The impact on most of you will be minimal. The e-mail server will be shut down for a short time, but no e-mail will be lost.
Those of you using NeoMail, or those of you using another IMAP client, may feel the effects of this a bit more. After the conversion, NeoMail will no longer be available. If you are using another IMAP client, you will need to resubscribe to any user-created folders you may be using.
Those of you using SpamAssassin will also be affected. After the conversion, you will find this tool replaced in your hosting control panel with one called Mailscanner. Mailscanner still uses SpamAssassin “under the hood” to manage spam, but offers some additional options not available with the old package. Unfortunately, if you are currently using SpamAssassin, you will need to reconfigure this tool after the conversion takes place.
If you have any problems with your e-mail after the conversion, or if you have any other questions or concerns, please open up a support ticket on our help desk.
Here are the details:
We are moving from UW-IMAP, which uses the mbox convention, to Courier-IMAP, which uses the maildir convention. Under the current (mbox) system, all mail you receive is stored in a single file which you can then download via a POP client such as Outlook or view via an IMAP client such as Outlook or one of the webmail programs offered in the control panel.
From a server administrator as well as a user viewpoint, there are some issues with this type of implementation. When mail is stored in a single file, for instance, interrupted POP sessions can leave behind lockfiles that must be manually removed, can result in duplicate download of mail when the session is resumed, and is difficult to manage when corruption of a message is suspected. In addition, the resource usage associated with some of the processes, such as the current POP daemon, can be quite high. Mail delivery can also be impacted by the single-file convention: when the inbox is locked while mail is being retrieved via a POP session or when a lockfile remains after an interrupted session, new mail cannot be delivered until that session is completed and the lock ends when the process is completed or until the lockfile is manually removed.
The new system is a non-locking system. Mail is stored in individual files rather than in a single large mailbox. This eliminates both the lockfiles that are required under the current system and the problems associated with those locks if sessions are interrupted - an interruption under the new system will simply allow the user to resume retrieval when they connect again. Due to the manner in which mail is moved to the user’s space, mail can also be delivered even if retrieval is currently underway. The resource usage under the new system is also significantly lower than under the current system, and will solve solve administrative-related issues we can see on servers that handle the largest volume of mail within the network.
One of the largest impacts of the conversion will be in the area of webmail. Under the new system, Neomail will no longer be offered as a webmail client from the control panel. Although we understand many people use and like Neomail, it has issues of its own. Of the three webmail clients available, Neomail is the most resource-intensive and the most unstable. Neomail has a bad habit of corrupting the files it uses and has, on occasion, and completely randomly, lost all mail for a user if they tried to perform an operation within Neomail. In addition, Neomail has not been in active development since 2002. Both Horde and SquirrelMail will continue to appear in the control panel. Of the two, we recommend the use of Horde over SquirrelMail. Part of the conversion process will take the Neomail addressbook (if it exists) and convert it to the addressbook for Horde.
One other impact at the user level is related to the use of folders, and requires a post-conversion operation at the user level. This is discussed below in “after the conversion”.
During the conversion:
Various processses that run the current mail system will be shut down. This is to protect the integrity of the files being converted, and means mail-related activities will not be available while the conversion completes. Mail sent to the server during this time will be stored while the conversion process takes place, and will not be lost. The new system will be installed and all mail directories will be converted to the new format required by maildir. Existing mail will be processed and converted. Neomail address books will be converted to Horde.
After the conversion:
Once the conversion is complete, the mail processes will be restarted and will be immediately available. Any mail that was stored during the conversion will be delivered at this time. Neomail will no longer function and will not be available for login. Clients will need to take action if utilizing an IMAP client (any standalone IMAP client (such as Outlook), Horde, Squirrelmail, and so on) or if previously using Neomail. Due to the conversion of mail folders, those people will need to resubscribe to their folders before those folders will be visible. Note that the mail within these folders has not been lost - the folders simply are not immediately visible because of the conversion process, which prepends the folder name with a leading dot.
The simplest way to resubscribe is to log in to SquirrelMail for the user who has folders that require resubscription. Select Folders from the top menu. On that page, you will be presented with a list of folders available for subscription. Select the folders to which you wish to subscribe and submit the form. To select multiple folders, hold the ctrl key and click on the folders to which you wish to subscribe. Refresh your folder list by clicking the “check for new mail” option in the upper left navigation bar. The folders will now be visible in the folder listing and the mail within those folders available for viewing and other operations. This is a one time operation. Folders created after the conversion process will be named in the new format and will be immediately available. Users of Horde will need to follow this procedure as well, but again, this procedure must only be done one time following conversion.
There are some changes to the mail directory structure after the conversion has completed. You will find several new directories in /home/users/mail/domain.com/username: tmp, new, cur, and (depending on how you get your mail), several folders beginning with “courier”. The tmp, new, and cur folders are where incoming messages are processed by the system. The “courier” directories hold information specific to your mail account(s). These new directories should not be removed.