IMAP Client Coding HOWTO

By Timo Sirainen <[email protected]>, comments welcome.

Updates by Dave Cridland

NOTE: This HOWTO has been rewritten as Best Practices for Implementing an IMAP Client

Design Critical

These are the most important issues to understand, or your implementation will mostly likely be completely broken and the only way to fix it would be a complete rewrite:

• Read the RFC-3501. You can't possibly write a correctly behaving IMAP client just by looking at how a couple of clients and servers behave. If you don't fully understand the RFC, ask about it in the IMAP protocol mailing list rather than guess based on how some servers behave.
• IMAP allows simultaneous access for multiple clients to a mailbox. Don't expect that you're the only one accessing it. Send changes immediately to the server, and show changes sent by the server immediately to the user.
• Don't ignore unexpected untagged replies. Especially EXPUNGE replies must be remembered, and it's not only the EXPUNGE command that can create those replies. UIDVALIDITY changes must also completely invalidate your local mailbox cache, no matter when that happens. You might as well support handling most of the untagged responses, independently on what generated them. For example if a server sends you STATUS updates automatically, don't bother asking those mailboxes' status for a while.
• Commands can fail - deal with it. You may have requested a message that was just expunged from server by another connection. Someone might have deleted the mailbox you just tried to access. In any case, if server replies with NO for a command, you should try to deal with it or show the error message to user. RFC-2180 helps you with figuring out how to handle FETCH and STORE errors.

• Use token-based parsing. It quite likely makes it easier for you. Many commands can send atoms, quoted-strings or literals as parameters and you'll have to be able to handle all of them. Literals sent by the server are almost always small and it's easier for you to just read them into memory and handle them the same way as quoted-strings. Exceptions are the replies for FETCH BODY[], RFC822, RFC822.HEADER and RFC822.TEXT which can be large, try to avoid reading them into memory.

  Note that you can't really assume any size limits, the size of the returned replies depends on the size of mailbox and other things. You might however want to do at least some sanity checks for them, such as not allowing any token to be larger than 10MB or so.

Common Flaws

Miscellaneous points that you should get right:

• Don't do LIST "" *. You may end up listing thousands of mailboxes as a result. The correct way to do it is LIST "" % and list child mailboxes only when the user requests them. If you need to know if the mailboxes have children, there are several ways to do it:
  • LIST "" %/% lists all the immediate child mailboxes. This is the only way that works with all the servers.
  • Take advantage of \HasChildren and \HasNoChildren flags that servers supporting CHILDREN extension (RFC-3348) send.
  • LISTEXT isn't a RFC yet, but it's likely to be the future. If you see it in the server's capabilities list, use LIST (CHILDREN) "" % and again pick the \HasChildren and \HasNoChildren flags.
  • Note that the server may not always send either \HasChildren or \HasNoChildren. If \NoInferiors isn't sent either, fallback to LIST "" mailbox/%.
  • If your client receives some data that matches the syntax for an extension, ignore it if the extension isn't advertised. Some servers provide data from drafts which subsequently get changed, for instance. Similarly, be prepared to drop an extension if an attempt to use the extended command syntax returns BAD - typically, early drafts of LISTEXT have been widely implemented.
• LIST reply is very loosely defined. It may be very much out of order. It may even contain duplicates. Here's an example reply for LIST "" *:

  * LIST (\UnMarked) "." "foo.bar"
  * LIST (\Marked) "." "foo"
  * LIST () "." "baz"
  * LIST (\UnMarked \NoSelect) "." "baz"
  

• SELECT mailbox, STATUS mailbox behavior isn't well defined. Don't do STATUS for a selected mailbox. Except for UNSEEN count, you should get all that information as a reply to SELECT. If you really want to know the UNSEEN count as well, use SEARCH UNSEEN. Even that has to be done only once, since the server keeps you updated of message flag changes automatically. UIDNEXT changes aren't notified however, if you care about them. That's a bit problematic case.
• Mailbox names are in modified UTF7 character set. Don't use anything else.
• Make it secure against malicious servers. The most common security related problems with IMAP clients are integer overflows when handling huge literal sizes and huge amounts of messages. If you do eg. malloc(literal_size+1), it may overflow to malloc(0). The same problem happens with malloc(messages_count * sizeof(struct message)). See here and here for more information.

Behave Well

• Supporting NAMESPACE extension is good for you. See RFC-2342.
  • NAMESPACEs sometimes appear in the LIST, and sometimes don't. Sometimes they end with a seperator, and sometimes they don't. In particular, MDaemon gives NAMESPACE reply as Shared - , and the LIST will give you Shared - mailbox. The mailbox's name is mailbox.
• Fetch MIME mails in parts. If the mail has a large attachment, there's no need to download it unless the user actually wants to do something with it. Use BODY or BODYSTRUCTURE to find out the message body parts.
• Pipelining commands can make you faster. Just make sure you're not doing anything ambiguous (see RFC 3501). Server may execute the commands in any order it wishes, or it might execute them simultaneously mixing the replies together. If you have any doubt that it might not work, don't do it. One of the good uses for this is for checking mailboxes' STATUS. For example:

  1 STATUS INBOX (MESSAGES UNSEEN RECENT)
  2 STATUS foo (MESSAGES UNSEEN RECENT)
  3 STATUS bar (MESSAGES UNSEEN RECENT)
  * STATUS "INBOX" (MESSAGES 3 RECENT 2 UNSEEN 3)
  * STATUS "bar" (MESSAGES 5 RECENT 1 UNSEEN 1)
  1 OK
  * STATUS "foo" (MESSAGES 20 RECENT 0 UNSEEN 0)
  3 OK
  2 OK
  

• Don't try to modify read-only mailboxes. Don't EXPUNGE, it won't work. If PERMANENTFLAGS doesn't contain flag you want to update, don't bother sending it to the server since the server will either ignore it or store it only for the current session.

• Optionally support multiple connections to server. Depending on the server, this might be a huge performance benefit. If the user mostly switches between three mailboxes, keeping each one of them in a separate connection would require selecting the mailbox only once and leaving them open. This also makes it possible for the server to notify the client immediately about changes in those mailboxes.

  Note that this also has many potential problems that you should consider, although quite unlikely to ever occur. For example different connections to the same the server might actually have different capabilities because the server software was just upgraded or some configuration setting was changed. Or maybe you were just redirected to another server in a cluster, which had slightly different settings. Maybe the server limits the number of IMAP connections per user, and kills one of your old connections while you create a new one. Or maybe it doesn't let the new one in at all.

User Interface

Not that much IMAP specific, but things you should consider:

• Handle EXPUNGE replies safely. Suppose a user sees two messages and wants to expunge them. He quickly hits DELETE + DELETE + EXPUNGE keys. Now suppose someone else had just expunged the second message and notified your client about it. If the client moved the selection to the third message after expunging the second one, your client just expunged a message accidentally. The safest way to handle this would be to simply mark the message deleted instead of removing it from list. Only after the user hits the EXPUNGE-key would the messages be removed.
• Don't block the user interface just because the user is uploading or downloading a large mail. Wouldn't it be nice to continue reading your mails while waiting for that 10MB attachment download to finish? You can do this by creating multiple IMAP connections to the server, or you can just fetch the attachments in smaller pieces (preferrably user or auto-configurable (eg. one second per piece)). Note that accessing a mailbox already opened in another connection may not work with all servers, so you should prepare for that too.

• Try not to fetch the entire mailbox's metadata. If the user sees only 20 newest messages in the screen, there's no reason to ask about any others from the server. If the user has a slow network connection and a large mailbox, it can be frustrating to wait for headers of messages to download which are never seen.

  This shouldn't be too difficult to implement even with standard GUI toolkits. You know the number of messages in the mailbox, so create that many dummy items (eg. empty or "wait..") in your messages list. When the user scrolls for more data, fetch the data for the messages that are then visible. The user may see the dummy items temporarily in the screen until the server has replied, but with some smart prefetching you can avoid that for most cases (ie. download a bit more than just the visible messages). You could also optionally fetch all the other messages as well, but do it in smaller blocks on the background so the user doesn't have to wait.