IMAP Elixir Adventures The complete guide to handling IMAP e-mails from within Elixir

The IMAP CLOSE command closes the INBOX correctly. Then via the IMAP Logout command we tell the server we’re logging out. Using these two commands is indicated as per the IMAP specifications so that both parties know that a connection ended. We wouldn’t want to have corrupt data anywhere in the stack in a long running production application due to not gracefully closing a connection!

In my own Elixir application I used a scheduler (cron job) based on a process which runs every 5 minutes and verifies if there are any new e-mails.

For this step I’ve taken the liberty to assume that you’ve set up ExImap properly (see https://hexdocs.pm/eximap/readme.html).

Before you can do anything like getting an e-mail with IMAP you need to select a folder or the INBOX folder. (NOTE: POP3 doesn’t know the concept of folders.)

Whenever we select an folder like INBOX IMAP sends us some nifty information.

We get the UIDNEXT which is the next predicted unique id of the next mail.
We have the UIDVALIDITY it informs us how many unique valid i’s exist. There is also a unique id for a folder.
Why do we need 2 different id’s?
Each e-mail has it’s own ID however this ID is just the order in the INBOX/FOLDER. By deleting an e-mail the id’s will change.
Each e-mail gets it’s unique id or UID. This UID is unique for a specific folder, when we move mails around they change.
In order to know the specific data about an e-mail we use the UIDVALIDITY for the folder we’re in.

We also get the information about recent "new" unread e-mails that exist in our folder/inbox. Recent means since we last selected (Point 2) the folder/inbox.
We also know how many e-mail exist in that folder.

We get the permanent flags which tells us which flags are permitted. I won’t be using this. We can search for e-mails which contain these flags. Searching for seen e-mails that contain a specific text may be helpful in certain situations.
Usually, we also get informed that the previous mailbox was closed.

The most interesting part of the response is UNSEEN. This indicates ID (not UID) of the first unread e-mail. I’ve parsed this with Regular Expressions to extract the data.
This ID can be used to fetch and download the headers and the e-mail. It’s important to convert this ID to it’s equivalent UID.
What happens if there are multiple new e-mails?
Just select the INBOX/folder again doing it recursively until there are no more UNSEEN e-mails.

This way we always know if new changes have been made since the last select due to the fact that we can analyze the rest of the discussed details IMAP sends us.
It’s pretty interesting to select the inbox each time.

We need to select a folder in order to get or fecth data about an e-mail

There are two very similar ways how we can fetch email data.
We have the normal list of functions like FETCH which uses the ID. Or we have the UID type commands like UID FETCH which use the UID. Remember, the ID Is the order of the inbox/folder the UID is the unique e-mail id unaffected by the order in the mailbox.

The first time I prototyped I used the ID to do anything. However if e-mails get deleted, moved then this will give us problems down the line.
The logical thing to do is to convert the ID to an UID and then work with that UID.

Getting the UID based on the mail sequence (ID) number is easy:
UID SEARCH id_sequence
Or in Elixir code with the ExIMAP library:

Maybe sometimes you have the UID and want to find the ID. The reverse is SEARCH UID <UID> and we get the ID.
Elixir Code:

There are many ways to get the headers of an e-mail
One I like is using the UID Fetch and specifying which flags we want to select. Date, FROM, TO.. etc
This allows me to select just the information I need.

Notice the BODY.PEEK[1]. That command gets the first multipart body of the e-mail. Which can be either text/plain or text/html. This is useful when the e-mail is really large and we just want the plaintext body.

Getting the full e-mail can be done in various ways. I’ve experimented and I’ve found that there is a very small command which we can use. All we need is to have read an RFC document RFC822.
UID FETCH <uid> RFC822

The response contains the header, body and some other information like parts.

Eximap returns the IMAP information as is and processes it in Elixir. Normally we’d have to verify the status. IF the request completed we get a success status. I skipped this since I’ve tested the commands and they should work.
Plus, from the list of the body I usually select the last one.

How do we get information about attachments if there are any? IMAP comes to the rescue and provides us with a very nifty way to get information about the attachments by using the command
UID FETCH <uid> (BODYSTRUCTURE.ATTACHMENTS)

As you can see we get a lot of information regarding the attachments. The names, the mime type, the size in bytes. This can be easily extracted with Regular Expressions.

Now that we’ve done all of these, we need to structure the data somehow.
I’ve looked into this and experimented a little bit. The data we get is plain text IMAP format which is not really compatible with Elixir.
Using Regular expressions to extract what we need is a good hack however the structure is different each time.
Doing it manually is time intensive and error prone.

There was a specific package in Elixir called "MIMEMAIL" but it doesn’t really work as expected. It’s based on a very old version of Elixir
I tried modifying it.

if you use HTTPOISON you have a mimemail erlang module. This can be used (YES!) to parse an e-mail or it’s headers.
Here I’ve had some extra bad luck because it uses iconv. Installing ICONV proves that it’s a different version.
I included a small hack, or a module which implements the iconv open, close and convert functionality which makes :mimemail work.

By just using the :mimemail.decode() we get a erlang data structure tuple which contains all the information we need to process it further
This tuple consists of 5 elements. The 4th tuple consists again of a list with the 5 elements.

The ExImap library has certain flaws which I’ve attempted to solve earlier. One of these problems is processing and downloading large e-mails. By large e-mails I mean e-mails with attachments where the total size is larger than 1 MB.

It crashes. I’ve set the timeout to 40 and 60 seconds. I tried tweaking it but it didn’t help. It remained somewhat "slow" and processing resulted in timeouts for large e-mails at around 4-7 MB.
This seems pretty slow.
I’d recommend not using ExImap when downloading the full e-mail data if it has large attachments.
I’d recommend using external applications to process

You can use mutt, alpine even curl can be used to download emails via IMAP.

Downloading e-mails which have attachments with curl is way faster than anything else I’ve tested. The great thing about this is that we get a text file which can be later decoded with mimemail to get the data structures we need.
This way we externalize the process of downloading e-mails to another system process and leave the processing to our application once it’s done.

You can get the ExImap version with my modifications from this page.

In the future I’ll create and provide a small library which processes this automatically.