IMAP Elixir Adventures The complete guide to handling IMAP e-mails from within Elixir

The complete guide to handling the processing of IMAP e-mails with Elixir.
The functionality to receive and process emails automatically is extremely important for any business to succeed. All entrepreneurs depend on being able to automate.

I've did the legwork on how this works so you can just focus on processing everything:)

Recently I wanted to do some IMAP e-mail processing in Elixir. The first step was simple. I started researching to see if there are certain libraries which can process incoming e-mails.
Processing IMAP e-mails with more mainstream programming languages seems to be simpler on first sight. There are more libraries available yet those programming languages lack the power and elegance of Elixir.

Why not POP3?

I've found certain POP3 libraries in Erlang/Elixir. However I'm not going to use POP3 and I will explain why you shouldn't use it for processing e-mails automatically (unless you really have to).
The POP3 protocol downloads a copy of the full e-mail of the request. So you have no way of inspecting the headers. If it's a big e-mail with huge attachmens you will download 20 MB of data.
Those libraries, especially the POP3 elixir/erlang one stated that they have problems with downloading large attachments.
Downloading a 2 to 5 MB e-mail crashes most of those libraries. And if they don't crash they use around 200 MB of RAM for the download! This isn't what I want.

POP3 isn't what you would want to use to process data. IMAP is far better

Now why IMAP you may ask?

IMAP allows you to inspect the mail. To search for certain keywords and properties that a mail contains. You can download specific data of the mail. The full headers, a specific portion of the headers, information about attachments etc.
Say you have a 15 MB e-mail with attachments. Downloading it at once will eat up precious resources. Network resources and memory/CPU resources needed to process the data.
If you need to process multiple such e-mails then there is a high strain on your servers resources.
Why not let the server send us what we need so we can process each individual thing separately?
What if I want the text version only? What about the HTML version? How about if I want to get just information about the attachments without downloading them? This can all be done by using IMAP.

ExImap

I found ExImap. An Elixir library which simulates basic IMAP. I went on to test it. It wasn't really difficult to install and make a connection via code to an IMAP e-mail account.
After playing around with it I came to the conclusion that I need to do everything manually via IMAP commands. This proved to be a good experience in experimenting with the protocol. Luckily for me the ExIMAP library takes care of the authentication and encryption of password challenges.
Looking at the library I saw one little small flaw. It creates one process and then it leaves the process open. The problem is that after 10 minutes the process will crash because the sever will close the connection without
any activity. The other problem is that there is only one GenServer for one account which is hardcoded in the Elixir configuration.

This is probably not intended. I did some modifications to the ExImap library as a patch.
My patch does 3 extra things:

• It allows multiple accounts and multiple connections. For me, any library or application must have a way to allow for MULTIPLE accounts/versions of itself running at the same time. It doesn't matter if the developer has to do some extra legwork it's just a requirement. I can't imagine using an IMAP library for just one account at a time!
• Then I've implemented a way to do the logout and close command.
• Also closing the socket and genserver so there are no errors.

A simple solution to an annoying bug

The IMAP CLOSE command closes the INBOX correctly. Then via the IMAP Logout command we tell the server we're logging out. Using these two commands is indicated as per the IMAP specifications so that both parties know that a connection ended. We wouldn't want to have corrupt data anywhere in the stack in a long running production application due to not gracefully closing a connection!

In my own Elixir application I used a scheduler (cron job) based on a process which runs every 5 minutes and verifies if there are any new e-mails.

The IMAP Flow

The implementation flow with IMAP looks like following:

1. Connect to the account
2. Select the folder/inbox you want to work on
3. Search for mails, get portions of data,
4. Process the data in your app maybe download attachments, etc
5. Go to step 3 untill you are done then proceed to step 6
6. Issue IMAP CLOSe, LOGOUT
7. Close the socket
8. Close the GenServer

Connect to the account

For this step I've taken the liberty to assume that you've set up ExImap properly (see https://hexdocs.pm/eximap/readme.html).

Select the folder/inbox you want to work on

Before you can do anything like getting an e-mail with IMAP you need to select a folder or the INBOX folder. (NOTE: POP3 doesn't know the concept of folders.)

Whenever we select an folder like INBOX IMAP sends us some nifty information.

reqsel = Eximap.Imap.Request.select("INBOX")Eximap.Imap.Client.execute(pid, reqsel)%Eximap.Imap.Response{  body: [    %{},    %{message: "[UIDNEXT 35] Predicted next UID", type: "OK"},    %{message: "[UIDVALIDITY 1555495275] UIDs valid", type: "OK"},    %{message: "[UNSEEN 23] First unseen.", type: "OK"},    %{message: "RECENT", type: "1"},    %{message: "EXISTS", type: "23"},    %{      message: "[PERMANENTFLAGS (\\\\Answered \\\\Flagged \\\\Deleted \\\\Seen \\\\Draft NonJunk Junk \\\\*)] Flags permitted.",      type: "OK"    },    %{      message: "(\\\\Answered \\\\Flagged \\\\Deleted \\\\Seen \\\\Draft NonJunk Junk)",      type: "FLAGS"    },    %{message: "[CLOSED] Previous mailbox closed.", type: "OK"}  ],  error: nil,  message: "[READ-WRITE] Select completed (0.002 + 0.000 + 0.001 secs).",  partial: false,  request: %Eximap.Imap.Request{    command: "SELECT",    params: ["INBOX"],    tag: "EX2"  },  status: "OK"}iex(4)> 

We get the UIDNEXT which is the next predicted unique id of the next mail.
We have the UIDVALIDITY it informs us how many unique valid i's exist. There is also a unique id for a folder.
Why do we need 2 different id's?
Each e-mail has it's own ID however this ID is just the order in the INBOX/FOLDER. By deleting an e-mail the id's will change.
Each e-mail gets it's unique id or UID. This UID is unique for a specific folder, when we move mails around they change.
In order to know the specific data about an e-mail we use the UIDVALIDITY for the folder we're in.

We also get the information about recent "new" unread e-mails that exist in our folder/inbox. Recent means since we last selected (Point 2) the folder/inbox.
We also know how many e-mail exist in that folder.

We get the permanent flags which tells us which flags are permitted. I won't be using this. We can search for e-mails which contain these flags. Searching for seen e-mails that contain a specific text may be helpful in certain situations.
Usually, we also get informed that the previous mailbox was closed.

The most interesting part of the response is UNSEEN. This indicates ID (not UID) of the first unread e-mail. I've parsed this with Regular Expressions to extract the data.
This ID can be used to fetch and download the headers and the e-mail. It's important to convert this ID to it's equivalent UID.
What happens if there are multiple new e-mails?
Just select the INBOX/folder again doing it recursively until there are no more UNSEEN e-mails.

This way we always know if new changes have been made since the last select due to the fact that we can analyze the rest of the discussed details IMAP sends us.
It's pretty interesting to select the inbox each time.

We need to select a folder in order to get or fecth data about an e-mail

Before you start fetching and downloading e-mail data know the difference between UID and ID usage

There are two very similar ways how we can fetch email data.
We have the normal list of functions like FETCH which uses the ID. Or we have the UID type commands like UID FETCH which use the UID. Remember, the ID Is the order of the inbox/folder the UID is the unique e-mail id unaffected by the order in the mailbox.

The first time I prototyped I used the ID to do anything. However if e-mails get deleted, moved then this will give us problems down the line.
The logical thing to do is to convert the ID to an UID and then work with that UID.

Getting the UID based on the mail sequence (ID) number is easy:
UID SEARCH id_sequence
Or in Elixir code with the ExIMAP library:

  @doc  """  Get the mail UID based on the  mail sequence number.  The sequence number is the number of the email relative to the folder  The UID should be an unique identifier 2^32 of the mailbox  """  def get_uid(pid,seq) do    req = Eximap.Imap.Request.uid(["SEARCH", seq])    response = Eximap.Imap.Client.execute(pid, req)    |> Map.get(:body)    |> List.last()    |> Map.get(:message)  end

Maybe sometimes you have the UID and want to find the ID. The reverse is SEARCH UID <UID> and we get the ID.
Elixir Code:

    req = Eximap.Imap.Request.search(["UID #{uid}"])    response = Eximap.Imap.Client.execute(pid, req)

Getting the headers

There are many ways to get the headers of an e-mail
One I like is using the UID Fetch and specifying which flags we want to select. Date, FROM, TO.. etc
This allows me to select just the information I need.

  req = Eximap.Imap.Request.uid(["FETCH",id, "(FLAGS BODY[HEADER.FIELDS (DATE FROM TO RECEIVED SUBJECT LIST-UNSUBSCRIBE)] BODY.PEEK[1])"])

Notice the BODY.PEEK[1]. That command gets the first multipart body of the e-mail. Which can be either text/plain or text/html. This is useful when the e-mail is really large and we just want the plaintext body.

Downloading the full Headers with PEEK

Downloading the full headers only can be done by using BODY.PEEK[HEADERS]

\treq = Eximap.Imap.Request.uid(["FETCH",uid, "BODY.PEEK[HEADER]"])

There are two downsides to this method and to any method which gets the e-mails.

• We don't get any information about the attachments which may or may not exist including name, size, etc. (We'll get to this later!)
• We won't get any information about the size of the whole e-mail.

Yes, you can get th full size of an e-mail!

There is also a command which fetches the full size of an e-mail. We just need to
UID FETCH <uid> FAST

req = Eximap.Imap.Request.uid(["FETCH",id, "fast"])

Using regular expressions to extract the size~r/RFC822.SIZE (\\d+)/

Next we can implement a system to download the full version of the e-mail or just a smaller version. If the size is greater than a few hundred KB then for sure there are attachments.

Getting the full e-mail

Getting the full e-mail can be done in various ways. I've experimented and I've found that there is a very small command which we can use. All we need is to have read an RFC document RFC822.
UID FETCH <uid> RFC822

The response contains the header, body and some other information like parts.

But how do we process this?

Eximap returns the IMAP information as is and processes it in Elixir. Normally we'd have to verify the status. IF the request completed we get a success status. I skipped this since I've tested the commands and they should work.
Plus, from the list of the body I usually select the last one.

Attachments

How do we get information about attachments if there are any? IMAP comes to the rescue and provides us with a very nifty way to get information about the attachments by using the command
UID FETCH <uid> (BODYSTRUCTURE.ATTACHMENTS)

 req = Eximap.Imap.Request.uid(["FETCH",uid,"(BODYSTRUCTURE.ATTACHMENTS)"]) response = Eximap.Imap.Client.execute(pid, req)%Eximap.Imap.Response{  body: [    %{},    %{      message: "FETCH (UID 14 BODYSTRUCTURE ((\\"text\\" \\"plain\\" (\\"charset\\" \\"utf-8\\" \\"format\\" \\"flowed\\") NIL NIL \\"7bit\\" 37 2 NIL NIL NIL NIL)(\\"audio\\" \\"mpeg\\" (\\"name\\" \\"FiRMA - Ultimul Dans (AUDIO)_d3ea11aee93c06e80266414a7a0ccd3.mp3\\") NIL NIL \\"base64\\" 3771726 NIL (\\"attachment\\" (\\"filename\\" \\"FiRMA - Ultimul Dans (AUDIO)_d3ea11aee93c06e80266414a7a0ccd3.mp3\\")) NIL NIL)(\\"application\\" \\"pdf\\" (\\"name\\" \\"mafia_history.pdf\\") NIL NIL \\"base64\\" 4166652 NIL (\\"attachment\\" (\\"filename\\" \\"mafia_history.pdf\\")) NIL NIL)(\\"image\\" \\"jpeg\\" (\\"name\\" \\"photo_2019-04-17_23-56-35.jpg\\") NIL NIL \\"base64\\" 117474 NIL (\\"attachment\\" (\\"filename\\" \\"photo_2019-04-17_23-56-35.jpg\\")) NIL NIL) \\"mixed\\" (\\"boundary\\" \\"------------8977D37A0D18BAEAA8F9748F\\") NIL (\\"en-US\\") NIL))",      type: "3"    }  ],  error: nil,  message: "Fetch completed (0.001 + 0.000 secs).",  partial: false,  request: %Eximap.Imap.Request{    command: "UID",    params: ["FETCH", 14, "(BODYSTRUCTURE.ATTACHMENTS)"],    tag: "EX15"  },  status: "OK"}

As you can see we get a lot of information regarding the attachments. The names, the mime type, the size in bytes. This can be easily extracted with Regular Expressions.

How to get the data structured in easy to process Elixir data structures?

Now that we've done all of these, we need to structure the data somehow.
I've looked into this and experimented a little bit. The data we get is plain text IMAP format which is not really compatible with Elixir.
Using Regular expressions to extract what we need is a good hack however the structure is different each time.
Doing it manually is time intensive and error prone.

There was a specific package in Elixir called "MIMEMAIL" but it doesn't really work as expected. It's based on a very old version of Elixir
I tried modifying it.

if you use HTTPOISON you have a mimemail erlang module. This can be used (YES!) to parse an e-mail or it's headers.
Here I've had some extra bad luck because it uses iconv. Installing ICONV proves that it's a different version.
I included a small hack, or a module which implements the iconv open, close and convert functionality which makes :mimemail work.

defmodule :iconv do  def open(a,b), do:     {:ok,b }  def close(a), do: {:ok, a}  def conv(_a,b), do:    {:ok,b } end

I've tried codepagex however I've seen that by using the beforementioned module it doesn't convert anything. It works fine.

By just using the :mimemail.decode() we get a erlang data structure tuple which contains all the information we need to process it further
This tuple consists of 5 elements. The 4th tuple consists again of a list with the 5 elements.

Flaws to ExImap and my solutions

The ExImap library has certain flaws which I've attempted to solve earlier. One of these problems is processing and downloading large e-mails. By large e-mails I mean e-mails with attachments where the total size is larger than 1 MB.

It crashes. I've set the timeout to 40 and 60 seconds. I tried tweaking it but it didn't help. It remained somewhat "slow" and processing resulted in timeouts for large e-mails at around 4-7 MB.
This seems pretty slow.
I'd recommend not using ExImap when downloading the full e-mail data if it has large attachments.
I'd recommend using external applications to process

You can use mutt, alpine even curl can be used to download emails via IMAP.

Downloading large e-mails with curl

Downloading e-mails which have attachments with curl is way faster than anything else I've tested. The great thing about this is that we get a text file which can be later decoded with mimemail to get the data structures we need.
This way we externalize the process of downloading e-mails to another system process and leave the processing to our application once it's done.

You can get the ExImap version with my modifications from this page.

In the future I'll create and provide a small library which processes this automatically.