Receiving new messages from POP3 server (Part 2)

Advanced topics Summary: Presents advanced code sample of receiving new messages. Highlights performance improvements, responsive user interface, error-checking, old POP3 servers support and simple implementation of message identifiers database.

Tutorial map:

Part 1 - Getting new messages
Part 2 - Advanced topics

Typical scenario: The mail client checks whether new messages have arrived since last connection and wants to download new messages (if any). The mail client allows user to abort operation, handles errors, keeps user interface responsive during email operation.

Limitations: None.

Performance improvements

In Part 1 code sample, Unique-IDs for all the messages in the mailbox are looked up in the database. In real-world situations, mailboxes usually contain only a small number of new messages beside total number of the messages. Another important fact is that new messages are always appended to the end of the mailbox. It's impossible that new message is followed by not new message. In other words:

Number of not new messages is usually larger than number of new messages
Once not new message is found, the mail client can expect that all messages from this message to the beginning of the mailbox are not new

According to the above, it is better to iterate through the messages in reverse order, from the last message to the first. If the message is new, retrieve it and proceed with previous message. Once already downloaded (not new) message found, the entire loop can be stopped - there are no more new messages in the mailbox.

Keeping user interface responsive when performing email operation

By default, Windows events are not processed when MailBee is busy with certain email operation. This gives maximum performance to your application, and fits fine for web applications such as ASP.

For desktop applications, however, it's common to give user ability to interact with your application even when it is busy. Mailer objects (POP3, SMTP, IMAP4) provide EnableEvents property for this purpose. Particularly, to enable events processing by POP3 object, you need to set POP3.EnableEvents=True before connecting to the mail server.

Checking for errors

Connection to the POP3 server can be suddenly broken for a variety of reasons. Well-written application should take care of such situation. Also, your application should consider that not all POP3 servers support the command required for extracting Unique-IDs of the messages - Unique-IDs support is an optional extension of the POP3 protocol.

Search new messages when the POP3 server does not support Unique-IDs

If the POP3 server does not support Unique-IDs, smart mail client may consider using backup method to download new messages. This method uses Message-ID values taken from "Message-ID:" field of the messages' headers. Its disadvantage is the fact that headers for all the messages in the mailbox must be retrieved. This results in larger network traffic (relatively to retrieving Unique-IDs). However, in situation where Unique-IDs cannot be used, this method is still better than downloading all the messages entirely.

The idea is obvious:

Retrieve headers for the first message in the mailbox
Lookup Message-ID of the message in the database. If the Message-ID is not found, the message is new and should be retrieved entirely
In the same manner, proceed with subsequent messages till the end of the mailbox

Simple text-file database implementation for single email account

In the previous part, sample code has no implementation of the database, and IsNewMessage() function always returns True.

In this part, simple database implementation is provided. The database is stored in a text file. Each line of the file holds one Unique-ID value. If the server does not support Unique-IDs, Message-ID values are used instead.

Sample code description

The sample code below receives new messages on Command1_Click method call. This occurs when user clicks CommandButton named Command1. "Subject:" field of each new message is added to ListBox control named List1.

GetFirstOfNewMessages_UID helper function is used to get message number of the first of new messages in the mailbox. Also, this function returns the list of Unique-IDs of all the messages currently being in the mailbox.

GetFirstOfNewMessages_MID helper function is analog of GetFirstOfNewMessages_UID with only difference that it operates with Message-IDs instead of Unique-IDs. It is used when the POP3 server does not support Unique-IDs.

Besides that, the sample code includes performance optimizations (Unique-IDs are scanned in reverse order), error-checking, Windows events processing.

Any time user can shutdown the application so the connection to mail server will be immediately aborted (POP3.Abort method is called in Form_Unload routine). The application does not "freezes" during performing certain email operation.

License key, mail server name, user account name and password, database filename, and other settings are declared as constants at the top of the source.

Code example:

' MailBee License key
Const LICENSE_KEY = "put your license key here"

' POP3 server name
Const SERVER_NAME = "mail.server.com"

' POP3 server port (usually 110)
Const SERVER_PORT = 110

' POP3 account name
Const ACCOUNT_NAME = "jdoe"

' POP3 account password
Const ACCOUNT_PASSWORD = "secret"

' Path to a file that will be used as database
Const DATABASE_FILENAME = "C:\database.txt"

' Succesful (No error)
Const ERR_OK = 0

' Error: The server does not support Unique-IDs
Const ERR_POP3_COMMAND_NOT_SUPPORTED = 216

' POP3 object declared global because it is used in
' more than one routine.
Dim objPOP3

' If True, email operation was aborted by user
Dim blnAborted



' Displays a message describing the error occurred
Sub ReportError(objPOP3)
  MsgBox "Error #" & objPOP3.ErrCode & "," & objPOP3.ErrDesc & _
    ", Server responded: " & objPOP3.ServerResponse
End Sub



' Returns message number of first of new messages in the mailbox.
' The search is performed using Unique-IDs of the messages
'
' Parameters:
' objPOP3 - POP3 object in the connected state.
'
' strStoredUIDs - CRLF-separated list of Unique-IDs of all messages
'   in the mailbox as of moment of the previous connection.
'
' strCurrentUIDs - (output parameter) filled by this
'   function with CRLF-separated list of Unique-IDs of all messages
'   in the mailbox as of moment of the current connection.
'
Function GetFirstOfNewMessages_UID(objPOP3, strStoredUIDs, ByRef strCurrentUIDs)
  Dim I, J

  ' Array of Unique-IDs of all messages in the
  ' mailbox as of moment of the previous connection.
  Dim arrStoredUIDs

  ' Array of Unique-IDs of all messages in the
  ' mailbox as of moment of the current connection.
  Dim arrCurrentUIDs
  
  ' Convert CRLF-separated list into array
  arrStoredUIDs = Split(strStoredUIDs, vbCrLf)

  strCurrentUIDs = ""
  
  ' Get Unique-IDs of all messages from POP3 server
  arrCurrentUIDs = objPOP3.Search

  If objPOP3.ErrCode = ERR_OK Then
    ' Successfully got Unique-IDs array from POP3 server
  
    ' Convert array into CRLF-separated list
    strCurrentUIDs = Join(arrCurrentUIDs, vbCrLf)

    ' Iterate through old messages from last to first
    For I = UBound(arrStoredUIDs) To LBound(arrStoredUIDs) Step -1
      ' Lookup old message in current messages
      For J = UBound(arrCurrentUIDs) To LBound(arrCurrentUIDs) Step -1
        If arrStoredUIDs(I) = arrCurrentUIDs(J) Then
          ' Old message was found in the list of current messages.
          ' All messages after found message are new.
          GetFirstOfNewMessages_UID = J + 1
          Exit Function
        End If
      Next
    Next
    
    ' All messages have been scanned, but no matches with old
    ' messages have been found. All messages in the mailbox are new.
    GetFirstOfNewMessages_UID = 1
  ElseIf objPOP3.ErrCode = ERR_POP3_COMMAND_NOT_SUPPORTED Then
    ' Non-fatal error, must rollback to Message-ID
    GetFirstOfNewMessages_UID = 0
  Else
    ' Fatal error
    GetFirstOfNewMessages_UID = -1
  End If
End Function



' Returns message number of first of new messages in the mailbox.
' The search is performed using Message-IDs of the messages
'
' Parameters:
' objPOP3 - POP3 object in the connected state.
'
' strStoredMIDs - CRLF-separated list of Message-IDs of all messages
'   in the mailbox as of moment of the previous connection.
'
' strCurrentMIDs - (output parameter) filled by this
'   function with CRLF-separated list of Message-IDs of all messages
'   in the mailbox as of moment of the current connection.
'
Function GetFirstOfNewMessages_MID(objPOP3, strStoredMIDs, ByRef strCurrentMIDs)
  Dim I, J

  ' True if new message was found
  Dim blnNew

  ' Message object that holds message headers
  Dim objMsg
  
  ' Array of Message-IDs of all messages in the
  ' mailbox as of moment of the previous connection.
  Dim arrStoredMIDs

  ' Convert CRLF-separated list into array
  arrStoredMIDs = Split(strStoredMIDs, vbCrLf)

  strCurrentMIDs = ""

  ' No new messages found to the moment
  blnNew = False

  ' No new messages yet, so mark that first new
  ' message is outside the valid range.
  GetFirstOfNewMessages_MID = objPOP3.MessageCount + 1

  ' Iterate through all messages in the mailbox
  For I = 1 To objPOP3.MessageCount

    ' Get message headers from POP3 server
    Set objMsg = objPOP3.RetrieveSingleMessageHeaders(I)
  
    If objPOP3.ErrCode <> 0 Then
      ' Return error
      GetFirstOfNewMessages_MID = -1
      Exit Function
    End If
    If Not blnNew Then
      ' New messages not found yet

      ' Initially assume the message is new
      blnNew = True

      ' Try to find just retrived Message-ID
      ' in the array of old messages.
      For J = LBound(arrStoredMIDs) To UBound(arrStoredMIDs)
        If arrStoredMIDs(J) = objMsg.MessageID Then
          ' Message-ID was found, the message is not new
          blnNew = False
          Exit For
        End If
      Next
      If blnNew Then
        ' Message-ID was not found, the message is new.
        ' All subsequent messages will be new too.
        GetFirstOfNewMessages_MID = I
      End If
    End If

    ' Add Message-ID to the list of Message-IDs of current messages
    If Len(strCurrentMIDs) > 0 Then
      strCurrentMIDs = strCurrentMIDs & vbCrLf & objMsg.MessageID
    Else
      strCurrentMIDs = objMsg.MessageID
    End If
  Next
End Function



' Checks for new messages and displays Subject field
' of each new message in the ListBox.
Sub Command1_Click()
  Dim objMsg, I
  
  Dim fso, f
  
  ' strStoredIDs - CRLF-separated list of IDs (Unique-IDs or
  ' Message-IDs, if Unique-IDs are not supported) of all
  ' messages in the mailbox as of moment of the previous connection.
  Dim strStoredIDs
  
  ' strStoredIDs - CRLF-separated list of IDs (Unique-IDs or
  ' Message-IDs, if Unique-IDs are not supported) of all
  ' messages in the mailbox as of moment of the current connection.
  Dim strCurrentIDs
  
  ' Message number of first of new messages. This message is newer
  ' than any old message, and older than any other new message.
  Dim lFirstNew

  Set objPOP3 = CreateObject("MailBee.POP3")
  objPOP3.LicenseKey = LICENSE_KEY

  ' Make user interface responsive
  objPOP3.EnableEvents = True

  ' Connect to POP3 server
  If objPOP3.Connect(SERVER_NAME, SERVER_PORT, ACCOUNT_NAME, ACCOUNT_PASSWORD) Then
  
    ' Open text file that holds the database
    Set fso = CreateObject("Scripting.FileSystemObject")
    Set f = fso.OpenTextFile(DATABASE_FILENAME, 1, True)

    If Not f.AtEndOfStream Then
      ' Read array of IDs from the file
      strStoredIDs = f.ReadAll
    Else
      ' The file was empty
      strStoredIDs = ""
    End If

    f.Close
  
    ' Determine first of new messages using Unique-IDs
    lFirstNew = GetFirstOfNewMessages_UID(objPOP3, strStoredIDs, strCurrentIDs)

    If lFirstNew = 0 Then
      ' Unique-IDs are not supported by the POP3 server

      ' Determine first of new messages using Message-IDs
      lFirstNew = GetFirstOfNewMessages_MID(objPOP3, strStoredIDs, strCurrentIDs)
    End If

    ' Exit the routine if user wants to close the app
    If blnAborted Then Exit Sub

    If lFirstNew < 0 Then
      ' Report error to user
      ReportError objPOP3
    Else

      ' Loop from the first of new messages to the end of the
      ' mailbox. If lFirstNew > objPOP3.MessageCount, the loop
      ' is executed zero times (no new messages were found).
      For I = lFirstNew To objPOP3.MessageCount

        ' Download entire message
        Set objMsg = objPOP3.RetrieveSingleMessage(I)
        
        ' Exit the routine if user wants to close the app
        If blnAborted Then Exit Sub
  
        ' Display Subject of the message
        List1.AddItem objMsg.Subject
      Next
  
      ' Open the text file for writing
      Set f = fso.OpenTextFile(DATABASE_FILENAME, 2, True)
      
      ' Store IDs in the file
      f.Write strCurrentIDs

      f.Close
    End If

    ' Disconnect on finish
    objPOP3.Disconnect
  ElseIf Not blnAborted Then
    ' Report connection error to user
    ReportError objPOP3
  End If
End Sub



' Initialize
Sub Form_Load()
  blnAborted = False
  Set objPOP3 = Nothing
End Sub



' Abort operation when user wants to exit the app
Sub Form_Unload(Cancel As Integer)
  If Not objPOP3 Is Nothing Then
    blnAborted = True
    objPOP3.Abort
  End If
End Sub