Receiving new messages from POP3 server (Part 2)

Advanced topics Summary: Presents advanced code sample of receiving new messages. Highlights performance improvements, error-checking, old POP3 servers support and simple implementation of message identifiers database.

Tutorial map:

Part 1 - Getting new messages
Part 2 - Advanced topics

Typical scenario: The web-based mail client checks whether new messages have arrived since last connection and wants to download new messages (if any). The web-based mail client also handles all possible errors.

Limitations: None.

Performance improvements

In Part 1 code sample, Unique-IDs for all the messages in the mailbox are looked up in the database. In real-world situations, mailboxes usually contain only a small number of new messages beside total number of the messages. Another important fact is that new messages are always appended to the end of the mailbox. It's impossible that new message is followed by not new message. In other words:

Number of not new messages is usually larger than number of new messages
Once not new message is found, the mail client can expect that all messages from this message to the beginning of the mailbox are not new

According to the above, it is better to iterate through the messages in reverse order, from the last message to the first. If the message is new, retrieve it and proceed with previous message. Once already downloaded (not new) message found, the entire loop can be stopped - there are no more new messages in the mailbox.

Checking for errors

Connection to the POP3 server can be suddenly broken for a variety of reasons. Well-written application should take care of such situation. Also, your application should consider that not all POP3 servers support the command required for extracting Unique-IDs of the messages - Unique-IDs support is an optional extension of the POP3 protocol.

Search new messages when the POP3 server does not support Unique-IDs

If the POP3 server does not support Unique-IDs, smart mail client may consider using backup method to download new messages. This method uses Message-ID values taken from "Message-ID:" field of the messages' headers. Its disadvantage is the fact that headers for all the messages in the mailbox must be retrieved. This results in larger network traffic (relatively to retrieving Unique-IDs). However, in situation where Unique-IDs cannot be used, this method is still better than downloading all the messages entirely.

The idea is obvious:

Retrieve headers for the first message in the mailbox
Lookup Message-ID of the message in the database. If the Message-ID is not found, the message is new and should be retrieved entirely
In the same manner, proceed with subsequent messages till the end of the mailbox

Simple text-file database implementation for single email account

In the previous part, sample code has no implementation of the database, and IsNewMessage() function always returns True.

In this part, simple database implementation is provided. The database is stored in a text file. Each line of the file holds one Unique-ID value. If the server does not support Unique-IDs, Message-ID values are used instead.

Sample code description

The sample code below receives new messages on ASP page execution. For each new message found, "Subject:" field is printed to the output.

GetFirstOfNewMessages_UID helper function is used to get message number of the first of new messages in the mailbox. Also, this function returns the list of Unique-IDs of all the messages currently being in the mailbox.

GetFirstOfNewMessages_MID helper function is analog of GetFirstOfNewMessages_UID with only difference that it operates with Message-IDs instead of Unique-IDs. It is used when the POP3 server does not support Unique-IDs.

Besides that, the sample code includes performance optimizations (Unique-IDs are scanned in reverse order), and comprehensive error-checking.

License key, mail server name, user account name and password, database filename, and other settings are declared as constants at the top of the source.

Code example:

<%
' MailBee License key
Const LICENSE_KEY = "put your license key here"

' POP3 server name
Const SERVER_NAME = "mail.server.com"

' POP3 server port (usually 110)
Const SERVER_PORT = 110

' POP3 account name
Const ACCOUNT_NAME = "jdoe"

' POP3 account password
Const ACCOUNT_PASSWORD = "secret"

' Path to a file that will be used as database
Const DATABASE_FILENAME = "C:\database.txt"

' Succesful (No error)
Const ERR_OK = 0

' Error: The server does not support Unique-IDs
Const ERR_POP3_COMMAND_NOT_SUPPORTED = 216

' POP3 object
Dim objPOP3



' Displays a message describing the error occurred
Sub ReportError(objPOP3)
  Response.Write "Error #" & objPOP3.ErrCode & "," & objPOP3.ErrDesc & _
    ", Server responded: " & objPOP3.ServerResponse
End Sub



' Returns message number of first of new messages in the mailbox.
' The search is performed using Unique-IDs of the messages
'
' Parameters:
' objPOP3 - POP3 object in the connected state.
'
' strStoredUIDs - CRLF-separated list of Unique-IDs of all messages
'   in the mailbox as of moment of the previous connection.
'
' strCurrentUIDs - (output parameter) filled by this
'   function with CRLF-separated list of Unique-IDs of all messages
'   in the mailbox as of moment of the current connection.
'
Function GetFirstOfNewMessages_UID(objPOP3, strStoredUIDs, ByRef strCurrentUIDs)
  Dim I, J

  ' Array of Unique-IDs of all messages in the
  ' mailbox as of moment of the previous connection.
  Dim arrStoredUIDs

  ' Array of Unique-IDs of all messages in the
  ' mailbox as of moment of the current connection.
  Dim arrCurrentUIDs
  
  ' Convert CRLF-separated list into array
  arrStoredUIDs = Split(strStoredUIDs, vbCrLf)

  strCurrentUIDs = ""
  
  ' Get Unique-IDs of all messages from POP3 server
  arrCurrentUIDs = objPOP3.Search

  If objPOP3.ErrCode = ERR_OK Then
    ' Successfully got Unique-IDs array from POP3 server
  
    ' Convert array into CRLF-separated list
    strCurrentUIDs = Join(arrCurrentUIDs, vbCrLf)

    ' Iterate through old messages from last to first
    For I = UBound(arrStoredUIDs) To LBound(arrStoredUIDs) Step -1
      ' Lookup old message in current messages
      For J = UBound(arrCurrentUIDs) To LBound(arrCurrentUIDs) Step -1
        If arrStoredUIDs(I) = arrCurrentUIDs(J) Then
          ' Old message was found in the list of current messages.
          ' All messages after found message are new.
          GetFirstOfNewMessages_UID = J + 1
          Exit Function
        End If
      Next
    Next
    
    ' All messages have been scanned, but no matches with old
    ' messages have been found. All messages in the mailbox are new.
    GetFirstOfNewMessages_UID = 1
  ElseIf objPOP3.ErrCode = ERR_POP3_COMMAND_NOT_SUPPORTED Then
    ' Non-fatal error, must rollback to Message-ID
    GetFirstOfNewMessages_UID = 0
  Else
    ' Fatal error
    GetFirstOfNewMessages_UID = -1
  End If
End Function



' Returns message number of first of new messages in the mailbox.
' The search is performed using Message-IDs of the messages
'
' Parameters:
' objPOP3 - POP3 object in the connected state.
'
' strStoredMIDs - CRLF-separated list of Message-IDs of all messages
'   in the mailbox as of moment of the previous connection.
'
' strCurrentMIDs - (output parameter) filled by this
'   function with CRLF-separated list of Message-IDs of all messages
'   in the mailbox as of moment of the current connection.
'
Function GetFirstOfNewMessages_MID(objPOP3, strStoredMIDs, ByRef strCurrentMIDs)
  Dim I, J

  ' True if new message was found
  Dim blnNew

  ' Message object that holds message headers
  Dim objMsg
  
  ' Array of Message-IDs of all messages in the
  ' mailbox as of moment of the previous connection.
  Dim arrStoredMIDs

  ' Convert CRLF-separated list into array
  arrStoredMIDs = Split(strStoredMIDs, vbCrLf)

  strCurrentMIDs = ""

  ' No new messages found to the moment
  blnNew = False

  ' No new messages yet, so mark that first new
  ' message is outside the valid range.
  GetFirstOfNewMessages_MID = objPOP3.MessageCount + 1

  ' Iterate through all messages in the mailbox
  For I = 1 To objPOP3.MessageCount

    ' Get message headers from POP3 server
    Set objMsg = objPOP3.RetrieveSingleMessageHeaders(I)
  
    If objPOP3.ErrCode <> 0 Then
      ' Return error
      GetFirstOfNewMessages_MID = -1
      Exit Function
    End If
    If Not blnNew Then
      ' New messages not found yet

      ' Initially assume the message is new
      blnNew = True

      ' Try to find just retrived Message-ID
      ' in the array of old messages.
      For J = LBound(arrStoredMIDs) To UBound(arrStoredMIDs)
        If arrStoredMIDs(J) = objMsg.MessageID Then
          ' Message-ID was found, the message is not new
          blnNew = False
          Exit For
        End If
      Next
      If blnNew Then
        ' Message-ID was not found, the message is new.
        ' All subsequent messages will be new too.
        GetFirstOfNewMessages_MID = I
      End If
    End If

    ' Add Message-ID to the list of Message-IDs of current messages
    If Len(strCurrentMIDs) > 0 Then
      strCurrentMIDs = strCurrentMIDs & vbCrLf & objMsg.MessageID
    Else
      strCurrentMIDs = objMsg.MessageID
    End If
  Next
End Function



' Main part: check for new messages and print their "Subject:" field

Dim objMsg, I
  
Dim fso, f
  
' strStoredIDs - CRLF-separated list of IDs (Unique-IDs or
' Message-IDs, if Unique-IDs are not supported) of all
' messages in the mailbox as of moment of the previous connection.
Dim strStoredIDs
  
' strStoredIDs - CRLF-separated list of IDs (Unique-IDs or
' Message-IDs, if Unique-IDs are not supported) of all
' messages in the mailbox as of moment of the current connection.
Dim strCurrentIDs
  
' Message number of first of new messages. This message is newer
' than any old message, and older than any other new message.
Dim lFirstNew

Set objPOP3 = CreateObject("MailBee.POP3")
objPOP3.LicenseKey = LICENSE_KEY

' Connect to POP3 server
If objPOP3.Connect(SERVER_NAME, SERVER_PORT, ACCOUNT_NAME, ACCOUNT_PASSWORD) Then
  
  ' Open text file that holds the database
  Set fso = CreateObject("Scripting.FileSystemObject")
  Set f = fso.OpenTextFile(DATABASE_FILENAME, 1, True)

  If Not f.AtEndOfStream Then
    ' Read array of IDs from the file
    strStoredIDs = f.ReadAll
  Else
    ' The file was empty
    strStoredIDs = ""
  End If

  f.Close
  
  ' Determine first of new messages using Unique-IDs
  lFirstNew = GetFirstOfNewMessages_UID(objPOP3, strStoredIDs, strCurrentIDs)

  If lFirstNew = 0 Then
    ' Unique-IDs are not supported by the POP3 server

    ' Determine first of new messages using Message-IDs
    lFirstNew = GetFirstOfNewMessages_MID(objPOP3, strStoredIDs, strCurrentIDs)
  End If

  If lFirstNew < 0 Then
    ' Report error to user
    ReportError objPOP3
  Else

    ' Loop from the first of new messages to the end of the
    ' mailbox. If lFirstNew > objPOP3.MessageCount, the loop
    ' is executed zero times (no new messages were found).
    For I = lFirstNew To objPOP3.MessageCount

      ' Download entire message
      Set objMsg = objPOP3.RetrieveSingleMessage(I)
        
      ' Display Subject of the message
      ' We use HTMLEncode to convert '<' and '>' into
      ' their HTML equivalents
      Response.Write Server.HTMLEncode(objMsg.Subject) & "<br>"
    Next
  
    ' Open the text file for writing
    Set f = fso.OpenTextFile(DATABASE_FILENAME, 2, True)
      
    ' Store IDs in the file
    f.Write strCurrentIDs

    f.Close
  End If

  ' Disconnect on finish
  objPOP3.Disconnect
Else
  ' Report connection error to user
  ReportError objPOP3
End If
%>