SMTP Message Switch

The (on disk) message queue is the centre of M-Switch. Arriving messages are written to the disk and secured, using fsync() to ensure no possibility of message loss, which is central to reliable store and forward messaging. Messages may then be checked and processed locally, before being sent on and removed from the disk. Typically, messages will remain in the queue for a very short while, often for just a few milliseconds.

Most M-Switch processes are channels. The exceptions are QMGR (see below) and three processes associated with the audit database. Channels perform many functions, including bringing messages into M-Switch, delivering or transferring messages out of M-Switch, and processing messages within M-Switch.

QMGR is the operational core of M-Switch, communicating with channels and with management interfaces. Key QMGR tasks:

• Starting and stopping channel processes.
• Scheduling active processing and delivery of messages.
• Controlling number of connections to the SMTP Server.
• Receiving status information from channels, on connections and message transfer status.
• Responding to information and control requests from management processes.
• Ensuring that higher priority messages are handled first.

To achieve this, QMGR makes use of three protocols:

• An internal channel communication protocol.
• AgentX to enable SNMP monitoring.
• SOM (Switch Operational Management Protocol), which is used by Isode’s management tools and provides strong authentication and data confidentiality using TLS and SASL.

QMGR is a multi-threaded event driven process that holds an internal representation of the status of all queued messages, channels and connections. By handling all of this with a single process that does not need to look at the queues on disk, very sophisticated scheduling, control and status reporting is available. QMGR is designed to efficiently manage very large message queues (hundreds of thousands of messages). More detailed information can be found on the M-Switch Queue Manager page.

sode provides full support for Simple Mail Transfer Protocol (SMTP) as specified in RFC 5321 within relevant M-Switch products, conforming to the Internet host requirements for messaging (RFC 1123) including message submission (RFC 6409). SMTP support includes a number of auxiliary Internet Standard related to general capabilities and Lemonade support. A full list is given on the M-Switch SMTP conformance page.

LMTP (RFC 2033) is a variant of SMTP optimized for delivery to local message stores, also supported by Isode’s M-Box IMAP/POP server.

Message Transfer by SMTP out of M-Switch is provided by the SMTP channel and Message Delivery by LMTP is provided by the LMTP channel. M-Switch may run many instances of each of these channels.

Inbound SMTP is handled by a single multi-threaded SMTP Server process (channel), so that new SMTP connections do not require new processes.

Message submission uses the SMTP protocol. The Internet Standard for Message Submission (RFC 6409) gives a special port for this (587), distinct from the SMTP port used for message transfer (25). Message submission is frequently authenticated, whereas this is unusual for message transfer. The SMTP Server can be configured to listen on both ports, or two SMTP servers can be used (one for each port).

Security for SMTP includes TLS (using START-TLS) and SASL based authentication, supporting a wide range of security options including client strong authentication using SASL EXTERNAL.

The Message Submission Library provides a standard internal service for putting messages onto the message queue, which ensures that per message functions such as authentication, routing, and processing calculations are applied consistently and correctly. Key functions provided by the submission library are:

• Sender validation.
• Recipient validation and message routing.
• Determining what internal processing is needed.
• Policy and authorization checks.
• Writing the message to disk.

M-Switch generally uses directory configuration. Table based configuration is also provided, as it is useful for some specialized deployments, particularly where a very simple static configuration is needed. The M-Switch GUI for managing directory configuration is MConsole, further information can be found on the configuration management page. Table based configurations are managed by editing text files.

M-Switch configuration is used by all of the M-Switch processes. Logically, configuration information from the directory is used by all processes. As changes to configuration are generally infrequent, this is implemented by QMGR obtaining configuration information from the directory using LDAP, checking for updates, and sharing it with the other processes.

In order to optimize performance, the number of channels running, number of each channel type running, number of connections to a single MTA, will vary dependent on many factors, including remote system, network and local hardware performance. Manual optimization is complex, and in any event the optimal configuration is likely to vary over time. For this reason, M-Switch does a sophisticated “self test”, to measure performance and adapts configuration accordingly. This enables M-Switch to adapt its operational parameters to different hardware and network environments.

Routing of messages to other SMTP servers uses the domain part of the email address and is generally handled using the DNS (Domain Name System), but may also use locally configured host resolution. M-Switch also allows for connections to be configured to specific SMTP hosts using MConsole. Reasons for doing this include:

• Internal routing that overrides DNS based routing.
• Configuring security options, such as TLS for data confidentiality and SASL for peer authentication using passwords or strong authentication.
• Configuring one or more permanent SMTP connections, which will reduce message switching latency by removing the time needed to establish a connection.

M-Switch can also be set up to route based on the whole email address. This is referred to as LASER routing, after the Internet working group that developed a specification with this name. M-Switch broadly follows that specification, with a number of extensions, and options to deal with directory servers that store information in a different way.

The key concept in LASER routing is that you search in an LDAP directory for a mailbox of the form joe.user@mycorp.com. Routing succeeds when the search for the address returns in exactly one result for an Entry which holds suitable delivery information. Additional attributes in the entry can control with channel and host the message is then sent to. Typically, the match will use a default LMTP connection to a local M-Box server. This architecture works well for organizations with large “white pages” directories.

There are three basic ways that LASER routing may be used in conjunction with domain based routing:

• Single domain system, such as a consumer ISP or enterprise. If the domain is local, LASER routing is used, otherwise domain routing is used.
• Hosted service, handling multiple domains on one server. LASER routing is used to a single directory server. If there is a match, the message is handled according to the match. If not, domain routing is used for authorized message submission.
• Boundary switching to multiple departmental servers. Domain routing looks to identify locally configured domains. A match will identify the departmental LDAP server against which to do the LASER lookup, and the SMTP server to which matched addresses should be routed. This approach allows the boundary switch to fully validate email addresses by making use of existing LDAP servers.

M-Switch has flexible configuration that can support these setups and others. In addition to the core LASER routing, M-Switch also provides:

• An aliasing mechanism.
• A mechanism to specify alternate mailbox addresses.
• A list channel, that expands managed distribution lists.
• Sub-addressing support (joe+XXX@myorg.com).

Internet messages are structured according to the MIME (Multipurpose Internet Message Extensions) Standard defined in RFC 2045. M-Switch has a mimeshaper channel that understands MIME format and can perform conversion on messages and components within messages.

Individual body parts can be converted from one to another using conversion filters. Typically this is used for converting a text body part from one character set to another. The necessary conversions are calculated when a message is first submitted and they may be re-evaluated when a message is ‘exploded’.

It is often desirable to rewrite header information – in particular, to ‘normalize’ addresses by rewriting the address in some canonical form, rather than one of the multiple addresses that can be used to reach a specific recipient. Mimeshaper provides options for the normalization of Internet message headers. This capability can be used to provide a coherent view of addresses for local users, or to manage addresses to give an external view in a boundary messaging configuration.

Reputation services provide a mechanism for organizations originating messages to provide information that enables message recipients to verify that the message comes from its claimed source. Ideally, all email should make use of reputation services. In practice, reputation services can help differentiate in anti-spam checks, by making reputation information available as input to the anti-spam checks. M-Switch supports two reputation services.

DKIM

M-Switch provides DKIM (DomainKeys Identified Mail) signing of messages, to verify the originating domain and message integrity. This provides a digital signature across message content and selected message headers to provide secure reputation support, which can be used to help protect against phishing attacks and spam.

SPF

SPF (Sender Policy Framework) makes use of DNS (Domain Name Service) configured information. Setting up SPF is part of DNS configuration, independent of M-Switch. M-Switch can perform SPF checks on inbound messages. There are two approaches to handling:

1. Reject messages at the SMTP server when SPF checks fail.
2. Mark the message with a special header, which can be used in subsequent anti-spam checks.

Operationally, the second approach is usually more useful.

M-Switch provides anti-virus checks on some or all messages being handled, using third party anti-virus packages. The following anti-virus packages are supported:

• Sophos, a commercial product that is optimized for this type of checking. This can be purchased directly from Sophos.
• ClamAV is an open source anti-virus checker, that is specifically designed to target email-borne viruses and malware.

Setup of M-Switch to use the anti-virus package of your choice is straightforward.

What does M-Switch do to support Anti-Virus checking?

The basic function of M-Switch to handle viruses is very simple. It takes an inbound stream of SMTP messages, separates out the message content to hand to a virus checker, and then sends the messages onward by SMTP (once they have passed the virus check). M-Switch can be easily inserted into an SMTP message stream, to add anti-virus capability. The more detailed process is:

• M-Switch has the concept of “channels” which perform specific functions on messages in the internal queue. A content checking channel drives the anti-virus capabilities which M-Switch uses. This is programmable, so different content checking channels may be invoked (by the same instance of M-Switch) with different parameters in different situation, or even with different virus checkers.
• M-Switch can be configured to invoke the anti-virus checking on all messages, or on selected messages (e.g., “all inbound”, “all outbound”, “all messages from organization X”, “all messages to user X”).
• M-Switch can control virus checking by size. In particular, virus checking can be skipped for very small messages (which are common and will be too small to carry a virus).
• The virus checking can do various things on detecting a virus, including one or more of:
  • Sending a customizable message back to the sender
  • Sending a customizable message on to the intended recipient (example below)
  • Removing the infected body part, and then replacing it with another body part (typically one that says “there was a virus infected thing here”)
  • If the virus checker can clean up the virus, the channel can replace the infected body part with a clean one
  • Generate a non-delivery report to the originator of the message
• The virus checking audit logs all activity, which can be processed into management reports as needed.
• Anti-virus statistics can be displayed in MConsole, when the audit database is used.
• The virus channel has a framework which can be used with any virus checker that provides an API or command line interface. Integration is straightforward. While the virus checker is usually run on the same machine as M-Switch, it can also be set up to run remotely.

S/MIME (Secure MIME) is a widely used standard (RFC 5751) for providing end to end message digital signature based on X.509 PKI and message encryption.

M-Switch products can check S/MIME signatures on message submission, to validate message integrity and origination. These checks are integrated with the authorization system, so messages can be controlled based on signature presence and validity.

S/MIME headers may be signed following the S/MIME procedures. When this is done M-Switch marks the S/MIME signed headers following “Considerations for protecting Email header with S/MIME“. This enables correct reverse mapping by M-Switch when header signing is used. There is also an option to not sign the message header, which gives better interoperability but poorer security.

M-Switch content conversion can strip S/MIME signatures, and can add an S/MIME signature. So M-Switch can be used to add S/MIME signatures at a boundary, or to add signatures where clients cannot do this, to provide onward message integrity services. This content conversion works without restriction on messages that are signed but not encrypted.

For triple-wrap encrypted messages, M-Switch can validate and convert the outer S/MIME signed wrapper. Isode’s Harrier product can generate S/MIME triple wrap. However, M-Switch cannot decrypt or generate triple wrap S/MIME. This feature may be added in a future version of M-Switch

S/MIME Encryption is supported by M-Switch Encryption, which is a capability that may be added to all M-Switch products. This supports S/MIMEEnveloped encryption and STANAG 4406 triple wrap encryption.

M-Switch products use Transport Layer Security (TLS) for data confidentiality and Simple Authentication and Security Layer (SASL) for authentication. SASL is also used to map simple identifiers onto directory names for authentication. A wide range of SASL authentication mechanisms are supported, including GSSAPI (Kerberos). SASL and TLS are supported for SMTP Message Submission, and may be optional or mandatory.

SASL authentication identifiers are usually managed in the directory, but may also be configured in an XML database, which may be suitable for small deployments. TLS, SASL and S/MIME Cryptography may be configured to be compliant with FIPS 140-2.

Operator Authentication & Rights

M-Switch provides authentication for both configuration and operation. Users are authenticated against the directory. Isode’s recommended model is that (human) users authenticate as themselves, and that each user is given appropriate rights. The benefit of this approach is that all actions can be audit logged according to the user (not the role) which improves accountability when activity is analysed.

Configuration is held in the directory, and so the operator will authenticate to the directory. Directory access control is used to give users full rights on the configuration, read only, or no access. Operational access uses the SOM protocol, which uses SASL authentication against the directory, so that common authentication is used for configuration and operation. Operator rights for SOM usage are configured using a directory attribute.

M-Switch products include a comprehensive rule based authorization mechanism, for controlling which messages can be sent and how they are sent. Capabilities include controls based on:

• Sender and recipient, including address pattern matching.
• Destination channel, which enables the control of protocols and options used.
• Message size and message priority.
• S/MIME signatures and security labels.
• Submitting IP address and sub-network and on submitting host and authentication.
• Subject Indicator Code (SIC) count.
• Security classification.

Recipient Addition

M-Switch products provide an authorization controlled mechanism to add additional recipients to a message. This is implemented as an extension to the Archive Capability, so that inbound messages may be “archived” by sending email to a configured recipient. This can be used for archiving, but also provides a mechanism to get selected messages sent to additional recipients. The recipient addition capability can also be used for a variety of purposes, including:

• Sending select messages to an additional backup address.
• Monitoring traffic to or from selected local users or remote destinations.

M-Switch provides support for Security Labels in a number of formats:

• ESS Labels using the standard RFC 2634 encoding in S/MIME.
• ESS Labels, base64 encoded in a configurable X- Header
• Text encoded labels, represented in X-Header, Subject for First Line of Text (FLOT).

M-Switch can convert between label formats, using a Security Policy (SPIF) based approach to map between the various supported label formats. This conversion can also use configured equivalent labels to map between different Security Policies.

M-Switch can make authorization decisions based on presence and validity of a security label. It can also make Access Control decisions (checking against Security Clearance in the context of a governing Security Policy) based on destination channel, MTA, and user. For more information see the whitepaper [Security Label Capabilities in M-Switch].

M-Switch message processing, subsequent to message arrival, is controlled by the Queue Manager. This processing includes transfer-out or delivery, and also other tasks for messages, such as content conversion, delivery report generation and removing messages from the on-disk queue.

M-Switch has a multi-process architecture, giving some significant advantages:

• It is more robust, as the different tasks which the MTA must perform are separated into appropriate modules. Failure in one part does not affect other processes.
• It is also more flexible, as arranging concurrent processing is straightforward, using operating system provided features.

A multi-process architecture can have some disadvantages. One issue is that the startup cost for a process can be large. There is both the operating system cost of creating a new process, and there is the application cost for initializing this process and perhaps creating inter-process communication links with other processes. Another issue arises if there are many processes competing for system resources. This can become very inefficient. The operating system can spend a significant amount of time switching between process contexts.

The primary task of the Queue Manager is to start processes which perform the required functions on messages, and pass messages for processing to these processes. A single process can handle more than one message and can handle messages for transfer to different MTAs. It can close the association with one MTA and open an association with another.

Another Queue Manager function is the handling of non-permanent errors on message processing tasks, tasks which can be re-tried at some point in the future. The Queue Manager uses a ‘back-off’ strategy for this. On each failed attempt, the next attempt is scheduled for a time in the future. The time between attempts increases linearly with the number of attempts, up to a maximum time interval.

When errors occur on one peer MTA, this does not prevent the transfer of messages to other peer MTAs. Similarly, if a message-specific error arises for one message, this does not prevent the transfer of messages for the same peer MTA.

The routing process assigns to each recipient of a message the next-hop MTA (or for local delivery) and a channel. The channel is the object which processes the appropriate protocol. A channel runs as a separate process started by the Queue Manager. The Queue Manager can run more than one process for a given channel.

The Queue Manager organizes the messages in a tree structure. The different channels are at the top. Attached to each channel are the peer MTAs reached by that channel. Then attached to each MTA is a list of “Mlists”. An Mlist is the internal name given to the list of recipients for a particular message to be transferred to a particular peer MTA.

If a message has multiple recipients and the recipients are to be transferred to different peer MTAs, there are multiple Mlists. These are treated separately, as if from separate messages, so the Mlists can be processed in parallel. Whether there is parallel processing depends upon the scheduling decisions made by the Queue Manager.

If there are several Mlists available of the same priority but for different MTAs on a given channel, the Queue Manager will choose for processing an Mlist for the MTA which has the smallest number of messages currently being processed.

If there are no processes running for a channel, then the Queue Manager will need to start such a process if there is at least one Mlist for that channel.

If there is at least one process running for a channel, but there are Mlists for the channel which are available for processing, then the Queue Manager has a choice: start a new channel process or wait for one of the currently running channel processes to finish the current message it is processing, so that it can then process another message. This is the fundamental scheduling decision.

It is apparent that there are trade-offs to be made here between the rapid processing of messages and the use of system resources. There are two extremes:

1. One process per channel: This makes good use of system resources, but it means that messages for the channel are processed only serially.
2. A process for each message: This is very inefficient, in that the competition for system resources is high, and the system may thrash. Also, if there are messages for the same peer MTA, this may have problems with many messages being transferred in parallel.

The optimum throughput for the MTA lies somewhere between these two extremes. There is some point when increasing the number of channel processes does not gain any increase, and may decrease the throughput as there is contention for various resources. However, it is hard to calculate or discern the actual optimum. It depends significantly on a number of factors, including the character of the traffic, for instance number of messages for different MTAs, the size of the messages, and also factors like the speed and latency on network links to peer MTAs.

The Queue Manager in M-Switch currently uses some simple heuristics for making the scheduling decisions. These were derived some time ago and have been well tested in operation, having been used in the MTA under a number of different workload patterns. This mechanism has proved robust.

Number of channel processes

The first control is over the overall number of channel processes which can be run (qmgr_maxchans). This has the effect of limiting possible thrashing. However, the limit on processes depends upon the message priority. The Queue Manager is prepared to run more processes when the priority of the available message is urgent, than when it is non-urgent. The effect of this is that, even if no more processes would be started for a non-urgent or normal priority message, if an urgent message arrives, a new process for that message can be started. The number of channel processes, and the number to reserve for urgent (qmgr_reserve_urgent) and normal priority messages (qmgr_reserve_normal) can be configured by the MTA administrator.

Start rate of processes

The second control is over the rate at which the Queue Manager starts new processes for a given channel. The idea here is that if the channel processes the messages quickly, then it is more efficient to have few process than to have many processes competing for resources. There are two different rates which apply here, which are expressed as time intervals from the time when a channel process has been started. The first time interval (qmgr_chan_time2) is a short time, and no new processes for the channel will be started in this time. The second, longer time interval for preventing additional channel processes (qmgr_chan_time3) applies if the load on the channel is not great. Both these times can be configured.

Single message control

The third control stops the Queue Manager continually starting and stopping channel processes for single messages, which would be very inefficient, given the system cost of process creation. This is done by another time interval (qmgr_chan_time1). If all processes for a channel are stopped, the Queue Manager will not start another process within this time. This gives time for messages to accumulate for the channel, which then can be processed more efficiently by a single process. Again, this time can be configured.

If the Queue Manager is running different channel processes, and the total number of processes is restricted, there is a mechanism which attempts to balance the processes between the different channels. That is, if there is demand for processes, some channel processes will be stopped (when they have finished processing the current message).

The trade-off is between starting processing messages quickly (by decreasing the time between allowed channel process starts) and processing messages efficiently (by using a smaller number of processes to avoid system resource contention). This balance is best determined by tuning a system with the actual load.

Also, tuning the maximum number of processes is also best done in the light of actual circumstances. If large messages are transferred over slow links, the system will be less stressed with many processes (which are spending most of the time waiting for the remote end to respond), than if short messages are transferred over fast links, when a smaller number of processes will be more efficient.

Basic queuing theory tells us that if the average load (i.e. the message arrival rate) does not exceed the system capacity, then the average queue size of messages which can be processed immediately will be small (there may be queuing of messages which cannot be processed as a result of, for example, failure to connect to a peer MTA). Under these circumstances, the choices made by the Queue Manager in fact have little effect. Messages are processed in a timely fashion.

The main circumstance where you see the effect of the choices made by the Queue Manager are when there is an abnormally high arrival rate. The Queue Manager then has to adjust to attempt to clear the resulting queue in a timely fashion. We believe the current mechanisms for this are a reasonable and produce good results for average message throughput.