Directory Replication by Email and over ‘Air Gap’

Directory Replication by Email Overview

The diagram below shows a high level overview of Isode’s approach to performing efficient directory replication using email. Directory data is replicated from one directory server (the supplier) to another (the consumer). Isode’s Sodium Sync product will access the supplier directory servers using LDAP or X.500 DAP, and determine changes made since the last update. These changes are written out to an LDIF (LDAP Data Interchange Format) file. Sodium Sync will then attach the file to an email and send it to one or more recipients. At the consumer directory server end, another Sodium Sync will extract the LDIF file from the email. It will then check that the update is the ‘next in sequence’ and apply the changes to the consumer directory server.

It can be seen that this provides a flexible mechanism for directory replication, and that the volume of data to be transferred is minimized.

This paper looks at the directory components of the solution (the email transfer elements are considered in more detail in another white paper [File Transfer by Email]) and also looks in detail at use of end to end acknowledgements to ensure reliability.

Scenarios

Directory replication by email, and similar approaches using different transfer mechanisms are more complex than use of direct server to server replication such as that provided by X.500 DISP (Directory Information Shadowing Protocol) and M-Vault multi-master replication. This section looks at scenarios where direct server to server replication is not possible or desirable.

HF Radio

HF Radio is an important communications technology for military communication. Directory is critical infrastructure, and so replication across the slow HF links is important. Direct server to server replication would be inefficient, because of the nature of HF Radio. It would not work when a receiver is in radio silence (EMCON – Emission Control), and does not take advantage of the broadcast nature of radio. It would be possible to define a new directory replication protocol working over STANAG 5066, but this would be complex.

Messaging can be efficiently supported over HF Radio, both Internet Mail and STANAG 4406 formal messaging. The messaging infrastructure provides:

• Reliable Data Transfer
• Compression
• Support of multicast and EMCON
• Security – in particular Message Origin Authentication and Content Integrity

Use of email infrastructure to support directory replication is a natural fit.

Satellite

Although standard directory replication can be used over satellite, it does not take advantage of the multicast nature of satellite. The same messaging infrastructure used for HF Radio, but operating over IP rather than STANAG 5066, can be used to provide multicast directory replication. This can be important for supporting servers that have copies of the same data.

Email only Boundary

Security boundaries often restrict access and functionality available over the boundary. Many military networks provide secure email gateways, which may follow ACP145, but do not permit directory protocols.

Where email is allowed, and directory protocols are prohibited, directory replication by email is a natural approach. Some cross domain deployments mandate this approach.

Directory Gateway

A related scenario to the email only boundary is a security boundary where there are stringent security requirements on data flow. In this scenario, there are requirements to check replicated data for things such as:

• Leaks of sensitive information to a lower security environment.
• Checks for malware and security threats within the replicated data.

A system performing such checks would generally need to be security accredited, according to Common Criteria or similar measure. Use of ‘air gap’ replication provides a straightforward way to achieve this.

Although the above architecture is quite complex, it gives a key benefit of making the Security Accredited process that checks the directory data a very simple one. Essentially it needs to perform security checks and copy data across the security boundary. It enables the bulk of the directory replication functionality to not require (expensive) security accreditation. In keeping the accredited process simple, it also increases the overall security of the system.

Tactical Directory

In a tactical environment, directory data is often replicated to many similar directory servers that operate on tactical units that may have irregular network access. Rather than having a supplier directory push updates to each server, email replication can be used to send updates to an email server. Then each tactical unit can retrieve these updates from the email server at a convenient time, and apply the updates locally. If the email server also holds “total updates”, this approach can also be used to restore broken directory servers or to provision new ones. The real benefit of using email here is that it decouples the consumer directory server from the supplier ones, and allows multiple consumes to easily share a single supplier.

The Isode Solution

We now look at how Isode’s Sodium Sync product provides the capabilities described, in order to address the above scenarios.

Architecture

The basic architecture of Isode’s solution separates the directory synchronization component from the underlying transport. The reason for this is to enable this system to be used with a variety of transports, including:

• Internet email, using SMTP submission and transport with IMAP or POP delivery.
• X.400 (STANAG 4406) message transport.
• File copying for ‘air gap’ approaches.
• Custom transport.

To achieve this, Isode provides a clean separation between synchronization and transport, and allows use of Isode provided or customer provided transport mechanisms. This is shown below.

The diagram above shows how sending works. Sodium Sync will access the supplier directory using LDAP or X.500 DAP and determine if any changes have been made, by efficiently comparing data in the directory to a reference copy of the data. If changes have been made, Sodium Sync will generate an LDIF (change) file representing these changes, and place it into a file system directory. Then it will (optionally) run a ‘transport process’ that will cause the LDIF file to be transferred (e.g., sent by email) to one or more consumer systems. The transport process will log activity, and on successful operation (e.g., correct submission of messages to transfer) will rename the LDIF. This means that if there is a temporary failure, that the file can be picked up next time the transport process runs.

The diagram above illustrates the reverse process. Sodium Sync runs a consumer transport process which places LDIF files into a directory (e.g., by retrieving email messages using an access protocol and extracting the LDIF files). Sodium Sync then looks in the directory to find the ‘next’ LDIF change file, based on sequence numbering within the LDIF file name. The overall system needs to deal with:

• Duplicates (e.g., a message being sent twice. Message transfer has ‘at least once’ and not ‘exactly once’ semantics). The current release flags duplicates for operator attention. A future version will check that if duplicate updates (i.e., two with the same sequence number) are identical, and if this is the case will ignore the duplicate.
• Out of sequence. Messages may not be delivered in order. Sodium Sync will not process a change until the previous one has been processed.
• Missing updates. Sodium Sync will report this, and the operator will observe that processing is ‘stuck’. A future version may automatically request resending an update.

Updates are simply applied to the consumer directory, and this keeps the consumer directory up to date.

Monitoring & Logging

Sodium Sync provides a monitoring view, so that an operator can check on progress. This is shown above and includes:

• All configured synchronization agreements and schedule.
• Time each sync was last run and success/failure status.
• Success/failure status of the transport process.
• Status of the underlying transport (e.g., if a delivery report has been received from the other end). (planned)
• Peer acknowledgement. If the underlying transport is bi-directional, application level acknowledgement is shown (planned).

Sodium Sync provides logging of all activity, that may be used for both diagnostic and audit purposes.

Scheduling & Total Updates

Updates may be scheduled to occur at specific times each day, or at an interval (specified in minutes) since the previous update. An LDIF update is always provided, even if there are no changes. This allows a consumer to watch for scheduled updates, and enables it to quickly detect if a scheduled update is not sent. The first synchronization will contain all of the data (a total update).

Theoretically, this initial transfer is the only time that a total update is needed. In practice it is often desirable to use total updates from time to time (for example, a user or operator might delete an entry in the consumer directory) and Sodium Sync gives and option for a total update to be sent immediately or on the next synchronization.

Filtering and Mapping

Sodium Sync includes sophisticated capabilities for filtering, mapping and transforming information synchronized. These are all available for use with directory replication by email, as part of the supplier process. Two are of particular importance:

• Filtering information, such as removing specific attributes, or entries. This is important to minimize transfer of data.
• Moving information to a different part of the directory information tree.

Email transport and End to End Acknowledgements

Sodium Sync sends and receives data as files, which means that it can be used with any underlying transport system that can handle files. Sodium Sync can also run external processes to pull and push files.

Isode provides a system to transfer files by email, described in the white paper [File Transfer by Email], and Sodium Sync integrates with this.

• Testing and demonstration.
• Use in situations not requiring email security.
• A model for customers to provide transports.

Isode plans to provide further transport capabilities in future releases, including:

• SMTP and X.400 (STANAG 4406) email transport products.
• GUI configuration of email transports.
• Message Security based on digital signature. This will include Message Origin Authentication and Content Integrity, and may include Content Confidentiality.
• Message splitting, to handle very large updates.

Isode would like to receive input on product requirements for email transports.

Conclusions

Directory replication over email or air gap can be used to address a number of important specialized scenarios, in particular directory support over HF Radio, tactical directory, and various boundary replication scenarios. Isode’s Sodium Sync product implements directory synchronization over email.