Sodium Sync

Syncs are configured and scheduled using a Wizard interface which offers immediate access to three common directory to directory sync profiles, four common LDIF transformations as well as access to an Advanced Wizard view giving fine-grain control of the synchronization process. Simple syncs occur as independent events but more complex scenarios exist where it makes sense for syncs to have relationships to each other and to external events. Sodium Sync allows for the grouping of syncs and external events into a Directory Synchronization Workflow. For more information see the page on sync configuration, scheduling and workflow.

Sodium Sync incorporates extensive functionality addressing the complexities of data transformation & mapping, merging & correlation encountered when synchronizing data from multiple sources and in certain scenarios. You can read more about this on the page on data transformation, mapping and merging.

Sodium Sync’s primary goal is to synchronize directories supporting LDAP or X.500 DAP access.

LDIF (LDAP Data Interchange Format) defines a text format for representing directory data. LDIF files will generally be used to hold data corresponding to a directory subtree. Sodium Sync treats an LDIF file as equivalent to an LDAP or DAP directory subtree, and an LDIF file can be used as source or target for a sync. LDIF can be used to represent a set of actions on a directory, which can also be consider as a ‘delta’. The term ‘change LDIF’ is used to distinguish this type of file from an LDIF file that represents a sub-tree. Change LDIFs are used to support Sync by Email. Change LDIFs can be useful for ad hoc directory management, and so Sodium Sync enables loading of change LDIF, and comparison of two sources and generation of a change LDIF to represent the delta.

There is often the requirement to handle data from other sources with directory synchronization, Sodium Sync provides support for data import and export using two widely used interfaces:

• CSV (Comma Separated Value) format files (RFC 4180).
• SQL Databases.

Sodium Sync shares directory access with Isode Sodium directory management GUI, and details are shared with Sodium. Sodium offers a number of security options for the two primary protocol access mechanisms (X.500 and LDAP).

There are a number of situations where normal directory replication protocols cannot be used. For example:

• In constrained bandwidth environments such as HF radio, where performance will be poor.
• Across secure boundaries, where directory replication and access protocols may not be used.
• At firewalls with ‘air gap’ requirements.
• Over data diodes.

Sodium Sync provides a number of related solutions for these environments. Email is often a practical communication mechanism when directory replication cannot be used, so directory replication over email is a key building block.

Sodium Sync provides capabilities to generate LDIF changes relative to an automatically stored reference copy. This enables Sodium Sync to generate a sequence of change LDIFs, and for another copy of Sodium Sync to robustly apply them, checking for duplicates, missing updates and out of order changes. It can also drive ‘transport’ programs before or after it runs. This provides capability to replicate between directory servers using email, or to provide directory replication across an ‘air gap’ gateway.

More information on synchronization by email and scenarios that require it are given in the whitepaper [Directory Replication by Email and over ‘Air Gap’]. Information on using this capability with M-Switch is given in the whitepaper [File Transfer by Email]. Sodium Sync can also be used to perform directory replication over a data diode, to support directory replication in secure environments with one way data flow.

Commonly used Sodium Sync features are available for GUI configuration. Custom features can be configured using XML templates, with further extensibility by use of scripting languages including Javascript.

Attribute syntax checks and custom data mappings can be defined in XML. XML provides a flexible mechanism for mapping information, that includes selection and transformation of attribute values using regular expressions. When this is insufficient, mappings can be extended by the user of scripting

Sodium Sync supports scripting languages using the JSR 223 interface. Javascript is built in (with sample profiles written in Javascript provided) and a wide range of other scripting languages can be loaded including JACL, jRuby and jython. This support enables complex mappings to be specified, for example mappings requiring access to an external database.

The basic operational mode of Sodium Sync does a “full update” on each run. This works by reading data from both directories, and then applying any necessary changes to the target directory. This is robust, straightforward, and works with any LDAP or DAP directory. Sodium Sync works by streaming data, and minimizes the amount of data it holds at any time. This enables it to scale to synchronize very large directory information trees. On a modern Core i5 machine, between fast directory servers such as Isode’s M-Vault, typical performance for Sodium-Sync is around 300 entries per second. This makes it practical to synchronize several thousand entries with updates at very short intervals.

New syncs are configured via a wizard which offers immediate access to a number of synchronization options:

• 3 common directory to directory syncs.
• 4 common LDIF transformations.
• Go straight to Advanced view
• Group syncs into a directory replication workflow.

Some of these sync options will lead to a Simplified view of the Sync Wizard, consisting of Source & Target and Scheduling tabs. Others will go directly to an Advanced view which has additional tabs, as shown in the following screenshot.

The front profile screen gives a workflow diagram of the core functions, the diagram will adapt as different options are chosen in the tabs which define options, including those described below.

Mode

A sync can be configured to operate in a number of different modes:

• Source Only: This sync operates without a target, the primary use of this mode if for checking data.
• Complete Scan: The standard mode of operation used for directory to directory syncs. Source and target (which can be independently configured as directory, LDIF, CSV or SQL) are compared and changes applied to the target.
• Cached Scan: This varient uses a cached copy of the target directory, which the sync will update. This has the advantage of removing the need to read data from the target, which reduces network overhead. This mode is essential for sync over email and related modes where it is not possible to read data from the target directory.
• Recreate: Where a standard sync will calculate changes and apply only necessary changes to the target, this mode will delete the target and fully load each time.
• LDIF: This mode takes a change LDIF (an LDIF representing a set of changes to be applied to a directory) as input.
• Queues: This mode takes as input a sequence of LDIF file and is designed to support sync by email and related modes

Source and Target

Source and target tabs control where data is coming from and where it is going to. Where to source or target is a directory server, the server is referenced by a bind profile that is configured to connect to that directory at the point of the DIT (directory information tree) used. For the source, options exist to handle aliases either as a mechanical copy or be de-reference (so that the target gets the data from the entry that is pointed to by the alias).

Checks

A number of checks can be configured including a referential integrity check, constraint on update size and a check that a given attribute is unique. Attribute syntax checks can also be configured by defining custom XML.

Output

The Output tab gives a number of options for the final step of the sync:

• Discard changes
• Apply changes to the target (this is the standard approach).
• Apply changes to another directory.
• Generate a change LDIF file.
• Generate a change LDIF file for a queue, this is in support of synchronizations by email.
• For a source-only scan, the set of output options is LDIF, CSV and writing to a different area of the source directory.

Trace

The final tab, Trace, gives a number of trace, logging and debug options.

Sodium Sync enables independent scheduling of multiple syncs. Regular syncs can be scheduled daily, hourly or on specified days of the week or month. Syncs can also be specified with an interval so that a sync will start a configurable time after the previous sync has finished.

When unscheduled tasks are run from the Sodium GUI, status and errors are shown interactively. Sodium Sync Manager (below) can be run at any time to check on the status of synchronizations.

When there are problems this screen can be used as an entry point for diagnosing problems and accessing error logs. Any errors are also logged as Isode events. These events can be monitored using any of the approaches available as part of the Isode event system.

Simple syncs occur as independent events but more complex scenarios exist where it makes sense for syncs to have relationships to each other and to external events, for example:

1. To sync data inwards from an external source before sending a sync outwards to multiple targets.
2. Running a database report generation program to generate a file, prior to loading via a sync.
3. Load in data, check its validity, and publish automatically via another sync if the checks succeed. It the checks fail, publication will not happen. This will ensure operator checking, and that the published data has always been checked.

Grouped Syncs

This type of capability is referred to as ‘Directory Replication Workflow’. The first feature to support this is Group Syncs. A sequence of syncs and checks can be grouped together, so that they are run in order and subsequent syncs will only run if the previous ones succeed.  This can help with situations where data is brought in from multiple sources and then sent onwards. It can also bring in remote data, perform processes, and then only transmit onwards if a series of checks succeed.

This enables the definition of a new group and defines error handling proceedures. Groups are shown on the management screen and syncs in a group can be run independently or as a group. Commands can be specified to run before and after a sync, allowing external processes to be integrated with the directory replication workflow.

The Checks tab allows the configuration of a number of standard checks:

• Referential Integrity is applied to all attributes with Directory Name syntax, to verify that they point to a directory entry that exists.
• Update size can be constrained, primarily to ensure that unintentional major changes or deletions are not propagated.
• Check that a given attribute is unique, for example to ensure that no two users have the same telephone number. This is important for syncs into Active Directory as email address must be unique for Exchange.

Attribute syntax checks can also be configured by defining custom XML. Selection of this text is then done in the mapping tab.

Sodium Sync can be used to provide a simple sync process such as from a sub-tree in one Directory (or other data source) to another Directory. However Sodium Sync’s merging and correlation functions allow it to be used to handle more complex setups such as bringing data from multiple different directory services into a single central directory.

While this sort of task can be addressed by setting up a series of independent synchronizations, it’s often desirable to merge data. Sodium Sync supports data merging based on a model that every piece of data has a clear master location, ensuring that the overall state can be determined definitively.

• Every entry must have a directory which is authoritative for naming.
• Every attribute within an entry must have a directory which is authoritative for the value of that entry.

The setup described above assumes consistent naming between all of the sources and target. As this will not usually be the case, Sodium Sync includes correlation functionality needed to provide consistency between data sources. Correlation is also important when using non-directory sources in order to manage the mapping of that data onto a directory structure.

Managed by a GUI Interface correlation allows a specific 1:1 mapping for data items to be configured. Correlations can be extended by the scripting interface to support specialized situations.

A sync correlation report can be generated (above) which shows:

• Correlated entries.
• Entries that were correlated but have become problematic due to data changes.
• Suggested matches (and misses) for data that is not correlated.

Correlated entries can be ignored (if on one side only), discarded or approved.

The following LDAP standards are supported by Sodium Sync.