Architecture for IP Application Services over HF Radio Point to Point, Multicast and Broadcast

Applications & Services over HF Today

The range of applications used extensively in military HF deployments is quite limited. The main applications are:

1. ACP127. This provides formal military messaging and is the BRASS application. Broadcast operation is “direct to modem” whereas point to point will usually use STANAG 5066.
2. Battle Force Email (BFEM) is specified as Compressed File Transfer (CFTP) in STANAG 5066. BFEM provide simple support of informal SMTP messaging for HF.
3. HF Operator Chat is specified in STANAG 5066 and used for basic real time chat between operators.
4. Link 11/16/22 is an independent family of C2 protocols used over HF.
5. Custom C2 applications are also commonly used, particular for “man pack” type deployments.

Applications are quite limited and closely tied to the HF systems.

HF Challenges & STANAG 5066

HF communication provides some difficult challenges for building applications:

• Low Speed. Narrow Band HF range is 75 – 9600 bps.
• High Latency gives more technical difficulties than the low speed. There are a number of reasons for high latency:
• Physical switching limits.
• Transmission synchronization, including crypto sync.
• Use of long interleavers (which reduces data loss) increases latency, particular with synchronous crypto.
• Use of Annex L and Annex K to enable frequency sharing.
• Traffic load as a consequence of low speeds or congestion.
• Connection Quality can be poor and highly variable over different time ranges.
• Frequency Selection is tricky as HF propagation is unpredictable and varies with time and location.
• Limited Frequencies. There are only 9,000 HF 3kHz channels, most of which will not propagate in a given scenario.
• Antennae selection: shore stations need to select best antenna for sending to and receiving from a given destination or set of destinations.

STANAG 5066 is a link layer optimized for HF that sits between modem and applications. It has been designed to present a service that enables applications to operate as efficiently as  possible over HF and enables multiple applications to share underlying HF communication.

Broadcast, EMCON & BRASS

HF radio is a broadcast medium and it is important to enable applications to make use of this. The BRASS service is based around a core broadcast service, in addition to three types of point to point link that generally operate over STANAG 5066 (Ship to Ship; Ship to Shore; and Maritime Rear Link).

A key goal of BRASS broadcasts is to transmit messages to ships that are in EMCON (emission control) and so will not be transmitting. This is achieved by:

• FLASH messages are sent three times to minimize risk of loss.
• The broadcasts are sent on multiple frequencies (typically 4-5) to maximize probability that they are received by a given ship.
• Transmissions are slow (typically either 300 or 600 bps) to minimize risk of data corruption.

This is an important and distinct service that needs to be supported as we move to new services.

Goals for Deployment of Modern IP Applications & BRE1TA

Most applications currently being run over HF are quite tightly coupled to HF systems. It is desirable to use a range of modern applications over HF. Modern applications invariably operate of IP and so this desire to run modern applications over HF may be thought of as running IP over HF. This paper looks at the best way to achieve this.

There are a number of HF technology developments that can facilitate this:

• Wideband HF that can enable speeds of up to 240 kbps, by using up to 16 3kHz channels.
• Advances in Automatic Link Establishment (ALE) technology allows faster linking and thus improved frequency selection and utilizations.
• New work on STANAG 5066 will enable improved application use of the underlying HF channel. A summary of potential changes is given in the Isode white paper [STANAG 5066 Update Plan].

While these are useful enhancements, it should be remembered that fundamental HF characteristics remain unchanged and these new developments do not make an HF channel behave like an Ethernet.

It is desired to take advantage of improved performance and characteristics to give a better service over higher speed WBHF links. However, when the HF service is slow (down to 75bps narrow band) it is important that basic services can be maintained.

BRASS deployments have some specific additional requirements and these are addressed in the NATO BRE1TA (BRASS Enhancement One Technical Architecture) program. There is a need to replace the ageing ACP127 technology and continue key broadcast and EMCON services and to introduce new applications into the BRASS environment.

IP Models of Service Deployment

There are two models for deploying IP Applications over HF: "direct" and "proxy". This sections looks at these approaches.

IP Directly over STANAG 5066

Standard IP applications communicate "end to end" as shown below. A middle layer protocol is always used between the application and the IP layer which is invariably either TCP or UDP. Other middle layer protocols can be used, and the architectures discussed here could be applied. This paper focusses on TCP and UDP, which are the most commonly used end to end protocols.

IP datagram packets are sent end to end over a series of subnets. The above diagram shows four subnets (two LANs, a WAN and HF). Subnets are interconnected by IP routers that switch IP packets between subnets.

STANAG 5066 Annex F defines a subnet mapping of IP onto STANAG 5066, which enables and HF network to act as an IP subnet. IP datagrams may be mapped onto STANAG 5066 ARQ (reliable) or onto non-ARQ (not reliable). This is a clean and flexible integration of HF into the standard IP architecture.

Unfortunately, this architecture generally gives poor performance. The broad reason is that the application and middle layers generating IP packets are decoupled from the HF subnet and so there are no clear mechanisms to optimize performance. More detailed considerations:

• There is no flow control between the HF subsystem and the point where the decision is being made to send or to not send packets. This makes is difficult to ensure that the HF network is used efficiently. This is a particular problem for bulk transfer protocols operating over UDP, such as ACP 142.
• The TCP windowing mechanism interacts very badly with HF Subnetwork characteristics. The Isode whitepaper [Performance Measurements of Applications using IP over HF Radio] gives performance measurements, showing utilizations as low as 1%. This is problematic, as most IP applications run over TCP.
• IP and TCP have overheads which are significant at lower and intermediate HF speeds. IPv4 header is 20-60 bytes, IPv6 header is 40 bytes, and TCP headers are 20-60 bytes. Note that TCP needs to exchange PDUs that do not carry data (e.g., SYN and ACK) which also incur both IP and TCP overheads.
• There is no viable mechanism for prioritizing data within and between applications. This is a particular problem when the HF link becomes congested, which is likely given that it is interconnecting faster networks and applications. Without priority controls, a situation can easily arise where lots of applications are trying to send data and none is working usefully.

There are some situations where the direct architecture can give reasonable performance, for example:

• Where the application load is light relative to the HF link and HF conditions are good and TCP parameters have stabilized or been pre-configured.
• For streaming video where data rate is less than link speed and no other data is flowing.

Although the architecture can perform reasonably, in general it will not.

Alternate IP Routes

An architectural benefit of HF Subnet is that where there are multiple routes to a destination (e.g., HF and Satcom) that IP packets can use different routes dependent on network availability. This should give resilience and transparent failover. In practice this will rarely be achievable.

Proxy & Edge Server Architecture

The second architecture is to make use of proxies to bring control closer to the HF network. There are two variants of this architecture.

Application Proxy Architecture

An HF application proxy works by terminating the application protocol at the edge of the HF network and then running an optimized application protocol over the HF network. For example for a messaging, the end application might talk SMTP over port 25 to the application proxy. This will then get converted to one of the HF optimized messaging protocols described later (e.g., ACP 127 over COSS over STANAG 5066).

Middle Layer Proxy Architecture

An application proxy will optimize performance for a given application, but it is impractical to provide an application proxy for every application. A middle layer proxy terminates TCP or UDP at the HF boundary and then uses an optimized protocol over the HF link. This approach makes TCP/IP viable over an HF link and gives useful performance and control benefits for UDP/IP. This approach will work for some applications but not for all. For example a TCP application with extensive handshaking and/or short application timeouts will not work well. For such applications, and application proxy is needed.

Routing & Transparency

The use of HF Proxies can be made completely transparent to the end applications. Domain Name Service (DNS) lookup is used to determine the IP address of the final destination. IP packets will be routed to the IP address of the destination, but will be picked up by the HF Proxy. This requires that the HF Proxy works at IP level (effectively as an IP router).

A simpler approach, appropriate for smaller systems would be to use either table based address lookup or separate DNS on each side of the HF to ensure correct routing of traffic to the proxies.

Which Model to Choose?

It is clear from technical and operational analysis that use of proxies is the best way to go. The rest of this paper will analyze the proxy approach in more detail. Although the direct approach can work reasonably in some specialized configurations, the proxy approach will always give superior performance.

Application Proxies Directly using STANAG 5066

Application proxies will give the best performance, and so it will make sense to specify application proxies for the most critical applications where optimum performance is required.

Military Messaging

Messaging is a critical service and the sole application in current BRASS deployments. There are two standard IP protocols for military messaging:

1. STANAG 4406 using STANAG 4406 Annex A is the NATO standard for military messaging over IP.
2. SMTP messaging operates over IP and military extensions can be provided using RFC 6477 as described in [Military Messaging (MMHS) over SMTP].

Either of these protocols can be used on the IP side of the proxy. There is a choice of protocols operating over STANAG 5066 that can be used on the HF side of the proxy.

1. ACP 127 over COSS, as defined in STANAG 5066 Annex F.
2. Battle Force Email (BFEM) using CFTP specified in STANAG 5066 Annex F provides a basic SMTP service.
3. STANAG 4406 Annex E over ACP 142 is the NATO standard for constrained bandwidth operation.  A mapping onto STANAG 5066 is specified in STANAG 5066 Annex F.
4. MULE (Multicast Email) over ACP 142 provides advanced SMTP and uses the same STANAG 5066 mapping as STANAG 4406 Annex E.

MULE offers the following benefits over BFEM:

• Use of ACP 142 enables multicast operation and supports EMCON.
• Integrates cleanly with STANAG 4406 messaging and shares STANAG 5066 configuration.
• Supports NOTARY, which enables control of Delivery Status Notifications, and in particular the ability to request positive DSNs.
• Support 8BITMIME and BINARYMIME which will optimize performance for binary data.
• Removes the 128 MByte maximum message size.
• Enables higher priority messages to "overtake" lower priority messages that have started transmission. While CFTP can support this, it may only be used with peer agreement (which is a nuisance to configure), and is not generally supported.
• Extensible choice of compression algorithm, which provides a framework for significant performance optimization in some deployments, by use of custom compression or custom compression libraries.
• Slightly smaller messages, due to a more compact encoding.
• Use of MTA to MTA acknowledgments is a cleaner protocol architecture, and protects against message loss in some scenarios.

Isode's M-Switch products can operate as a messaging proxy and support all of the above protocols.

Services Operating over Messaging

Some applications can be usefully deployed over messaging, which avoids the need to define specific HF protocols. In particular:

• File transfer as described in the whitepaper [ File Transfer by Email].
• Directory Synchronization/Replication, as described in the whitepaper [Directory Replication by Email and over 'Air Gap'].

XMPP

Real time text messaging is increasingly important for military deployments and XMPP is the NATO standard for this. XEP-0365: Server to Server communication over STANAG 5066 ARQ specifies how to operate XMPP over STANAG 5066. This approach eliminates handshaking and reduces data volumes relative to the standard XMPP server to server protocol.This enables low latency XMPP communication over HF, which would not be possible with the standard protocols and a TCP PEP Proxy. Isode's M-Link product implements standard XMPP protocols as well as XEP-0365, and so can act as an application proxy for XMPP.

Voice

Voice is a critical service for HF. Current systems will use HF links for either voice or data but not both. The faster HF networks provided by WBHF give opportunity to mix voice and data. While it will be possible to use standard IP protocols for voice (VOIP) there will be clear benefit to develop an optimized proxy protocol for voice support that will allow data sharing and ensure sensible use of the HF link.

Middle Layer Proxies

TCP PEP Proxy

The architecture for using a proxy with TCP is well understood and is set out in "Performance Enhancing Proxies Intended to Mitigate Link-Related Degradations" (RFC 3135), which is the "Transport Layer PEP" that is the focus of most of the RFC. Communication between the end applications and the proxy will use standard TCP and the standard application will operate end to end.  TCP PEP is commonly used on Satellite networks.

Communication between the proxies will need a special HF protocols that is designed to support TCP proxy over HF. This could be a proprietary protocol or a new open standard operating over STANAG 5066. There are some current PEP implementations, which have used proprietary protocols.

UDP Proxy

A similar architecture is also useful for UDP. A basic benefit is that the overhead of IP and UDP headers can be significantly reduced.   Further benefits can be taken by control of behavior based on UDP port. A basic choice is use of ARQ (reliable) mapping:

• For protocols supporting management, such as SNMP or DNS, it makes sense to use ARQ data, to ensure that the data is transferred across the network.
• For real time protocols such as Voice or Video, a non-ARQ (unreliable) mapping is preferable as you want to maintain order of transfer and smooth delays.

Another option that will be chosen on a per-application basis is handling of queues:

• For low volume management and control protocols it will make sense to queue data for a significant period of time, to ensure that it gets sent.
• For real time data, it will be preferable to drop data from the queue relatively quickly to prevent undue buildup of traffic.

It may also make sense to have application specific behavior in the UDP proxy. For example, DNS queries will be repeated if there is no answer. In order to avoid duplicate traffic over the HF link, the sender can discard duplicate requests and the receiver can repeat sending over the second link if no answer is received.

Icon PEP

Isode is developing Icon PEP, a future product that will support both TCP PEP Proxy and UDP Proxy. Isode will publish the proxy protocols used by Icon PEP for communication over STANAG 5066 and will make them available to NATO as input to possible future standards for supporting proxy architectures.

Traffic Selection & Prioritization

The HF network will generally sit between much faster networks. There is potential for congestion when HF performance degrades and/or more load is applied than can be carried. To deal with this, it is important that traffic is prioritized. When bandwidth is limited, it is preferable to block lower priority traffic than to send some data and have the application fail to work.

STANAG 5066 Level

STANAG 5066 provides flexible support for priority based control. All STANAG 5066 data is marked with a priority and STANAG 5066 ensures that data is handled so that higher priority data is sent first. This is particularly beneficial for applications such as military messaging where messages are marked by priority (ROUTINE; FLASH; etc) and this priority can be reflected down to the STANAG 5066 level. This will ensure that the highest priority data will be sent first and lower priority data will only be sent when the higher priority data has been handled.

There are standardized controls (Rank) that handle relative priorities of applications. This can be used to set relative priority of different applications, for example to ensure that real time XMPP traffic is given priority over bulk messaging data. This baseline approach can be improved upon by vendor-specific controls.

TCP PEP Proxy & UDP Proxy

The TCP PEP and UDP proxies can also apply controls to ensure sensible priority handling of TCP and UDP applications which will be identified by the port used. Desirable controls include:

• Ensuring that only selected application are allowed and other applications blocked.
• Control of relative priorities of different applications.
• Allow applications based on minimum link speed (for example, there is not point trying to do streaming video the HF network speed is 600 bps).
• Control maximum number of connections for a given type of application (for example, allowing too many voice calls would lead to none being usable).

Capabilities of this type will be associated with specific products and do not need explicit standardization.

ALE & Link Sharing

Older HF and BRASS systems rely on extensive operator support to function. In a modern environment with IP Applications there is an expectation that communication works with a minimum of operator support. ALE (Automatic Link Establishment) and ALM (Automatic Link Maintenance) are key technologies to enable this.

Point to Point ALE

Basic use of ALE enables establishment of point to point links. When a node wishes to communicate with another node, the first step is to establish and ALE link between the nodes. Then STANAG 5066 communication using CAS-1 for ARQ links can take place between the nodes.

ALE enables a frequency to be chosen that enables communication; a key HF problem is the propagation is not predictable. ALE also allows a set of nodes to share multiple HF frequencies, so that pair-wise communication between multiple pairs of nodes can happen at the same time. This is a good baseline architecture for basic communication.

Sharing Frequencies with Annex K & Annex L

There is commonly a shortage of viable HF frequencies at any given time. STANAG 5066 provides two mechanisms to address this by enabling  a set of three or more nodes share a single frequency:

• Annex K specifies a CSMA (Carrier Sense Multiple Access) approach, which gives a simple mechanism to allow multiple nodes to share a frequency and minimize risk of conflict. This approach is suitable for situations where there is relatively low utilization of the frequency.
• Annex L specifies a Token Ring mechanism, which provides a mechanism where each node transmits in turn. This approach is good where there is high channel utilization.

These mechanisms can be used with ALE or fixed frequency. Using ALE to negotiate a frequency for a set of nodes to use enables the system to adapt to changing conditions and select an optimal frequency for the set of nodes to use.

ALE for Shore Stations

A typical BRASS shore station will have split transmit and receive sites with a number of antennae at each site. This allows for multiple simultaneous HF communications with ships. Where there are more ships than antennae/modems (the usual case) there is a need to share modems between ships. Older systems will typically manage this allocation under operator control. For a modern systems, it is desirable to manage links with ALE and for automatic selection and switching.

To make this work, there is a need to correctly route application queues to the correct peer and allocate modems in turn to each peer. To address this, it is necessary to have a STANAG 5066 server support multiple modems and ALE units. This approach is described in more detail in the Isode HFIA Presentation "USING ALE WITH STANAG 5066" (February 2017).

Multicast, Broadcast & EMCON

Broadcast and EMCON are key requirements for BRASS deployments, which are not supported by general purpose application protocols. These requirements need to be addressed by applications designed for HF and constrained network environments.

ACP142 & Multicast

ACP 142 "P_Mul – A Protocol for Reliable Multicast Messaging in Constrained Bandwidth and Delayed Acknowledgement (EMCON) Environments" enables broadcast and multicast messaging and is key to replacing ACP 127 which is currently used for BRASS messaging. ACP 142 with STANAG 4406 Annex E is the NATO standard approach for constrained networking. ACP 142 can also be used with SMTP messaging using the MULE standard.

ACP 142 enables efficient unicast and multicast messaging over HF with direct operation over STANAG 5066 specified in STANAG 5066  Annex F. Key benefits that an ACP 142 approach gives over ACP 127 include:

• Support for modern messaging features including:
• Attachments. (ACP 127 is text only).
• Message forwarding.
• Delivery Reports and Read Receipts.
• Digital Signatures
• Multicast message transmission (ACP 127 has broadcast, but not multicast). This gains useful efficiency when messages are sent to multiple HF destinations.
• Checksums to ensure integrity. (ACP 127 gets this for point to point transfers by use of COSS over STANAG 5066, but broadcast is direct to modem).
• Rapid acknowledgements and error retransmissions for unicast and multicast messages (ACP 127 broadcast relies on hourly recap messages).
• Compression, which gives improved performance relative to ACP 127.

ACP 142 over STANAG 5066 can be deployed over Annex K (CSMA) or Annex L (Token Ring) networks, which may be operated fixed frequency or ALE.

his provides an efficient and reliable basis for unicast and multicast HF military messaging, which can be deployed as an application proxy to support end systems using standard IP messaging protocols (STANAG 4406 Annex A and/or SMTP).

Broadcast & EMCON

Multicast messaging is where a message is sent to an explicit set of destinations with reliable delivery ensured by explicit acknowledgement for each destination. Broadcast messaging is where messages are sent out without explicit expectations of acknowledgement.   Not requiring acknowledgement is essential to support EMCON as ships in EMCON will not be able to send acknowledgements.

ACP 127 broadcast just sends out messages which are each addressed to one or more ships. This uses a circuit which just broadcasts. Every hour, the sender will broadcast a “recap” of all the messages sent in the last hour. This will enable ships to work out if any messages have been missed and they can send requests to retransmit (if they are not in EMCON).

Broadcast and EMCON can also be supported by ACP 142 using a broadcast only transmission. ACP 142 breaks (compressed) messages into PDUs so that the PDUs can be retransmitted. Messages will usually have all PDUs transmitted a number of times on a broadcast network to minimize risk of loss. If a message recipient is not in EMCON it may ACK the message or NACK missing PDUs using a ship to shore link established by ALE to send these ACKs/NACKs. Where a message is ACK’d to all recipients, there is no need for further retransmission.

This support gives an approach to move forward from ACP 127 and improve broadcast and EMCON services with ACP 142 operation over STANAG 5066.

Frequencies for Broadcast

ACP127 broadcasts are usually sent out over several frequencies (typically 4 or 5) so that ships can choose which frequency provides the best reception for them. Transmissions may be measured using OTAM, which uses a ground station positioned at "reception distance" from the transmitter. The data received is compared to the data sent, in order to determine reception quality. This measurement is used to help select the best set of frequencies to broadcast on.

A similar approach could be used to manage broadcast frequencies for ACP 142. Because data broadcast using STANAG 5066 would have checksums on each DPDU, a reception station can measure reception quality by checking the checksums to determine Frame Error Rate, without an need to compare data with the sending stream. This would simplify this quality testing.

Another approach would be to use ALE to select the broadcast frequencies to be used. 3G ALE provides a generic mechanism to set up broadcast frequencies and 2G ALE defines two distinct mechanisms:

• Allcall: this sets up an ALE link without any responses.
• Anycall: this sets up an ALE link with randomized responses.

Isode notes that there are options as to how this is managed, but does not make any recommendation.

Mobile HF Support

HF transmissions cover long distances and wide areas, but exact coverage depends on many factors. Countries operating HF services will often have a number of sites that give different coverage and countries collaborate so that facilities with different coverage can be shared as ships and other mobile units move to different locations. Making this communication work is referred to as "mobile support".

This section describes how the application proxy model can provide mobile support using messaging as an example. The diagram above illustrates the problem. There is a system originating a message that needs to send to a ship which is moving between two national and a partner HF network.

A very simple approach might have the originating system send the message with SMTP. When the ship is in range on one of the HF networks, the associated messaging application proxy can be configured to send messages over HF to the ship. DNS can then be configured using MX records so that the originating system will send messages for the ship to the correct application proxy.

While this approach might be viable for switching between national networks, it is unlikely to work for partner networks due to intervening firewalls and message guards. The details of a system to support this is going to need to take many factors into consideration. A likely pragmatic approach is that routing configured for a ship should always take it to a default national location; this will ensure that movement of the ship impacts a minimum number of systems. Then message routing for the ship will need to be configured on this default system, and on the message application proxy that handles the ship. The default system should explicitly route the message at SMTP level to the desired message application proxy. This will mean that intermediate guards do not need to have routing modified. Support for this is likely to need the default messaging system and the application proxies to have flexible routing configuration.

Conclusions

This paper has covered a broad range of topics related to supporting IP Applications over HF and moving from BRASS to BRE1TA. A proxy architecture is strongly recommended, and a mix of application, TCP PEP and UDP proxies is needed.