STANAG 5066 IP Client & Providing IP Services over HF Radio

What is IP Client

IP Client provides a mechanism for transferring IP packets over an HF subnet following the classic IP architecture shown above. It achieves this by a simple mapping of IP packets on to the STANAG 5066 Unidata service.

UDP Protocols

UDP (User Datagram Protocol)  is a other primary layer protocol over IP that provides an unreliable datagram service directly over IP.   IP Client is the natural mechanism to support UDP protocols.

DNS

DNS (Domain Name Service) is the major protocol that operates over UDP. It can be important for Mobile Units to look up domain names of shore services. Mobile Unit domain information will typically be available on shore side services, so the reverse direction is less important.

DNS is an important application to run over IP Client.

SNMP

SNMP (Simple Network Management Protocol) is a significant standard application that operates over UDP. The use case for SNMP is less clear, as the benefits of performing monitoring over and HF link do not seem significant.  If SNMP were used over HF,  IP Client would be the best approach.

Other Standard UDP Applications

Some use of UDP has been made by specialized peer to peer applications. It is unclear if these or other UDP applications are of interest to operate over HF.

It seems likely that there are only a limited number of standard UDP applications of interest to run over HF.

Custom Military Protocols 

A number of custom military protocols operate over UDP.  These protocols typically treat UDP as a reliable datagram protocol (which it is not) to transfer small messages.    These protocols can sensibly be transferred over HF using IP Client with ARQ mode.

ICMP & ICMP Ping

ICMP (Internet Control Message Protocol) is an infrastructure protocol that is important for IP operation.  Most ICMP based protocols do not make sense to transfer over HF, although any ICMP traffic over HF should use IP Client.

An exception is ICMP Ping, which is useful for testing and determining peer status. ICMP Ping works well over IP Client.

Other IP Services

This section covers a range of applications for which IP Client is not the best approach.   It summarises the best approach and rationale for the approach.

Key Applications: Messaging and XMPP

Some mission critical applications define mappings directly onto STANAG 5066. There are a number of options for messaging protocols, as set out in Isode whitepaper [Messaging Protocols for HF Radio]. Performance of these protocols can give better than 90% link utilization as described in the Isode whitepaper [Measuring Performance of Messaging Protocols for HF Radio].

The approach for XMPP is described in the Isode whitepaper [Operating XMPP over HF Radio and Constrained Networks]. This approach has been demonstrated in UK MoD trials to work well down to 300 bps, with low latency and high reliability.

These optimized protocols obviate the requirement to operate these services using IP Client, which would be significantly less efficient.

TCP and Web Services

TCP is the dominant protocol for applications running over the Internet, and it is clearly important to operate a range of standard and special protocols over TCP (e.g., Database Access).

HTTP (Web) operates over TCP. Web is important in its own right, particularly for access from Mobile Units to shore. Many standard and most modern proprietary protocols operate over HTTP, rather than directly over TCP.

The best approach for TCP based services is to use STANAG 5066 Annex X “(HF-PEP: TCP Performance Enhancing Proxy Protocol” .  Technical summary and analysis is provided  in the Isode white paper “Providing TCP Services over HF Radio”.

Support of TCP and Web applications is a key target for operation over HF.

QUIC

QUIC is a new protocol developed by Google for fast Web access using UDP.

It performs very badly over IP Client.  If it was desired to support QUIC over HF Radio, a new proxy mechanism would be needed.

General Purpose Multicast Protocols

Use of multicast protocols has been a research topic, but in practice, IP applications are deployed point to point. Multicast protocols over the Internet remains a research topic. Specialized multicast protocols for most applications are used on a single subnetwork (e.g., ARP – Address Resolution Protocol) or streaming media distribution. There does not appear to be benefit in operating such protocols over HF.

IP Routing Protocols

IP routing protocols operate over IP in various ways (TCP, UDP, Special), dependent on routing protocol choice. It does not make sense to operate these routing protocols directly over HF.  This is discussed in detail in the Isode white paper “IP Routing over HF”.  The recommended approach to IP Routing is the HF-RIP protocol specified in STANAG 5066 Annex AB (“HF Routing Information Protocol – HF-RIP”).

Voice, Video and Streaming Media

Traditionally Voice has been deployed over HF by dedicated use of the link.

Use of Streaming Video has been made to demonstrate IP operation over WBHF. This shows the link capacity, with received video delayed and buffered. While this can be made to work, it does not seem to be a sensible operational approach. It would be preferable to record the video and then transfer it as a file (e.g., as an email attachment). This will make more efficient use of the link and avoid issues of the stream being impacted by outages or reduced performance.

Interactive Voice and Video conferencing are attractive services for the target audience. Superficial analysis suggests that operation of VoIP, Video and other IP streaming media services over HF using IP Client is not going to work in a useful manner. This is not considered further in this paper. Deployment of Voice and Data simultaneously over HF and basic Video over HF are topics that Isode believe warrant serious investigation.

Other IP Applications

There are other protocols that operate over IP, some of which are used for streaming media and routing (covered separately). We are not aware of any other protocols over IP that may be useful to operate over HF.

Notes on IP Client Operation

IP Client is a simple and straightforward specification. IP packets are mapped simply to STANAG 5066 Unidata. Both IPv4 and IPv6 is supported. Use of either ARQ or non-ARQ services are allowed. Some specific observations are made in the following subsections.

IP Client Overhead

IPv4 has a per packet overhead of 20-60 bytes (IPv6 has a fixed 40 bytes overhead, but a different extension mechanism may lead to other overheads). Although this is not particularly large for what IP is doing, it is a significantly larger overhead than typical for protocols that are optimized for HF. TCP headers are a further 20-60 bytes, which applies to both data and control packets. These overheads can be significant

ARQ and non-ARQ

IP Client allows operation using both ARQ and non-ARQ services. Annex U allows both options and recommends to have the choice configurable.

In Order

STANAG 5066 allows specification of data to be delivered “In Order” or ‘As They Arrive’ for ARQ. Non-ARQ is always ‘As They Arrive’.

MTU Size

An IP packet is carried in a single STANAG 5066 Unidata (max size 2048 bytes).  The MTU (Maximum Transmission Unit) of an IP packet over HF must be small enough for the complete PDU to fit into a standard STANAG 5066 Unidata.  Common setups with 500 and 1500 byte IP data fit comfortably.   IP Routing configuration must ensure that this is the case.   To facilitate this, an IP Client implementation is required to support MTU path discovery, which will ensure that compliant implementations will observe MTU limits.

Priority

IP Client can set the STANAG 5066 priority of each IP packet, which will control the relative priority to be given to each packet relative to other IP packets and other applications.

TTL

IP Client can control the Time To Live (TTL) for each IP packet sent over STANAG 5066.  For services such as ICMP Ping, it is desirable to set this to a short value so that old traffic does not cause congestion at 5066 layer.   For services where it is desired to maximize reliability, a longer TTL is desirable.

Queuing Strategy

STANAG 5066 can flow control an IP Client Service, which will lead to IP packets getting queued. At some point, it will become sensible to discard packets.   The choice of when to do this and which packets to discard is an implementation choice, as it does not impact IP Client interoperability.

Configuration Options

For each IP packet handled,  and IP Client implementation can make a number of choices.  A basic choice is whether to discard or transmit a packet.   Then choices can be made for all of the parameters noted above:  ARQ vs Non-ARQ; Priority; TTL;  Queueing strategy.

The choices are an implementation choice, as it does not impact IP Client interoperability.   STANAG 5066 Annex U recommends to make this configurable.  Factors in this choice can include general conditions (e.g., current transfer speeds) and information on the IP packet including: Destination Address; IP Protocol (e.g., ICMP, GRE, UDP, etc); UDP Port.

Addressing

IP Client needs to direct each IP Packet to the correct IP peer using the STANAG 5066 address of the peer.  This is essentially an IP routing choice, as the packet is being directed to the next IP router.

A simple approach is configuration by IP Subnet, with each subnet mapping to a STANAG 5066 address.   A more sophisticated approach would be to use the HF-RIP routing protocol specified in STANAG 5066 Annex AB.

Measurements

Test Architecture

The above architecture is used to measure performance. Components as follows:

1. HF Network simulated by Isode’s MoRaSky tool.
2. Icon-5066 (Isode STANAG 5066 product) provides STANAG 5066 service.
3. Icon-PEP (Isode product) provides IP Client. It connects two IP Routers to provide an IP Subnet service over HF.
4. LH router connects to test host.
5. RH router connects to Internet.

The following tests can be run from the LH Host:

1. ICMP Ping to any node on the RHS.
2. DNS Lookup using NSLookup tool to the Internet.

STANAG 4539 was used for speeds of 9600 bps and below. STANAG 5069 with 48 kHz bandwidth was used for the 240 kbps tests. All tests used short interleaver.

Use of ARQ

For most services, it is clear that reliability is needed and ARQ is the sensible way to achieve this. For this reason, most measurements make use of ARQ. Where measurements do not use these options, this is explicitly noted.

ICMP Ping

Ping tests were performed at a range of speeds using ARQ. 5 pings were done at each speed, with a gap between each ping so that only one ping was outstanding.

Notes on these results:

1. Initially 75 bps was planned as lowest speed, but it was not possible to get this to work at all. It was not clear why pings failed at this speed.  Therefore 300 bps was used as the bottom speed. 150bps was not tested.
2. It is clear that ICMP Ping, which makes very simple use of IP can work over a wide range of speeds.
3. As noted below, it proved difficult to eliminate all “background” IP traffic from the test system. There was a low level of IP traffic being generated in addition to the test traffic. We believe that this had the following impact on test results:
   1. It is likely the reason that tests at 75bps did not work.
   2. It means that a CAS-1 soft link was permanently open, so that this traffic did not have the overhead of establishing a CAS-1 soft link.
   3. The interaction with this traffic is a likely explanation for the wide variation of response times.
4. The 9600bps seem slow, in context of the other results. The reason for this is unclear.
5. The results show that basic use of IP Client is viable across a wide range of HF and WBHF speeds.

The following test was made with non-ARQ:

This showed a response time that was better than ARQ at the same speed but was broadly in line with the ARQ results. The improvement in performance of non-ARQ is not understood.

DNS

DNS (Domain Name System) measurements were made using the standard nslookup tool to look up the isode.com domain from the public Google DNS server (8.8.8.8) accessed over the test HF network. Response times were measured with manual stop watch.

DNS needs reliable response, so ARQ mapping was chosen.

Notes:

1. DNS worked across the speed range with acceptable response times in line with ping results. The times are dominated by turnaround effects, rather than data transfer.
2. Apart from the fastest speed, response times are significantly longer than the typical DNS repeat timer (5 seconds on Linux, 1 second on Windows).   This means that queries and responses were sent over the network several times. This is undesirable, particularly at lower speeds.

Notes on IP Client for Target Applications

The following sections provide some analysis and notes arising from the measurements made.

ARQ

It is clear that services such as DNS need reliable data and that use of ARQ is a preferable way to achieve this to using DNS repeat transmissions. Similarly, it is desirable to not lose ICMP messages.

For most target applications, it seems clear that use of ARQ is the best choice.

IP MTU Size & IP Fragmentation

The IP MTU needs to be small enough so that all IP packets can fit into a 2048 byte STANAG 5066 maximum Unidata size. Given ARQ transmission, keeping IP MTU as large as possible is desirable to minimize relative overhead of IP and TCP headers. An MTU of 1500, which is the standard Ethernet size, seems a sensible choice.

We found in tests that applications (running on LAN) would select much larger IP MTU sizes, which were then fragmented to go over HF. We view that use of IP fragmentation is undesirable. For IPv6, fragmentation is not allowed. Icon-PEP constrains MTU size to the specification and support MTU discovery. However, if an application uses a larger MTU, the router we used performs fragmentation. This appears difficult to avoid in practice. IP fragmentation did not appear to significantly affect performance.

IP Client Queuing

When STANAG 5066 SIS service flow controls IP Client, the IP Client can choose to queue or discard incoming IP packets.

Discarding packets is problematic for TCP performance, as TCP retransmission is control by RTO (Retransmission Timeout) which is three times the RTT (Round Trip Time). The RTT is very large for HF, which means that retransmissions will take a long time and lead to delays. Retransmission will also lead to reduction of TCP window size, which will reduce throughput.

When there is no data loss, window size increases. We observed that most of this increased window size is handled by OS buffering. The IP Client implementation needs to consider how to handle STANAG 5066 flow control. It is clear that IP packet discard should be avoided for the reasons noted above.

However, large IP queues are problematic. This is particularly the case when delays lead to application timeouts, when transmission of the queued IP packets will lead to problems,

Isode’s basic strategy with Icon-PEP is to queue everything, to avoid the problems of discard. Then there are configurable limits to queue size and oldest queued IP packet. When these limits are exceeded, the entire queue is discarded. At this point, the application over IP will need to deal with the loss. Making all of the losses at one point, will minimize the number of such resets. This will typically be done by TCP layer, but UDP applications such as DNS will generally retransmit if there is no response.

CAS-1 Soft Link Failure

Under poor conditions and extended fades, the CAS-1 soft link that under-pins ARQ data transfer will break. The STANAG 5066 server will reject all queued data.

Icon-PEP strategy is based on the likelihood that CAS-1 failures will often be long enough to lead to application failure. So, Icon-PEP will not retransmit rejected messages. It will also discard all queued IP packets for the STANAG 5066 peer. This approach forces recovery up to the level above IP. If TCP is the layer above IP Client, it will handle this recover, but performance is likely to be significantly degraded.

Spurious IP Traffic

The test configuration was set up with one node as a Mobile Unit, with default routing to the “shore” system. This is desirable, as it will lead to correct behaviour of IP applications on the Mobile Unit. A good deal of IP traffic just appeared over the IP link. This included:

• Routers sending ICMP traffic to validate link up state.
• Applications on the (default Centos 7) Linux server sending UDP packets to all sorts of IP addresses.

This traffic was surprisingly difficult to eliminate. On a fast link, this traffic would not be noticed. It would impact a slow HF link.
Traffic can be eliminated by any of:

• System configuration to not generate the traffic.
• Router configuration.
• Icon-PEP configuration.

This needs to be considered when setting up an IP system operating over HF. IP connectivity between networks is designed to be “always up” and this would impact operation if the intent is “on demand” behaviour.

Performance Summary

Notes on performance for the following applications:

1. ICMP.
   1. Traffic levels are expected to be low, and IP Client is suitable.
   2. ARQ mapping recommended.
2. DNS.
   1. ARQ mapping recommended.
   2. IP Client performance acceptable at higher speeds.
   3. Duplication of queries and responses undesirable.
3. UDP Point to Point Applications.
   1. ARQ mapping likely to be best.
   2. IP Client likely to be suitable.

Icon-PEP support of IP Client

Icon-PEP contains  Isode’s implementation of IP Client.  Icon-PEP is descibed in the product overview https://www.isode.com/product/performance-enhancing-proxy/.

Icon-PEP provides flexible rules to configure traffic handling.  An example rule is shown below, that specifies that NTP (Network Time Protocol) packets should be discarded.

Conclusions

This white paper has described how IP Client (STANAG 5066 Annex U) works and presents measurements.   It also described how Isode’s Icon-PEP product supports IP Client.  It shows that IP Client works well for UDP Protocols and ICMP Ping.

Measurements of TCP using IP Client validate that this performs poorly and that the best approach for supporting TCP is HF-PEP (STANAG 5066 Annex X).