Measurements of Skywave HF Radio Intermediate Term Variation and Implications for Optimizing Link Performance

Background

STANAG 5066 and Application Performance Optimization

The nature of an HF Radio channel has characteristics that need consideration at every level from radio to application in order to achieve optimal application performance. STANAG 5066 provides a flexible link level optimized for HF Radio, and in particular to deal with the long turnaround times arising from the simplex nature of HF radio communications. Applications need to be optimized to operate efficiently over STANAG 5066, and this is considered in other Isode whitepapers. This paper looks at optimizing STANAG 5066 performance, while considering key application requirements, in particular low latency for interactive applications such as XMPP and optimizing bulk data throughput for applications such as messaging.

This problem has been considered in detail in the Isode whitepaper [Optimizing STANAG 5066 Parameter Settings for HF & WBHF], which provides analysis based on standard channel models of HF Radio Performance. In particular it looks at the CCIR models which are considered to be a good representation of HF Skywave and AWGN (Additive White Gaussian Noise) which is considered to be a good representation of HF Groundwave.

The whitepaper did some theoretical analysis of ITV and concluded that measurements were vital to draw any useful conclusions. This whitepaper describes measurements of this type and builds on the analysis of the earlier whitepaper.

Intermediate Term Variation

HF radio signal variation is classed in three groups:

1. Short Term Variation. These are changes which last for fractions of a second or a few seconds, such as Rayleigh Fades. It is generally agreed that short term variation is well represented by standard HF models and the HF modems are optimized to deal with short term variation. In particular use of long interleavers can effectively remove the effects of short term variation.
2. Intermediate Term Variation. This is variation from 10 seconds to a few minutes, which are considered in this paper. ITV will interact significantly with typical HF transmissions which for STANAG 5066 last from a few seconds to just over two minutes. These transmissions use fixed parameters, which need to be selected before the transmission starts. This whitepaper builds on considerations in the previous whitepaper to optimize this choice.
3. Long Term Variation. This is variation of order minutes or hours. It is generally anticipated that STANAG 5066 and associated systems will be able to react to such changes and modify parameters in light of long term variation.

For Skywave HF, ITV leads to significant variation in signal quality; papers on the subject suggest a typical variation with a standard deviation of 4dB (which is in line with the measurements described in this paper). It seems likely that this will have significant impact on the error characteristics of data being transferred over a Skywave link and consequent effect on application performance.

Measurement Approach

The high level measurement approach is simply to send "known" data over a link and to record errors. At the same time SNR is recorded, so that the relationship between signal quality and errors can be seen. The known data has the property that the receiving end can synchronize with the pattern even in the presence of bit errors and missing or inserted data.

Although the goal is to improve application performance, it makes most sense to make measurements at the modem level. This data can then be used to analyse consequent application performance and to tune performance.

Tools and Configuration

The tests are done using Isode's "HF Tool" which operates directly over the modem drivers, written in the Lua scripting language, used in Isode's Icon-5066 product, a STANAG 5066 server. 

For the measurements, an instance of HF Tool is run with each of the two modems used. One HF Tool will send data, typically using a selection of transmission speeds and interleaver settings. The second HF Tool will receive the data and record it in a set of text log files. These text logs contain all the information needed for analysis, so do not need to be correlated with the sender.

Result Analysis

HF Tool produces a number of text log files. Samples of some of these are shown below.

**START 2873948 (analysis dump of 2873948 received bytes)
* ::
*651630 -3325423-----------343362-----------------------------------------1734
*651700 44326343457433-------35346562643443453453563237432-------3464384264542
*651770 5534523655643---------------------------------------------------------
* ::
*651910 -------------------1335426542-----------------------------------------
*651980 ----------------------244-------------------------------------------34
*652050 3446633336341---------------3432465331--------------------------------
*652120 --------------331545653654414253653325571572443335432---------------44
*652190 316382544431------------------------------1543532---------------342436
*652260 66547231---------------------------------------------------------14516
*652330 341------------142543616454334542423------5331---------------------145
*652400 63546334554-----------------------------------------------------------
* ::
*652610 1532-----------------2455431----------------343521---------13---------

This format shows data received. “::” indicates a period without errors. Other lines show errors, with “-“ indicating a correct byte and an integer indicating the number of bit errors in the byte. This can be determined as the receiver knows the pattern of data sent.

**QUALITY 14:40:03.795 snr=27
**QUALITY 14:40:04.310 snr=29
**QUALITY 14:40:04.825 snr=29
**QUALITY 14:40:05.339 snr=30
**QUALITY 14:40:05.839 snr=30
**QUALITY 14:40:06.353 snr=29

This format records SNR. Tools can then be used to analyse this data and to provide graphical representation.

Over The Air Tests

Rockwell Collins ran the tests over a mid-latitude channel between Cedar Rapids, Iowa and Las Cruces, New Mexico on 26-29th August 2014. This used Rockwell Collins RT-4800 WBHF Modems. Tests were performed with:

• Narrow Band HF using STANAG 4539.
• Wide Band HF using US MIL-STD-188-141C Appendix D "Data Waveform Suite" with 3 kHz, 12 kHz and 24 kHz bandwidths.

OTA Measurement Results

The bulk of the measurements were five minute runs with variations of speed, bandwidth, and interleaver. We did a small number of thirty minute runs. Speeds were chosen across a range which lead to zero errors and to very high data loss. Some sample results from these runs are shown below. It was striking that the patterns seen in the graphs were similar for variations of all the parameters.

These graphs plot the SNR (white) and data loss (red). Red reaching the bottom of the graph shows a period where all data is lost. Where STANAG 5066 (or a similar protocol with no error correction) is operated over the link, any red will cause the data frame covering the red to be corrupted and thus rejected.

A selection of 300-second graphs and one 1800-second graph is shown.

Looking at these results a number of observations can be made:

• There is very significant variation of SNR over periods of 10s of seconds to a few minutes.   This validates that ITV is a very significant effect.
• There is no observable pattern to SNR variation that would enable prediction over the next minute or so.
• Data loss is very strongly correlated to SNR.  Data loss occurs when SNR is poor and does not occur when it is better.  
• This variation leads to patterns where there a "blocks" of transmission without loss interspersed with periods where data is lost.
• The variation of SNR is so large that quite a number of speed changes are needed to transition for “all red” to “all green”.   The samples above  show various patterns on the range from mostly green to mostly red.

Comparison to Standard Channel Simulation

It is useful to compare these OTA measurements with the measurements obtained from a channel simulation. Some sample graphs are included.

On quick inspection a number of graphs from simulation, such as the one above have patterns very similar to those seen OTA. There is variation of SNR with timeframe of order 10 seconds with similar magnitude, although in these graphs a bit less (perhaps a range of 10-15dB vs 15-20dB in the real traffic). However, this looks like a quite reasonable approximation of the sort of graphs seen OTA.

The next two channel simulation graphs look rather different:

These graphs have a much more "stripy" pattern with many red/green transitions. The channel simulation does not lead to any "blocky" graphs often seen OTA with large areas of red and green. This shows that ITV has a quite significant effect on the "above modem" characteristics that are observed.

Use of classic channel simulator has significant deviation from HF Skywave over periods of 10s of seconds and minutes, and so results based on use of channel simulators need to be considered with care.

Implications for STANAG 5066

Based on these observations of the impact of ITV on modem performance a number of observations can be made on tuning performance for operation over HF Skywave.

DPDU Size

In the analysis presented in [Optimizing STANAG 5066 Parameter Settings for HF & WBHF], Isode suggested that even at the fastest WBHF speeds, there would be performance benefits from using quite short (200 byte) DPDU size. This analysis was based on characteristics of modem operation over channel simulator.

Using the OTA test results, we analysed for each five minute run, the optimum DPDU size. For speeds of 9600 bps and faster (i.e., all WBHF speeds) it was very clear that use of 1024 bytes (the max value) was the best choice. There was only one run where this was not the case.

The reason for this different result is the "blocky" nature of errors. For a channel that behaves like a channel simulator, errors will be spread fairly evenly. Use of smaller DPDUs will enable more DPDUs to "fit into the gaps". With the more blocky nature of the OTA channel, DPDUs will only be lost at the edge of blocks. Very few DPDUs will be lost, as there are few blocks. At higher speeds, where large numbers of blocks can be sent, the tradeoff between data loss from blocks lost and higher efficiency of large blocks means that maximum size blocks give the best performance.

It also seems clear that, given the level of variation of conditions, even if there was a short period where shorter DPDU size was optimal, it would not be possible to detect and adapt to this. It will be important to select an appropriate DPDU size for the conditions.

We did not have any measurements at lower speeds. Therefore we undertook a calculation based on block patterns of the measurements we made to calculate optimum block size at slower speeds. This is represented in the table below.

CPDU seg:	25	50	75	100	150	200	300	400	600	800	1000
75 bps	5	8	10	8	7	5		2
150 bps	1	7	5	11	7	6	5	2	1
300 bps	1	1	2	9	6	11	6	5	2	1
600 bps		1	1	6	6	9	7	6	6	1	2
1200 bps		1		2	5	7	3	12	6	7	2
2400 bps			1	1	2	5	6	10	5	6	9
4800 bps				1	2	1	4	6	8	5	18
9600 bps					1	3	1	6	5	7	22
19200 bps					1	2	1	3	3	4	31
38400 bps					1	1	1	1	6		35

This suggests that for the HF speed range of 75 bps to 9600 bps, the optimum DPDU size ranges from 75 bytes to 800 bytes. This makes logical sense in context of the above analysis. It also fits with the recommendation of STANAG 5066 Annex G to use 2-300 byte DPDU size in typical HF operating speeds of 300-2400 bps. Although we feel reasonably comfortable with this conclusion, it would seem highly desirable to make OTA measurements across the range of narrow band HF speeds to validate them.

Data Rate Change

The classic algorithms of STANAG 5066 data rate change (Trinder/Brown and Trinder/Gillespie) and subsequent literature has a basic model of "use the last period of transmission (typically 1-2 minutes), and use Frame Error Rate (FER) and SNR to predict best choice for the next transmission".

It can be seen that this model is flawed, as this time period is too short to allow for ITV. By basing speed selection on the last transmission, you will choose sub-optimal speeds and end up jumping between speeds. This problem is borne out by operational experience and problems noted with the algorithms. The key difficulty is that ITV will lead to “bad patches” which should not be reacted to.

The effects of ITV mean that longer term analysis needs to be done to determine the optimal parameters. The best speed needs to be determined over a period of at least 5-10 minutes. There may be transmissions that have no loss followed by transmissions with high loss. The system needs to observe SNR (and to a lesser extent FER) over this period and work out best data rate from this. Icon-5066 will hold significant history information in order to facilitate such calculations. It may be preferable to use historical data (e.g., from 24 hours ago or last successful transmission) rather than guessing based on a few seconds of SNR sampling.

DPDU Repeats

A consequence of the very wide SNR variation is that a data transmission speed where loss of data is unlikely will be a lot slower than the speed for optimum data rate (i.e., not just one step lower). This slower speed will still be sensible for very short transmissions where data loss is highly undesirable (e.g., just transmitting an ack).

This means that in many situations at least some DPDU loss is inevitable. A good way to address this for longer transmissions it to repeat a DPDU within a transmission. This can often be achieved by using "spare" space in the final block. It will be desirable to do this for data where loss should be minimized (e.g., traffic such as XMPP where low latency is important or acknowledgements).

There seems to be significant potential to optimize application QoS requirements by using this technique.

Dealing with Long Term Variation

It seems clear that ITV needs to be addressed (for Skywave) by determining best transmission speed using data measured over a period of at least 5-10 minutes.

There are also longer term variations in performance over hours. It is clear that these longer term changes are important to react to. However, ITV is going to make these longer term changes harder to observe.

What Next?

These measurements are a starting point for further work.

Further Measurements

These measurements have been made for mid latitude Skywave in the day time. It seems clear that further measurements are highly desirable:

• Simply to make additional measurements to see how repeatable these results are.
• Measurements at different geographical locations.
• Measurements at classic narrow band HF speeds.
• NVIS (Near Vertical Incidence Skywave) measurements.
• Measurements at night.
• Measurements of Groundwave, including measurements over water for long and short distances (including BLOS propagation with diffraction). 

Measurements over water are of particular interest to Isode, in part because of the importance for Naval communication and in part because results significantly different to Skywave seem likely.

Isode is happy to make HF Tool available to partners and other organizations interested to make such measurements.

Use of Results in Icon-5066

Isode plans to use the results of this work to optimize performance of Icon-5066, Isode's STANAG 5066 server product.

Isode provides MoRaSky as to simulate modem behavior, and support application testing using Icon-5066. We plan to use this measurement data to enable MoRaSky to simulate ITV in line with the measurements obtained in these OTA tests.

Conclusions

This paper has described and analysed the Intermediate Term Variation measurements of HF radio and behavior "above the modem". It describes how these new results can be used to optimize performance of STANAG 5066 servers, and how Isode will address this in the Icon-5066 product.