Providing Web Services over HF Radio

Why Web Services over HF?

Web Services and Web browsing are not traditional HF services and some have argued they are not needed.   NATO, as a part of BRE1TA (BRASS  Enhancement 1 Technical Architecture) have included Web as a target service.   The rationale is that Messaging and XMPP chat services are primarily “push” and enable transfer of information from short to MUs (Mobile Units).   Web browsing provides a complementary “pull” service that allows MUs to access critical information on shore.

NATO are addressing the BRE1TA requirements with the BRIPES (BRASS IP Enhanced Software) which uses a number of Isode products.    The Web service (T-Web) uses Isode’s Icon-PEP product.

TCP over HF

The Web has a huge family of protocols which run over TCP with the lowest layers being HTTP and HTTPS.   The approach described here is to use TCP over HF Radio.   The approaches taken are summarized in the Isode white paper “Providing TCP Services over HF Radio”.  This is based on the HF-PEP architecture and protocols specified in STANAG 5066 Annex X (“HF-PEP: TCP Performance Enhancing Proxy Protocol”).

The HF-specific engineering is primarily in this layer and not described further in this white paper.

Deployment Architecture

The above diagram shows a deployment architecture which can be used to  measure performance and also outlines a simple operational deployment.  Components as follows:

1. Icon-5066 (Isode STANAG 5066 product) provides STANAG 5066 service.
2. Icon-PEP provides the HF-PEP services and also IP Client service.   It conncts two IP Routers to provide an IP Subnet service over HF.
3. Web Browser connects to LH IP router.
4. RH IP router connects to Web sites via the Internet or other IP network.

This architecture has been used in the lab, with the HF Network simulated by Isode’s MoRaSky tool.  It has also been used with reat modems and OTA operation.

What can be Achieved

There are many variables that affect performance, which are discussed later.  This makes it awkward to publish specific timings that have useful and comparable meaning.   This section gives a qualitative summary of what can be achieved.

Over HF, it is never possible to achieve the almost instantaneous response that is expected over a fast network.   Two assumptions are made:

1. The service is being used to access critical information, where the user is happy to wait seconds or tens of seconds to get a response.
2. The key target is Web sites that have been designed with an intent to be accessed over HF.

To support testing and provide an example of a Web site optimized for HF,  Isode has created http://hf-browse.com.

Observations from testing:

1. Web browsing is not viable at the slowest HF speeds. There is too much data for anything useful to work at 75bps.
2. At top end narrowband HF speeds (9600bps) browsing of an optimized Web site is quite viable.
3. At mid-range narrowband HF speeds, Web browsing of optimized sites is quite viable, albeit slow. The hf-browse.com site has a section suitable for access at 1200bps.
4. Some standard Web sites are usable at top end narrowband HF speeds. For example, the Wikipedia site is just about usable at 9600bps.
5. Wideband HF helps a lot with Web browsing. The higher speeds improve performance and widen the range of standard Web sites that can be usefully accessed.

There are a number of factors that are critical to performance, summarized here:

1. Structure of the Web site is crucial. If you want a Web site to be accessed over HF, there are many choices that can be made to enable HF access to work well.
2. Web site parameter choices can be made to improve HF performance.
3. Choice of Browser. Performance does vary between Web Browser, but this does not seem to be a critical choice.
4. Browser and other system components doing real time updates.
5. Domain Name Service (DNS) usage.
6. Use of Transport Layer Security (TLS).

These are considered in the rest of this white paper.

Web Site Choices

Notes on Web Browsing over HF

Typical Web sites are built using many files (HTML, CSS, JavaScript etc) that are cross-referenced and retrieved by a (Web) browser.   When accessing a Web page, a browser will start by retrieving one file identified by the initial URL and will them retrieve further files based on references in files retrieved.  A Web site will typically have files referencing multiple domains (e.g., to load standard scripts and fonts).

Browsers will generally open several TCP connections to retrieve elements of a Web site in parallel; two connections is typical.  This enables multiples files to be downloaded over a single TCP connection and minimizes the overhead associated with opening a TCP connection.

Modern Web browsers typically work by retrieving files incrementally, with file retrieval decoupled from presenting the Web page to the user.   The browser will then render the files that have come back as best they can. This can lead to partial loading, where some files (typically large graphics) are not loaded.  When designing a Web site for HF access, structuring the site so that key information can be rendered early is helpful.

The low speeds provided by HF have a significant impact on performance when accessing standard sites, as graphics and other large data blocks are commonly used.   However, the impact of “hand shaking” is often more significant due to the high latency of data over HF.  Some specific points are noted:

1. Partial Load, caused by failure to download a file. This is a particular problem over HF. For the isode.com site, failures are seen on JavaScript files as well as graphics.  This led to a situation where the site worked, but not exactly as intended.
2. Some browsers appear to retry downloads, and this can happen even when the initial retrieval later succeeds.   This leads to sub-optimal performance.
3. Web sites are not generally compressed, although some modern HTTP protocols do provide compression.  The HF-PEP mechanism of STANAG 5066 Annex X provides compression, which gives a generic performance improvement.

Web Site Recommendations

It is anticipated that the primary use of Web over HF will be to access Web sites optimized for this, rather than general purpose browsing of standard sites.   This section provides some recommendations for such a site:

1. Structure the Site in an hierarchical manner, keeping pages fairly small (ideally less than a few kBytes).
2. Minimize the number of files used for each page and avoid lots of small files for a page.
3. Structure the web page so that the core loads as quickly as possible. Users like interactivity.
4. Put graphics and larger files into separately downloadable files, giving information on file size. Note that 9600 bps (top Narrowband HF speed)  is about 1 kByte/sec.  When operating at 1200 bps (mid range Narrowband HF speed), it is significantly slower.
5. When including images in larger pages, consider use of lazy loading (e.g. <img loading=”lazy” src=… />), which defers fetching of off-screen images until they become visible.
6. Do not download fonts – use the browser defaults.
7. Hold all resources with the same DNS name. This will mean that the Web client will not need to do (inefficient) DNS lookups as part of the page retrieval.

The hf-browse.com provides a good example of a site optimized for HF, while remaining pleasant and usable.

Web Server Recommendations

HTTP Keep-Alive Timer is an Web Server configuration that controls when TCP connections are closed.   This is commonly set to 10 seconds, which means that for HF access, TCP connections get closed and do not get re-used.

For a site accessed by HF, the HTTP Keep-Alive timer should be set to several minutes.   This significantly improves performance by allowing TCP connection re-use.

It is also desirable to configure the Web server with HTTP cache headers such as Cache-Control or Expires, so that re-used resources are only fetched once by the browser.  Note that if a resource’s URL includes a hash of the resource, it may be appropriate to mark the response as “immutable” in order to cache the response indefinitely. For further information, see https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching

Browser Considerations

Browser Choice

The detailed way that a browser behaves has potential to  make significant difference to performance of Web browsing over HF.   To optimize performance for browsing general purpose Web sites at narrowband HF speeds, it is likely that a tuned browser will be needed.

Most Isode testing has been done with  Chrome and Firefox.   Some tests were tried with Opera, which did not offer better performance and might be inferior.

Chrome works reasonably well.   At 9600, partial loading is observed quite a bit.   There is no option to configure timeouts in Chrome, so performance has to be accepted “as is”.   In practice, Chrome appeared the better option.

Firefox also works reasonably well.  Its partial loading behavior appears slightly better than Chrome.   However, it appears to retry loading in a manner which means that a lot of redundant traffic is created.   Firefox provides extensive customizability of timers.   Some experimentation did not lead to improvements over default behavior.  There appears to be potential for tuning.

At 9600 bps, Chrome was faster than Firefox to complete a partial load of isode.com (4 minutes vs 6:30 minutes).  Firefox led to HF traffic for many minutes after the partial load was completed.  In both cases, the isode.com site was quite usable, although it was not presented as designed.

QUIC

QUIC.  Some browsers and Web sites use QUIC, which is a new protocol to access Web sites, intended to improve performance.  QUIC is not suitable for operation over HF using IP Client (STANAG 5066 Annex U) and so needs to be disabled in the browser.

Domain Name Services (DNS)

DNS is used for standard Web site deployment.  DNS queries are made over UDP and mapped by Icon-PEP onto STANAG 5066 IP Client.    The default DNS timeout used in the browsers tested was a bit shorter than typical retrieval times.   This meant that DNS queries were sent several times and several responses come back.  Delayed DNS responses also lead to ICMP traffic being sent back over HF.   These effects are exacerbated when DNS queries are made during Web site retrieval, as DNS traffic is delayed by HTTP traffic. Slow DNS lookup impacts user experience.

A simple option for deploying MUs to access special Web sites is to configure IP addresses in “hosts” file and not use DNS.   This is a simple way to optimize user experience and performance.

If DNS is used, a  DNS Caching Server on the client system is important to achieve reasonable performance.  When configuring a Web site, it is important to set long TTLs (ideally several days) for DNS records.   This will enable DNS to be used with quite moderate overhead.

Note that many sites use short TTLs on DNS records, so the benefit of DNS caching is much less for such sites.

Real Time updates for Browsers and other Components

Many modern software components automatically update over the Web.   The traffic caused by this needs to be avoided when an HF link is used.    Web Browsers perform such checks by default, including associated DNS lookups.

If DNS is not used over HF, this update traffic automatically gets blocked.  If DNS is used, either a special browser is needed (e.g., one with locked down configuration) or the update overhead needs to be accepted.

Transport Layer Security (TLS)

The modern Web operates almost entirely using Transport Layer Security (TLS) with the https: Web protocol option. Use of TLS for Web access adds two overheads:

1. An extra handshake
2. About 5 kByte of data dues to X.509 information carried.

This has negligible impact on a fast network, but both are significant for HF.   In practice, this means the TLS makes Web Access unusable at low speeds -both client and server will frequently time out connection initiation with TLS.

The simple solution is to operate without TLS.   The hf-broswe.com site intentionally does not use TLS.   This approach is recommended where it is operationally acceptable.

Given the ubiquity of TLS and the likely desire for non-HF systems to access Web sites which have been configured to allow HF access, it may well be important to use TLS.    This can be achieved using options in STANAG 5066 Annex X to provide a TLS Proxy.   This is described in the Isode white paper “Providing TCP Services over HF Radio”

Use of TLS Proxy provides two additional advantages over end to end TLS:

1. Because the HTTP stream is decrypted, it can be compressed using the stream compression capabilities of STANAG 5066 Annexes W and X. This is a useful performance win.
2. The HF operator can monitor the HTTP traffic, to ensure that usage is appropriate.

A consequence of the TLS Proxy is that the Web browser will see the certificate provided by the TLS Proxy.   Without special configuration, the browser will treat the certificate as untrusted and will issue warnings to the end user, leading to an uncomfortable user experience.   To address this two things needs to be done:

1. The system running on the browser needs to be configured to trust the TLS Proxy certificate.
2. The TLS Proxy Certificate needs to include the X.509 Subject Alternate Names (SANs) of all Web sites to be accessed.

Icon-PEP Support for Web Browsing

Icon-PEP is Isode’s product for supporting IP Services and TCP over HF Radio.   Detailed descriptions are found in the product overview https://www.isode.com/product/performance-enhancing-proxy/ and in the Isode white paper “Providing TCP Services over HF Radio”.

Icon-PEP provides some capabilities oriented to support of Web browsing over HF, which are shown here.   Icon-PEP provides UI that shows all current and recent TCP connections, which gives the operator a clear view as to what is going on.  All the screenshots were taken using a 9600bps simulated HF link.

The first two show access to the hf-browse.com Web site (above), looking first at the main page and then the STANAG 4406 white paper.

This is the first TCP connection opened by the browser.  It is used to retrieve the requested Web page, which is kept quite small.

The second TCP connection is opened at the same time.   On the initial access this downloads:

1. CSS file, which is used to structure presentation of all the pages.
2. The Isode logo, which is used on all pages.

The second Web page is loaded on this link – just the single file.

The site has a long HTTP keep-alive and so only these two connections are needed.

This is one of four TCP connections used to download the initial Wikipedia page.   A shorter HTTP keep-alive means that the early connections get closed by the server.   This is being done using TLS Proxy, so the Wikipedia pages and HTML are visible.

The somewhat larger pages give an experience that is slow, but usable at 9600 bps.

A second monitoring capability is provided for DNS, which is very helpful in setups where DNS is used.    This screen shows the flow of DNS queries and answers going over HF.

Conclusions

This paper has shown how Web browsing can be achieved over HF and provides a number of recommendations as to how best to deploy this.  It also shows use of Isode’s Icon-PEP product to provide and monitor Web browsing over HF.   While this can work with general purpose Web sites, the expectation is that it will generally be used to give access to Web sites specifically developed to perform reasonably over HF.

It is anticipated that this service will be used by Mobile Units (Ships; Planes; Field HQs) which only have HF communication, to enable access to critical information on a central network.  This provides a helpful complement to XMPP and Messaging applications.