This chapter exists in three parts: proxy structure, proxy implementation and supporting information. It begins with a four sections (\ref{section:implementation-proxy} to \ref{section:implementation-prod-cons}) looking at the structure of the application, with a focus on future growth. Following this are four examples of concrete implementations and specifics in sections \ref{section:implementation-tcp} to \ref{section:implementation-security}. Finally, details on the automated evaluation are given and a repository overview.
The central structure for the operation of the software is the \verb'Proxy' struct. The proxy is defined by its source and sink, and provides methods for \verb'AddConsumer' and \verb'AddProducer'. The proxy coordinates the dispatching of sourced packets to consumers, and the delivery of produced packets to the sink. This follows the packet data path shown in figure \ref{fig:proxy-start-data-flow}.
The proxy is implemented to take a consistent sink and source and accept consumers and producers that vary over the lifetime. This is due to the nature of producers and consumers, as each may be either ephemeral or persistent, depending on the configuration. An example is a device that accepts TCP connections and makes outbound UDP connections. In such a case, the TCP producers and consumers would be ephemeral, existing only until they are closed by the far side. The UDP producers and consumers are persistent, as control of reconnection is handled by this proxy. As the configuration is deliberately intended to be flexible, both of these can exist within the same proxy instance.
The structure of the proxy is built around the flow graph in figure \ref{fig:proxy-start-data-flow}. The flow graph demonstrates the four transfers of data that occur: packet source to source queue, source queue to consumer, producer to sink queue, and sink queue to packet sink. For the former and latter, these exist once for an instance of the proxy. The others run once for each consumer or producer. Basic examples of the logic applied for each flow are given in figure \ref{fig:proxy-loops}.
Although the pseudocode given in figure \ref{fig:proxy-loops} is incredibly simple, aside from error handling, this is as implemented in the Go code. Go's cooperative scheduler and lightweight Goroutines make this an efficient implementation. However, given the expected quantities of simultaneously connected consumers and producers is low, heavier OS threads would also be effective here. The queues are further trivial to implement in Go, as channels provide all of the necessary functionality, but can also be implemented in other languages. The lifetime of producers and consumers are controlled by the lifetime of the aforementioned loops, and are only referenced within them, such that the garbage collector can collect any producers and consumers for which the loops have exited.
Finally is the aforementioned ability for the central proxy to restart consumers or producers that support it (thus far, those initiated by the proxy in question). This causes the wrapping of the loops shown in figure \ref{fig:proxy-loops} in an additional layer. Pseudocode for the expansion to the consumer is shown in figure \ref{fig:proxy-loops-restart}, with the providers being expanded similarly.
\begin{figure}
\begin{minted}{python}
do:
while is_reconnectable(consumer) and not is_alive(consumer):
reconnect(consumer)
while is_alive(consumer):
packet = source_queue.popOrBlock()
consumer.consume(packet)
while is_reconnectable(consumer)
\end{minted}
\caption{Pseudocode for a consumer, supporting reconnection.}
The configuration format chosen was INI, extended with duplicate names. Included is a single Host section, followed by multiple Peer sections specific to a method of communicating with the other side. Processing the configuration file is split into three parts: loading the configuration file into a Go struct, validating the configuration file, and building a proxy from the loaded configuration.
Validation of the configuration file is included to discover configuration errors prior to building an invalid proxy. Firstly, this ensures that all parts of the program built from the configuration are given values which are invalid in context and easily verifiable, such as a TCP port of above 65,535. Secondly, catching errors in configuration before attempting to build the proxy constrains the errors of an invalid configuration to a single location. For a user, this might mean that an error such as \verb'Peer[1].LocalPort invalid: max 65535; given 74523' is shown, as opposed to \verb'tcp: invalid address', which shows the user's error as opposed to a bug in the code.
Once a configuration is validated, the proxy is built. This is a simple case of creating the proxy from the given data and adding the producers and consumers for its successful running.
This builder structure is also useful for a Go project, as it helps avoid circular imports, which are banned in Go. An example is a TCP flow implementing \verb'proxy.Consumer'. The proxy package cannot import TCP to create flows, so this must be delegated to another package. A builder package can bridge the gap, while maintaining a close link between the configuration and a single place where it is built.
% ------------------------- Sources and Sinks ------------------------------ %
\section{Sourcing and Sinking Packets}
A source and sink of packets are the names given to the method of entry for packets to tunnel and the method of exit for packets received. In the current implementation, both of these are achieved with a TUN adapter. A TUN adapter is a virtual network interface provided by the kernels of multiple unices, allowing userspace code to interact with Layer 3 networking.
% ---------------------- Providers and Consumers --------------------------- %
Producers and consumers were designed with flexibility in mind. A producer produces packets from the far side proxy. A consumer sends packets to the far side proxy. It has already been mentioned that they can be either ephemeral or persistent, and this is achieved by implementation of an additional interface, \verb'Reconnectable'. The three relevant interfaces for producers and consumers are given in figure \ref{fig:producer-consumer-interfaces}. A basic consumer or producer must simply provide a method to consume a packet or produce one, given the relevant security method, and a method to decide if it is still alive. This is an interface which can be applied to both TCP and UDP based connections, with others also feasible.
The base implementation for producers and consumers takes advantage of TCP. The requirements for the load balancing given above to function are simple: flow control and congestion control. TCP provides both of these, so was an obvious initial solution. However, TCP also provides unnecessary overhead, which will go on to be discussed further.
TCP is a stream oriented connection, while the packets to be sent are discrete datagrams. That is, a TCP flow cannot be connected directly to a TUN adapter, as the TUN adapter expects discrete and formatted IP packets while the TCP connection sends a stream of bytes. To resolve this, each packet sent across a TCP flow is prefixed with the length of the packet. On the sending side, this involves writing the 32-bit length of the packet, followed by the packet itself. For the receiver, first 4 bytes are read to recover the length of the next packet, after which that many bytes are read. This successfully punctuates the stream oriented connection into a packet based connection.
However, using TCP to tunnel TCP packets (known as TCP-over-TCP) can cause a degradation in performance in non-ideal circumstances \citep{honda_understanding_2005}. Further, using TCP to tunnel IP packets provides a superset of the required guarantees, in that reliable delivery and ordering are guaranteed. Reliable delivery can cause a decrease in performance for tunnelled flows which do not require reliable delivery, such as a live video stream - a live stream does not wish to wait for a packet to be redelivered from a portion that is already played, and thus will spend longer buffering than if it received the up to date packets instead. Ordering can limit performance when tunnelling multiple streams, as a packet for a phone call could already be received, but instead has to wait in a buffer for a packet for a download to arrive, increasing latency unnecessarily.
Although the TCP implementation provides an excellent proof of concept and basic implementation, work moved to a second UDP implementation, aiming to solve some of these problems. However, the TCP implementation is functionally correct, so is left as an option, furthering the idea of flexibility maintained throughout this project. In cases where a connection that suffers particularly high packet loss is combined with one which is more stable, TCP could be employed on the high loss connection to limit overall packet loss. The effectiveness of such a solution would be implementation specific, so is left for the architect to decide.
To resolve the issues seen with TCP, an implementation using UDP was built as an alternative. UDP differs from TCP in that it provides almost no guarantees, and is based on sending discrete datagrams as opposed to a stream of bytes. However, UDP datagrams don't provide the congestion control or flow control required, so this must be built on top of the protocol. As the flow itself can be managed in userspace, opposed to the TCP flow which is managed in kernel space, more flexibility is available in implementation. This allows received packets to be immediately dispatched, with little regard for ordering.
Congestion control is most commonly applied in the context of reliable delivery. This provides a significant benefit to TCP congestion control protocols: cumulative acknowledgements. As all of the bytes should always arrive eventually, unless the connection has faulted, the acknowledgement number (ACK) can simply be set to the highest received byte. Therefore, some adaptations are necessary for TCP congestion control algorithms to apply in an unreliable context. Firstly, for a packet based connection, ACKing specific bytes makes little sense - a packet is atomic, and is lost as a whole unit. To account for this, sequence numbers and their respective acknowledgements will be for entire packets, as opposed to per byte. Secondly, for an unreliable protocol, cumulative acknowledgements are not as simple. As packets are now allowed to never arrive within the correct function of the flow, a situation where a packet is never received would cause deadlock with an ACK that is simply set to the highest received sequence number, demonstrated in figure \ref{fig:sequence-ack-discontinuous}. Neither side can progress once the window is full, as the sender will not receive an ACK to free up space within the window, and the receiver will not receive the missing packet to increase the ACK.
I present a solution based on Negative Acknowledgements (NACKs). When the receiver believes that it will never receive a packet, it increases the NACK to the highest missing sequence number, and sets the ACK to one above the NACK. The ACK algorithm is then performed to grow the ACK as high as possible. This is simplified to any change in NACK representing at least one lost packet, which can be used by the specific congestion control algorithms to react. Though this usage of the NACK appears to provide a close approximation to ACKs on reliable delivery, the choice of how to use the ACK and NACK fields is delegated to the congestion controller implementation, allowing for different implementations if they better suit the method of congestion control.
Given the decision to use ACKs and NACKs, the packet structure for UDP datagrams can now be designed. The chosen structure is given in figure \ref{fig:udp-packet-structure}. The congestion control header consists of the sequence number and the ACK and NACK, each 32-bit unsigned integers.
The first algorithm to be implemented for UDP Congestion Control is based on TCP New Reno. TCP New Reno is a well understood and powerful congestion control protocol. RTT estimation is performed by applying $RTT_{AVG}= RTT_{AVG}*(1-x)+ RTT_{SAMPLE}*x$ for each newly received packet. Packet loss is measured in two ways: negative acknowledgements when a receiver receives a later packet than expected and has not received the preceding for $0.5*RTT$, and a sender timeout of $3*RTT$. The sender timeout exists to ensure that even if the only packet containing a NACK is dropped, the sender does not deadlock, though this case should be rare with a busy connection.
To achieve the same curve as New Reno, there are two phases: exponential growth and congestion avoidance. On flow start, using a technique known as slow start, for every packet that is acknowledged, the window size is increased by one. When a packet loss is detected (using either of the two aforementioned methods), slow start ends, and the window size is halved. Now in congestion avoidance, the window size is increased by one for every full window of packets acknowledged without loss, instead of each individual packet. When a packet loss is detected, the window size is half, and congestion avoidance continues.
\mynote{Left for now. I have a bit of an idea about why this might/might not be necessary, and want to discuss it with slightly more context (it differs massively between use cases).}
\mynote{Left for now. I have some specific debugging I want to do on the real connection to see if I can come up with some better reasons for this. Currently it's along the lines of "setting it to 0 solves one problem but might make another worse".}
The security implementation (beyond layered security, see section \ref{section:layered-security}), is implemented in three parts: the \verb'Exchange' interface for initial exchanges across a flow, the \verb'RepeatBlocker' interface for preventing repeated packets, and the \verb'MacGenerator' and \verb'MacVerifier' interfaces for providing and verifying message authenticity.
Cryptographic exchanges are represented by the interface given in figure \ref{fig:crypto-exchanges-interface}. The interface was chosen to be absolutely flexible, while maintaining simplicity. The \verb'Next' method returns two byte arrays. The first contains the message that should be sent back to the far side, if there is one. The second contains data taken from the packet, if it contains any. This allows for exchanges such as the TCP three way handshake, in which the final message contains data as well as setup. The first exchange implemented is the \verb'None' exchange, which should implement no exchange at all, and is shown in figure \ref{fig:crypto-exchanges-none}.
The exchanges are designed to be applicable to any packet transport method. The flow of using an exchange is shown in pseudocode in figure \ref{fig:crypto-exchanges-pseudocode}. The flow manages all packets exchanged by the connection, until the exchange confirms that it is done, or fails. The exchange can also produce data during the exchange, which is handed off to the normal packet handling mechanism, represented in the pseudocode by the \verb'yield' keyword.
As discussed in section \ref{section:preparation-repeated-packets}, the algorithm used to prevent repeated packets is the \emph{IPsec Anti-Replay Algorithm without Bit Shifting}\citep{tsou_ipsec_2012}. In the referenced work, a \verb'C' implementation is provided, which could be easily adapted to Go.
Although the repeated packet protection is primarily targeted at avoiding repeated IP packets, it is possible for it to be used elsewhere, as mentioned in exchanges. As such, the interface for repeat protection is made concurrent, such that it can be used anywhere in the program it's deemed useful. Given that initial flow exchanges should happen exceedingly rarely compared to packet flow, this does not affect the size of the repeated packet window significantly.
\mynote{Discuss the specific implementation of this algorithm once finalised.}
\subsection{Message Authentication Codes}
The message authentication interfaces, shown in figure \ref{fig:message-authenticator-interface}, are designed to simply support any MAC generator, including digital signatures. The MAC algorithm that is implemented at this point is BLAKE2s \citep{hutchison_blake2_2013}, a fast and secure algorithm that supports prefix-MAC generation. BLAKE2s is available in OpenSSL\footnote{\url{https://openssl.org}} and the Go \verb'crypto'\footnote{\url{https://github.com/golang/crypto}} library, making it accessible. It is also an extremely fast algorithm, that provides the necessary guarantees for this software, so was chosen for its low performance overheads. Any other MAC which can be generated in a similar fashion could also be used.
The security features presented in this section form a hierarchy within the data flow of a packet. Given in figure \ref{fig:udp-packet-dataflow} is an example of the growth of a packet with a specific configuration of UDP, replay protection, and a message authentication code. The process of sending a packet via a consumer is given right to left, and the reverse process of receiving a packet from a producer is given left to right. For a packet to consume, the first element added is the data sequence number. This data sequence number is closest to the proxy, as it is global and not unique to a flow. This is followed by the congestion control header, added before the MAC such that it receive integrity and authenticity protection, preventing tampering. Finally, the UDP header is prepended before the packet is dispatched. The same process is repeated in reverse in the other direction. This figure represents the flow of data through the software, and importantly, where each portion of the security must be implemented to achieve its goals.
The project has a particular focus on automatic evaluation. Although some testing can be performed with mocked connections, the evaluation of the success criteria revolve around virtual hardware. The benefit of virtual hardware is the ability to spin up and spin down entire testing environments when required. As such, automatic evaluation software can be built to create the required environments and gather the information required. This allows each code change to be verified by confirming that the graphs show the required trends.
The application is split into two parts: data gathering and data processing. There are two reasons for this. Firstly, the data gathering takes significantly longer than data processing, so being able to gather data once and then process it multiple times is a benefit for rapid development. Secondly, the choice of language is different for each stage, as discussed in \ref{section:preparation-language-choices-evaluation}. To complete the full process of automatic evaluation, the data is first updated by running the data gathering application, after which the data processing is completed. As the only item which needs to cross the language barrier is test output data, this can be achieved by using the file system. Any loss of speed by switching from memory to files is irrelevant for testing purposes, so this is an acceptable trade-off.
Discussion of data gathering will be given in this section, while the data processing will be discussed in the evaluation chapter, in section \ref{INVALID}. The data gathering grew to be a significant part of this project, and was built alongside the implementation to gather the requisite data. In fact, it developed from a custom script created to make testing this project easier, into a larger Java library that can be used for a much wider variety of application testing.
The system for automated data gathering is built using Java. It involves three layers: the Java application itself, the \verb'virtualtests'\footnote{\url{https://github.com/JakeHillion/virtual-tests}} library, which depends on the \verb'proxmox'\footnote{\url{https://github.com/JakeHillion/proxmox-java}} library. Each of those elements were developed by me within the constraints of this project.
The data gathering system was designed to make adding data points as simple as possible. As such, the pseudocode of the program's main method follows that in figure \ref{fig:data-gathering-pseudocode}. This succinctly demonstrates that each testing environment exists only as long as test are running within it. The majority of tests are standard tests, with a network structure shown in \ref{fig:data-gathering-standard-network-structure}. The structure connects the local portal via $n$ connections to the remote portal. The test then specifies the initial rate limit for each connection, changes to the rate of each connection over time, and changes to the liveness of each connection. Once this structure is created, tests can be run between the speed test server and proxied client.
The tests proposed here are expensive to compute. They involve creating Virtual Machines, installing the required software, and most involve running speed tests that take multiple seconds. As such, methods are taken to keep repeats of test small, while maintaining sufficiently small uncertainty to conclusively demonstrate effects. To achieve this, each test is run a minimum of 5 times, then run until the coefficient of variance ($CV =\mu/\sigma$) falls below a provided value. Further, the tests support a draft mode, which runs the tests fewer times in order to produce results more quickly, albeit with a higher level of uncertainty.
After performing the required data analysis to confirm the level of uncertainty for each result, the result can be saved directly to a file. For the majority of tests these are JSON files. To ensure correct communication between the Java and Python softwares, a matching structure exists on both sides that can be compiled to a directory name. Within this directory, each repeat of test data can be stored by its numeric index.
A directory tree of the repository is provided in figure \ref{fig:repository-structure}. The top level is split between \verb'code' and \verb'evaluation', where \verb'code' is compiled into the application binary, and \verb'evaluation' is used to verify the performance characteristics and generate graphs.
The program overall is structured for future growth, and to provide flexibility for network architects to implement it as they see fit. This chapter makes clear how this is achieved using interfaces that are flexible, before providing details on the concrete implementations. TCP provides a proof of concept with less implementation effort, but with varying performance outside of ideal environments, allowing the structure to be thoroughly tested before a more complex UDP implementation was developed. Security allows for either external or internal solutions to be used, with future support built for more complex internal initial exchanges, allowing for security measures such as digital signatures. Overall, this chapter shows a highly flexible solution for a multi-path proxy. In the next chapter, it will be shown to be highly performant.
