Meshdynamics
First, Second and Meshdynamics Third Generation Mesh Architectures   
                                         First, Second and Third Generation Mesh Architectures (Published 2005)

Abstract
.

Evolving from ad hoc 802.11 networking, earlier generations of wireless mesh provided basic networking over extended outdoor areas. With the emergence of demanding data applications along with video and voice, single-radio "First Generation" single-radio wireless mesh solutions are proving unsatisfactory in many of these demanding environments. Third Generation wireless mesh solutions are based on multi-radio backhauls and deliver 50-1000X better performance, but some custom hardware-oriented approaches limit flexibility and create deployment challenges. Software- oriented Third Generation wireless mesh based on distributed dynamic radio intelligence delivers the same high performance but with the additional benefits of easier installation, better avoidance of interference, and the added flexibility of easy mobility. These new capabilities are enabling many new types of applications beyond the traditional wireless mesh metro/muni environment.

Introduction

Mesh network requirements have evolved from their military origins as requirements have moved from the battlefield to the service provider, and residential networking environments. Today, to cover large areas with a single wired Internet link, more cost effective and efficient means of bandwidth distribution are needed. This implies more relay nodes (hops) than were needed before. Further, growing demands for Video and Voice- over-IP require packets to be moved over the mesh at high speeds with both low latency and low jitter. These new mesh requirements (more hops to cover large areas, more efficient bandwidth distribution and better latency and jitter for Video and VOIP) has given rise to the third-generation of mesh architectures.

Three Generations of Mesh Architectures

                 

Figure. 1: (L-R): Ad Hoc, 1-Radio Meshed Backhaul, 3-radio Structured Mesh

Three generations of evolving mesh architectures are depicted above. They are (Left to Right):

• First Generation: 1-Radio Ad Hoc Mesh (left). This network uses one radio channel both to service clients and to provide the mesh backhaul. The ad hoc mesh radio, marked AH, provides both services – client access and backhaul. This architecture provides the worst services of all the options, as expected, since both backhaul and service compete for bandwidth.

• Second Generation: Dual-Radio with Single Radio Ad-Hoc meshed backhaul (center). This configuration can also be referred to as a “1+1” network, since each node contains two radios, one to provide service to the clients, and one to create the mesh network for backhaul. The “1+1” appellation indicates that these radios are separate from each other – the radio providing service does not participate in the backhaul, and the radio participating in the backhaul does not provide service to the clients. These two radios can operate in different bands. For example, a 2.4 GHz IEEE 802.11 b/g radio can be used for service and an IEEE 802.11a (5.8 GHz) radio can be used exclusively for backhaul. Though this configuration is sometimes called a Dual Radio Mesh, only one radio participates in the mesh. Performance analysis indicates that separating the service from the backhaul improves performance when compared with conventional ad hoc mesh networks. But since a single radio ad hoc mesh is still servicing the backhaul, packets traveling toward the Internet share bandwidth at each hop along the backhaul path with other interfering mesh backhaul nodes - all-operating on the same channel. This leads to throughput degradations which are not as severe as for the ad-hoc mesh, but which are sizeable nevertheless.

• Third Generation: 3-Radio Structured Mesh (right). The last architecture shown is one that provides separate backhaul and service functionality and dynamically manages channels of all of the radios so that all radios are on non-interfering channels. Performance analysis indicates that this provides the best performance of any of the methods considered here. Note that the two backhaul radios for the 3-radio configuration shown in Figure 1 are of the same type - not to be confused with 1+1 so-called dual radio meshes where one radio is for backhaul) and the other for service. In the 3-radio configuration, 2 radios are providing the up link and down link backhaul functionality, and the third radio is providing service to the clients.

Bandwidth degradation on Single Channel Backhauls
 

Figure 2: Single vs. Dual Channel Backhauls.
Figure 3: Live Feed from a multi-hop dual channel backhaul of 9 hops.

With one backhaul radio available for relaying packets, all nodes communicate with each other on one radio channel. For data to be relayed from mesh node to mesh node, that node must repeat it in a store-and-forward manner. A node first receives the data and then retransmits it. These operations cannot occur simultaneously because, with only a single radio channel, simultaneous transmission and reception would interfere with each other (Figure 2).

This inability - to simultaneously transmit and receive - is a serious disadvantage. If a node cannot send and receive at the same time, it loses ½ of its bandwidth as it attempts to relay packets up and down the backhaul path. A loss of ½ with each hop implies that after 4 hops, a user would be left with (½*½*½*½) = 1/16 of the bandwidth available at the Ethernet link. This is 1/(2N) relationship defines the fraction of the bandwidth available to a user after N hops.

Third generation mesh products eliminate bandwidth degradation with a dual channel backhaul. There is been no measurable bandwidth degradation and this is the significant departure from both first and second-generation mesh architectures.

Figure 3 shows live video from an IP camera part of a 9-hop mesh network at a ski resort. A single channel backhaul would be incapable of delivering this video feed beyond 1-2 hops: typical video bandwidth requirements would cripple the system. Latency and jitter would be unacceptable.

Latency/Jitter Degradation on Single Channel Backhauls



Figure 4a (left): Latency for 36 simultaneous VOIP calls over a 4 hop Mesh running VOIP Concatenation over the backhaul.
Figure 4b (right): Jitter for 36 simultaneous VOIP calls over a 4 hop Mesh running VOIP Concatenation over the backhaul.

Latency is inversely related to available bandwidth: thin pipes can provide only so much flow. Single channel backhauls suffering from bandwidth degradation also suffer from poor latency and jitter over multiple hops. This is primarily due to the need for a single radio to serve both backhaul and client traffic. The result is that most First Generation single-channel backhaul networks provide reliable video- or voice service over only one- or two hops. As a result many more costly wired or fiber Internet or intranet drops are needed to deliver adequate service, increasing the ongoing total cost of ownership.

Third Generation mesh products do not suffer from bandwidth degradation – they use multiple backhaul radios to obviate radio channel interference. Field tests indicate a latency of less than 1 millisecond per hop even under heavy traffic. Since latency is thus not a factor of traffic or user density, jitter (variation in delay) is also very low. This makes Third Generation products suitable for networks serving large number of users, demanding applications, video, and voice -- even simultaneously.

These capabilities of Third Generation wireless mesh networks make them especially useful where video surveillance is part of a metro/muni requirement, for expanding coverage into under-served areas with limited high speed wired or fiber infrastructure, and for border and perimeter networking, where bandwidth must be extended node-to-node as if in a long string of pearls.

Additionally, Meshdynamics multi-radio backhauls also incorporate VOIP concatenation for timely VOIP packet delivery. VOIP packets are small – typically less than 300 bytes but sent frequently, generally once every 20 milliseconds.

Networking protocols like CSMA/CA don’t transport small and time sensitive packets well. The VOIP concatenation engine aggregates small VOIP packets into a larger packet for more efficient delivery. This aggregation takes place every 5-10 milliseconds. USAF tests (Figure 4) show overall latency is less than 10 ms + 1 ms per hop over a 4 hop network. Jitter is less than 1 ms. per hop.

USAF tests found comparable latency/jitter for single channel backhauls to be an order of magnitude higher. Some reasons:

• Bandwidth degrades with each hop. As an analogy to water pipes, the smaller the pipe, the slower the flow.
• With all radios on the same channel, there is compounded contention will packets fighting each other, all on the same channel.
• The overall efficiency of the CSMA/CA protocol used degrades exponentially as number of clients on one channel increase.

Frequency Agility through Distributed Intelligence

Fig 5(a): Channel Agility at every section of the mesh network.  Fig 5(b) Field Tested Mobile Backhaul and Extender Nodes

Wireless is a shared medium. Radios communicating on the same channel and within range of each other contend for available bandwidth. In single channel backhauls, there is one radio acting as the repeater between nodes: all backhaul radios must be on the same radio channel. The entire network is susceptible to channel interference/jamming. System performance is compromised. Figure 5 shows how the dual channel backhaul can switch channels to avoid debilitating external interference effects. This first step of providing two-radio backhauls provides an obvious theoretical advantage over single-channel backhauls, but management of channel selection and interference avoidance becomes a challenge addressed in a number of different ways.

One of the first approaches applied to this problem was to segregate the backhaul links with hardware radio switching and sectored directional antennas. Since each sectored antenna "sees" only a narrow field of view, radio emissions from adjacent nodes do not create interference. The limitations of this hardware-centric approach are the costs of sectored antennas and custom-developed radio hardware, the relative inflexibility of the system in dealing with perturbations and new external interference sources, and the complexity of site-surveying and installation. The alignment of the directional antennas must be precise and installation must include some manual determination of channel choices to maximize the efficient use (and re-use) of channels as well as a manual configuration of network topology.

A more-recent alternative approach utilized by Meshdynamics automates the channel-selection and topology-definition tasks by distributing dynamic radio intelligence in each node, in effect creating multiple "RF robots". Sophisticated algorithms allow each node to listen to its environment continuously to determine its relationship with neighboring nodes as well as extraneous and potentially interfering radio sources. Based on this analysis of the environment, an individual node selects the best channels to use to connect to the optimal nearby node for highest performance.

In this distributed dynamic radio intelligence approach, the network forms a tree-like structure emanating from one or more "root" nodes that have the wired or fiber connection to the Internet or intranet. As the branches of the "tree" radiate outward, eventually they become geographically distant enough from one another for nodes to begin re-using channels. This greatly increases the data-carrying capacity of the network, since it makes better use of the scarcest resource in an outdoor WiFi environment: the fixed number of unlicensed channels.

It may be obvious that this approach works with both sectored directional antennas and with omnidirectional antennas. The greatest flexibility comes with the use of omnidirectional antennas, since no pre-engineering of paths must be done. Each node ascertains the best connection and coordinates that connection with adjacent nodes through the periodic exchange of routing and other information.

Although the structure that results gives the appearance of a tree, the software intelligence in each node permits it to function exactly as the single- radio First Generation wireless mesh. A failure of any node prompts immediate coordinated reconnections around the network to bypass the failed node, as in the case of a traffic accident felling the light pole on which the node is mounted.

When the missing node is returned to service, its neighbors recognize its presence and recalculate the best connections once again. This capability also makes additions and expansions to the network very straightforward, as new nodes may be simply configured with the proper security information, then powered-up. New nodes automatically are added to the network based on an exchange of information between the existing nodes, which are continually monitoring the environment.

The independent but coordinated "RF robots" in each node are also useful in detecting and avoiding external interference sources. Because 802.11 WiFi is an unlicensed medium, new independent and uncontrollable RF courses may appear in the network unpredictably and at any time. First- Generation wireless mesh networks are challenged by such sources since every device is sharing the same channel, but all may not be close enough to "hear" the offending point interference source. Hardware-oriented Third Generation networks may also be affected if an unexpected interference source appears in one of the sectored node-to-node paths.

Such an interference source is easily with managed distributed dynamic radio intelligence. Nodes close to the offending RF source may move in a coordinated fashion to a new and unused channel. By coordinating this movement, the impact to end users may be minimized. In extreme cases where a powerful interference source blankets an area, nodes cut off from communication with the rest of the network simply repeat the start-up process of listening for other nodes on non-interfered channels and the network rapidly reconverges to a stable state.

Quick and easy deployment without significant pre-engineering, automated and rapid avoidance of interference sources, and fast additions and reconfigurations of the network by the simple addition of nodes are all distinctive features of Third Generation architectures delivered through distributed dynamic radio intelligence as developed by Meshdynamics. An emerging application of this technology is in temporary and event-driven networking such as that needed for sporting events. An additional benefit of this software-based distributed intelligence is the ability to place a node in motion relative to other nodes in the network.

One use of mobility has been in security and other types of rapid-response applications. Nodes may be mounted on vehicles, even man-carried to new locations while remaining in communication with the overall wireless mesh network at all times. No management or other user interaction is needed, and with the use of omni-directional antennas, the mobile nodes may move in any direction around the perimeter or through the middle of the geographic area supported by the fixed wireless mesh node. Mobile nodes have even been mounted in unmanned aerial vehicles or tethered balloons to provide coverage of an area.

Another mobile application supported by Meshdynamics' distributed dynamic radio intelligence is support for networking in rail corridors. Mobile nodes installed in commuter trains link with a series of fixed nodes installed along the rail line. As the train moves, the mobile node is constantly listening to the environment. Typically, the signal from the fixed node being approached will be increasing while the signal from the fixed node recently passed will be decreasing. In a coordinated fashion, the "RF robots" in all three nodes coordinate the hand-off from one fixed node to another, assuring seamless connectivity for commuters in the railcar, whether the train is in motion or halted at a station.

Radio Agnostic Mesh

Figure 6, 7: Meshdynamics 4 radio slot module (left) supports 5.8 GHz and 2.4 GHz Backhaul Interoperability (right).

Frequency agility is taken one step further in Meshdynamics Modular Mesh products. The mesh control "RF robot" software runs above the MAC layer of the radio: the same mesh control software supports radios operating on different frequency bands. Decoupling the logical channel-selection and topology-definition processes from the specific physical radio in this fashion delivers distributed dynamic radio intelligence benefits for current as well as emerging radio standards.

Figure 6 shows how this level of flexibility is supported. There are 4 mini-PCI slots on the board, two on the bottom and two on top. Each of the four slots can house a different frequency radio. This opens up some interesting possibilities including 2.4 GHz backhaul systems being part of a mesh with 5.8 GHz backhauls. Since the service and backhaul radios are distinct, it is possible to use a service radio to bridge over from a 5.8 GHz backhaul to 2.4 GHz backhaul – as shown in Fig 7. The 4325 Mobility Relay node on the bottom left has joined the mesh – even though the upper links are 5.8 GHz (blue) – through the service radio (pink).

Switching to another channel contains local interference at one section of the network. With one radio backhauls, this is not possible: the entire network is on the same channel and switching to another channel is simply not practical. The performance of single channel backhauls is therefore heavily compromised in RF polluted environments or under malicious attacks. Military field trials with dual channel backhaul have demonstrated that frequency agility ensures the mesh is running even with malicious RF interference- the backhaul radios (blue) simply switched to non-interfering channel.

One advantage of this level of flexibility includes supporting longer range and lower bandwidth 2.4 GHz 802.11b radios with shorter range but more bandwidth capable 5.8 GHz 802.11a radios, all part of the same mesh network. The longer range enables the edge of the network – where bandwidth requirements are low – to be serviced adequately by 2.4 GHz edge/mobility nodes, (Figure 7). Bandwidth is thus distributed more efficiently and cost effectively managed: the node spacing is adjusted for the subscriber density based on the range of the radios used.

In the future, this flexibility will also permit the incorporation of new radio types such as WiMAX (802.16) as an adjunct to the 802.11 WiFi mesh. These potentially higher-speed links may offer a flexible way to inject bandwidth into the mesh along with a cost-effective distribution strategy for high-speed WiMAX links.

One additional benefit of decoupling distributed the dynamic radio intelligence software controlling channel selection and topology configuration from the radio hardware is the potential to incorporate commercial-off-the-shelf (COTS) radio and other modules. This substantially decreases time- to-market and greatly increases manufacturing scale for some components, reducing both development- and unit cost over the use of custom hardware development.

Greater Network Scale

Two elements allow Meshdynamics-based networks to scale to much larger overall sizes than networks based on competitors' products. The first is simply the nature of Third Generation technology. Because of the multiple radios in each node, it is possible to use separate, non-interfering channels for upstream links (toward the wired or fiber connection) and downstream links. The use of separate uplink and downlink creates the characteristic "tree" topology within a logical mesh network. Each of these uplinks and downlinks is a separate collision domain, vastly improving performance in terms of higher bandwidth, lower delay, and minimized jitter. The result is that much larger number of hops (node-to-node connections) may be supported, with production networks operating over twelve hops and twenty-hop networks being well within performance capabilities. This translates to high-performance networking over very large areas including hundreds or thousands of nodes, well beyond the capabilities of First- or Second-Generation technology. The difference between Third Generation and earlier technologies is somewhat akin to the vast difference between Ethernet switch and Ethernet hub technology: the segmentation of collision and contention domains creates more determinism in the network and permits much greater scale.

In addition, when using 802.11a for the backhaul protocol, the greater number of available channels permits re-use of channels as the network expands a distance beyond the wired or fiber connection. This re-use of channels is crucial to overcoming the scarcest limitation in the unlicensed wireless spectrum: the paucity of available channels. With virtually unlimited channel re-use in larger networks, they may scale to very large numbers of nodes without compromising performance.

The difficulty some users have experienced with earlier implementations of Third Generation technology is the requirement to manually define channel selection across the network. For very small networks this is less critical, but in larger-scale networks it rapidly becomes unwieldy. By contrast, Meshdynamics' distributed dynamic radio intelligence automates the process of channel selection and topology configuration. The distributed "radio robots" in each node automatically find other nodes in the network, choose the optimal channel and network uplink path, and continually monitor the environment to update these choices. This allows very large networks to be built without extensive pre-engineering, manual channel mapping, or antenna selection and aiming. Instead, new authorized nodes added to the network automatically join the existing network when power is applied, permitting rapid build-outs and expansion. Importantly, the network also dynamically avoids new interference sources and adjusts to other changes (such as a traffic accident downing a light pole supporting a mesh node).

In order to scale to larger sizes with adequate performance for high-demand data, video, and voice, both Third Generation and distributed dynamic radio intelligence are necessary. Meshdynamics provides the best choice for high performance without placing excessive pre-engineering and support demands upon network providers.

New Applications Enabled by Third Generation Wireless Mesh
 
Applications of earlier generations of wireless mesh technology were limited to a small number of node-to-node hops due to performance limitations, particularly the limited support for video- and voice-over-IP. This has created the perception that wireless mesh is suitable only for the delivery of basic networking such as casual web surfing. The emergence of Third Generation wireless mesh, with its inherent higher performance, is engendering many new and useful applications. Chief among these are applications requiring bandwidth to be extended across long distances by use of the wireless mesh backhauls alone. In these cases, it is impossible or impossibly costly to add additional wired or fiber network drops every two or three hops.

In one example, Meshdynamics wireless mesh nodes are being used along a national border to provide connectivity to distant locations in a long "string of pearls" configuration. In this case, there is no need to provide blanket WiFi coverage for client PCs or PDAs over the entire area, so the links between nodes are up to 14 miles in length. Mobile nodes mounted in security forces vehicles join the network dynamically and while in motion. Service radios in the vehicles provide connectivity for staff in the vehicles and operating nearby.

Similarly, a video surveillance application required extension of bandwidth into an undeveloped area with no installed high-speed infrastructure. Cameras are cabled directly to Meshdynamics nodes and linked back to a central site via many hops. Third Generation technology provides high performance and minimal delay and jitter to support the high fidelity video CODECs in use.

Meshdynamics' distributed dynamic radio intelligence technology, combined with Third Generation performance, is also key to a military application deploying sensors on moving combat vehicles with no fixed root wired or fiber connection. These vehicles are in constant motion in relation to one another, but information from sensors mounted on each must be brought together for threat analysis. The network topology must constantly and automatically adapt to the varying distances between vehicles while the mesh must deliver high performance with very low latency and jitter to permit the sensor data to arrive in a timely fashion. This combination of high performance and automated topology flexibility is enabling many other mobility applications that are not possible with other hardware-oriented Third Generation wireless mesh solutions.

Even in more traditional metro/muni applications, Third Generation technology is being used where earlier generations of wireless mesh have failed. In some localities, high speed internet infrastructure is not yet available from cable or telco providers or is prohibitively expensive. A single high- speed connection at the root node must be extended over a broad area using only the node-to-node connections for a backbone. This requirement for many hops resulted in the removal of earlier-generation wireless mesh and replacement with Meshdynamics equipment. The ease of deployment that comes with distributed dynamic radio intelligence can also be a benefit in these underserved areas where skilled RF engineers may be in short supply. And the reduced number of wired or fiber drops contributes to a lower total cost of ownership, permitting these networks to be deployed more quickly and more broadly.

Conclusions

New mesh requirements (more hops to cover larger areas, more efficient bandwidth distribution, better latency and jitter for Video and VOIP) have given rise to the Third Generation of mesh architectures. Third Generation multi-radio backhaul architectures deliver the higher bandwidth and more-deterministic performance necessary to meet these new requirements. While all Third Generation solutions deliver demonstrably higher performance, some custom hardware-oriented solutions come with requirements for extensive pre-engineering and offer limited capabilities for mobility and avoidance of interference. By contrast, software-oriented approaches based on distributed dynamic radio intelligence support frequency agility, automated channel selection, dynamic topology configuration, and radio agnostic meshes, providing more effective single framework solutions for larger scale and diverse application environments. This combination of features delivers higher performance for traditional metro/muni applications, but is also opening many new applications for wireless mesh.

About Meshdynamics

The Meshdynamics team began developing wireless mesh technology in 2002, with production unit shipments late in 2005. Meshdynamics' customers worldwide deploy our products in many applications: municipal (metro) networking; video surveillance, homeland security/defense; transportation; military; and public safety. In addition to these sales of the standard products, Meshdynamics maintains substantial custom development activity for key defense applications. Much of the software pioneered for these demanding customers has been utilized as the basis for Meshdynamics' standard product line. The Meshdynamics design is radio-manufacturer independent, allowing the rapid addition of new radio frequencies, radio system suppliers, and new technologies such as WiMAX and other emerging standards. Meshdynamics' technology is patented and patent-pending, developed by teams of engineers in the USA and India.