News

How and why we evaluated new border routers

When it comes to high-performance routers for large networks, the names of industry giants such as Brocade, Cisco, or Juniper quickly come to mind. Routing and forwarding usually takes place in specifically designed hardware in combination with proprietary software of the respective manufacturer. However, as a cloud provider that has open-source embedded in its DNA, we wanted to expand the field of potential solutions.

The main requirement for our new border routers was to have 6-8 ports with a throughput of at least 10 Gbps – here we reached the limits of our existing solution, which was one of the main reasons for a replacement. We also needed several 1 Gbps ports to connect the management infrastructure and other network components. Redundant power supplies and the possibility of out-of-band management were of course a prerequisite. With regard to the software, our catalogue of requirements contained the usual protocols: OSPF and BGP.

Thanks to the preceding evaluation of a new leaf-spine setup, we became aware of Free Range Routing (FRRouting): This open-source project is backed by several vendors (including Cumulus Networks) with the goal of developing a stable and scalable routing daemon. Since spring 2017, FRRouting has been under the umbrella of the Linux Foundation.

Thus, with the right hardware, FRRouting could be a promising candidate for our border routers as well. We considered hardware from Intel (our partner for compute and storage nodes) and from Lanner, whose x86-based appliances stand out with their high density of modular network ports.

PoC and migration in two steps

After a proof of concept with FRRouting (first virtualized with Linux KVM and then bare-metal on Lanner hardware) we were convinced: This setup not only covers our performance and reliability needs, but also integrates perfectly into our general architecture, thanks to running on Ubuntu. Ubuntu, the operating system of our choice for the infrastructure at cloudscale.ch, is also FRRouting's reference test platform for Linux.

An important advantage of the new setup is that we can virtualize our entire network topology, including the new border routers, in our lab. This helped us with the planning of the upcoming migration, because that way we were able to test all installation processes and the developed configuration as often as we wanted without disturbing productive operation.

In a second step, we replaced the existing routers with the new devices during a maintenance window and thereby successfully completed the migration.

Advantages and possibilities of the new architecture

Thanks to solid performance, the possibility of expansion by adding interface modules and the active development of FRRouting by its broad community, our new routers are ready for the future. They also fit seamlessly into the aforementioned leaf-spine setup from Cumulus Networks, which we will put into operation in summer 2018. Using an identical software basis not only guarantees best compatibility, but also efficiency in maintenance. Finally, in addition to the completely redundant setup, the attractive price of the new hardware also allowed us to purchase a fully equipped spare device.

With the successful migration, we have also established the basis for integrating our border routers into our configuration management even better – in the future, we will make all changes to the network configuration via this central tool. Once all network components are running on FRRouting, we will be ready for the next milestone: the conversion of our network to BGP unnumbered to reduce its complexity and the consumption of IPv4 addresses.

Open and optimally connected.
Your cloudscale.ch team

PS: On November 9, 2017, our CEO Manuel Schweizer gave a presentation about FRRouting and BGP unnumbered at the SwiNOG-Meeting in Berne.