Smart network redundancy for a smart city

A common practice for evaluating network security and redundancy initiatives is to balance the costs against potential losses from a loss of connectivity. But when you’re literally embedded in the functioning of public services like education and healthcare in the entire municipality, it can be difficult to put a price tag on the peace of mind. The IT-department of Borås Stad is one such example – one of its primary roles is to take care of the network infrastructure for more than 30,000 users in schools, care homes, libraries, theaters, public transport, emergency services, even the city hall itself.

“We run a big, complex environment and, being a municipality, we own an incredible number of IP addresses. Basically speaking, a lot of public entities are relying on our Internet connection for their day-to-day operations, so if something happens to it – we have a problem”, says Andréas Alfinsson, Business Development Manager at the IT-department of Borås Stad. “At some point, the cost of adding redundancy is a pretty small thing to consider. The price in itself may not be the only important factor, however we are required to have the best possible solutions for the least amount of taxpayer’s money”.

So what can you do when the Internet connection is so critical for your organization? If you have a single connection and it goes down, you’ll be left high and dry until it is repaired. But let’s assume you do have diverse routing to the ISP – dark fiber, MPLS, and so on. What if the service provider itself goes down? ISPs can have problems too, especially in today’s troubled times of economic unrest, power shortages, and general instability. Fibers can get cut, and if you’re really unlucky – in a way that you don't have any redundancy.

The IT-department of Borås Stad had one such experience: due to what most probably was a configuration error, the backup connectivity didn’t kick in after the primary fiber was lost. So when Andréas Alfinsson landed the job at the municipality in December 2021, one of the very first tasks put before him was to figure out how to further increase the redundancy. At that point in time, the notion of using a secondary ISP had already been put in motion. But then there was the matter of the design.

“I come from an ISP environment, I’ve worked for 5 years at the biggest Internet Service Provider in the country. In Sweden, we have quite a lot of very reliable providers and the usual practice is to buy a connection and the diversity from the same company. Multihoming is not a common sight”, says Andréas Alfinsson. “However, I have a strong background in BGP and once I learned that our partner, NetNordic Sverige, could provide us with a powerful BGP technology, I knew what I wanted to do!”

Now, let’s entertain the following idea. Multihoming allows you to connect the organization’s network and IT infrastructure to two or more providers via the Border Gateway Protocol (BGP), the primary routing protocol used on the Internet. BGP basically enables you to hold the Internet routing table – several million entries saying who owns what on the Internet. In other words, not a feat that can be pulled off with your regular, “office” switch. So what if a municipality of the size of Borås Stad decided to act as its own “ISP”, or at least allow itself to handle all that complicated routing in case of emergency?

Of course, we need to consider the law which forbids public entities in Sweden from competing the private sector companies for delivering Internet services. However, when the neighboring municipality is doing the same thing, e.g. they have acquired an AS number and they're also using BGP technology, it’d be perfectly fine to connect your two networks together and dual-home your Internet connectivity to two separate providers. So in other words, if the ISP in your region would go down, you could divert all your traffic through the other commune.

Double the benefit, half the cost? Yes, but on one condition.

“A scenario like that poses significant technical challenges since not only are you holding the Internet routing table, which is already big, you're holding it twice because you have to route in both directions. The switch would need to decide where to send all of the traffic that's flowing through, every single packet, and if something goes wrong, all of that routing will need to immediately shift”, says Albert Kiippa, Account Manager at Extreme Networks. “It may sound like a simple task, but when we're talking about the sheer amount of traffic flowing in the municipality like Borås Stad, you really need specialized equipment”.

Fortunately, Borås Stad was more than equipped for the job.

They say “when in Rome, do as the Romans do”. So when you’re gunning for the same capabilities as ISPs have, you need to use similar technology. The pair of SLX 9640 router boxes deployed at the two data centers owned by Borås Stad proved to be the perfect match for what the municipality needed.

“The SLX is a very powerful compact deep buffer platform, designed primarily for ISP applications to cost-effectively deliver the scale and performance needed to address the explosive growth in network bandwidth, devices and services. Thanks to the exceptional computing power and flexible architecture, the solution supports diverse deployment options that require deep buffering for lossless forwarding, such as data center interconnect and automation, collapsed border routing or, like in the case of Borås Stad, Internet border”, Simon Inns, Senior Systems Engineer at Extreme Networks explains.

Since the implementation of the SLX 9640 boxes, the IT-department of Borås Stad has not experienced any unexpected incidents. But that’s not to say that the investment has not started to pay off already.

“It feels good to know that we’re one step ahead. I also think that these machines allow us to expand into the future because, just like with bandwidth, we have a lot of space. Particularly now, since we can use both our primary and secondary ISPs, we are utilizing about 10-15% of that capacity normally, in our daily use. But when there's a big peak, probably in about 5 years, it will be automatically dispersed”, says Andréas Alfinsson. “We do not, at the moment, steer the data. So it always takes the shortest path for everything. At the moment it's actually using our secondary ISP for 90% of the traffic. But should our primary ISP do something and change, or the services that we're using change, it will just automatically steer off, so we always have the shortest path. And that's another advantage of having two ISPs because, especially in healthcare, some delays are actually important”.

The engineers at the IT-department of Borås Stad didn’t have much experience with BGP prior to the implementation – after all, it’s not a typical piece of technology for a municipality. Even Andréas Alfinsson, with his ISP background, didn’t have a chance to work with actual SLX switches before. But instead of going through a long and painful exercise of setting up this quite specialist thing alone, Andréas and his colleagues could count on their partners at NetNordic Sverige, who in turn recommended leveraging Extreme Professional Services.

“A highly skilled Extreme specialist came up and showed us how to set up everything and get the SLX switches up and running. Once he produced the configuration, I was like ‘okay, it seems simple now!’. The professional jokingly asked if we even needed his services because we handled it so well, but the truth is, he helped us a lot. There are always the small details, and since we're running a society-critical infrastructure, it's really nice to have professional services so you don't miss that tiny gap that something can slip through. Plus, if anything should happen, we still have our NetNordic consultant with BGP experience and background”, says Andréas Alfinsson.

Complex technology doesn’t have to be difficult to manage. Today, the dual SLX platform deployed at Borås Stad is being handled solely by Andréas Alfinsson, who is responsible for the periodic maintenance and backups. But other than that, it’s basically operating on its own.

“BGP is really nice in that way – once you have it up and running, it just keeps running. It's what drives the entire Internet. You don't need to do anything more complex. Usually, with this kind of technology, you must work with the equipment on a daily basis, but so far we haven't had a need to interact with it at all”, Andréas Alfinsson sums up.

“Extreme technologies and solutions keep providing Borås Stad with new, advanced capabilities that enhance its digital transformation to better engage and serve their workforce and citizens. Together with our partners from NetNordic, we have established one of the largest cloud-managed network infrastructures in Sweden, pushing the boundaries of what the Borås Stad’s smart city status truly means”, Albert Kiippa sums up.

NetNordic’s focus is on creating and delivering smart intelligent integrations, specializing in solutions and services for mission critical infrastructure within the area of Cyber Security, secure Data Centers, secure Networks and secure Integrated Communication. The company acts as an enabler for enhancing the mission critical business, and is a leading Nordic provider of cloud and infrastructure. NetNordic simplifies complexity.

NetNordic’s mission is to build and deliver solutions and services for a better society. The company has approximately 500 employees, represented in subsidiaries in Norway, Sweden, Denmark and Finland. In total, NetNordic has 13 offices across the Nordic region.