The Google SDN WAN An Open Networking Foundation Interview

Transcription

1 The Google SDN WAN An Open Networking Foundation Interview How has this pioneering OpenFlow deployment shaped up over two years? Jim Wanderer, Director of Engineering, Platforms Networking, Google, gives his opinion. There s intense curiosity about Google s inter-datacenter SDN deployment using OpenFlow. That s hardly surprising given that OpenFlow is so recent and has, on the one hand, been unfairly dismissed by some as an academic exercise yet to bear fruit, while on the other we have here an example of not only a large scale deployment by a giant among Cloud service providers, but also one that has already clocked up two years of operational experience. I can sum up that experience by saying OpenFlow SDN has worked really well for us, that Google couldn't have achieved the results it has without SDN. We could have used other approaches, but there s no way that the results would have been as effective. Finally, as a result of this success, Google is now committed to further SDN deployments. So, to satisfy some of that curiosity, and to answer those who might think as a member of the ONF board, Jim would say that, this article gives some more detail, and spells out the challenges and the benefits that have convinced us that SDN really is the future of networking. The opportunity Google is such a universally familiar service offering with Google+, Gmail, Google Maps, YouTube, as well as the best known web search engine that you might not realize that it has not one but two main backbones the Internet facing network for all that user traffic, but also an internal WAN linking several large datacenters in Europe, North America and the APAC region see Fig. 1. Fig. 1

2 The two backbones have very different requirements and traffic characteristics. In particular the internal WAN was not constrained by the severe SLAs of a public-facing service, and that allowed our development team more freedom to explore innovative approaches. With a dozen key datacenters linked across three continents it presented an exciting opportunity to put the then-nascent open SDN concept to the test in early The problems we wanted to address were all too familiar to those managing large networks: Big networks don t behave predictably enough. Failure response and performance are suboptimal. Difficulties remain in configuring and operating large networks. Dependency on manual, error-prone operations persists. In addition, we did not have the advantage of starting from scratch as in an academic pilot project we were needing to connect to existing networks. When the project began, OpenFlow-enabled hardware scalable to the network s traffic levels was not yet available, so Google built its own network switch from merchant silicon and open-source routing stacks with OpenFlow support. Our aim was to optimize the WAN routing for high performance and network utilization, while being able to monitor and control network behavior from a central point. A vendor-agnostic solution was needed because we already had a massive installation of equipment from nearly every major vendor. At each site we had multiple switch chasses allowing scalability to multi-terabit bandwidth as well as providing fault tolerance. The approach Centralized control is a feature of the SDN concept, but when spread over an intercontinental WAN it made more sense to have central controllers at each datacenter linked into an overall traffic engineering controller for the whole WAN. This is analogous to having an intelligent police presence directing traffic at each major road junction providing fast, low latency decision making with each linked by radio to a central traffic office that optimizes overall traffic strategy and passes instructions back to the police on duty. In addition, multiple OpenFlow controllers ensured no single point of failure. The centralized traffic engineering (TE) service collects real-time use and topology data from the network and calculates bandwidth demand from the applications and services. It then computes the best traffic flow path assignments and uses the OpenFlow protocol to program those into the switches. As demand fluctuates, or unanticipated events happen in the network, the TE service re-computes and reprograms the system.

3 With this OpenFlow control system in place, we now had enormous potential for radical re-thinking and experimentation with new networking concepts, but chose instead to begin with a fairly traditional Quagga BGP and ISIS system compatible with existing networks and familiar to the operations staff. The technology was new, so it was a safer bet to begin with a traditional feature set and then evolve from there, using the network s software definable capability. One initial advantage was that, as an internal network, it did not need to carry the full Internet routing table. Fig. 2 summarizes the approach: centralized traffic engineering (TE) layered on top of base system and connected with high traffic priority to OpenFlow controllers (OFC) at each site. Fig. 2 The first quarter of 2011 saw the SDN WAN up and running, and it worked! From that point the expansion has been ongoing, adding sites, increasing capacity and adding features. Test and development The first benefit of SDN for developing the network was that it worked really well in a multichassis environment. This can be difficult for vendors, getting an application to work consistently across a range of devices. With the control sitting outside, we could make multiple devices function as one logical device. Also, rather than being limited to the relatively small processing power within in a router, we could do much more intelligent management with the power of high performance servers hosting the local controller. We also used existing distributed-system software to support controller failover. Then, with SDN now in place, it enabled ongoing updating and addition of new traffic engineering and security applications see Fig. 3.

4 Fig. 3 When it came to testing, SDN offered a load of benefits, mostly around simulation. The OpenFlow agent, using V-Switch, is already a simulated network device and it is easy to run a mass of v-switches as a very large network. Even when the hardware is not yet available, control software can be tested on a very large scale, and new configurations or control tools can be trialed in advance without compromising a working network. An extension of this that we found very useful was to combine real and simulated network elements in the test, verifying the behavior with real devices while scaling the test to a greater network. One possible extension of this, yet to be tried out, would be to create an entire synchronized mirror network, subjected to the same real-world event stream but without executive action on the actual data flow so that changes could be tested in real conditions but without putting real data at risk. I outlined these powerful test opportunities made easy by OpenFlow at a recent presentation and one service provider came up to me and said you ve got it exactly right meaning however much work goes into the design of big networks, testing is the real key to success. Tests like those described could also be generated using traditional routers, but they become a whole lot easier with OpenFlow, and that is very important. Network testing is a vital but Cinderella task, and the more it is made easy and quick, the greater chance it won t be passed over. SDN in action Members of our operations team were the ones to appreciate the next set of benefits. One we had not fully anticipated was the way it made it so much easier to implement reliable failover. Using SDN and proven distributed system software we quickly got the system working with many fewer problems than normal in setting up reliable failover.

5 As well as preserving availability, it made upgrades especially tidy. Normally it would be done using two control cards, upgrading the backup card and then failing over to it while the main card is being upgraded. With SDN, however, you can do the equivalent by failing over to a second upgraded controller, without even touching the actual router. What s more, the OpenFlow protocol made it easy for the application state to be re-constructed accurately. Just as in the development and test situation, operations also want to trial new software under real conditions but without compromising the working network. With the control plane separate from the network hardware, it is possible to run the new version on a separate controller, exposed to the actual event stream but in a no operation mode where no actual changes take place. You can see how it behaves under the impact of real conditions, but without risking the actual data flow. Something else that would hardly be practicable in a traditional network would be to carve it into two parts running different versions of an application to see what difference an upgrade makes maybe trialing it on a quarter of the network while leaving the remaining three quarters with the old software see Fig. 4. Fig. 4 Network configuration on this scale is a major challenge, and it would be wrong to sell today s SDN as a magic potion to cure every configuration headache but it does offer significant advantages. For a start it means configuring networks and fabrics rather than hundreds of individual devices. Rather than pushing out a complex configuration over CLI, never a simple task, we build and run a management application to configure the whole network. And, as already explained, SDN made it much easier to pre-test configurations before roll out. So far, I ve run through the benefits that helped the operations team running the network, but what is a whole networking operation really about? It must be

6 about delivering the best possible WAN service for the users, and that would be the ultimate test for our SDN solution. Even in the early stages of development it was clear that centralized, nearoptimal traffic engineering was meeting the system s high performance demand while keeping costs down. High availability was assured by rapid response from local controllers, using the greater compute power of fast servers and with centralized traffic engineering then optimizing overall performance. Google s application teams and services have high expectations for the internal technical infrastructure, and the internal WAN is now helping us meet those expectations. Also, controlling ten or so devices as a unit so that the traffic engineering system no longer needs to be aware of each separate device gives us far better scalability. As demand changes the system can recalculate and push several new traffic solutions per hour in the past it could take hours to develop a single new solution. Challenges still exist With the above set of benefits one thing is sure for us: we have no regrets about taking the OpenFlow SDN path for this very large and crucial internal WAN. What s more, the development, testing, and operations experience has reinforced our belief that SDN is the future for networking. All the same, there have been some challenges, and it is up to the SDN community to share this sort of learning. For a start, managing this network is different. Operations staff are no longer just looking at network devices, they are looking at a system made of devices, controllers, and applications, and this requires new thinking. This highlighted the need for new tools that would consolidate this diversity of data and present it in a useful, graphic way to help the operations staff visualize and work with systems, rather than just devices. SDN does furnish the base structure to create such tools, but we still need to develop them. The basic issues of any traditional network latency, hardware programming speed, queuing, and packet I/O performance still need to be addressed in an SDN system, but it should by now be clear that we had greater flexibility to work around these classic problems in a more innovative manner. One example was a problem with our specific Open VSwitch implementation that arose early on as a consequence of having both protocol packet I/O and flow updates running over a single TCP stream. Our OpenFlow agent serialized the processing of all the OpenFlow commands, so you got large sets of flow updates with hundreds of route changes and these were holding back the time sensitive protocol I/O packets. Time-sensitive protocols then began to expire, creating further flow updates and blocking even more data. The solution was pretty simple we just added some advanced queue

7 prioritization but it shows that you do need to be aware of basic networking issues when developing SDN solutions and vendors should bear this in mind. SDN and the future of networking OpenFlow in particular, and SDN in general, are still in the early stages of development, but I can say with confidence that, even at this stage, OpenFlow has proved its value to us in optimizing traffic efficiency on an extremely large real-world WAN. The benefits we got from implementing a pioneering OpenFlow solution can be summarized in terms of: Unified network-centric approach Leveraging high performance servers for more sophisticated traffic control Faster, more consistent failover Quicker development and easier testing to accelerate deployment Low impact system upgrades Flexibility, no longer constrained by individual hardware device limitations Although we have only been describing benefits on Google s internal facing WAN, I can now truly claim that these are beginning to impact the Cloud services we provide in terms of faster response, better search, greater reliability and so on. The world needs a faster, better, and more reliable Internet, and it will continue to evolve. SDN and OpenFlow have an important contribution to make towards realizing these needs. For my part I will conclude with two main items on my wish list. First, we need more carrier-class OpenFlow-enabled devices, please. Google has gone a long way as a pioneer developing its own solutions, but we want a marketplace where we can shop for future developments. Secondly, as a member of the ONF board I am obviously a keen supporter of standards to ensure interoperability and build market confidence in SDN. However, in my personal opinion, we should focus on standardizing at the ground level getting a rock solid OpenFlow protocol but above that we must preserve freedom to innovate. The greatest gift of software-defined networking is that it liberates rigid structures to become flexible, programmable, and organically evolvable systems. Let s build that on a firm foundation, but let s also maintain a fertile, open environment for the exciting future of networking.