Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 SER1729BE Networking for Virtual Environments What s New and What s Next for vsphere networking world 2017 Content: Not for publication Sudhanshu (Suds) Jain and Pushkar Patil #world2017 #SER1729BE

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from ware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #SER1729BE CONFIDENTIAL 2

3 vsphere Networking ADDRESSING VARIED USE-CASES Storage Low latency, highbandwidth & predictable performance NSX High packet rate, fast packet manipulation, hardware offloads, All-IP networks vsphere Networking Telco & HPC Ultra-low latency, packet intensive workloads, scale-out Transforming World of Enterprise Scale, HPC-like and Highly Distributed Cloud Multi-tenancy, zero-trust, Specialized hardware and customization #SER1729BE CONFIDENTIAL 3

4 Agenda 1 vsphere Networking vsphere Networking What s New? 3 Innovations Moving Forward #SER1729BE CONFIDENTIAL 4

5 vsphere Networking 101 Basic Concepts Multicast NIOC v3.0 L3-vMotion with multiple TCP/IP stack world 2017 Content: Not for publication Network Monitoring and Troubleshooting Guest Introspection #SER1729BE CONFIDENTIAL 5

6 Virtual Switch Concepts 1 PG-A vsphere Virtual Switch VLAN tagged traffic Host A 2 PG-B 1 PG-A vsphere Virtual Switch Host N 2 PG-B Basic Concepts Virtual NIC vnic Virtual Port/s Virtual Switch PortGroup/s (PG/s) - vmnic / dvuplink - Physical NIC / Uplink #SER1729BE CONFIDENTIAL 6

7 vsphere Distributed Switch 1 PG vsphere distributed Switch vsphere Virtual Switch vsphere Virtual Switch Host A 2 vcenter dvuplink-pg Data Center 1 PG Host N 2 dvport Basic Concepts dvportgroup Central management dvuplink/dvuplinkpg #SER1729BE CONFIDENTIAL 7

8 VDS Architecture: Local Components 1 dvpg-a VDS (local component) dvuplink-pg Host A 2 dvpg-a VDS (local component) dvuplink-pg Host B dvpg-b 3 Host C A local component of the VDS is instantiated on each host vcenter dvportgroups, dvuplink PortGroup configuration on each host is pushed from vcenter VDS (local component) dvuplink-pg vcenter representation of the data center Physical data center Basic Concepts #SER1729BE CONFIDENTIAL 8

9 Vmkernel Interfaces Several services create dedicated interfaces, called vmkernel interfaces or vmknics. Example: 1 MGMT HA Management, High Availability, vmotion, NSX VTEP, etc dvpg-a dvpg-mgmt dvpg-ha dvuplink-pg vmotion VTEP dvpg-vmotion dvpg-vtep dvpg-a 2 vsphere Distributed Switch 3 MGMT HA Basic Concepts vmotion dvpg-b dvpg-mgmt dvpg-ha dvpg-vmotion dvuplink-pg Host A Host B #SER1729BE CONFIDENTIAL 9

10 VLAN Backed VDS One dvportgroup one VLAN on the physical infrastructure Coupling of virtual/physical is achieved with ToR integration Require End-to-end Layer 2 network Host A End-to-End L2 Physical Network vsphere Distributed Switch Host B VLAN 2 VLAN 1 Legend : dvpg-a backed by VLAN 1 dvpg-b backed by VLAN 2 dvuplink PG dvuplink Basic Concepts Physical infrastructure needs to be configured for VLAN 2 #SER1729BE CONFIDENTIAL 10

11 Host Uplink Connectivity Recommendations vsphere Distributed Switch Host A vmnic0 vmnic1 Avoid single point of failure Connect to separate network devices Up to 32 uplinks possible, Recommend 2x10G rather than N x 1G When using a limited number of physical uplinks Don t dedicate physical uplinks to vmknics Move all uplinks to VDS and share infrastructure and data traffic Enable NIOC different classes of service sharing uplink VXLAN HW capable NICs recommended (Emulex, Intel) Configure Port Fast and BPDU guard on Switch Ports No STP running on VDS VDS never bridge traffic between its uplinks cannot loop traffic Basic Concepts #SER1729BE CONFIDENTIAL 11

12 VDS Uplink Connectivity A Host is typically connected via several uplinks for redundancy and added bandwidth Redundancy The host can sustain the loss of an uplink Link failure results in degraded bandwidth (and this might have an impact on operations) Added Bandwidth First, remember that 2x10Gbps uplinks is not equivalent to a 20Gbps uplink 20Gps: Lucky VDS 10Gps: Unlucky VDS vmnic0 vmnic1 vmnic0 vmnic1 Basic Concepts ToR 0 ToR 1 ToR 0 ToR 1 2x10Gbps uplinks provide between a theoretical 20Gbps down to 10Gpbs Efficiency depends on even packet load balancing, VDS provides several options (detailed later) #SER1729BE CONFIDENTIAL 12

13 Load Balancing Options in Traditional VLAN Backed VDS Teaming options only differ on how they spread traffic. vmnic0 1 2 dvpg A dvpg B VDS vmnic1 Explicit Failover Granularity: Port group All traffic to/from a given port group is pinned to an uplink vmnic0 1 dvpg A VDS 2 vmnic1 Source port hash Granularity: All traffic to/from a particular is pinned to an uplink vmnic0 Flow A 1 dvpg A VDS vmnic1 Flow B IP Hash/LACP Granularity: flow A particular flow is pinned to an uplink Basic Concepts #SER1729BE CONFIDENTIAL 13

14 vsphere Multicast Up to vsphere 5.5, multicast support is achieved based on mac address filtering: registers for a certain IP multicast group Programs the vnic to receive the corresponding L2 multicast mac address The VDS only delivers matching mac addresses Limitations: Mac address collision inefficiency world 2017 Implementation allows for 32 mac address filter per vnic, beyond that multicast is delivered to uninterested s No source specific filtering possible e0b.0102 destip: destmac: e0b.0102 Multicast e0b.0102 VDS on Host A Content: Not for publication #SER1729BE CONFIDENTIAL 14

15 vsphere Multicast vsphere 6.0 introduces: IGMP v1-3 snooping for IPv4 multicast MLD v1-2 snooping for IPv6 multicast (S2, ) (S2, ) (S2, ) (S1, ) (S1, ) No collision (does not rely on mac addresses) Capable of source specific filtering Up to 256 multicast groups can be handled per port (S1, ) world 2017 VDS On Host A Multicast Content: Not for publication The implementation allows for precise and efficient multicast delivery within the host*. (*IGMP/MLD snooping information not used to prune traffic between hypervisors on a VXLAN backed logical switch) #SER1729BE CONFIDENTIAL 15

16 Goal for NIOC Protect and prioritize important traffic when there is contention on the uplinks Gold Gold dvpg VDS Bronze Bronze dvpg Contention NIOC VDS/Manage/Resource Allocation #SER1729BE CONFIDENTIAL 16

17 NIOCv3: Reservation Appears NEW Minimum bandwidth can be reserved for system traffic or NIOC distribution VDS/Manage/Resource Allocation/System Traffic #SER1729BE CONFIDENTIAL 17

18 System Traffic Rules Enforced at the Uplink world 2017 Content: Not for Gold 1 Gold dvpg VDS Gold 2 NIOC publication VDS/Manage/Resource Allocation/System Traffic #SER1729BE CONFIDENTIAL 18

19 Configuration Enforced at the vnic! Hardware Configuration world 2017 Content: Not for Gold 1 Gold dvpg VDS Gold 2 NIOC publication VDS/Manage/Resource Allocation/System Traffic #SER1729BE CONFIDENTIAL 19

20 Integration with the Distributed Resource Scheduler (DRS) NIOCv3 allows DRS to take network requirements into account Newly created s are dispatched to hosts that can accommodate their bandwidth reservation If the network cannot meet the constraints of a, that latter cannot be started NIOC A can be moved dynamically to a different host by DRS if its bandwidth constraints are not met any more (ex. following uplink failure.) #SER1729BE CONFIDENTIAL 20

21 Example: DRS in Action with NIOCv3 Uplinks have been configured to reserve 500 Mbps of traffic 2 s, each reserving 300 Mbps are up on Host 1 An uplink on Host 1 fails: the reservation for the is not met 300Mbps 500Mbps Uplink Failure Gold 2 Gold dvpg VDS Gold Mbps = 600 Mbps Host1 500Mbps 500Mbps Gold dvpg VDS Gold 1 Host2 300Mbps Mbps Demo Cluster NIOC #SER1729BE CONFIDENTIAL 21

22 Example: DRS in Action with NIOCv3 Uplinks have been configured to reserve 500 Mbps of traffic 2 s, each reserving 300 Mbps are up on Host 1 An uplink on Host 1 fails: the reservation for the is not met The DRS evacuates automatically one to Host 2 300Mbps Gold 2 Gold dvpg VDS Gold 3 Host1 300Mbps 500Mbps 500Mbps Gold dvpg VDS Gold 1 Host2 300Mbps Mbps NIOC #SER1729BE CONFIDENTIAL 22

23 What Do We Call L3 vmotion and Why Would I Want It? L3 vmotion: achieved leveraging Layer 3 connectivity only, no Layer 2 needed Why? ESXi Host1 L3 Network Infrastructure No Layer 2 Required ESXi Host2 Because I am expanding my network with hosts in a different subnet (with no L2 connectivity) Because I don t want to stretch L2 in my network L3 vmotion #SER1729BE CONFIDENTIAL 24

24 vmotion over a Layer 3 Network Two kinds of traffic are involved during vmotion: 1 vmk1 ESXi The guest traffic: 2 L2 Host1 L3 Network Infrastructure There need to be L2 adjacency between source and destination This is achieved by running L2 over VXLAN VXLAN The vmotion infrastructure traffic used to sync the state: This traffic is IP already routed from a vmkernel interface L3 ESXi vmk1 Host2 L3 vmotion #SER1729BE CONFIDENTIAL 25

25 So, What is the Problem? We want vmkernel interfaces on different networks for security reason But (most) vmkernel interfaces share the same IP stack The default gateway for this stack is configured for the management traffic Default IP Stack default vmk0 mgmt ESXi vmk1 vmotion L3 Network Infrastructure Default IP Stack default An easy solution is to have all the vmotion vmknics in the same subnet vmotion vmknics don t need a default gateway if all their peers are in the same subnet vmk0 mgmt ESXi L2 vmk1 vmotion L3 vmotion #SER1729BE CONFIDENTIAL 26

26 How about Putting vmotion vmknics in Different Subnets? Because the default gateway is already used by the management vmknic, we need to add more specific static routes for remote networks Default IP Stack default vmk0 mgmt ESXi If there are many remote networks, several static routes must be entered On all the hosts Error prone vmk1 vmotion L3 Network Infrastructure Default IP Stack default ESXi world 2017 Not supported by ware, unless the customer goes through the RPQ process vmk0 mgmt vmk1 vmotion L3 vmotion Content: Not for publication #SER1729BE CONFIDENTIAL 27

27 Solution: Multiple TCP/IP Stacks Provide several IP stacks: this way, vmotion vmknic can have its own default gateway Default IP Stack default vmk0 mgmt ESXi vmotion IP Stack default vmk1 vmotion L3 Network Infrastructure Default IP Stack default vmk0 mgmt ESXi vmotion IP Stack default Other benefits: better isolation (memory heap, ARP tables etc ) VXLAN already had its own TCP/IP stack vmk1 vmotion L3 vmotion #SER1729BE CONFIDENTIAL 28

28 Configuration L3 vmotion #SER1729BE CONFIDENTIAL 29

29 Network Operation & Visibility Gain Valuable Insight for Efficient Operation - Poll data - Receive events - Collect network and application flows Integration Options SNMP MIBs APIs IPFIX NetFlow Use cases Monitor interface and service statistics Discover network topologies Application and network performance Capacity planning Network Operation & Visibility - Capture packet data - Collect logs Port Mirroring Logs Traffic Copy Packet level troubleshooting and root cause analysis Log analysis #SER1729BE CONFIDENTIAL 30

30 Host Level Packet Capture Host A 1 Logical SW A dvuplink-pg VXLAN VTEP dvpg-vtep Generic IP Fabric vsphere Distributed Switch CLI based lowest level troubleshooting tool Can operate at any stage in the packet s life like: dvport Vmknic Uplink Extensive range of filters such as source/destination mac, IP, Protocol, VLAN, VXLAN, ports etc pcap format output for use with protocol analyzers such as Wireshark Network Operation & Visibility #SER1729BE CONFIDENTIAL 32

31 Leverage vsphere Data for Operation & Visibility e.g. vrealize Network Insight with vds, IP FIX, vcenter, NSX Manager Data Web App DB Network Operation & Visibility #SER1729BE CONFIDENTIAL 33

32 Guest Introspection Windows 10 Support Guest Introspection File, user identity, process (application), network connections, registry keys, etc. Simplified Deployment Automated deployment of 3 rd party appliance to all selected clusters in data center. Virtualization Platform Overview Guest Introspection support for varied GOS, leveraging agent less approach for security Get all asset inventory and context - file, process, registry key - via Guest Introspection Aligned with NSX EPSec partner support Benefits Real-time security without administrative and memory overhead using Agentless approach Increased security for VDI use cases Guest Introspection #SER1729BE CONFIDENTIAL 34 34

33 End-point Security: Leverage NSX Manager Available as part of vsphere Download Group Guest Introspection #SER1729BE CONFIDENTIAL 35

34 End-point Security: Third-party Solutions Available on ware Compatibility Guide Guest Introspection #SER1729BE CONFIDENTIAL 36

35 Agenda 1 vsphere Networking vsphere Networking What s New? 3 Innovations Moving Forward #SER1729BE CONFIDENTIAL 37

36 vsphere 6.5 What s New? 25/50/100G RDMA & pvrdma Performance Improvements SR-IOV Enhancement Default Gateway per vmknic Scaling Limits #SER1729BE CONFIDENTIAL 38

37 1G 10G 40G 50G 100G Datacenter is getting ready for next forklift in infrastructure world 2017 Good market traction for 25G Volume and cost curve is in favor 25/50/100G Many drivers analytical workloads, large datasets, all-flash storage, HPC, NFV Many challenges need hardware acceleration, stack optimization to deliver on promise Content: Not for publication OCP Project Olympus is designed with 50G standard connectivity for servers #SER1729BE CONFIDENTIAL 39

38 100G Host & 100G Bring your networking appetite over vsphere XNET3 ESXi Guest OS 1 Guest OS Gbps 100 Gbps K ESXi 1 K 100 Gbps 100 Gbps vsphere I/O is continuously evolving along with NIC speeds In vsphere 6.5, support for 100G bandwidths is added 25/50/100G Unlocking new possibilities for vsphere services/ apps ESXi can now achieve close to 100G with Single K interface A single can achieve 100G in ESXi 100 Gbps #SER1729BE CONFIDENTIAL 40

39 Tech Preview 25/50/100G vmotion on Steroids Faster Host Evacuations; Reduced Maintenance Window! Challenges Size is growing Monster is common deployment on vsphere vmotion could be challenging to perform for large Customers are engineering network with multiple vmknic ; complex networking, additional port, cabling cost vsphere vmotion over 100G Ethernet Approach Accelerate vmotion using higher speeds/feeds; 50/100G Ethernet Performance optimize vsphere networking stack and kernel to leverage single vmknic to achieve higher throughput No more multiple Ethernet ports just for vmotion, simplified networking; lower TCO 41

40 Demo: 100G vmotion Faster Host Evacuations; Reduced Maintenance Window! Tech Preview 25/50/100G

41 Upping the Stake in Networking Stack Strengthening the core stack for 25/50/100G vnic1 Virtual Switch PNIC vnic1 vnic2 vnic1 Virtual Switch vnic1 PNIC vnic2 vnic2 Improved vmxnet3 data-path Single thread per is a bottleneck Performance Improvements Virtual switch optimization improve caching mechanism, lock enhancements Build data-path option to address workloads needs Virtual Switch Virtual Switch PNIC PNIC * - All items listed in green are in vsphere 6.5 #SER1729BE CONFIDENTIAL 44

42 100G Demo Throughput Benchmark NetPerf NetPerf 25/50/100G #SER1729BE CONFIDENTIAL 45

43 100G Demo Throughput Benchmark NetPerf NetPerf 25/50/100G

44 GENEVE Offloads Addressing NFV and NSX Needs Build Logical Flow with Confidence world 2017 Performance Improvements Single Tunnel mechanism for all your flows Build complex networking flows for mutliservice traffic Content: Not for publication Software Defined, on commodity Ethernet, no need of specialized hardware lower TCO Data Source: FMS, 2016 #SER1729BE CONFIDENTIAL 47

45 Default Gateway per vmknic Eliminate the need for static routes! world 2017 Default Gateway per vmknic Gateway setting overridden at kernel port level No need for a separate netstack instance, no need for static routes Content: Not for publication Independent L3 connectivity for services using vmkernel ports Initial support for vmotion and provisioning services #SER1729BE CONFIDENTIAL 48

46 Introducing RDMA over Converged Ethernet (RoCE) Brings true convergence over single fabric RDMA over Converged Ethernet (RoCE): Provides the benefits of RDMA for existing Ethernet data center infrastructure Why RoCE? Classical NIC Deployment: Most widely deployed RDMA solution over Ethernet Link-Speeds: Available for all Ethernet speeds, including 25/50/100G RDMA NIC OS Ubiquity: Drivers available in Red Hat, SUSE, Microsoft Windows and other common operating systems Low Latency: Lowest latency for Ethernet in the industry Tremendous Ecosystem Support: IBTA standard, major NIC vendors, all major OEMs, all major OSs Build single Ethernet fabric to deliver all your datacenter needs! RDMA & pvrdma #SER1729BE CONFIDENTIAL 49 49

47 Para-virtualized RDMA (PVRDMA) Industry s First Virtualized RDMA Solution Guest OS 1 RDMA App libvrdma PVRDMA NIC Buffers libibverbs PVRDMA Driver PVRDMA Device Emulation ESXi RDMA Stack vsphere Guest OS 2 RDMA App libvrdma Buffers libibverbs PVRDMA Driver vsphere PVRDMA Device Emulation ESXi RDMA Stack RDMA & pvrdma Ultra-low latency near bare-metal RDMA performance Auto-detection of peer end-points and connection management Best of both world ultra-low latency with live vmotion and DRS (future) Improve application performance and infrastructure efficiency == $$$ HCA Device Driver HCA Device Driver Works without underlying RDMA hardware perfect tool developer environment NIC RoCE (RDMA over Converged Ethernet) NIC Control Path Data Path #SER1729BE CONFIDENTIAL 50

48 SR-IOV Addressing the Market Needs Bring SR-IOV mainstream for networking and other use-cases SR-IOV Enhancements To configure s, needs to specify the exact PCI slot location (Bus:Device.Function)/PF Flexible Provisioning: Automated assignment of physical functions. Enable automation of provisioning #SER1729BE CONFIDENTIAL 51

49 Tech Preview SR-IOV Enhancements Enable I/O Technologies via SR-IOV Faster Route to Market via Generic SR-IOV Certification PCIe NIC GPU/GPGPU SR-IOV Interconnect like IB and OPA Hardware Accelerations e.g QAT FPGA Evolving landscape of connectivity and I/O technologies Need to deliver faster route to market for workloads to take advantage of it SR-IOV is one such enabler technologies Building a program for technologies to be enabled as generic SR-IOV based I/O technologies Storage Controller Ne SSD PCI SSD #SER1729BE CONFIDENTIAL 52

50 Tech Preview Scaling Limits Scaling NIC Hardware Limits Enable with Server Vendors Partnerships world 2017 Content: Not for vsphere Maximum Configuration Publish Limits limited configuration available Extending the published limits using sever vendor certification program publication Aligned with Server vendor GTM plan to support specific configuration on vsphere platform #SER1729BE CONFIDENTIAL 53

51 Agenda 1 vsphere Networking vsphere Networking What s New? 3 Innovations Moving Forward #SER1729BE CONFIDENTIAL 54

52 EXPLOSIVE GROWTH OF DATA Key Infrastructure Challenges Build Capacity world 2017 NEAR REALTIME Need efficient access of capacity HIGHLY DISTRIBUTED Need efficient access of capacity everywhere Content: Not for publication #SER1729BE CONFIDENTIAL 55

53 Explosive Growth of Data Stressing your storage and network benchmarks Cloudera Zookeeper Source: Intel IDF 2016 #SER1729BE CONFIDENTIAL 56

54 Near Real-time Need to act fast What we do with data is changing Act fast to gain business value Source: #SER1729BE CONFIDENTIAL 57

55 Highly Distributed Build, deploy and manage interconnected, collaborative workloads & infrastructure Big Data Cloud-Native App Multi-tier App Distributed Storage world 2017 Content: Not for Virtual SAN Deploy multiple workloads with strong demand for inter- traffic Optimize data delivery to applications publication Policy driven application management for network and security policy #SER1729BE CONFIDENTIAL 58

56 Moore s Law is Transforming Power is delivered via Convergence and Consolidation CPU Traditional Processors Why the change? Keep buying power from consolidating Multi Core Processors HPC Hardware Market Shift Workload innovations demand more value from hardware Industry Impacts: N Compression Crypto Fast IO FPGA Remoting Container Security Application & Infrastructure Optimization No more CPU focused platform; need to invest in hardware offloads and convergence Require more vertical integrations GPGPU Convergence Smart NIC Opportunity to expand and differentiate SDDC offering with hardware innovation and integration #SER1729BE CONFIDENTIAL 59

57 Storage Industry Under Transformation Evolving space of compute, storage and interconnect Technology Trends CPU densities continues to increase Hi-Density Flash and NVDIMM will dominate Enterprise Storage High speed interconnect to keep up with fast storage world 2017 Content: Not for publication Higher CPU density, faster data access and high-speed interconnects, all are changing the paradigm of IT infrastructure designs These trends are already changing how enterprise storage solutions designed and deployed; disrupting the complete ecosystem of SAN as well as DAS market Source:

58 What Constitutes Modern Network Fabric? True convergence with datacenter as a computer Easy to scale and manage Performant with various acceleration Inherently secure #SER1729BE CONFIDENTIAL 61

59 Smart Ethernet Fabric Build the Data-centric Infrastructure Pools of Compute Ethernet rnic 10G/25G/50G/100G Ubiquity: Ethernet is everywhere High performance: 25/50/100G Ethernet and low latency Data Source: FMS, 2016 Ethernet Fabric provides multiple benefits High Scalability: Non-block topology and multiple Ethernet connections Deterministic: QoS & advanced congestion control Ethernet Fabric 10G/25G/50G/100G world 2017 Content: Not for Cost/Performance Optimized: Single fabric for all your scale-out needs Concept Storage/GPUs/ FPGAs/Accelerator ESXi CPU publication ESXi CPU ESXi CPU #SER1729BE CONFIDENTIAL 62

60 Accelerating Virtual Switch using DPDK High Performance Stack world 2017 Content: Not for Tech Preview Increasing network speed need better handling of resources in the stack cache, memory, interrupt etc publication Today s DPDK solutions have limited support Build the performant stack without sacrificing the agility and flexibility of virtualization Transparent to applications, make it easier to consume DPDK like concept without the need of engineering your application #SER1729BE CONFIDENTIAL 63

61 Application-Aware DRS and vmotion Build Network Intelligence at Every Layer Service A On-Premise Infrastructure Service B1 DRS and Netflow automatically identify and co-locate cooperating s Service B2 Stretch or Public Cloud Infrastructure Service C1 Tech Preview Service A1 Service A2 Service C2 Workloads are migrated as a single unit to reduce Cross-Cloud traffic #SER1729BE CONFIDENTIAL 64

62 Secure Fabric Zero-trust is key to success Security Simplified Management Security Management Compute Security world 2017 Data Security Concept Network Security Content: Not for publication Secure Boot, vtpm Forensic, App Defense, Remote Host Attestation encryption Encryption, VSAN vmotion Encryption Data-in-flight encryption Micro-segmentation #SER1729BE CONFIDENTIAL 65

63 vsphere Host Fabric Build Your Infrastructure with Confidence vsphere Host Local Storage FS Shared Storage FS/NFS vvol DataStore L2 Switch Firewall Load Balancer L3 Router vsan Datastore VPN Internet Software Defined Best of Breed Converged Performant * - It is a logical representation and may vary for different services #SER1729BE CONFIDENTIAL 66

64 vsphere Networking: Summary and Takeaways Rich integration and optimization to address varied needs Best of breed innovations and architecture to deliver value Foundation to deploy Software Defined Infrastructure 67

65

66