Hyper-scaling on Openstack with Open Source tooling. A use case in deploying hyper-scale grid computing on Open Telekom Cloud

Transcription

1 Hyper-scaling on Openstack with Open Source tooling A use case in deploying hyper-scale grid computing on Open Telekom Cloud

2 Why Open Source? Open Standards Standards Organizations: IEEE, ISOC, W3C, IETF, IAB, ETSI, OASIS, NIST, CSCC, ANSI Community Lead Contributor Model Community Tested Open Design Cloud Design Patterns Application Design Patterns Distributed Systems Design Patterns Open Choice No-Lock in Interchangeable Components Interoperability Reusability Invest in People not Software

3 Why Openstack? User Benefits Community: The OpenStack community is nearing 10,000 members in nearly 100 countries, with foundation support by more than 200 companies. Mixed Hypervisor: OpenStack is the only solution that allows for mixed hypervisor IT environments Open Source: Open source solutions are now widely accepted and adopted after 20 years in the enterprise, paving the way for OpenStack success. Standards: provides a common, open standard through its API and allows portability between cloud environments. T-Systems Hybrid Cloud and Public Cloud based on Openstack (Same Architecture) providing interoperability and ease of migration or Burst scenarios. Benefits to Cloud Provider Community: Sharing ideas with the community allows our Developers to learn fast and deliver quicker. Contributing advances the entire ecosystem Research & Development: Lowers our R&D cost and speeds time to market for new PaaS services. Open Source: Interoperability with most open source projects such a Kubernetes, Spark, Etc providing us the ability to deliver standardized services. Standards: Standardization allows us to delivery Public Cloud and Private Cloud solutions architected exactly the same. We can deliver new PaaS services within days of requirements from customers.

4 Openstack Contribution Source: stackalytics.com Adaptability and access to the source code The open architecture design gives in-house IT more scope to tailor the technology to specific needs Participating in the community enables us to influence developments Community: many providers and solutions that enhance functionality Establishment of multi-cloud scenarios Hardware neutrality: connectors are available for almost all storage, server and network components on the market High degree of standardization (open APIs) Resources can be provisioned and marketed across provider boundaries Data and applications can be transferred from provider to provider

5 How do we contribute to the Open Source community We developed the Tricircle project which allows Openstack to be a viable Public Cloud architecture. We contribute regularly to the project listed to the right as leader or team member. We are on the board of governors for Openstack The Openstack and Open Source community benefit from much of the Research and Development work we do for the public cloud and private cloud services. We have circa 5,000 engineers dedicated to the Openstack projects for Public and Private Cloud Solutions who are leading the way on AI, ML, IOT, Kubernetes Container services. Corporate member of the Linux Foundation and on governing board of CNCF, Member LF Deep Learning Foundation. Zaqar Kuryr Tacker Rally Ceilometer Search Light Karbor Manila Neutron Zun Heat Tricircle

6 Hybrid Cloud Solution based on OpenStack Small scale O&M Team Budapest TSI Datacenter Public Cloud based on Openstack Datacenter of choice Public Cloud Architecture on-premise at customer site or hosted location Agility and scalability built in Cloud Native capability Data link (Hybrid/Private Region to public OTC) E.g. Ethernet Connect or MPLS Management Link (OPS public OTC) Management Link (OPS Hybrid/Private Region)

7 Deployment Tooling with Terraforms and Ansible

8 What is Terraform? Terraform is a multi-cloud DevOps tool for building, changing, and versioning infrastructure safely and efficiently. Available as Open Source or Enterprise License. Enables Infrastructure-As-Code deployments Open Telekom Provider developed for Terraform

9 What is Ansible? Application Deployment Tooling Agentless Architecture Integrates seamlessly with Terraforms

10 Grid Computing in Finance Industry - Use Case for OTC

11 What is Grid Computing Grid computing is a distributed architecture of large numbers of computers connected to solve a complex problem. In the grid computing model, servers or personal computers run independent tasks and are loosely linked by the Internet or low-speed networks. Computers may connect directly or via scheduling systems.

12 Finance Use Case (Grid Computing) Business Requirements Bank was looking to cloud computing to carry out their complex calculations, marking a shift from investing heavily in in-house systems that are expensive to develop, maintain and upgrade. Regulatory and Business demands such as Fundamental review of the trading book (FRTB) Derivatives Valuation Adjustments (XVA) End-Of-Day & Monte Carlo IT Requirements Transforming IT Operations to DevOPS methodology. Standardized on Open Source tooling and systems such as Ansible, Terraforms, Kubernettes, Kafka, etc) Need for massively scalable cloud services for Grid Computing, Big Data, and Devops( Kubernetes, ML, Analytics, Open Banking) Opex payment model. Do not want to invest in on premise servers that are not used much of the time or ever have enough capacity. Must speed up time to results to the business (Currently 36 to 48 hours) on FRTB and XVA calculations.

13 Grid Computing Requirements and Challenges Technical Requirements Capacity requirements 26,000 vcpus minimum 50,000 vcpus Burst Primary Benefit to Bank Time is money Improvements in Processing time Grid Software: Tibco Gridserver Infrastructure deployment with Terraform Server Configuration with Ansible Elastic Compute (ECS) and Elastic Volume Service (EVS) Challenges to Delivery API Gateway overloaded Rearchitectected Openstack API services to handle up to 10,000 API calls a second. Server provisioning to slow due to delay in image services deployment of images. Improved storage network capacity and latency as well image distribution across pools). Required lots of very large hosts which were not already available. Deployed 2000 additional host to meet demand.

14 Deploying the Grid using Terraforms Full Grid deployed to OTC Terraforms Plan resource "opentelekomcloud_compute_instance_v2" "engine_node1" { count = "${var.vm_engine_count}" availability_zone ="eu-de-01" name = "${var.project}-node${format("%02d", count.index+1)}-az1" image_name = "Enterprise_Windows_STD_2012R2_XEN" flavor_name = "s2.medium.4" key_pair = "${openstack_compute_keypair_v2.keypair.name}" security_groups = [ "AD" ] #ANSIBLE scripts user_data = "#ps1\nget-netadapter\nnet user ansible Test1234+ /add\nnet localgroup administrators ansible /add\n#ps1\nget- NetAdapter -name 'Local Area Connection' Set-DnsClient - ConnectionSpecificSuffix cloudgrid.bank.com - RegisterThisConnectionsAddress:$true - UseSuffixWhenRegistering:$true\nSet-NetFirewallProfile -Profile Domain,Public,Private -Enabled False " OTC API Circa 2 Hours to deploy 15,000 vcpus