Improving efficiency of Twitter Infrastructure using Chargeback

Transcription

1 Improving efficiency of Twitter Infrastructure

2 AGENDA Brief History Problem Chargeback Engineering Challenges The product Impact Future

3 Getty Images from

4 Getty Images from

5 3283 Tweets Per Sec (TPS) Getty Images from

6 3283 Tweets Per Sec (TPS) 5X increase on avg. TPS Getty Images from

7 The Simpsons

8

9 MONOLITH SERVICES

10 FENCING & OWNERSHIP Clear isolation of services & its ownership. RELIABILITY Failure isolation and graceful degradation (Micro) Services Oriented Model SCALABILITY & EFFICIENCY Scale independently ensuring efficient use of infrastructure DEVELOPER PRODUCTIVITY Make it simple for engineers to build and launch services quickly and easily

11 2013

12 August 2 at 7:21:50 PDT

13 143,199 Tweets Per Sec (TPS) August 2 at 7:21:50 PDT

14 143,199 Tweets Per Sec (TPS) 28X increase on avg. TPS August 2 at 7:21:50 PDT

15 Hundreds and thousands of #events at any given instant

16 Most Retweeted Tweet in History

17 RELIABILITY DEVELOPER AGILITY SCALABILITY EFFICIENCY

18 Do More with Less

19 Fast forward to 2016

20 MANAGEMENT TOOLS SELF SERVE MONITORING AND ALERTING CACHE MESSAGING & QUEUES SOCIAL GRAPH USERS WORKFLOW MANAGEMENT INTERACTIVE QUERY FRAMEWORK/ LIBRARIES FINAGLE (RPC) SERVICE DIRECTORY SEARCH TWEETS DATA DISCOVERY SCALDING (Map Reduce in Scala) CHARGEBACK REVERSE PROXY PLATFORM SERVICES CORE APPLICATION SERVICES DATA & ANALYTICS PLATFORM JVM DEPLOY (Workflows) CONFIG INFRASTRUCTURE SERVICES C O M P U T E AURORA (Scheduler) HADOOP (Map-Reduce) MESOS (Cluster Manager) S T O R A G E MANHATTAN (Key-Val Store) HDFS (File System) BLOBSTORE GRAPH STORE HERON (Streaming Compute) INFRASTRUCTURE AND DATACENTER MANAGEMENT

21 MANAGEMENT TOOLS SELF SERVE MONITORING AND ALERTING CACHE MESSAGING & QUEUES SOCIAL GRAPH USERS WORKFLOW MANAGEMENT INTERACTIVE QUERY FRAMEWORK/ LIBRARIES FINAGLE (RPC) SERVICE DIRECTORY SEARCH TWEETS DATA DISCOVERY SCALDING (Map Reduce in Scala) CHARGEBACK REVERSE PROXY PLATFORM SERVICES CORE APPLICATION SERVICES DATA & ANALYTICS PLATFORM JVM DEPLOY (Workflows) CONFIG INFRASTRUCTURE SERVICES C O M P U T E AURORA (Scheduler) HADOOP (Map-Reduce) MESOS (Cluster Manager) S T O R A G E MANHATTAN (Key-Val Store) HDFS (File System) BLOBSTORE GRAPH STORE HERON (Streaming Compute) INFRASTRUCTURE AND DATACENTER MANAGEMENT

22 MANAGEMENT TOOLS SELF SERVE MONITORING AND ALERTING CACHE MESSAGING & QUEUES SOCIAL GRAPH USERS WORKFLOW MANAGEMENT INTERACTIVE QUERY FRAMEWORK/ LIBRARIES FINAGLE (RPC) SERVICE DIRECTORY SEARCH TWEETS DATA DISCOVERY SCALDING (Map Reduce in Scala) CHARGEBACK REVERSE PROXY PLATFORM SERVICES CORE APPLICATION SERVICES DATA & ANALYTICS PLATFORM JVM DEPLOY (Workflows) CONFIG INFRASTRUCTURE SERVICES C O M P U T E AURORA (Scheduler) HADOOP (Map-Reduce) MESOS (Cluster Manager) S T O R A G E MANHATTAN (Key-Val Store) HDFS (File System) BLOBSTORE GRAPH STORE HERON (Streaming Compute) INFRASTRUCTURE AND DATACENTER MANAGEMENT

23 MANAGEMENT TOOLS SELF SERVE MONITORING AND ALERTING CACHE MESSAGING & QUEUES SOCIAL GRAPH USERS WORKFLOW MANAGEMENT INTERACTIVE QUERY FRAMEWORK/ LIBRARIES FINAGLE (RPC) SERVICE DIRECTORY SEARCH TWEETS DATA DISCOVERY SCALDING (Map Reduce in Scala) CHARGEBACK REVERSE PROXY PLATFORM SERVICES CORE APPLICATION SERVICES DATA & ANALYTICS PLATFORM JVM DEPLOY (Workflows) CONFIG INFRASTRUCTURE SERVICES C O M P U T E AURORA (Scheduler) HADOOP (Map-Reduce) MESOS (Cluster Manager) S T O R A G E MANHATTAN (Key-Val Store) HDFS (File System) BLOBSTORE GRAPH STORE HERON (Streaming Compute) INFRASTRUCTURE AND DATACENTER MANAGEMENT

24 MANAGEMENT TOOLS SELF SERVE MONITORING AND ALERTING CACHE MESSAGING & QUEUES SOCIAL GRAPH USERS WORKFLOW MANAGEMENT INTERACTIVE QUERY FRAMEWORK/ LIBRARIES FINAGLE (RPC) SERVICE DIRECTORY SEARCH TWEETS DATA DISCOVERY SCALDING (Map Reduce in Scala) CHARGEBACK REVERSE PROXY PLATFORM SERVICES CORE APPLICATION SERVICES DATA & ANALYTICS PLATFORM JVM DEPLOY (Workflows) CONFIG INFRASTRUCTURE SERVICES C O M P U T E AURORA (Scheduler) HADOOP (Map-Reduce) MESOS (Cluster Manager) S T O R A G E MANHATTAN (Key-Val Store) HDFS (File System) BLOBSTORE GRAPH STORE HERON (Streaming Compute) INFRASTRUCTURE AND DATACENTER MANAGEMENT

25 THOUSANDS OF SERVICES HUNDREDS OF TEAMS

26 What is the overall use of infrastructure & platform resources across Twitter s services?

27 What is the overall use of infrastructure & platform resources across Twitter s services? How to attribute resource consumption to teams/organization?

28 What is the overall use of infrastructure & platform resources across Twitter s services? How to attribute resource consumption to teams/organization? How do you incentivize the right behavior to improve efficiency of resource usage?

29 CHARGEBACK Ability to meter allocation and utilization of resources per service, per engineering team and charge them accordingly

30 SERVICE Ads Shard SERVICE Tweet Service SERVICE Who To Follow MULTI-TENANCY tenant management using canonical identifiers RESOURCE unit of abstraction PLATFORM AND OTHER SERVICES COMPUTE STORAGE

31 SERVICE Ads Shard SERVICE Tweet Service SERVICE Who To Follow SERVICE IDENTITY MULTI-TENANCY tenant management using canonical identifiers RESOURCE CATALOG RESOURCE unit of abstraction PLATFORM AND OTHER SERVICES COMPUTE STORAGE

32 SERVICE Ads Shard SERVICE Tweet Service SERVICE Who To Follow SERVICE IDENTITY MULTI-TENANCY tenant management using canonical identifiers RESOURCE CATALOG RESOURCE unit of abstraction METERING AND CHARGEBACK PLATFORM AND OTHER SERVICES COMPUTE STORAGE

33 SERVICE METADATA SERVICE Ads Shard SERVICE Tweet Service SERVICE Who To Follow SERVICE IDENTITY MULTI-TENANCY tenant management using canonical identifiers RESOURCE CATALOG RESOURCE unit of abstraction METERING AND CHARGEBACK PLATFORM AND OTHER SERVICES COMPUTE STORAGE

34 SERVICE METADATA SERVICE Ads Shard SERVICE Tweet Service SERVICE Who To Follow SERVICE IDENTITY MULTI-TENANCY tenant management using canonical identifiers RESOURCE CATALOG RESOURCE unit of abstraction METERING AND CHARGEBACK PLATFORM AND OTHER SERVICES COMPUTE STORAGE UNIFIED CLOUD PLATFORM

35 SERVICE IDENTITY

36 A canonical way of identifying a service that consumes resources on various platform infrastructure.

37 PROBLEM Project: chargeback Team: Cloud Infra Mgmt Source code: /cim Disparate identifiers across infrastructure and platform services COMPUTE BATCH COMPUTE role: cim-service job_name: ui; env: prod id: <role>.<env>.<job_name> role: cim-service pool: etl_pipe_prod job_name: compute_cost id: <role>.<pool>.<job_name> Multiple provisioning workflows (Self-Serve, Tickets) Disparate Ownership trackers ( , LDAP) STORAGE app_id: cost_reporting id: <app_id> Lack of support for public cloud Identity and Access Management systems (IAM)

38 OUR APPROACH DASHBOARD Designed an Entity Model that Define canonical identifier scheme across infrastructure and platform services IDENTITY MANAGER Define ownership structure with org PROVISION Single pane of glass for every developer to CONSUMPTION manage their project IDs (including abstracting out public cloud IAM systems) API INFRASTRUCTURE SERVICE Provider APIs for infrastructure services to provision and manage identity

39 IMPACT Source of truth for identifier to org structure mapping improving Service ownership within the Org Enables service to service authentication/authorization

40 BUSINESS OWNER 1:N TEAM ENTITY MODEL FOR SERVICE IDENTITY 1:N PROJECT 1:N SERVICE/ SYSTEM ACCOUNT Model that provides canonical identifier across infrastructure and platform service and ties it to an org structure 1:N <INFRA, CLIENTID>

41 ENTITY MODEL: EXAMPLE BUSINESS OWNER REVENUE 1:N TEAM ADS SERVING ADS PREDICTION 1:N PROJECT adshard prediction 1:N SERVICE/ SYSTEM ACCOUNT 1:N adshard ads-prediction <INFRA, CLIENTID> <Aurora, adshard.prod.adshard> <Aurora, adsprediction.prod.campaign-x> EXAMPLE of services running (on Aurora/Mesos)

42 RESOURCE CATALOG

43 Consistent way of identifying and inventorying of resources of various platform infrastructure.

44 PROBLEM Lack of clarity on what is available & how many resources are consumed COMPUTE STORAGE BATCH COMPUTE CPU MEMORY DISK STORAGE IN GB WPS RPS CPU FILES ACCESSED STORAGE IN GB Need to capture resource fluidity across infrastructure and platform services Better support to model abstract resources (ex, QPS, Tweets per Second) Need to define TCO (Total Cost of Ownership) of a resource per unit time

45 CORES MEMORY DISK application = Task( name = 'application', resources = Resources(cpu = 1.0, ram = 512 * MB, disk = 1024 * MB), processes = [stage_application, run_application], constraints = order(stage_application, run_application))

46 CORES MEMORY DISK application = Task( name = 'application', resources = Resources(cpu = 1.0, ram = 512 * MB, disk = 1024 * MB), processes = [stage_application, run_application], constraints = order(stage_application, run_application)) GPU NETWORK Need for Fluidity!

47 Defining unit price for a resource Framework to price resources. Ensure Total Cost of Ownership. Eg. License cost, chargeback cost from other services, human cost etc. Support for Time Granularity. Eg. Machines/VMs used per day, Cores used per day

48 Operational Overhead Headroom Excess Quota and Reservation Underutilized Quota Allocation Disaster Recovery & Event Spikes Container Size Buffer (Underutilized Reservation) Total Cost of Ownership Twitter Compute Platform $X core-day Non-Prod Used Cores Used Cores

49 PROVIDER 1:N INFRASTRUCTURE SERVICE 1:N OFFERINGS 1:N ENTITY MODEL FOR RESOURCE CATALOG Model that supports Resource Fluidity and captures and manages unit price of a resource over time. OFFER MEASURES 1:N OFFER MEASURE COST

50 ENTITY MODEL: EXAMPLE PROVIDER 1:N INFRASTRUCTURE SERVICE TWITTER DC/ PUBLIC CLOUD AURORA TWITTER DC HADOOP 1:N OFFERINGS COMPUTE STORAGE PROCESSING CLUSTER 1:N OFFER MEASURES CORE-DAYS GB- RAM FILE ACCESSES GB- RAM FILE ACCESSES 1:N OFFER MEASURE COST $X $X $Y $M $N EXAMPLE of Resource Catalog

51 METERING PIPELINE

52 HIGH LEVEL ARCHITECTURE

53 The Product

54 TEAM/ORG BILL

55 INFRASTRUCTURE PNL

56 ORG/TEAM BUDGET

57 CUSTOM REPORTS Infrastructure & Platform Owners Overall Cluster Growth Allocation v/s Utilization of resources by Customer Team Service Owners Allocation v/s Utilization of resources across each Infrastructure & Platform Finance Budget Management (Budget v/s Spend) Execs Efficiency Trends

58 What has been the Impact?

59 Twitter Compute Platform (Aurora/Mesos) Allocated Quota Jun 1, 2015 Sept 1, 2015 Utilized Cores 3 months (Jun - Sep, 2015)

60 Twitter Compute Platform (Aurora/Mesos) Allocated Quota Sept 1, 2015 Jan 1, 2015 Utilized Cores 4 months (Sep, Jan, 2016)

61 33% More core usage against reservation compared to May 2015

62 IMPACT Ensures true to the cost unit price computation Input for capacity planning and budgeting Visibility into the organizational spend and enables accountability Improved utilization of infrastructure service resources Enables comparison with Public Cloud Offerings Improved Service Ownership

63 Kite - Unified Cloud Platform A cloud agnostic service lifecycle manager

64 DASHBOARD (SINGLE PANE OF GLASS) REPORTING SERVICE LIFECYCLE WORKFLOWS SERVICE IDENTITY MANAGER RESOURCE PROVISIONING MANAGER IDENTITY METADATA RESOURCE QUOTA MANAGEMENT DEPLOY METERING & CHARGEBACK PROVIDER APIS & ADAPTERS INFRASTRUCTURE INFRASTRUCTURE & PLATFORM SERVICE SERVICE

65 @igb

66