Big Data Fuels IT Architecture Evolution

Size: px

Start display at page:

Download "Big Data Fuels IT Architecture Evolution"

Pierce Bell
5 years ago
Views:

1 Big Data Fuels IT Architecture Cludera Cpyright 2014 Cludera, Inc.

2 Data Re-Thinking Drivers Multitude f new data types Internet f Things We live nline Insights lead yur Business Cpyright 2014 Cludera, Inc.

3 Where we are Heading INFORMATION-DRIVEN Cpyright 2014 Cludera, Inc.

The Need t Rethink Data Architecture Thusands f Emplyees & Lts f Inaccessible Infrmatin Hetergeneus Legacy IT Infrastructure EDWs Marts Servers Dcument Stres Strage Search

4 The Need t Rethink Data Architecture Thusands f Emplyees & Lts f Inaccessible Infrmatin Hetergeneus Legacy IT Infrastructure EDWs Marts Servers Dcument Stres Strage Search Data Archives Sils f Multi- Structured Data Difficult t Integrate ERP, CRM, RDBMS, Machines Files, Images, Vide, Lgs, Clickstreams External Data Surces Cpyright 2014 Cludera, Inc.

New Categry: The Enterprise Data Hub (EDH) Infrmatin & data accessible by all fr insight using leading tls and apps Enterprise Data Hub Unified Data Management Infrastructure EDH EDWs Marts

5 New Categry: The Enterprise Data Hub (EDH) Infrmatin & data accessible by all fr insight using leading tls and apps Enterprise Data Hub Unified Data Management Infrastructure EDH EDWs Marts Servers Dcuments Strage Search Archives Ingest All Data Any Type Any Scale Frm Any Surce ERP, CRM, RDBMS, Machines Files, Images, Vide, Lgs, Clickstreams External Data Surces Cpyright 2014 Cludera, Inc.

6 Hadp et al Enabling an EDH Applicatins Cpyright 2014 Cludera, Inc.

7 Frm the Mnlith t Micr-Services Randy linkedin.cm/in/randyshup

8 The Mnlithic Architecture Single, vertically-integrated unit The System The System

9 The Mnlithic Architecture Prs Cns Simple at first Crdinatin verhead as team grws In-prcess latencies Pr enfrcement f mdularity Single cdebase, deply unit Pr scaling (vertical nly) Resurce-efficient at small scale All-r-nthing deply (dwntime, failures) Lng build times

10 The Mnlithic Architecture, v2 Set f mnlithic tiers The frnt-end, The app server, The database Presentatin Applicatin Database

11 The Mnlithic Database Prs Cns Simple at first Cupling ver time Jin queries are easy Pr scaling and redundancy (all-rnthing, vertical nly) Single schema, deplyment Difficult t tune prperly Resurce-efficient at small scale All-r-nthing schema management

12 Micr-Services Single-purpse Simple, well-defined interface Mdular and independent Mre graph f relatinships than tiers Fullest expressin f mdularity and encapsulatin A B C D E

13 Micr-Services Prs Cns Each unit is simple Many cperating units Independent scaling and perfrmance Independent testing and deplyment Can ptimally tune perfrmance (caching, replicatin, etc.) Many small reps Requires mre sphisticated tling and dependency management Netwrk latencies

14 Ggle Clud Datastre Clud Datastre: NSQL service Highly scalable and resilient Strng transactinal cnsistency SQL-like rich query capabilities Megastre: ge-scale structured database Multi-rw transactins Synchrnus crss-datacenter replicatin Bigtable: cluster-level structured strage (rw, clumn, timestamp) -> cell cntents Clssus: next-generatin clustered file system Blck distributin and replicatin Cluster management infrastructure Task scheduling, machine assignment Clud Datastre Megastre Bigtable Clssus Cluster manager

15 Pr-Tips: Building a Micr-Service Cmmn Chassis Make it trivially easy t build and maintain a service Define Service Interface (Frmally!) Prpse, Discuss, Agree Prttype Implementatin Simplest thing that culd pssibly wrk Client can integrate with prttype Implementr can learn what wrks and what des nt Real Implementatin Thrw away the prttype (!) Rinse and Repeat

16 Transitin t Service Relatinships Vendr Custmer Relatinship Friendly and cperative, but structured Clear wnership and divisin f respnsibility Custmer can chse t use service r nt (!) Service-Level Agreement (SLA) Prmise f service levels by the prvider Custmer needs t be able t rely n the service, like a utility Charging and Cst Allcatin Charge custmers fr *usage* f the service Aligns ecnmic incentives f custmer and prvider Mtivates bth sides t ptimize

17 Why Enterprises are Embracing the Clud Randy linkedin.cm/in/randyshup

18 Embracing the Clud (The Obvius) Prvisining Speed Minutes, nt weeks Autscaling in respnse t lad Near-Infinite Capacity N need t predict and plan fr grwth N need t defensively verprvisin Pay Fr What Yu Use N utilizatin risk frm wning / renting If it s nt in use, spin it dwn

19 Embracing the Clud (The Less Obvius) Instance Optimizatin Opprtunities Instance shapes t fit mst parts f the slutin space (cmpute-intensive, IO-intensive, etc.) If the shape des nt fit, try anther Service Quality Amazn and Ggle knw hw t run data centers Battle-tested and highly autmated Wrld-class netwrking, bth cluster fabric and external peering Unstppable Ecnmics Almst impssible t beat Ggle / Amazn buying pwer r perating efficiencies 2010s in cmputing are like 1910s in electric pwer

20 Sn it will be just as cmmn t run yur wn data center as it is t run yur wn electric pwer generatin -- me

21 Big Data Fuels IT Architecture Cludera Cpyright 2014 Cludera, Inc.

22 Data Re-Thinking Drivers Multitude f new data types Internet f Things We live nline Insights lead yur Business Cpyright 2014 Cludera, Inc.

23 Where we are Heading INFORMATION-DRIVEN Cpyright 2014 Cludera, Inc.

24 The Need t Rethink Data Architecture Thusands f Emplyees & Lts f Inaccessible Infrmatin Hetergeneus Legacy IT Infrastructure EDWs Marts Servers Dcument Stres Strage Search Data Archives Sils f Multi- Structured Data Difficult t Integrate ERP, CRM, RDBMS, Machines Files, Images, Vide, Lgs, Clickstreams External Data Surces Cpyright 2014 Cludera, Inc.

New Categry: The Enterprise Data Hub (EDH)

using leading tls and apps Enterprise Data Hub

Marts Servers Dcuments Strage Search Archives

ERP, CRM, RDBMS, Machines Files, Images, Vide,

25 New Categry: The Enterprise Data Hub (EDH) Infrmatin & data accessible by all fr insight using leading tls and apps Enterprise Data Hub Unified Data Management Infrastructure EDH EDWs Marts Servers Dcuments Strage Search Archives Ingest All Data Any Type Any Scale Frm Any Surce ERP, CRM, RDBMS, Machines Files, Images, Vide, Lgs, Clickstreams External Data Surces Cpyright 2014 Cludera, Inc.

26 Hadp et al Enabling an EDH Applicatins Cpyright 2014 Cludera, Inc.

27 Why Enterprises are Embracing the Clud Randy linkedin.cm/in/randyshup

28 Embracing the Clud (The Obvius) Prvisining Speed Minutes, nt weeks Autscaling in respnse t lad Near-Infinite Capacity N need t predict and plan fr grwth N need t defensively verprvisin Pay Fr What Yu Use N utilizatin risk frm wning / renting If it s nt in use, spin it dwn

29 Embracing the Clud (The Less Obvius) Instance Optimizatin Opprtunities Instance shapes t fit mst parts f the slutin space (cmpute-intensive, IO-intensive, etc.) If the shape des nt fit, try anther Service Quality Amazn and Ggle knw hw t run data centers Battle-tested and highly autmated Wrld-class netwrking, bth cluster fabric and external peering Unstppable Ecnmics Almst impssible t beat Ggle / Amazn buying pwer r perating efficiencies 2010s in cmputing are like 1910s in electric pwer

30 Sn it will be just as cmmn t run yur wn data center as it is t run yur wn electric pwer generatin -- me

31 Frm the Mnlith t Micr-Services Randy linkedin.cm/in/randyshup

32 The Mnlithic Architecture Single, vertically-integrated unit The System The System

33 The Mnlithic Architecture Prs Cns Simple at first Crdinatin verhead as team grws In-prcess latencies Pr enfrcement f mdularity Single cdebase, deply unit Pr scaling (vertical nly) Resurce-efficient at small scale All-r-nthing deply (dwntime, failures) Lng build times

34 The Mnlithic Architecture, v2 Set f mnlithic tiers The frnt-end, The app server, The database Presentatin Applicatin Database

35 The Mnlithic Database Prs Cns Simple at first Cupling ver time Jin queries are easy Pr scaling and redundancy (all-rnthing, vertical nly) Single schema, deplyment Difficult t tune prperly Resurce-efficient at small scale All-r-nthing schema management

36 Micr-Services Single-purpse Simple, well-defined interface Mdular and independent Mre graph f relatinships than tiers Fullest expressin f mdularity and encapsulatin A B C D E

37 Micr-Services Prs Cns Each unit is simple Many cperating units Independent scaling and perfrmance Independent testing and deplyment Can ptimally tune perfrmance (caching, replicatin, etc.) Many small reps Requires mre sphisticated tling and dependency management Netwrk latencies

38 Ggle Clud Datastre Clud Datastre: NSQL service Highly scalable and resilient Strng transactinal cnsistency SQL-like rich query capabilities Megastre: ge-scale structured database Multi-rw transactins Synchrnus crss-datacenter replicatin Bigtable: cluster-level structured strage (rw, clumn, timestamp) -> cell cntents Clssus: next-generatin clustered file system Blck distributin and replicatin Cluster management infrastructure Task scheduling, machine assignment Clud Datastre Megastre Bigtable Clssus Cluster manager

39 Pr-Tips: Building a Micr-Service Cmmn Chassis Make it trivially easy t build and maintain a service Define Service Interface (Frmally!) Prpse, Discuss, Agree Prttype Implementatin Simplest thing that culd pssibly wrk Client can integrate with prttype Implementr can learn what wrks and what des nt Real Implementatin Thrw away the prttype (!) Rinse and Repeat

40 Transitin t Service Relatinships Vendr Custmer Relatinship Friendly and cperative, but structured Clear wnership and divisin f respnsibility Custmer can chse t use service r nt (!) Service-Level Agreement (SLA) Prmise f service levels by the prvider Custmer needs t be able t rely n the service, like a utility Charging and Cst Allcatin Charge custmers fr *usage* f the service Aligns ecnmic incentives f custmer and prvider Mtivates bth sides t ptimize

Managing Data in Microservices. Randy linkedin.com/in/randyshoup

Managing Data in Microservices. Randy linkedin.com/in/randyshoup Managing Data in Micrservices Randy Shup @randyshup linkedin.cm/in/randyshup Backgrund VP Engineering at Stitch Fix Using technlgy and data science t revlutinize clthing retail Cnsulting CTO as a service