Google Cloud Bigtable. And what it's awesome at

Transcription

1 Google Cloud Bigtable And what it's awesome at

2 Jen Tong Developer Advocate Google Cloud

3 Agenda 1 Research 2 A story about bigness 3 How it works 4 When it's awesome

4 Google Research Publications

5 Google Research Publications

6 Managed Cloud Versions Bigtable Flume Dremel

7 Managed Cloud Versions Bigtable Bigtable Flume Dataflow Dremel BigQuery

8 Cloud Bigtable

9 Google s mission is to organize the world s information and make it universally accessible and useful.

10 Bigness

11 Google Internal Bigtable in Numbers Storage: 100s of PB Throughput: 1,000,000s of QPS Bandwidth: 100s of GB/sec

12 How much is that? Several Datas worth Photo credit: jdhancock

13 How much is that? Millennia of DVD video Photo credit: illinoislibrary

14 Influence Bigtable Plus Hundreds of Internal Services Google is not affiliated or endorsed by any of these companies. Apache HBase, Apache Cassandra and Apache Accumulo are trademarks are of The Apache Software Foundation. Hypertable is the trademark of Hypertable Inc.

15 Engineering

16 Engineering Hundreds of engineer-years worth

17 Bigtable - The early years Jeff and Sanjay decided to build a database service that could scale linearly across thousands and thousands of commodity servers Systems will fail, retain performance at scale Abandon traditional relational model The first generation was about: Prototyping and build the service to do its first scaling Migrate initial applications to Bigtable Figure out replication, and first multi-tenant version of Bigtable

18 Bigtable - Stabilized From batch only, to serving web traffic Low latency for 99th percentile of requests Polish the Bigtable service React better to abusive usage Mixed media clusters - mixture of SSD + HDD storage with configurable affinity Bring tablet server recovery time from 10s of seconds to 1 second or less Easier replication

19 Google Cloud Bigtable A fully-managed service Focus more on your business, less on infrastructure Straightforward pricing model

20 Data Model

21 Data model

22 How it works

23 HBase Architecture Region Server Bloomfilter HBase Client HBase Cluster Memory Table WAL Region Server Region Server Master ZooKeeper Region Region Region Server Region Server Region Region Block Cache HDFS

24 Bigtable Architecture Tabletserver HBase Client Bloomfilter Bigtable Cell Memtable Sharedlog Tabletserver Tabletserver Master Chubby Tablet Tablet Tabletserver Tabletserver Tablet Tablet Block Cache Colossus

25 Bigtable System Architecture Tabletserver HBase Client Bloomfilter Bigtable Cell Memtable Sharedlog Tabletserver Tabletserver Master Chubby Tablet Tablet Tabletserver Tabletserver Tablet Tablet Block Cache Colossus

26 Bigtable Architecture Tabletserver HBase Client Bloomfilter Bigtable Cell Memtable Sharedlog Tabletserver Tabletserver Master Chubby Tablet Tablet Tabletserver Tabletserver Tablet Tablet Block Cache Colossus

27 Bigtable Architecture Tabletserver HBase Client Bloomfilter Bigtable Cell Memtable Sharedlog Tabletserver Tabletserver Master Chubby Tablet Tablet Tabletserver Tabletserver Tablet Tablet Block Cache Colossus

28 Life of Bigtable data

29 Life of Bigtable data

30 Life of Bigtable data

31 Life of Bigtable data

32 Demo: Wikipedia Stats Code on Github

33 When it's awesome

34 Management Who in the audience have used HBase before? Things you will not see in Cloud Bigtable: Compactions Pre-splitting Lots of configuration settings 1 minute regionserver outages Coprocessors (for now)

35 Throughput Mixed Read/Write Throughput(MB/s) Write Throughput (MB/s)

36 Latency latency (ms) at 99% read update

37 Internet of Things Sensor data dashboards and anomaly detection Marketing/ Digital Media User engagement, clickstream analysis, real-time adaptive content Financial Services Faster risk analysis, credit card fraud/abuse Cloud Bigtable Use Cases Energy Oil well sensors, anomaly detection, predictive modeling Telecommunications Sampled traffic patterns, metric collection and reporting Biomedical Genomics sequencing data analysis

38 When not to use it Relational joins, like for online transaction processing Interactive querying Blobs over 10MB ACID transactions Automatic cross-zone replication You don't have much data yet

39 When not to use it Relational joins, like for online transaction processing - Cloud SQL Interactive querying - BigQuery Blobs over 10MB - Cloud Storage ACID transactions - Datastore Automatic cross-zone replication - Datastore You don't have much data yet - Datastore, Firebase, or Cloud SQL

40 Thank you! Jen Tong Developer Advocate Google Cloud little418.com

41