Location Agnostic Data

Size: px

Start display at page:

Download "Location Agnostic Data"

Edmund Simon
5 years ago
Views:

1 Arup Nanda Proligence It s 11 PM. Do you know where your data is? 2

2 SALES Schema A 3 Sales in last 3 months Sales in last 24 months SALES Schema A 4

3 SALES Schema A Schema B Datawarehouse 5 SALES Schema A Schema B Datawarehouse 6

4 SALES Schema A Schema B Datawarehouse HBase Hadoop 7 SALES OLTP Datawarehouse Hadoop 8

5 SALES OLTP Datawarehouse Hadoop 9 Cloud SALES OLTP Datawarehouse Hadoop 10

6 11 Sharding Location North America South America EMEA APAC Customer Importance Very Important Important Somewhat Imp Rest 12

7 Different Geography Different Location Different Technology 13 Challenges Need to know: The technology Oracle Database, Hadoop? SQL, or Map Reduce Code The location The data availability Do I have the right data in the same DB or got to get from somewhere else? 14

8 Solutions Put the business logic in the database But, which database? Which technology 15 Solution, contd. Put the logic in the app layer Independent of technology But is really technology independent? Definitely inconsistent Need to reinvent the wheel Still need to know the location, currency, etc. 16

9 Different Geography Different Location Different Technology 17 Data As a Service 18

10 Step 1 Gather a list of all high level data Not tables, columns; but data elements such as reservations, sales, etc. 19 Step 2 Map tables, columns, collections, etc. into these data elements Define an document business logic for each datastore Logic System of record, less than 3 months Less than 2 years More than 2 years Data store OLTPDB.SchemaA.TableA.ColumnA DWDB.SchemaA.ColumnA Hbase Define all other elements such as how often synced up. 20

Step 3 Identify datastores that can t tolerate

11 Step 3 Identify datastores that can t tolerate latency Customer Records Sales Records Unpredictable division across geographical boundaries Are generally called within ranges. Out of ranges can be slow. Replicate Shard 21 Example North America EMEA APAC Customer Customer Customer Sales Sales Sales 22

12 Sharded Data Strategies Use DB Links create database link newyork connect to gwuser/gwuser using newyork ; 23 Sharded Data, contd. Create View SALES as select * from sales union all select * from sales@newyork union all select * from sales@london 24

13 Sharded Data, contd. Create INSTEAD OF triggers create trigger instead of INSERT on sales for each row if country_code in ( US, CA, ) then INSERT into sales@newyork 25 Step 4 Write APIs to serve data elements Could be: Java Webservice Oracle Packages RESTful APIs 26

14 Services North America EMEA APAC Customer Customer Customer Sales Sales Sales Sales Service Customer Service 27 Service Has a fixed catalog of requests it can address Has knowledge about location, data dispersion, technology Example Accept StoreID Lookup country of the store Is data replicated? If so, get local data If not, locate the DB for that country Get the data from that DB 28

15 Services North America EMEA APAC Sales Service 29 Services North America Europe Africa APAC Sales Service 30

16 Services North America Europe Africa APAC Hadoop Sales Service 31 Services North America Europe Africa APAC Hadoop Sales Service 32

17 Suitability 1. Reference data country details, store details Static or almost static 2. Low variability customer preferences 3. Latency tolerated European customers generally call the European call centers 4. Read type of access 33 Bridging Technologies ServiceA Helper Service 34

Gateways Gateway 35 Summary Data will continue to exist in many places Single source of truth is next to impossible to attain Architect to live with diverse data, sometimes competing versions Decide

18 Gateways Gateway 35 Summary Data will continue to exist in many places Single source of truth is next to impossible to attain Architect to live with diverse data, sometimes competing versions Decide which data is to be manually replicated and sharded Use gateways or services to connect from one database to the other allowing location and technology transparency. In Oracle databases, DB Links, Views and Instead of Triggers can provide the location transparencies Gateways can bridge different technologies 36

19 Thank You! Blog: arup.blogspot.com Facebook.com/ArupKNanda Google Plus: +ArupNanda 37

Networking for the Cloud DBA. Arup Nanda Longtime Oracle DBA And Explorer of New Things

Networking for the Cloud DBA. Arup Nanda Longtime Oracle DBA And Explorer of New Things Networking for the Cloud DBA Arup Nanda Longtime Oracle DBA And Explorer of New Things Most Important Skill for a Cloud DBA 2 Netmask 3 Broadcast Address 4 Network ID IP Address 5 Most Important Skill