Introduction To Postgres. Rodrigo Menezes

Transcription

1 Introduction To Postgres Rodrigo Menezes

2 I joined in 2013, when we were ~20 people Acquired by Oracle during summer of 2017 Currently, we re about ~250 people I started off as a frontend developer

3 This Talk Why Postgres? Brief introduction to SQL ORM (Object relational mapper) Transactions Performance and indices Views and Materialized Views JSON and Postgres Scaling Moat

4 Why PostgreSQL?

5 Why PostgreSQL? Free + open source Really good community Fast bug fixes and frequent release cycle Amazing docs "Add features slowly, do them well" Great performance Ton of cool features SQL is great and people can pry it from my cold, dead hands

6 What is SQL?

7 SQL SQL is a language for inserting, updating, reading and deleting data SQL is meant for humans (pros and cons to this) SQL is everywhere. Has been since the 70s Every collection of data is a "table" Every new datapoint is a "row"

8 Creating a table postgres=# CREATE TABLE users( id INTEGER NOT NULL, TEXT NOT NULL, name TEXT ); CREATE TABLE Time: ms

9 Inserting a row postgres=# INSERT INTO users(id, ) VALUES( 0, 'rodrigo.menezes@oracle.com' ); INSERT 0 1 Time: ms

10 Selecting a row postgres=# SELECT id, , name FROM users WHERE id=0; id name rodrigo.menezes@oracle.com (1 row)

11 Selecting all columns postgres=# SELECT * FROM users WHERE id=0; id name rodrigo.menezes@oracle.com (1 row)

12 Case insensitive postgres=# select * from users where id=0; id name rodrigo.menezes@oracle.com (1 row)

13 Updating a row postgres=# UPDATE users SET ="larry.ellison@oracle.com" WHERE id=0; UPDATE 1 Time: ms

14 Deleting a row postgres=# DELETE FROM users WHERE id=0; DELETE 1 Time: ms

15 Dropping a table postgres=# DROP TABLE users; DROP TABLE Time: ms

16 Why SQL? It's great for human beings Standards are nice In the majority of cases, you want your data strongly typed Less accidents More assumptions => more performance / compression Compile-time errors in strongly typed languages

17 ORMs (Object Relation Mapper)

18 You can write a raw query # python import psycopg2 # Library to talk to Postgres def create_cursor(): conn = psycopg2.connect("dbname=test user=postgres") cursor = conn.cursor() def update_ (cursor, id, new_ ): query= ( 'UPDATE users SET ={}' 'WHERE id = {} LIMIT 1' ).format(id) cursor.execute(query)

19 SQL Injection... def update_ (cursor, id, new_ ): query= ( 'UPDATE users SET ={}' 'WHERE id = {} LIMIT 1' ).format(id) cursor.execute(query) update_ (create_cursor(), 1, 'NULL; DROP TABLE user; --`); # `--` is a SQL comment so it ignores the rest of the line

20 ORMs Library that handles all your SQL for you Handles user input for you to make it secure Neater syntax that hides SQL complexity (which is mostly a pro but sometimes a con)

21 Sequelize // db.js import { Sequelize } from 'sequelize'; sequelize = new Sequelize({ database: POSTGRES_DB, dialect: 'postgres', host: POSTGRES_HOSTNAME, password: POSTGRES_PASSWORD, username: POSTGRES_USER, });

22 Sequelize Model // models/user.js import { Sequelize } from 'sequelize'; const User = sequelize.define('users', { id: { type: Sequelize.INTEGER, autoincrement: true, primarykey: true }, { type: Sequelize.TEXT, unique: true } });

23 Model usage // services/createuser.js import { User } from './models/user.js'; function createuser() { const user = await User.create({ }); } function getuser() { const user = await User.find({ }); }

24 Transactions and constraints

25 Let's make a Venmo clone postgres=# create table users( id serial primary key, text not null, money bigint not null default 0 );

26 Aside: serials postgres=# CREATE TABLE users( id SERIAL NOT NULL, TEXT NOT NULL ); -- Now I don't need to specify ids postgres=# INSERT INTO users( ) VALUES ( 'rodrigo.menezes@oracle.com' ); postgres=# SELECT * FROM users; id rodrigo.menezes@oracle.com (1 row)

27 Sending $20 from user 1 to user 2 # Make sure user 1 has enough money postgres=# select money from users where id = 1; # Decrease user 1's money postgres=# update users set money=money 20 where id=1; # What happens if our webserver disconnects here? # Increase user 2's money postgres=# update users set money=money + 20 where id=2;

28 Transactions postgres=# begin; postgres=# select money from users where id = 1; postgres=# update users set money=money 20 where id=1; postgres=# update users set money=money + 20 where id=2; postgres=# commit;

29 Transactions postgres=# begin; postgres=# select money from users where id = 1; postgres=# update users set money=money 20 where id=1; postgres=# abort; -- revert all changes!

30 Transactions Transactions allow the effects of your code to happen in one go Achieve ACID compliance Atomicity everything in the transaction happens or none of it does Consistency after a transaction, the database will be in a valid state. Rules always apply. Isolation a newly committed transaction shouldn't affect your current transaction (there are multiple levels of this) Durability once a transaction is committed, it'll remain so even in case of power loss.

31 Wrap everything in a transaction // wrapview.js import sequelize from './db/sequelize'; export default function wrapview(fn) { return (req, res, next) => { sequelize.transaction(t => { return fn(req, res) }).catch(next); }; } // server.ts app.post("/api/user", wrapview(users.create));

32 Race condition -- Assume user 1 has $20 postgres=# select money from users where id = 1; postgres=# select money from users where id = 1; postgres=# update users set money=money 20 where id=1; postgres=# update users set money=money 20 where id=1; # Increase user 2's money postgres=# update users set money=money + 20 where id=2; # Increase user 3's money postgres=# update users set money=money + 20 where id=2; -- Now user 1 has -$20

33 Constraints postgres=# create table users( id serial primary key, text not null, money bigint not null default 0 check (money > 0) );

34 Race condition postgres=# begin; postgres=# begin; postgres=# update users set money=money 20 where id=1; -- will error out if necessary postgres=# update users set money=money 20 where id=1; -- will error out if necessary postgres=# update users set money=money + 20 where id=2; postgres=# update users set money=money + 20 where id=3; postgres=# commit; postgres=# commit;

35 Transactions make for safe testing # python import transaction from pyramid import testing from moatpro.web import add_routes class BaseTest(object): def setup_method(self, method): self.config = testing.setup() add_routes(self.config) transaction.begin() def teardown_method(self, method): testing.teardown() transaction.abort()

36 Performance and indices

37 Let's make a lot of fake data postgres=# CREATE TABLE users( postgres-# id SERIAL, postgres-# TEXT NOT NULL postgres-#); postgres=# INSERT INTO users( ) postgres-# SELECT 'test_' id::text '@ .com' postgres-# FROM generate_series(0, ) id; INSERT Time: ms postgres=# SELECT * FROM users; id test_0@ .com 2 test_1@ .com 3 test_2@ .com...

38 How slow do things get? postgres=# select * from users where ='test_0@ .com'; id test_0@ .com (1 row) Time: ms -- Can we make this faster?

39 Explain query postgres=# explain select * from users where ='something'; QUERY PLAN Gather (cost= rows=50000 width=36) Workers Planned: 2 -> Parallel Seq Scan on users (cost= rows=20833 width=36) Filter: ( = 'test_0@ .com'::text) (4 rows)

40 Indices postgres=# CREATE INDEX on users( ); CREATE INDEX Time: ms postgres=# explain select * from users where ='test_0@ .com'; QUERY PLAN Index Scan using users_ _idx on users (cost= rows=1 width=36) Index Cond: ( = 'test_0@ .com'::text) (2 rows)

41 Indices postgres=# select * from users where ='test_0@ .com'; id test_0@ .com (1 row) Time: ms -- Went from ~1s to ~1ms! x1000 speed up!

42 Unique index postgres=# CREATE UNIQUE INDEX ON users( ); postgres=# INSERT INTO users( ) INSERT 0 1 Time: ms postgres=# INSERT INTO users( ) VALUES ('rclmenezes@gmail.com'); ERROR: duplicate key value violates unique constraint "users_ _idx" DETAIL: Key ( )=(rclmenezes@gmail.com) already exists. Time: ms

43 Unique indices... wat? postgres=# CREATE TABLE users2( id SERIAL, TEXT -- NOT NULL ); postgres=# CREATE UNIQUE INDEX ON users2( ); postgres=# INSERT INTO users2( ) VALUES (null); INSERT 0 1 Time: ms postgres=# INSERT INTO users2( ) VALUES (null); INSERT 0 1 Time: ms

44 You can do indices on multiple columns postgres=# CREATE TABLE players( postgres-# id SERIAL PRIMARY KEY, postgres-# team TEXT NOT NULL, postgres-# jersey INTEGER NOT NULL, postgres-# first_name TEXT NOT NULL, postgres-# last_name TEXT NOT NULL postgres-#); postgres=# CREATE UNIQUE INDEX ON players(team, jersey);

45 A primary key is a not-null, unique constraint with an index postgres=# \d players Table "public.players" Column Type Modifiers id integer not null default nextval('players_id_seq'::regclass) team text not null jersey integer not null first_name text not null last_name text not null Indexes: "players_pkey" PRIMARY KEY, btree (id) "players_team_jersey_idx" UNIQUE, btree (team, jersey)

46 ETL with Postgres

47 moat ad search

48 Impressions table postgres=# create table impressions( brand_id integer not null, ad_id integer not null, created_at timestamp not null default current_timestamp ); -- For a specific brandand ad, we want results like this: -- day brand_id ad_id num_impressions

49 Simple group by postgres=# create table impressions( postgres-# brand_id int not null, postgres-# ad_id int not null, postgres-# created_at timestamp not null default current_timestamp postgres-#); postgres=# postgres-# postgres-# postgres-# select created_at::date, brand_id, ad_id, count(id) from impressions group by created_at::date, brand_id, ad_id where ad_id = foo and brand = bar; -- This query is a mouthful and it'd be a pain to type out all the time.

50 View postgres=# postgres-# postgres-# postgres-# postgres-# create view ad_brand_impressions_per_day select created_at::date as day, brand_id, ad_id, count(id) from impressions group by created_at::date, brand_id, ad_id where ad_id = foo and brand = bar; postgres=# select * from num_impressions_per_day postgres-# where brand_id=?; -- What if this is slow?

51 Number of creatives per brand? -- What if you want to see the number of ads a brand had in a daterange? postgres=# select brand_id, count(distinct ad_id) postgres-# from num_impressions_per_day postgres-# where brand_id=<blah> and day between ' ' and ' ' postgres-# group by brand_id; -- What if this is slow?

52 Materialized view create materialized view impressions_by_day as select created_at::date as day, brand_id, ad_id, count(id) as num_impressions from impressions group by created_at::date, hostname_id, ad_id; create unique index on impressions_by_day(day, brand_id, ad_id); select * from impressions_by_day where brand_id=foo and ad_id=bar;

53 Materialized view # Your ETL is now: refresh materialized view impressions_by_day; # But refreshes block reads, so if you have a unique index, # you can do: refresh materialized view impressions_by_day concurrently;

54 JSON and Postgres

55 latency analytics

56 Things to consider Our data is coming in via JSON because it's coming from a browser We don't know how our JSON schema will change We're going to have a lot of columns In this case, maybe it's fine to use JSON column type

57 JSON postgres=# create table ad_analysis( id serial primary key, analysis json not null, created_at timestamp not null default current_timestamp, updated_at timestamp not null default current_timestamp ); postgres=# insert into ad_analysis(analysis) values('{"size": 1234}'::json); postgres=# select analysis->>'size' from ad_analysis;?column? (1 row)

58 Normal JSON type isn't great postgres=# SELECT '{"c":0, json {"c":0, "a":2,"a":1} (1 row) postgres=# SELECT '{"c":0,?column? (1 row) "a":2,"a":1}'::json; "a":2,"a":1}'::json->>'a'; postgres=# create index on ad_analysis(analysis); ERROR: data type json has no default operator class for access method "btree"

59 Use JSONB postgres=# SELECT '{"c":0, "a":2,"a":1}'::jsonb; jsonb {"a": 1, "c": 0} (1 row) Time: ms postgres=# create index on ad_analysis(analysis); CREATE INDEX Time: ms -- Really good performance!

60 Scaling Moat

61 moat analytics

62 What we do Measure attention online Don't track cookies/ip addresses We're a neutral third party that publishers and advertisers use We process billions of events a day

63 lots of data

64 Databases Elmo: Last 33 Days Marjory: A few years Frackles: All of our historical data Decanter: aggregates our data

65 decanter

66 Foreign Data Wrapper create server other_db foreign data wrapper postgres_fdw options ( host 'other_db.moat.co', port '5432', dbname 'other_db' ); create user mapping for public server other_db options ( user 'user', password 'password' ); create schema foo; import foreign schema foo from server search_rds into foo; # works like a normal table select * from foo.table;

67 Foreign Data Wrappers Great way to talk to other databases Can talk to other types of data stores too (there's a mysql_fdw, elasticsearch_fdw, etc). Careful the query planner can be dumb. You can put the results in an mview if you want.

68 Decanter PostgreSQL instance that doesn't store data High CPU, high memory, no storage Gets results from the other databases via FDW and aggregates them Highly available (pgbouncer) If the databases down there change, the client doesn't notice

69 marjory

70 Our data considerations We have a lot of data We're basically bean-counters ("how many times did X happen?". So, we basically need to filter and sum. The rows are very wide and very sparse (a lot of NULLs). This makes columnar solutions not work well This is very compressable

71

72

73

74

75 Marjory 8-10x compression for our data CPU bound instead of IO bound Pretty great performance We use it for some historical data (a few years) Inflexible: we're trading generality for fit to our use case Probably not great if there's a lot of rows

76 frackles

77 Frackles Access ALL our historical data Needs to be cheap Support a lot of concurrency Be able to spin up new instances fast for scale Bonus: be faster than RedShift

78 What if Our database was just a series of SQLite files in S3 We used AWS Lambda to pull those S3 files and read from them We used a Python client to manage all of those lambdas and gave it a simple API We wrote our own FDW to talk to the Python client (I was really skeptical)

79

80 Frackles Surprisingly good performance It's dirt cheap you only pay for S3 and Lambdas Completely stateless (almost no ops!) Constant overhead S3 performance has high variance Capped on number of concurrent lambdas Better on queries with little aggregation

81

82 In summary

83 We re hiring. A lot. rodrigo.menezes@oracle.com