Nikolay Samokhvalov.

Transcription

1 Nikolay Samokhvalov

2 History Year of Birth: 1995

3 History 1995: Postgres95 POSTQUEL query language replaced with SQL

4 History 1995: Postgres95 POSTQUEL query language replaced with SQL 1996: Postgres95 departed from academia, renamed to PostgreSQL

5 History 1995: Postgres95 POSTQUEL query language replaced with SQL 1996: Postgres95 departed from academia, renamed to PostgreSQL 1998: PL/pgSQL added (PostgreSQL 6.4)

6 And a bit more history... Object Management in POSTGRES Using Procedures M. Stonebraker

7 What s now? - Postgres speaks a lot of PL languages: - - native : PL/pgSQL included: PL/Tcl, PL/Perl, PL/Python additional-traditional: PL/Java, PL/R, PL/sh, PL/v8 (JavaScript) not active: PL/Scheme, PL/PHP, PL/Ruby special/exotic/new: - PL/Proxy (sharding, from Skype), - PL/Container (Python, R), - plgo (Go), etc. - PgOpenCL (GPU!) Functions can also be created in: - C (anything is possible!) SQL (plain! standard! with [recursive] CTEs!)

8 What are Stored Procedures? In Postgres: Functions = UDFs (user-defined functions) = Stored Procedures (in other DBMSes: you can include your function/udf to a SELECT, while you can only PERFORM/EXEC/EXECUTE a stored procedure)

9 Functions & Triggers

10 Functions & Triggers

11 Why?

12 Reason #1: Data Clearness & Integrity Data Checks (format, constraints, etc) (Ruby or Python or PHP or )

13 Reason #1: Data Clearness & Integrity Data Checks (format, constraints, etc) in App (Ruby or Python or PHP or )

14 Reason #1: Data Clearness & Integrity Data Checks (format, constraints, etc) in App (Ruby or Python or PHP or )

15 Reason #1: Data Clearness & Integrity App (Ruby or Python or PHP or ) CHECKS

16 Reason #1: Data Clearness & Integrity App (Ruby or Python or PHP or ) CHECKS Control your Data Quality

17 Data Validation, an example: validate address Source:

18 Reason #2: Access Control - SECURITY DEFINER allows a user to do what she/he cannot usually do (but under strict control) GRANT/REVOKE a standard way to control permissions Good approach: forbid direct access to tables, provide functions and views with proper GRANTs Pay attention to: - objects (tables, views, functions) - columns (can REVOKE/GRANT individually!) - rows (check what Row-Level Security is)

19 Reason #3: speed (first of all, IO/network-related) DBMS (Postgres 9.6) AWS RDS, USA, Client (psql) somewhere in Germany. Getting all 10M rows is ~7x slower Use your RDBMS for Data Manipulation. It is not just a Storage.

20 Reason #3: speed There are a LOT of cases here. - ORMs (ActiveRecord, Hibernate, etc) and how people work with them Analytics (doing R or python calculations inside RAM, etc) Massive data updates (retrieve IDs and then DELETE rows? Doh. Just look around and you ll find more. Again: Work with Data Inside Database First. Pay attention to: - cardinality (how many rows you touch?) - RTT (round trip time), reduce network calls

21 Reason #4: Data Integration Data Manipulation Logic in App (Ruby or Python or PHP or ) Something* * ElasticSearch, Sphix, Analytics DBMS, etc

22 Reason #4: Data Integration Data Manipulation Logic in App (Ruby or Python or PHP or ) Something* * ElasticSearch, Sphix, Analytics DBMS, etc

23 Reason #4: Data Integration Data Manipulation Logic in App (Ruby or Python or PHP or ) Something* * ElasticSearch, Sphix, Analytics DBMS, etc

24 Reason #4: Data Integration App (Ruby or Python or PHP or ) Something* Data Manipulation Use: - functions, triggers, - Foreign Data Wrappers (FDW), - Logical Decoding (e.g. pglogical) * ElasticSearch, Sphix, Analytics DBMS, etc

25 #5: HTTP API w/o middleware, declarative - PostgREST Written in Haskell MIT license Actively developing chat: CREATE VIEW v1.person AS SELECT * FROM public.person; CREATE FUNCTION v1.myfunc(...) LANGUAGE...; /person /rpc/myfunc GET POST PATCH DELETE Only POST (write functions in any language: SQL, plpgsql, plpython, plr, plv8, etc!) SELECT INSERT UPDATE DELETE

26 #6: PL/Proxy: sharding - All work via functions Special functions (in PL/Proxy language ) are in the middle Developed in Skype, and still there Yandex.Mail migrated from Oracle to Postgres + PL/Proxy in (300+ TB, 250k RPS)

27 #6: PL/Proxy: sharding

28 #7: MADlib: Machine Learning inside your DBMS - A lot of ML algorithms implemented (added in each release) PL/Python Very easy and quick start to do machine learning with your Postgres data

29 Cons Tooling can be considered week (packaging, dependences, editors, debugging, profiling, etc) Version control and schema migrations Testing Stored Procedures consume resource in DBMS. Can be tricky to scale Example: call external API via plpythonu function and save data -- consumes CPU on your server unpredictably!

30 Cons - fixes Tooling can be considered week (packaging, dependences, editors, debugging, profiling, etc) vim+plpgsql highlighting; DataGrip, Debugger, Profiler (pgadmin) Version control and schema migrations Sqitch and others Testing pgtap Stored Procedures are consuming resource in DBMS. Can be tricky to scale Example: call external API via plpythonu function and save data -- consumes CPU on your server unpredictably! Avoid I/O things inside your master if you need to scale

31 Thank you! (new Postgres tweets daily!) RuPostgres.org