NOSQL FOR POSTGRESQL

Transcription

1 NOSQL FOR POSTGRESQL BEST PRACTICES DMITRY DOLGOV

2 1

3 2

4 2 Jsonb internals and performance-related factors

5 Jsonb internals and performance-related factors Tricky queries 2

6 Jsonb internals and performance-related factors Tricky queries Benchmarks 2

7 Jsonb internals and performance-related factors Tricky queries Benchmarks How to shoot yourself in the foot 2

8 Internals

9 4 Performance-related factors

10 Performance-related factors On-disk representation 4

11 Performance-related factors On-disk representation In-memory representation 4

12 Performance-related factors On-disk representation In-memory representation Indexing support 4

13 5

14 Jsonb document size node node... JEntry content 6

15 Jsonb document size node node... JEntry content 7

16 Jsonb Header type number of items JEntry length or offset? value type 8 value length or offset

17 JB_OFFSET_STRIDE JEntry may contains a value lenght or offset Offset = access speed Length = compressibility Every JB_OFFSET_STRIDE th JEntry contains an offset Rest of them contain length 9

18 Bson document size node node... Header Content 10

19 Bson document size node node... Header Content 11

20 Bson Header Value type Key name Value size 12

21 MySQL Json node node... Type Value 13

22 MySQL Json node node... Type Value 14

23 MySQL Json Object Count of elements Size Pointers to keys Pointers to values Keys 15 Values

24 Bson Jsonb/MySQL Json Key Value Key Key Key Value Value Key Key Value... Value... Value 16

25 17 { a : 3, b : xyz }

26 select pg_relation_filepath(oid), relpages from pg_class where relname = table_name ; pg_relation_filepath relpages -+ - base/40960/ (1 row) 18

27 19 bson.dumps({ a : 3, b : u xyz })

28 20 $ hexdump -C database/table.ibd

29 TOAST Jsonb Compression Chunks Toast table TOAST_TUPLE_THRESHOLD bytes (normally 2 kb) PostgreSQL and MySQL use LZ variation MongoDB uses snappy block compression 21

30 Alignment Variable-length portion is aligned to a 4-byte insert into test values( { a : aa, b : 1} ); insert into test values( { a : 1, b : aa } ); 22

31 In-memory representation Tree-like representation (JsonbValue, Document, Json_dom) Little bit more expensive but more convenient to work with Mostly in use to modify data (except MySQL) Most of the read operations use on-disk representation 23

32 Indexing support Postgresql single path, multiple paths, entire document MongoDB single path, multiple paths MySQL virtual columns, single path, multiple paths 24

33 PG indexing details jsonb_path jsonb_path_ops 25

34 Queries

35 Pitfalls No Json path out of the box (jquery, SQL/JSON) Queries with an array somewhere in the middle Iterating through document Update inside document 27

36 28 [{ items : [ { id : 1, value : aaa }, { id : 2, value : bbb } ] }, { items : [ { id : 3, value : aaa }, { id : 4, value : bbb } ] }]

37 WITH items AS ( ) SELECT jsonb_array_elements(data-> items ) AS item FROM test SELECT * FROM items WHERE item-» value = aaa ; item { id : 1, value : aaa } { id : 3, value : aaa } (2 rows) 29

38 { } items : { item1 : { status : true}, item2 : { status : true}, item3 : { status : false} } 30

39 WITH items AS ( ) SELECT jsonb_each(data-> items ) AS item FROM test SELECT (item).key FROM items WHERE (item).value-» status = true ; key - item1 item2 (2 rows) 31

40 Benchmarks

41 33

42 AWS EC2 m4.xlarge instance separate instance (database and generator) 16GB memory, 4 core 2.3GHz Ubuntu Same VPC and placement group AMI that supports HVM virtualization type at least 4 rounds of benchmark 34

43 35 PostgreSQL MySQL 5.7.9/8.0 MongoDB YCSB rows and operations AWS EC2

44 Configuration shared_buffers effective_cache_size max_wal_size innodb_buffer_pool_size innodb_log_file_size write concern level (journaled or transaction_sync) checkpoint eviction 36

45 Document types simple document 10 key/value pairs (100 characters) large document 100 key/value pairs (200 characters) complex document 100 keys, 3 nesting levels (100 characters) 37

46 Select, GIN simple document jsonb_path_ops where { key : value } ::jsonb 38

47 39

48 40

49 41

50 42

51 simple document btree Select, BTree 43

52 44

53 complex document btree Select, BTree 45

54 46

55 simple document journaled Insert 47

56 48

57 49

58 50

59 51

60 simple document Update one field journaled max wal size 1GB Update 50%, Select 50% 52

61 53

62 large document Update one field Update 50%, Select 50% 54

63 55

64 simple document btree insert JSON vs JSONB 56

65 57

66 simple document btree select JSON vs JSONB 58

67 59

68 simple document btree insert SQL vs JSONB 60

69 61

70 simple document btree select SQL vs JSONB 62

71 63

72 How to bring it down accidentally?

73 65

74 66 Update one field of a document DETOAST of a document (select, constraints, procedures etc.) Reindex of an entire document

75 Document slice large document One field from a document 67

76 select data-> key1 -> key2 from table; select data-> key1, data-> key2 from table; 68

77 69

78 large document Document slice 10 fields from a document 70

79 71

80 Document slice create type test as ( a text, b text); insert into test_jsonb values( { a : 1, b : 2, c : 3} ); select q.* from test_jsonb, jsonb_populate_record(null::test, data) as q; a b (1 row) 72

81 73

82 TOAST_TUPLE_THRESHOLD simple document 40 threads different document size select 74

83 75

84 Select, GIN simple document jsonb_path_ops where jsonb_build_object( key, value ) 76

85 77

86 78 Jsonb is more that good for many use cases

87 Jsonb is more that good for many use cases Benchmarks above are only hints 78

88 Jsonb is more that good for many use cases Benchmarks above are only hints You need your own tests 78

89 Questions? 9erthalion6 at gmail dot com 79