The DBA Survival Guide for In-Memory OLTP. Ned Otter SQL Strategist

Size: px

Start display at page:

Download "The DBA Survival Guide for In-Memory OLTP. Ned Otter SQL Strategist"

Abner Woods
6 years ago
Views:

1 The DBA Survival Guide for In-Memory OLTP Ned Otter SQL Strategist

2 About me SQL Server DBA since 1995 MCSE Data Platform Passionate about SQL Server Obsessed with In-Memory

3 Key takeaways Recovery (RTO) Corruption Container management

4 Deciphering Editions in SQL 2016/SP1+ Edition Enterprise Standard Web Express Max In-Mem/CCI OS limit/os limit 32 GB/32 GB 16 GB/16 GB 352 MB/352 MB

5 Deciphering Editions in SQL 2016/SP1+ Edition Enterprise Standard Web Express Max In-Mem/CCI OS limit/os limit 32 GB/32 GB 16 GB/16 GB 352 MB/352 MB per db per instance

6 Standard Edition (32 GB) Feature Per db Per instance In-Memory data

7 Standard Edition (32 GB) Feature Per db Per instance In-Memory data Yes No

8 Standard Edition (32 GB) Feature Per db Per instance In-Memory data Yes No Columnstore??????

9 Standard Edition (32 GB) Feature Per db Per instance In-Memory data Yes No Columnstore/In-Mem tbl Yes No

10 Standard Edition (32 GB) Feature Per db Per instance In-Memory data Yes No Columnstore/In-Mem tbl Yes No Columnstore/HD tbl No Yes

11 MEMORY-OPTIMIZED DATABASES

12 Data/delta files DATABASE

13 Data/delta files DATABASE FILEGROUP

14 Data/delta files DATABASE FILEGROUP CONTAINER

15 Data/delta files DATABASE FILEGROUP CONTAINER DATA/DELTA

16 Data/delta files DATABASE FILEGROUP CONTAINER DATA/DELTA STREAMING

17 Data/delta files DATABASE FILEGROUP CONTAINER NO MAXSIZE DATA/DELTA STREAMING

18 001 to to to to to 500 Checkpoint File Pairs (data/delta) DATA DELTA

19 Capacity planning, Memory Data Indexes Row versions

20 Memory allocation PHYSICAL MEMORY

21 Memory allocation OS PHYSICAL MEMORY

22 Memory allocation OS SQL Server max memory PHYSICAL MEMORY

23 Memory allocation OS SQL Server max memory PHYSICAL MEMORY

24 Memory allocation SQL Server max memory OS Buffer pool Proc Log Data cache cache structures PHYSICAL MEMORY

25 Memory allocation SQL Server max memory OS Buffer pool Proc Log Data cache cache structures In-Mem OLTP PHYSICAL MEMORY

26 Memory dance Buffer pool In-Mem OLTP PHYSICAL MEMORY

27 Memory dance ADD IN-MEM ROWS Buffer pool In-Mem OLTP PHYSICAL MEMORY

28 Memory dance PRESSURE ON BUFFER POOL Buffer pool In-Mem OLTP PHYSICAL MEMORY

29 Memory dance DELETE IN-MEM ROWS Buffer pool In-Mem OLTP PHYSICAL MEMORY

30 Memory dance WANTS TO GROW Buffer pool PRESSURE ON In-Mem In-Mem OLTP PHYSICAL MEMORY

31 Memory dance Buffer pool In-Mem OLTP PHYSICAL MEMORY

32 Memory dance Buffer pool In-Mem OLTP PHYSICAL MEMORY

33 Memory dance Buffer pool In-Mem OLTP PHYSICAL MEMORY

34 Memory dance Buffer pool In-Mem OLTP PHYSICAL MEMORY Buffer Pool Extension Standard Edition

35 Memory dance Buffer pool COMPRESSION In-Mem OLTP PAGING PHYSICAL MEMORY

36 Memory dance Buffer pool COMPRESSION In-Mem OLTP PAGING PHYSICAL MEMORY

37 RESOURCE GOVERNOR Enterprise Resource pool % Initial binding Monitoring

38 Windows Server max memory Windows 2012R2 = 4TB Windows 2016 = 24TB

39 Windows Server max memory Windows 2012R2 = 4TB Windows 2016 = 24TB

40 Logging Latency due to Logging In-Memory helps because No WAL (no UNDO) More efficient logging Index changes not logged

41 Logging Latency due to Logging In-Memory helps because No write ahead logging Logging efficiencies Index changes not logged*

42 Logging Latency due to Logging In-Memory helps because No write ahead logging Logging efficiencies Index changes not logged*

43 Logging Latency due to Logging In-Memory helps because No write ahead logging Logging efficiencies Index changes not logged*

44 RESTORE

45 Recovery and indexes Index type RANGE/NC HASH CCI Rebuilt

46 Recovery and indexes Index type RANGE/NC HASH CCI Rebuilt YES

47 Recovery and indexes Index type RANGE/NC HASH CCI Rebuilt YES YES

48 Recovery and indexes Index type RANGE/NC HASH CCI Rebuilt YES YES NO

49 RESTORE PHASES

50 RESTORE PHASES 1. CREATE files (MDF/NDF/LDF/DATA/DELTA)

51 RESTORE PHASES 1. CREATE files (MDF/NDF/LDF/DATA/DELTA) 2. COPY data from backup files (NORECOVERY)

52 RESTORE PHASES 1. CREATE files (MDF/NDF/LDF/DATA/DELTA) 2. COPY data from backup files (NORECOVERY) 3. STREAM data to memory (RECOVERY)

53 RESTORE PHASES 1. CREATE files (MDF/NDF/LDF/DATA/DELTA) 2. COPY data from backup files (NORECOVERY) 3. STREAM data to memory (RECOVERY) 4. RECREATE HASH/NC indexes (RECOVERY)

54 RESTORE

55 RESTORE

56 RESTORE durable in-mem data STREAM

57 RESTORE durable in-mem data STREAM

58 RESTORE durable in-mem data RESTORE FAILS! STREAM

59 RESTORE FILELISTONLY LogicalName PhysicalName Type FileGroupName Size MyDBData h:\sqldata\mydb.mdf D PRIMARY MyDBlog h:\sqldata\mydb.ldf L NULL MyDBFG1 h:\sqldata\fg1 S FG MyDBFG2 h:\sqldata\fg2 S FG

60 RESTORE FILELISTONLY LogicalName PhysicalName Type FileGroupName Size MyDBData h:\sqldata\mydb.mdf D PRIMARY MyDBlog h:\sqldata\mydb.ldf L NULL MyDBFG1 h:\sqldata\fg1 S FG MyDBFG2 h:\sqldata\fg2 S FG Does NOT account for: indexes logically deleted rows

61 Recovery process (STREAM) MEMORY-OPTIMIZED TABLES CONTAINER

62 Recovery process (STREAM) MEMORY-OPTIMIZED TABLES CONTAINER STREAM

63 Recovery process (STREAM) MEMORY-OPTIMIZED TABLES CONTAINER 1 CONTAINER 2 CONTAINER 3 CONTAINER 4 STREAM STREAM STREAM STREAM

64 Stream events

65 Stream events OFF/ONLINE

66 Stream events OFF/ONLINE RCSI ON/OFF

67 Stream events OFF/ONLINE RCSI ON/OFF R/O to R/W

68 Stream events OFF/ONLINE RESTORE RCSI ON/OFF R/O to R/W

69 Stream events OFF/ONLINE RESTORE RCSI ON/OFF SQL RESTART R/O to R/W

70 Stream events OFF/ONLINE RESTORE RCSI ON/OFF SQL RESTART R/O to R/W SERVER BOOT

71 Potential recovery issues CPU bound FCI/AG PIECEMEAL

72 CORRUPTION

73 Corruption: Detection and Recovery Method Harddrive Memory CHECKDB YES

74 Corruption: Detection and Recovery Method Harddrive Memory CHECKDB YES CHECKTABLE YES

75 Corruption: Detection and Recovery Method Harddrive Memory CHECKDB YES CHECKTABLE YES Page-level restore (mirroring/ag) YES

76 Corruption: Detection and Recovery Method Harddrive Memory CHECKDB YES CHECKTABLE YES Page-level restore (mirroring/ag) YES Recover data from NC indexes YES

77 Corruption: Detection and Recovery Method Harddrive Memory CHECKDB YES CHECKTABLE YES Page-level restore (mirroring/ag) YES Recover data from NC indexes YES Hack attach YES

78 Corruption: Detection and Recovery Method Harddrive Memory CHECKDB YES CHECKTABLE YES Page-level restore (mirroring/ag) YES Recover data from NC indexes YES Hack attach YES RESTORE REBUILD LOG YES

79 Corruption: Detection and Recovery Method Harddrive Memory CHECKDB YES NO* CHECKTABLE YES NO** Page-level restore (mirroring/ag) YES n/a Recover data from NC indexes YES NO Hack attach YES NO RESTORE REBUILD LOG YES NO

80 CHECKDB PRIMARY MDF LDF

81 CHECKDB PRIMARY MDF LDF MEMORY-OPTIMIZED Container1 DATA/DELTA Container2 DATA/DELTA

82 Corruption: Detection and Recovery BLOCK CHECKSUM

83 Corruption: Detection and Recovery SQL BACKUPS VALIDATE CHECKSUMS

84 Corruption: Detection and Recovery SQL BACKUPS VALIDATE CHECKSUMS FULL DIFF*

85 Corruption: Detection and Recovery SAN SNAPSHOTS: NO CHECKSUM VALIDATION

86 Corruption: Detection and Recovery CHECKDB Alternative: BACKUP <mem-optimized FG> TO DISK = nul WITH COPY_ONLY

87 CHECKSUM failure Corruption detection point Action Regular processing, i.e. file merge Sev 21, Error Backup/restore Sev 16, Error Database recovery Sev 21, Error 3414

88 Recovery model is...??? for memory-optimized DML irrelevant

89 Recovery model is... irrelevant for memory-optimized DML irrelevant

90 Recovery model is... irrelevant FULL BULK_LOGGED SIMPLE DML = FULLY LOGGED FOR DURABLE IN-MEM OBJECTS

91 Transaction log latency

92 Transaction log latency WRITELOG

93 Transaction log latency Persistent Memory Server Class Memory NVDIMM-N

94 Transaction log latency Persistent Memory Server Class Memory NVDIMM-N

95 Transaction log latency Persistent Memory Server Class Memory NVDIMM-N

96 Transaction log latency: NVDIMM-N DRAM

97 Transaction log latency: NVDIMM-N DRAM FLASH

98 Transaction log latency: NVDIMM-N DRAM FLASH POWER

99 Transaction log latency: NVDIMM-N LOG BLOCKS DRAM FLASH POWER

100 Transaction log latency: NVDIMM-N LOG BLOCKS DRAM FLASH POWER

101 Transaction log latency: NVDIMM-N LOG BLOCKS DRAM FLASH POWER

102 PERSISTENT MEMORY NVDIMM-N TDE

103 ALTER TABLE

104 ALTER TABLE Parallel/logs metadata only OFFLINE SCHEMABINDING 2x memory required

105 ALTER TABLE Parallel/logs metadata only OFFLINE SCHEMABINDING 2x memory required

106 ALTER TABLE Parallel/logs metadata only OFFLINE SCHEMABINDING 2x memory required

107 ALTER TABLE Parallel/logs metadata only OFFLINE SCHEMABINDING 2x memory required

108 ALTER TABLE exceptions Serially executed and fully logged

109 ALTER TABLE exceptions Serially executed and fully logged Add/alter variable length LOB Add/drop COLUMNSTORE index Column movement: on-row / off-row Create new off-row column

110 ALTER TABLE exceptions Serially executed and fully logged Add/alter variable length LOB Add/drop COLUMNSTORE index Column movement: on-row / off-row Create new off-row column

111 ALTER TABLE exceptions Serially executed and fully logged Add/alter variable length LOB Add/drop COLUMNSTORE index Column movement: on-row / off-row Create new off-row column

112 ALTER TABLE exceptions Serially executed and fully logged Add/alter variable length LOB Add/drop COLUMNSTORE index Column movement: on-row / off-row Create new off-row column

113 STATISTICS

114 STATISTICS Automatic/Manual creation Auto updated (compatibility level) Manually updated (UPDATE STATISTICS)

115 STATISTICS Automatic/Manual creation Auto updated (compatibility level) Manually updated (UPDATE STATISTICS)

116 STATISTICS Automatic/Manual creation Auto updated (compatibility level) UPDATE STATISTICS

117 STATISTICS Automatic/Manual creation Auto updated (compatibility level) UPDATE STATISTICS Native modules

118 INDEX MAINTENANCE

119 Index maintenance FRAGMENTATION

120 Index maintenance FRAGMENTATION FILLFACTOR

121 Index maintenance FRAGMENTATION FILLFACTOR REORGANIZE

122 Index maintenance HASH REBUILD: BUCKETS

123 CONTAINER MANAGEMENT

124 Don t touch those containers! No: File manipulation, i.e. copy/paste

125 Don t touch those containers! No: File manipulation, i.e. copy/paste Placing other files in containers

126 Don t touch those containers! No: File manipulation, i.e. copy/paste Placing other files in containers Trespassing!

127 ZERO FREE SPACE

128 Zero free space MEMORY-OPTIMIZED TABLES 0 FREE C 1/V 1 C 2/V 2 C 3/V 3 TLOG

129 Zero free space MEMORY-OPTIMIZED TABLES 0 FREE NO MAXSIZE C 1/V 1 C 2/V 2 C 3/V 3 TLOG

130 Zero free space MEMORY-OPTIMIZED TABLES 0 FREE NO MAXSIZE C 1/V 1 C 2/V 2 C 3/V 3 TLOG C 4/V 4

131 CONTAINER REDISTRIBUTION

132 Container redistribution You cannot drop a non-empty container or move data and delta file pairs to another container in the memory-optimized filegroup.

133 Container redistribution MEMORY-OPTIMIZED TABLES RTO = 30m C 1/V 1 C 2/V 2 C 3/V 3 NOW: 50GB USED, RESTORE = 15 minutes

134 Container redistribution MEMORY-OPTIMIZED TABLES C 1/V 1 C 2/V 2 C 3/V 3 RTO = 30m RTO LATER: 150GB USED, RESTORE = 45 minutes

135 Container redistribution MEMORY-OPTIMIZED TABLES C 1/V 1 C 2/V 2 C 3/V 3 RTO = 30m RTO C 4/V 4

136 Container redistribution DATA MEMORY-OPTIMIZED TABLES C 1/V 1 C 2/V 2 C 3/V 3 RTO = 30m RTO C 4/V 4

137 Container redistribution OLD NEW MEMORY-OPTIMIZED TABLES C 1/V 1 C 2/V 2 C 3/V 3 RTO = 30m RTO C 4/V 4 C 5/V 5 C 6/V 6 C 7/V 7 C 8/V 8

138 Container redistribution CREATE containers on NEW volumes DROP containers on OLD volumes Engine migrates data DB = ONLINE!

139 INDEX ISSUES

140 Index issues Dupes/collisions (100+) Wrong BUCKET_COUNT (HASH) Searching on subset of key columns (HASH) Inequality predicates (HASH) sys.dm_db_xtp_hash_index_stats (avg_chain_length)

141 NATIVE COMPILATION ISSUES

142 Native compilation issues EXEC

143 Native compilation issues EXEC

144 Native compilation issues EXEC

145 Native compilation issues Issue Named parameters Mismatched datatypes xevents reason named_parameters parameter_conversion natively_compiled_proc_slow_parameter_passing

146 Native compilation Parameter sniffing: not possible

147 Native compilation Parameter sniffing: not possible JOINs: NESTED LOOP

148 Native compilation Parameter sniffing: not possible JOINs: NESTED LOOP Execution: serial

149 Native compilation Parameter sniffing: not possible JOINs: NESTED LOOP Execution: serial Recompile NC after update stats/large DML

150 BEST PRACTICES

151 Best practices, Capacity planning Memory = 2x data set Total IOPS = Workload IOPS x 3 Storage = In-mem size x 4 (plus larger backups) Can tlog handle new throughput?

152 Best practices, Capacity planning Memory = 2x data set Total IOPS = Workload IOPS x 3 Storage = In-mem size x 4 (plus larger backups) Can tlog handle new throughput?

153 Best practices, Capacity planning Memory = 2x data set Total IOPS = Workload IOPS x 3 Storage = In-mem size x 4 Can tlog handle new throughput?

154 Best practices, Capacity planning Memory = 2x data set Total IOPS = Workload IOPS x 3 Storage = In-mem size x 4 Can tlog handle new throughput?

155 Best practices, Indexing Range index first Index ALL Foreign Keys Bucket count = Unique * 2 Verify chain length (HASH) Recompile NC after update stats/table grows

156 Best practices, Indexing Range index first Index ALL Foreign Keys Bucket count = Unique * 2 Verify chain length (HASH) Recompile NC after update stats/table grows

157 Best practices, Indexing Range index first Index ALL Foreign Keys Bucket count = Unique * 2 Verify chain length (HASH) Recompile NC after update stats/table grows

158 Best practices, Indexing Range index first Index ALL Foreign Keys Bucket count = Unique * 2 Verify chain length (HASH) Recompile NC after update stats/table grows

159 Best practices, Indexing Range index first Index ALL Foreign Keys Bucket count = Unique * 2 Verify chain length (HASH)

160 Best practices, Alerting Checksum failure: Regular processing Checksum failure: Backup/Restore Checksum failure: Database recovery

161 Best practices, Retry someone committed before you repeatable read validation failure serializable validation failure transaction dependency failure

162 Best practices, Planning 1. Database/Storage design changes 2. Test RTO with full data set Repeat until RTO achieved

163 Best practices, Maintenance Create documentation and test: Redistributing containers (failed RTO) Add container (out of space) PIECEMEAL restore

164 Best practices, ALTER TABLE Test ALTER TABLE

165 Connect items 1. Unknown memory requirements for restoring a backup containing memory-optimized data: 2. Excessive memory usage for memoryoptimized table variable with RANGE index:

166 Connect items (2) 3. DBCC CHECKDB should verify checksums on In- Memory OLTP tables:

167 KEY TAKEAWAYS Recovery (RTO) Corruption Container management

168 The DBA Survival Guide for In-Memory OLTP Ned Otter SQL Strategist

169 e: blog: nedotter.com The DBA Survival Guide to In-Memory OLTP Ned Otter SQL Strategist

Real world SQL 2016 In-Memory OLTP

Real world SQL 2016 In-Memory OLTP Ned Otter SQL Strategist #588 About me SQL Server DBA since 1995 MCSE Data Platform Passionate about SQL Server Obsessed with In-Memory KEY TAKEAWAYS ARCHITECTURE RTO