Jorge Dias Luís MCT Actual Training

Transcription

1 Jorge Dias Luís MCT Actual Training

2 2 Exchange storage background Disk storage technology Exchange Server 2010 (Beta) storage architecture Store innovations Extensible Storage Engine (ESE) database innovations Exchange Server 2010 (Beta) storage design Summary

3 3 Significant innovation in Exchange Server 2007 Reduce storage input/output (I/O) (70%) Use large amounts of memory (64 bit) Increased page size (4 kilobyte (KB) -> 8 KB) Lower storage costs Support large mailboxes (> 1 gigabyte (GB)) Provide fast search (CI) Continuous replication (log shipping) High Availability (HA) + fast recovery Eliminate single points of failure

4 SATA (3.5") Drive Capacity (GB) 750 2,000 8,000 RPM 7.2K 7.2K 10k Transfer Rate (Mb/sec) 930 2,000 5,000 Read Seek Time (ms) FC/SAS (3.5 ) Drive Capacity (GB) ,400 RPM 15K 15K 15K Transfer Rate (Mb/sec) 975 2,000 4,000 Read Seek Time (ms) Disk capacity trend predicted to continue Sequential throughput increasing linearly based on areal density (2010 SATA = 250 megabytes (MB)/sec) Random I/O performance not expected to improve substantially 4

5 Random IO Disk head has to move to process subsequent IO Head movement = High IO latency Seek Latency limits IO (IOPS) Sequential IO Disk head does not move to process subsequent IO Stationary head = low IO latency Disk RPM speed limits I/O per second (IOPS) 7.2K SATA Disk (20ms Latency) Random = 50 IOPS Sequential = +300 IOPS Disk Head 5

6 NAND Flash best utilized by Exchange Server 2010 (Beta) when used as a cache within storage stack PCM HBA / RAID NAND Enterprise SAN Array 6 Hybrid HDD Exchange Server 2010 (Beta) Mailbox Server SATA SSD

7 IO Reduction Sequential IO Storage Design Flexibility Large, Fast, Low-cost Mailboxes SATA/Tier 2 Disk Optimization RAID-less Storage (JBOD) 7

8 8 Store schema = the way the store organizes data in the Extensible Storage Engine (ESE) Database Exchange Server 2010 (Beta): One simple theme Move away from doing many, random, small size, disk IOs to doing fewer, sequential, large size, disk IO's Significant Benefits Fast/efficient Outlook Web Access (OWA)/Outlook Online Mode End user viewing for cold states/first time view creation Calendar operations Search performance Outlook cached mode/exchange Active Sync OST sync = sequential IO Exchange ActiveSync Server (EAS) sync = sequential IO Server management Move mailbox Content Index Crawls

9 Per Database Per Folder Mailbox Table Folders Table Message Table (Msg) Attachments Table Message/Folder Table (MFT) Exchange Server 2007 Jeff s Mbx Ann s Mbx Jeff:Inbox Ann:Drafts Joe:Msg10 Jeff:Msg32 Jeff:Excel.xls Ann:Pic.bmp Joe:Inbox:H1 Joe:Inbox:H2 Joe s Mbx Joe:Unread Ann:Msg180 Joe:Help.doc Joe:Inbox:H3 Secondary Indexes used for Views Per Database Per Mailbox Per View Exchange Server 2010 (Beta) Mailbox Table Jeff s Mbx Ann s Mbx Folders Table Joe:Inbox Joe:Drafts Message Header Table Joe:H10 Joe:H302 Body Table Joe:Msg10 Joe:Help.doc View Tables (e.g. From) Joe:H920 Joe:H302 Joe s Mbx Joe:Unread Joe:H920 Joe:Msg302 Joe:H10 9 New store schema = no more single instance storage within a database

10 14 Optimize for new store schema Allocate database space in contiguous manner (table space hints) Maintain database contiguity over time (online defrag refactored) Utilize space efficiently (database compression) Increase database (DB) IO Size DB page size increased from 8 kilobyte (KB) to 32 KB Improved read/write IO coalescing (Gap coalescing) Provide improved async read capability (pre-read) Increase cache effectiveness (milestone=r4) 100 megabyte (MB) checkpoint depth (HA configurations only) Cache compression (dehydration) DB cache priority (fast evict)

11 New Database Maintenance Architecture: ESE Function Exchange Server 2007 Service Pack 1 (SP1) Cleanup (deleted items/mailboxes) Space Compaction (deleted items/mailboxes) Maintain Contiguity (defragmentation) Cleanup performed during Online Defrag (OLD) which occurs during Online Maintenance (OLM) time window Database is compacted and space reclaimed during Online Defrag (OLD) N/A: Contiguity is compromised by space compaction Exchange Server 2010 (Beta) Cleanup performed at run time (when hard delete occurs) happens during Store dumpster cleanup (OLM), pages are zeroed by default Database is compacted and space reclaimed at run-time auto-throttled Database is analyzed for contiguity and space at run time and is defragmented in the background (B+Tree Defrag/OLD2) auto-throttled Database Checksum When configured, ½ of OLD maintenance window reserved for sequential scan (Checksum), manual throttle active DB copy only Two options (both Active and Passive copies): 1. Run DB Checksum in the background 24x7 (default). Sequential IO 2. Run DB Checksum during OLM window. Sequential IO Database B+Tree Defragmentation (aka OLD2): Background/throttled process that maintains space and contiguity of database tables 16

12 Exchange Server 2007 Message Header Table (aka MFT) FRAGMENTED DB Page Numbers Random deletes at the tail Exchange Server 2010 (Beta) Message Header Table (aka MsgHeader) CONTIGUOUS *Production/Dogfood database analysis 17 Blue = contiguous (good) Red = fragmented (bad)

13 Problem: Store Schema change, space hints, B+Tree Defrag and 32 KB page size combine to increase DB file size by 20% Solution: Growth is 100% mitigated by Database Compression Targeted compression for message headers and text/html bodies (7bit/Express) DB File Size Comparison DB Space Analysis 1,50 1,00 1,00 1,20 1,00 0,88 Counts E2K7 SP1 E2010 Mailbox Count Tables Internal Trees LV Trees 3 5 Msg Views 0,50 0,00 Secondary Indexes Pages Used Pages (%) 85.7% 86.7% Available Pages (%) 14.3% 13.3% Msg Table (% space) 84.9% 80.0% 32KB Pages E2K7/RTF E14/RTF E14/Mix E14/HTML 18 1 Database, 750 x 250MB mailboxes RTF = RTF Compressed, Mix = 77% HTML, 15% RTF, 8% Text Avg. Message size = ~50KB

14 IO Latency (ms) IO Latency increases with IO size 25 Random DB IO Latency Based on Size Exchange Server 2010 (Beta) Max IO Size = 256KB for Read 384KB for Write Write Read SqlIO Test, 1x 750GB 7.2k SATA, no caching array controller IO Size (KB) 22

15 Checkpoint depth = the amount of data that has yet to be committed to the database file (edb) Exchange Server 2010 (Beta) default checkpoint depth max is increasing from 20 MB to 100 MB only on databases within an HA solution (standalone still 20 MB) Deep checkpoint benefit = efficient DB writes (40% reduction) MB Checkpoint Depth = 40% DB write IO reduction Database Pages Repeatedly Written/sec DB Writes/sec (avg) Loadgen Test: 3000 Mailbox, 12 DB, Outlook 2007 Online Very Heavy Profile Checkpoint Depth (MB) Deep checkpoint risks = long store shutdown times, long crash recovery times Risk mitigation: shutdown databases in parallel, failover on store crash 23

16 DB IOPS Exchange Server 2007 Exchange Server 2010 (Beta) +70% Reduction! DB Read IO/Sec DB Write IO/Sec DB IO/Sec 3000 Mailboxes, 3MB DB Cache/user, Loadgen Outlook 2007 Online Very Heavy Profile, 250MB Mailbox Size (build 405)

17 1 1 DB IOPS/Mailbox +90% Reduction! 0,8 0,6 0,4 0,2 0 0,33 0,11 Exchange Server 2003 Exchange Server 2007 Exchange Server 2010 (Beta) Exchange Server 2003 Exchange Server 2007 Exchange Server 2010 (Beta) 25

18 Exchange Server 2010 (Beta) Smooth DB IO Benefit 49 3,7 10,1 Exchange Server 2010 (Beta) Baseline 34 0,7 5,1 Exchange Server 2010 (Beta) Smooth DB IO 50% Reduction! DB Read Latency (ms) Log Write Latency (ms) RPC Average Latency 3000 Mailboxes, 3MB DB Cache/user, 12 x 7.2k SATA disks (DB/Logs on same spindles), Loadgen Outlook 2007 Online Very Heavy Profile

19 Mailboxes/Disk (7.2K SATA) +4X Mailboxes/Disk! Exchange Server 2007 Exchange Server 2010 (Beta) MB Mailbox Size, 3MB DB Cache/user, 12 x 7.2k SATA disks (DB/Logs on same spindles), Loadgen Outlook 2007 Online Very Heavy Profile, measured at <20ms RPC Average latency

20 JBOD : 1 disk = 1 database (with logs) Requires Exchange Server 2010 (Beta) High Availability (3+ DB Copies) Annual Disk Failure Rate (AFR) = 5% JBOD Advantages Reducing Storage Costs/Complexity Eliminates unnecessary DB copies: server and storage redundancy can be symmetrical Reduces disk IO: eliminates RAID write penalty Enables simple storage design: 1 disk = 1 database (with logs) Enables simple storage failure recovery JBOD Challenges Exchange HA/storage must replace RAID functionality Disk striping performance (e.g. RAID10) cannot be leveraged Disk failure = database failover (~30 second outage) Re-enabling resiliency = spare disk assignment/partitioning/format/db reseed (scriptable) Soft disk errors (bad blocks) must be detected and repaired 30

21 31 Improve HA storage failure detection and failover Optimize HA failovers/switchovers Improve storage failure detection (bad blocks/corruption) Improve database seeding/repair HA now detects storage failures and automatically fails over (~30 seconds) ESE tuned to leverage DB cache between passive->active transitions (cache warming) Active/passive copy background scan (checksum) Active/passive copy lost write detection Utilize DB passive copy for seeding source (R4 feature) Avoid re-seed by using single page restore (active and passive) (R4 feature)

22 What is a lost flush? A DB write IO that the disk subsystem/os returned as completed did not actually get written to media or was written in the wrong location (aka lost write). Why are they so bad? Your database may be logically corrupt and you do not know it! How can they be detected in Exchange Server 2010 (Beta)? Two methods: 1. In memory flush map (active and passive): memory overhead of 2 bits/page event ID 530 is fired when detected (-1119) and page can be patched. Perfmon Counter: MSExchange Database - > Database Pages Lost Flush Detection %: The percentage of database pages which have valid lost flush detection information recorded. 2. Database recovery: event is fired (ID 516: timestamp mismatch, (- 567)) and database must be re-seeded. 32

23 1. Page corruption detected on Active Copy (e.g ) Database Availability Group (DAG) 2. Active DB places marker in log stream to notify passive copies to ship up to date page Mailbox Server Node 1 Mailbox Server Node 2 Mailbox Server Node 3 3. Passive receives log and replays up to marker, retrieves good page, invokes Replay Service callback and ships page DB1-Active Log Page1 DB1-CopyA Log Page1 DB1-CopyB Log Page1 4. Active receives good page, writes page to log, DB page is patched Page2 Page3 Database Page2 Page3 Database Page2 Page3 Database 5. Subsequent page repair from additional copies ignored 33

24 1. Page corruption detected on DB Passive Copy (e.g ) 2. Passive copy pauses log replay (log copying continues) 3. Passive retrieves the corrupted page # from the active using DB seeding infrastructure 4. Passive copy waits till log file which meets max required generation requirement is copied/inspected, then patches page Mailbox Server Node 1 DB1-Active Log Page1 Page2 Page3 Database Database Availability Group (DAG) Mailbox Server Node 2 DB1-CopyA Log Page1 Page2 Page33 Database Mailbox Server Node 3 DB1-CopyB Log Page1 Page2 Page3 Database Passive resumes log replay

25 SAN DAS (SAS) DAS (SATA) JBOD (SATA) HA = Shared Storage Clustering +1.0 IOPS/Mailbox K 146GB FC Disks RAID10 for DB & Logs Dedicated Spindles Multi-path (HBA s, FC Switches, SAN array controllers) Backup = Streaming off active Fast Recovery = Hardware VSS (Snapshots/Clones) HA = CCR.33 IOPS/Mailbox GB 10K SAS Disks RAID5 for DB RAID10 for Logs SAS Array Controller (/w BBU) Backup = VSS Snapshot Fast Recovery = CCR HA = DAG (2 DB copies).11 IOPS/Mailbox 3.5 2TB 7.2K SATA/SAS Disks RAID10 for DB & Logs SAS Array Controller (/w BBU) Backup = Optional/VSS Fast Recovery = Database Failover HA = DAG (3+ DB copies).11 IOPS/Mailbox 3.5 2TB 7.2K SATA/SAS Disks 1 DB = 1 Disk SAS Array Controller (/w BBU) Backup = Optional/VSS Fast Recovery = Database Failover 35 More options to reduce storage cost

26 36 Exchange Online archive provides mailbox storage flexibility One mailbox per user or two Exchange Server 2010 (Beta) optimized for DAS storage but SAN storage is supported IOPS reductions/sata optimizations enable lower performing storage Exchange Server 2010 (Beta) HA architected for DAS (simpler) JBOD* and RAID storage support Exchange Server 2010 (Beta)optimized for Tier 2 (SATA) disks but Enterprise disks are supported SSD/Flash storage supported but not recommended for mainstream due to high $/GB Max 100 databases/server, storage groups are gone Max recommended DB Size = 2 TB* Max recommended folder Item Count = 100 K** *3 copy High Availability only ** Assuming no 3rd party applications (OWA/Outlook Online)

27 Storage Guidance Stand Alone Exchange Server (Beta) 2010 HA (2 copies) Storage Type Disk Type DAS, SAN (Fibre Channel, iscsi) SAS, Fibre Channel, SATA, SSD Exchange Server 2010 (Beta) HA (3+ copies) RAID RAID recommended RAID optional RAID Type RAID-1/0, RAID-5, RAID-6 JBOD DB/Log Isolation Best Practice Not required Windows Disk Type Partition Type Partition Alignment File System NTFS Allocation Unit Size Encryption Support Basic (recommended), Dynamic GPT (recommended), MBR Windows 2008 Default (1MB) NTFS 64 KB for both database and log volumes Outlook Protection Rules, Bitlocker 37

28 38 Exchange Server 2010 (Beta) store has Reduced DB IOPS by +70%...again! Optimized for large mailboxes (+10 GB) and 100K item counts Optimized for large/slow/low-cost disks (SATA/Tier2) Made JBOD/RAID-less storage a viable option Enables unmatched storage flexibility to push storage Capex costs down

29 New Exchange Server 2010 (Beta) behavior 1. Delivery 2. Random Delete 3. Defragmentation Mailbox Messages Mailbox Messages M1 M1 M1 M2 M3 M2 M3 M3 M5 Contiguous M4 M4 M7 M5 M6 Contiguous M5 M6 Fragmented M10 M11 M7 M7 M12 M8 M8 M13 M9 M9 M14 M10 M10 M15 41

30 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.