TSM Node Replication Deep Dive and Best Practices

Transcription

1 TSM Node Replication Deep Dive and Best Practices Matt Anglin TSM Server Development

2 Abstract This session will provide a detailed look at the node replication feature of TSM. It will provide an overview of configuring replication, and explore various models in which replication may be deployed. Best practices and performance optimization will also be discussed. 2

3 Agenda Overview Preparing For Replication Performing A Replication Checking The Results Altering Replication 3

4 What Is TSM Replication? Source Server Target Server 1. Initial replication all objects are copied to target Backup, Archive, and Space Management Objects 2. Deleted objects are deleted from target 3. Modified objects are updated on the target 4. Newly stored objects are copied during next replication 4

5 Branch Office Model All branches replicate to central location 5

6 Disaster Recovery Model N4 N5 N6 N1 N2 N3 N1 N2 Nn Primary R/W Node N3 N4 Nn Replicated R/O Node N5 N6 6

7 Advantages Of TSM Replication Stgpool 1 DB DB Stgpool 1 Stgpool 2 AIX Windows 1. Different Operating Systems on source and target 2. Storage Pool layout can be different on each server 3. Meta-data and data always kept in sync 4. Data can be replicated from any storage pool 5. Objects without data are also replicated 7

8 Characteristics Highly parallel Can run multiple replications at once Examination of objects in a file space overlaps with batches being sent to target server Highly resilient Multiple levels of retry to achieve success Will attempt to get data from any storage pool Deduplication aware 8

9 Replication and Deduplication If the data on the source server is... Identified/ Deduplicated And the destination storage pool on the target server is... Enabled for deduplication The result is... Only extents that are not stored in the destination storage pool are transferred. Identified/ Deduplicated Not enabled for deduplication Files are reassembled by the source server and replicated in their entirety. Not Identified/ Deduplicated Enabled for deduplication Files are sent by the source server and replicated in their entirety. Server-side deduplication will occur on the target. Not Identified/ Deduplicated Not enabled for deduplication Files are replicated in their entirety. 9

10 Agenda Overview Preparing For Replication Performing A Replication Checking The Results Altering Replication 10

11 Hardware Requirements CPU/RAM minimum recommendations With deduplication, 8 CPU cores, 64GB RAM Best practice: 8 CPU cores, 128GB RAM Without deduplication, 4 CPU cores, 32GB RAM Best practice: 4 CPU cores, 64GB RAM Best practices assume complete server replication Requirements less if replicating less 11

12 Database And Log Requirements Active and Archive Log At least 64GB Active Log, preferably 128GB A file space with 15,000,000 objects can use 8GB log Archive log sized to 2 times daily log usage Database A 300GB DB on a source server will require an additional 300GB of DB on the target server In addition to current size of target DB Plan DB size and growth appropriately 12

13 Tasks Create/Verify the server definitions Set the replication target for the server Set server-level RULE defaults (optional) Determine which nodes, file spaces, and data types are to be replicated Assign appropriate rules, or use defaults Enable replication for the nodes Replicate! 13

14 Define Servers Replication uses SERVER-SERVER sessions, but they can also look like normal client sessions The servers must be cross-defined Similar to Library Client/Library Manager Session authentication performed at server level Can use SSL, if desired SET REPLSERVER to specify target server 14

15 Replication Rules ALL_DATA_HIGH_PRIORITY High Priority Backup, Archive, and Space Management ACTIVE_DATA_HIGH_PRIORITY High Priority Backup Only ALL_DATA Normal Priority Backup, Archive, and Space Management (Default) ACTIVE_DATA Normal Priority Backup Only Default None 15

16 Setting Default Rules Examples of setting rules for the server SET BKREPLRuledefault ACTIVE_DATA SET ARREPLRuledefault ALL_DATA_HIGH_PRIORITY SET SPREPLRuledefault ALL_DATA DEFAULT cannot be used for the server rules Examples of setting rules for a node UPDATE NODE XX BKREPLRULE=ALL_DATA UPDATE NODE YY ARREPLRULE=DEFAULT 16

17 Replication Rule Hierarchy 17

18 Populating Target Server Two basic methods to populate target server Method 1 Replicate from scratch Best if source and target are in close proximity All eligible data is sent Method 2 Synchronize and Replicate Best for large distances or if bandwidth is limited Use media-based Export/Import to populate target Replication with SYNC links the source and target objects 18

19 Sync Example SYNCSEND SEND SYNCRECEIVE S1 Linked Pair S1 T1 S2 Linked Pair S2 T2 S3 Linked Pair S3 S4 Linked Pair S4 S5 Linked Pair S5 Source Server Target Server 1. Export/Import used to transfer data to target server 2. REPLMODE set appropriately on each server 3. Replication will linked the common objects and change the REPLMODE 4. Subsequent replications will copy new objects 19

20 Replication Mode Determines role for a node (source/target) Normal modes SEND the node is the source of replication RECEIVE the node is the target of replication Cannot be set directly SYNC modes SYNCSEND the node is a synced source SYNCRECEIVE the node is a synced target 20

21 Replication State Indicates whether replication is enabled Used to temporarily stop replicating Applies to: Client Nodes Replication Rules Source Server Client Nodes 21

22 TSM Symposium 2013: Tivoli Storage Manager: Future Expectations Vendor Talks Managing The Policy Make sure target server has same policy constructs as the source for all nodes Arch CG MC_1 Arch CG MC_1 Bkup CG Domain_A Domain_B Domain_X Source Server Bkup CG Domain_A Domain_B Domain_Y Target Server Replication does not replicate the policy 22

23 Managing The Policy Replication does not replicate the policy Use use EXPORT/IMPORT or Enterprise Configuration If a policy construct is missing on the target server, the default construct is used 23

24 Version Control And Retention Managed by the source server only Expiration of replicated nodes disabled on target Source server instructs target to delete objects during replication If you don t replicate, objects don t get deleted Might be useful for taking snapshots 24

25 TSM Symposium 2013: Tivoli Storage Manager: Future Expectations Vendor Talks Target Management Class Fields Attributes Honored Attributes Ignored Management Class Attributes Management Class Attributes Default Mgmt Class Migration Destination Space Management Technique Auto-Migrate on Non-Use Migration Requires Backup Copygroup Attributes Copy Destination TOC Destination Copygroup Attributes Versions Data Exists Versions Data Deleted Retain Extra Versions Retail Only Version Copy Mode Copy Serialization Copy Frequency 25

26 Restrictions on Target Server A replicated node is Read-Only Cannot store new data from a client or application Expiration does not delete objects Cannot rename the node Data can be deleted from target with: DELETE VOL DISCARDD=YES AUDIT VOL FIX=YES DELETE FILESPACE **Data deleted from target will be sent during next replication 26

27 Planning Considerations Need to plan for the daily change rate The use of ACTIVE_ONLY doesn t alter this Are your RAM, CPU, and disks sufficient? How much data needs to be initially replicated to get to the steady-state? Do you have the time and bandwidth to replicate it from scratch? Would it be better to use Export/Import? 27

28 Agenda Overview Preparing For Replication Performing A Replication Checking The Results Altering Replication 28

29 Replicate Node Command The REPLICATE NODE command accepts: Multiple nodes and/or node groups Specific file spaces belonging to a node The data type to replicate The priorities to include in the replication REPLICATE NODE starts a single process Process ends when ALL nodes and file spaces are complete Can be scheduled as part of daily maintenance 29

30 Replicate Node Processing Each node and file space specified is examined List of node/file space/data-type is created Source and target exchange information Authentication validated Replication GUIDs exchanged Once set, can never be changed Can t clone a DB and start replication as new server 30

31 Replicate Node Processing For each node being processed Target node is registered, if necessary Replication State and Mode are verified Each node is enabled for replication Source in SEND mode, target in RECEIVE mode Target node is synchronized to source Attributes, including passwords, are replicated If you move a node to a different domain on the target, replication will move it back 31

32 Replicate Node Processing As file spaces are examined, batches created Options to control batch sizes: REPLSIZETHRESH option controls size of replication batches in terms of megabytes REPLBATCHSIZE option controls maximum number of objects in each replication batch Batches are processed even while file spaces are being examined Each batch uses 1 session MAXSESSIONS keyword controls number of batches 32

33 Cancelling Replication Replication can be cancelled only from the source server CANCEL PROCESS Will cancel a particular replication process CANCEL REPLICATION Will cancel all replications currently running 33

34 Agenda Overview Preparing For Replication Performing A Replication Checking The Results Altering Replication 34

35 Status of Current Replications As batches complete, statistics are updated Q PROCESS shows statistics as of last batch 6 Replicate Node Replicating node(s) IRONMAN3. File spaces complete: 8. File spaces identifying and replicating: 0. File spaces replicating: 1. File spaces not started: 0. Files current: 0. Files replicated: 0 of 5. Files updated: 0 of 0. Files deleted: 0 of 0. Amount replicated: 0 KB of 41,720 KB. Amount transferred: 0 KB. Elapsed time: 0 Day(s), 0 Hour(s), 1 Minute(s). 35

36 Other Statistics Completion messages: ANR0327I Replication of node IRONMAN3 completed. Files current: 0. Files replicated: 5 of 5. Files updated: 0 of 0. Files deleted: 0 of 0. Amount replicated: 41,720 KB of 41,720 KB. Amount transferred: 20,860 KB. Elapsed time: 0 Day(s), 0 Hour(s), 1 Minute(s). ANR0986I Process 6 for Replicate Node running in the BACKGROUND processed 5 items for a total of 42,721,280 bytes with a completion state of SUCCESS at 14:04:03. 36

37 Q PROCESS Output Details Replicating Node(s): list of nodes being processed in this replication File Spaces complete: file spaces that have completed replication File spaces examining file spaces that are determining what files to process and replicating: and are also replicating data File spaces replicating: file spaces that are replicating data only File spaces not started: file spaces that have not yet begun Files current: files that do not need replicated, updated or deleted Files replicated: files that have been replicated out of a total number to replicate Files updated: files that have been updated out of a total number to update Files deleted: files that have been deleted out of a total number to delete Bytes Replicated: bytes replicated out of a total number of bytes to replicate Bytes Transferred: bytes sent to the target server (shows dedup savings ) 37

38 QUERY REPLICATION Shows history of replications ANR2017I Administrator SERVER_CONSOLE issued command: QUERY REPLICATION ironman3 Node Name Filespace Name FSID Start Time End Time Status Phase IRONMAN3 /space 5 05/11/12 13:35:12 05/11/12 13:35:31 Ended None IRONMAN3 /space 5 05/11/12 13:42:59 05/11/12 13:43:10 Ended None IRONMAN3 /space 5 05/11/12 14:03:50 05/11/12 14:04:03 Ended None IRONMAN3 /hsmfs 6 05/11/12 13:35:12 05/11/12 13:35:31 Failed None IRONMAN3 /hsmfs 6 05/11/12 13:42:59 05/11/12 13:43:10 Ended None IRONMAN3 /hsmfs 6 05/11/12 14:03:50 05/11/12 14:04:03 Ended None IRONMAN3 /lspace 7 05/11/12 13:35:12 05/11/12 13:35:31 Failed None IRONMAN3 /lspace 7 05/11/12 13:42:59 05/11/12 13:43:10 Ended None IRONMAN3 /lspace 7 05/11/12 14:03:50 05/11/12 14:04:03 Ended None 38

39 Q REPLICATION F=D Node Name: IRONMAN Filespace Name: /space FSID: 2 Start Time: 02/08/11 21:44:19 End Time: 02/08/11 21:48:14 Status: Completed Process Number: 4 Command: replicate node ironman Phase: Completed Process Running Time: 0 Day(s), 0 Hour(s), 4 Minute(s) Completion State: Complete Reason for Incompletion: None Backup Last Update Date/Time: Backup Target Server: Backup Files Needing No Action: 0 Backup Files To Replicate: 0 Backup Files Replicated: 0 Backup Files Not Replicated Due To Errors: 0 Backup Files Not Yet Replicated: 0 Backup Files To Delete: 0 Backup Files Deleted: 0 Backup Files Not Deleted Due To Errors: 0 Backup Files To Update: 0 Backup Files Updated: 0 Backup Files Not Updated Due To Errors: 0 39

40 Q REPLICATION F=D Backup Bytes To Replicate (MB): 0 Backup Bytes Replicated (MB): 0 Backup Bytes Transferred (MB): 0 Backup Bytes Not Replicated Due To Errors (MB): 0 Backup Bytes Not Yet Replicated (MB): 0 Archive Last Update Date/Time: 02/08/11 21:48:14 Archive Target Server: NIGLINA Archive Files Needing No Action: 0 Archive Files To Replicate: 39,416 Archive Files Replicated: 39,206 Archive Files Not Replicated Due To Errors: 210 Archive Files Not Yet Replicated: 0 Archive Files To Delete: 0 Archive Files Deleted: 0 Archive Files Not Deleted Due To Errors: 0 Archive Files To Update: 0 Archive Files Updated: 0 Archive Files Not Updated Due To Errors: 0 Archive Bytes To Replicate (MB): 4,335 Archive Bytes Replicated (MB): 4,335 Archive Bytes Transferred (MB): 0 Archive Bytes Not Replicated Due To Errors (MB): 0 Archive Bytes Not Yet Replicated (MB): 0 40

41 Q REPLICATION F=D Total Files Needing No Action: 314,400 Total Files To Replicate: 39,416 Total Files Replicated: 39,206 Total Files Not Replicated Due To Errors: 210 Total Files Not Yet Replicated: 0 Total Files To Delete: 0 Total Files Deleted: 0 Total Files Not Deleted Due To Errors: 0 Total Files To Update: 0 Total Files Updated: 0 Total Files Not Updated Due To Errors: 0 Total Bytes To Replicate (MB): 4335 Total Bytes Replicated (MB): 4330 Total Bytes Transferred (MB): 1298 Total Bytes Not Replicated Due To Errors (MB): 5 Total Bytes Not Yet Replicated (MB): 0 Estimated Percentage Complete: 100 Estimated Time Remaining: Estimated Time Of Completion: 41

42 Agenda Overview Preparing For Replication Performing A Replication Checking The Results Altering Replication 42

43 Disabling Replication DISABLE REPLICATION Disables OUTBOUND replication (not inbound) ENABLE REPLICATION to resume Set REPLSTATE=DISABLED UPDATE NODE X REPLSTATE=DISABLED UPDATE FILESPACE X Y REPLSTATE=DISABLED UPDATE REPLRULE ALL_DATA STATE=DISABLED All file spaces that are disabled will be skipped when REPLICATE NODE is issued 43

44 Removing Replication REMOVE REPLNODE <nodename> Deletes all replication information from the DB Can be run on source, target, or both Sets the REPLSTATE and REPLMODE to NONE Does not delete any data 44

45 Removing Replication UPDATE FILESPACE X Y REPLST=PURGEDATA Puts a file space into a special purge state During next replication All objects in the file space are deleted from the target The file space REPLMODE set to NONE The file space REPLSTATE set to DISABLED If PURGEDATA not used, target data not deleted 45

46 Disabling vs Removing Replication Disabling is temporary All replication data is retained on both servers Nodes on target server are still Read-Only When you re-enable, it picks up where it left off Removing is permanent All replication information is deleted from both servers Nodes on target server become Read-Write Must do a SYNC operation to re-establish pair Or delete all data from the target server 46

47 Controlling Server Sessions Can enable or disable server sessions Useful on source and target to control replication DISABLE SESSIONS SERVER XX DIRECTION=OUT ENABLE SESSIONS SERVER XX DIRE=BOTH >>-DISAble SESSions > >--SERVer >.--DIRection--=--Both '--server_name ' +--DIRection--=--Both DIRection--=--INbound---+ '--DIRection--=--OUTbound--' 47

48 Reversing Direction On current source/new target REMOVE REPLNODE ZZZ UPDATE NODE ZZZ REPLMODE=SYNCRECEIVE REPLSTATE=ENABLED On current target/new source REMOVE REPLNODE ZZZ UPDATE NODE ZZZ REPLMODE=SYNCSEND REPLSTATE=ENABLED SET REPLSERVER=<new target>, if necessary REPLICATE NODE ZZZ 48

49 The DB Restore Problem S1 S2 S3 Linked Pair Linked Pair Linked Pair S1 S2 S3 Source Server Target Server 1. Start with 2 files on source and target 2. Back up database at time T1 3. Store another file at time T2 4. Replicate the file 5. Restore database to time T1 6. Replication will delete S3 on the target 49

50 Database Restore Two choices Accept the current situation ENABLE REPLICATION Copy the most recent data to the source server Follow the Reversing Direction instructions ENABLE REPLICATION REPLICATE NODE Follow the Reversing Direction instructions again To re-establish original source/target roles 50

51 Tuning For Performance Be sure to test replication throughput Adjust MAXSESSIONS to determine point of diminishing returns Network, CPU, and RAM will impact throughput Make sure sufficient mount points are available for replication For FILE, set mount limit to at least the product of NUMOPENVOLSALLOW and MAXSESSIONS 51

52 Best Practices If possible, allow sufficient time for IDENTIFY to process all data before replicating Allows replication to benefit from deduplication Run storage pool reclamation after replication Run storage pool migration after replication Faster to replicate from disk than tape Replication and storage pool backup can be run simultaneously 52

53 Best Practices Don t run all nodes in a single replication Replicate large nodes by themselves With a smaller value for MAXSESSIONS (1-3) Set up and use NODE GROUPS as way to manage replications Apply maintenance as it becomes available 53

54 Thank You 54