Better Living Through New Releases

Transcription

1 Better Living Through New Releases by Richard E. Baum Overview Server performance-related functionality Upgrades contain much more than bug fixes This is a high level view Close to chronological order By topic where appropriate Not an exhaustive list Release notes are your friend 1

2 Automatic labels (2006.2) Create labels based on changelist and path No p4 labelsync needed Label Revision: field applied to view lines Automatic labels (2006.2) Changelist alias: Label: nightly_build_01-apr-2009 Owner: buildmeister Descr: Nightly build process View: //depot/... Files in a single changelist: Label: patch Owner: mr-patchit Descr: Patch for bug 8471 View: //depot/... 2

3 p4 license (2006.2) Add licensed users without server restart From command line From Perforce Administration GUI IP change requires a restart +Sn filetypes (2007.2) Only store most recent n revisions Older revisions purged at submit Older metadata remains intact Values for n: 1-10, 16, 32, 64, 128, 256, 512 Improves handling of large binary objects 3

4 Journal file buffering (2006.2) Journal buffer flushed periodically No longer after every transaction Reduces writes Reduces locks taken Improves overall performance Faster journal / checkpoint rotation (2007.3) Rotation can now be a rename UNIX / Linux Rotation can be instant on UNIX / Linux p4d -jc or p4d -jj Server copies and truncates if -z flag used for compression Rename fails Non UNIX / Linux server used 4

5 Checkpoint / Journal locking (2007.3) Only read locks taken during: p4d jc p4d jj Most read operations can continue TCP/IP buffering (2008.1) Auto-discover / adjust buffering levels Make better use of buffering space Improved network performance Particularly over high latency connections 5

6 Proxy caches on submit (2007.3) Files immediately ready for sync via proxy No additional sync required MD5 hash values used for validation Timestamps used only as a fallback Improved Proxy cache load (2007.3) Update cache without sending files to client: p4 Zproxyload sync Intended to be used with dedicated client Have list assists with file selection Client workspace remains empty 6

7 Sensible proxy compression (2008.1) When syncing through Perforce Proxy Eliminates proxy decompress/compress Less overall work Saves CPU on proxy Speeds up overall processing Start server/proxy compression earlier Obliterate improvements (2008.1) We now compare: Number of lazy copies Number of revisions purged If equal, purge the archive file! Fewer scans of database Mapping optimizations 7

8 Locking (various releases) Overall huge improvements Incremental changes every release Targeted rewrites for better concurrency New locking algorithm (2008.1) MaxLockTime (2006.2) p4 group field Max time an operation can lock a db file Server request fails when limit is reached Group spec: Group: all_users MaxLockTime: Users: jerry User output: Operation took too long (over seconds); see 'p4 help maxlocktime'. Log output: --- killed by MaxLockTime 8

9 Group unlimited values (2008.1) p4 group limit maximums MaxScanRows, MaxResults, MaxLockTime, Timeout Allows use of finer granularity These both set no limit: unlimited other group settings don t apply unset other group settings apply Timeout of 0 no longer allowed (use unset ) Old behavior maintained on upgrade unlimited becomes unset New locking methodology (2008.1) Old, blocking methodology: Take locks as needed If needed lock is unavailable, wait until it is Perform operation Release locks Locks held during wait Blocks other processes 9

10 Old methodology locking log fragment: 'user-client -d build_client' -- db.domain -- locks wait+held read/write 0ms+0ms/0ms ms -- db.view -- locks wait+held read/write 0ms+1ms/0ms ms -- db.have -- locks wait+held read/write 115ms+10ms/3942ms ms -- db.label -- locks wait+held read/write 0ms+0ms/0ms ms -- db.resolve -- locks wait+held read/write 0ms+0ms/0ms ms -- db.locks -- locks wait+held read/write 0ms+0ms/0ms ms -- db.working -- locks wait+held read/write 0ms+0ms/0ms ms -- db.change -- locks wait+held read/write 0ms+0ms/252831ms+4ms 7 tables blocked 4+ minutes for 4ms of access to blocking table New locking methodology (2008.1) New methodology: Locks requested until a request blocks Release acquired locks Attempt to acquire blocking lock When granted, drop and start again After third try, use old behavior Only one lock held while waiting Does not block additional processes 10

11 Batched updates (various) Many database updates now batched Rather than updating revisions as changed Changes buffered to reduce I/O, locking Table update only every 100 records More commands added each release (sync, resolve, edit, revert, integ, submit, etc.) Server tunables (2008.2) Don t try this at home! Configure server limits/values Previously hard-coded values db.isalive 10K Rows scanned before maxlocktime check db.trylock 3 Attempts to avoid locks that block dm.flushtry 100 update buffer for sync, resolve, edit See documentation for more details: p4 help tunables p4 admin tunables 11

12 Enhanced p4d -xv (2008.2) Newly built db files include data checksums For db files built with and later server Reported by p4 admin dbstat Now, p4d -xv checks the checksum Previously only table structure was checked p4d -xr sets new checksums during repair Only repair with assistance from support! Enhanced p4d -xv (2008.2) Example: %p4 admin dbstat db.rev db.rev internal+leaf 0+1 page size 8k end page 1 generation 5 levels 1 fanout 0 Checksum %p4d -xv Validating db.counters... Validating db.rev **** Checksum mismatch **** possible data corruption 12

13 Summary Performance-related changes every release Release notes detail many modern marvels! Modernize your depot upgrade! Need help? Call! Perforce Technical Support Perforce Consulting Services Q & A 13

14 The End 14