Staying Out of the Swamp

Transcription

1 Staying Out of the Swamp Perforce User Conference 2001 Richard E. Baum

2 Introduction Perforce runs well when given proper resources. CPU requirements are quite small. A server s I/O bandwidth is generally the major performance limitation. A server that is responding poorly may in fact be swamped with requests.

3 What We Will Cover How to tell if your server is swamped Tools you can use to evaluate system performance External factors that cause swamp How Perforce can cause swamp

4 How Do I Tell If I m in the Swamp?

5 How Do I Tell If I m in the Swamp? Check for obvious signs of a problem. Excessive CPU utilization Abnormal memory usage I/O bandwidth problems Use available operating system tools to analyze system status. Standard OS tools provide most of the functionality required.

6 CPU Bottlenecks Check to see if there are any free processor cycles. If there are no free cycles see what is using the CPU time. It may not be perforce. Unix: use ps ef or ps axl Windows: Programs-> Administrative tools-> Performance monitor

7 CPU Bottlenecks See what Perforce process are running Note the parent/child relationships Abbreviated process table output (Unix): chinadoll:reb reb% ps -ef UID PID PPID C STIME TTY TIME CMD perforce :38:39 pts/4 0:00./p4d -p r. perforce :59:25 pts/4 0:33./p4d -p r. perforce :59:41 pts/4 0:09./p4d -p r.

8 Memory Bottlenecks How much RAM is in the system? How much swap space is defined? How much of these is available? Unix: use vmstat,, swap s, dmesg Windows: Use the task manager. (Ctl-Alt-Del)->Task Manager

9 vmstat: Memory Bottlenecks Swap column - free pages of swap Free column - free pages of RAM swap s: How much swap is configured?

10 Memory Bottlenecks Is the system swapping? swap -s output of a Solaris system: chinadoll:reb reb% % swap -s total: 22232k bytes allocated k reserved = 26752k used, k available vmstat output of a Solaris system: chinadoll:reb reb% vmstat 3 procs memory page disk faults cpu r b w swap free re mf pi po fr de sr dd dd f0 s0 in sy cs us sy id

11 Memory Bottlenecks How much memory is in the system? Partial dmesg output of a Solaris system: Sep 9 21:45:42 chinadoll unix: : [ID kern.info] mem = K (0x ) Sep 9 21:45:42 chinadoll unix: : [ID kern.info] avail mem =

12 What We Know About the System System has 640MB of RAM (655360kb) Between kb and kb of free memory. Percentage of user, system, and idle CPU time during vmstat run Use OS tools to establish a baseline of what normal values are.

13 Detecting Problems on Unix vmstat output of a Solaris system that is swapping: procs memory page disk faults cpu r b w swap free re mf pi po fr de sr s1 s2 s3 in sy cs us sy id

14 Detecting Problems on Unix Over 100mb of swap space in use CPU utilization peaks when system starts to swap, with zero idle cycles CPU utilization lower later, when paging memory in/out

15 Windows Task Manager Performance screen is most useful overall. Beware of deceptive information! CPU utilization is displayed as a percentage of total.

16 Windows Task Manager 256mb (262000kb) RAM Windows shows RAM + swap As total RAM kb in use This system is swapping!

17 Windows Task Manager Perforce server is using 11% of CPU Under 2mb of RAM

18 I/O Problems Perforce can only move data as fast as the slowest point in the data path Two main areas for problems: Disk Network

19 I/O Problems Disk access: Fast SCSI drives can transfer 45mb/sec RAID arrays can increase performance Multiple conflicting requests Disk errors

20 I/O Problems vmstat shows number of disk operations per second vmstat output of a Solaris system that is swapping: procs memory page disk faults cpu r b w swap free re mf pi po fr de sr s1 s2 s3 in sy cs us sy id

21 I/O Problems For more I/O detail try iostat Network: Network is not as fast as you think (10mb/100mb/1gb) Use of a hub on 100mb network (half duplex) Single network interface Same interface for NAS and users Physical problem (loose cable)

22 I/O Problems Network problems netstat output during large file transfer. netstat output of a Solaris system: chinadoll:reb reb% netstat -i -I le0 3 input le0 output input (Total) output packets errs packets errs colls packets errs packets errs colls

23 Windows Performance Monitor Allows monitoring of almost any part of the system. Monitor performance as it relates to: A particular process A particular thread The whole system

24 Windows Performance Monitor Example A Perforce operation that submitted a large binary file to the depot A scripted Perforce operation that performed many small submit operations. From the chart we can see that the network traffic is off the scale during the large submit. During the numerous small operations, the perforce process itself is performing much more work.

25 Windows Performance Monitor Number of bytes/sec processed by p4s Example Number of bytes written to/from the disk/sec Number of bytes/sec handled by the network card.

26 How Can Perforce Cause Server Swamp? There are many reasons your system can become overwhelmed Not all of these have to do with Perforce Your system may be used for other tasks as well.

27 Network Attached Storage Can allow faster disk access if properly configured Involves more complex configuration than local drives Opens additional areas for configuration problems

28 NAS Performance Issues Permission/locking problems NT service local user network access changed permissions Locking across the network can be slow Network topology problems Lack of adequate NAS server bandwidth Saturated network Improperly configured connection

29 Confusing / Complex Client Maps Confusing client maps may generate unexpected results. Perforce tries to do what you ve asked. In general, the last mapping wins. //depot1/... //client_name/... //depot2/... //client_name/subpath/...

30 Confusing / Complex Client Maps Client mappings do not use much memory unless multiple wildcards are used. Such views cause the server to do a lot of extra work mapping all combinations. //depot/.../subdir/... //client_name/.../subdir/...

31 Confusing / Complex Client Maps Mappings without a 1-to-1 relationship between client and server can be confusing. They can, given the proper conditions, also be harmful to Perforce servers.

32 Confusing / Complex Client Maps Example: With a new server create a new client with a default view and submit this file: //depot/subdir subdir/file Change the client view to this: //depot/... //client/... //depot/subdir/... //client/a/subdirectory/...

33 Confusing / Complex Client Maps Example: This command will then cause pre servers enter an infinite loop: p4 dirs //client/* displays this message instead: Operation: user-dirs Operation 'user-dirs' failed. Client map too twisted for directory list.

34 Confusing / Complex Client Maps Perforce can not check your client specifications for confusing, complex, or ambiguous mappings. You must do so by hand.

35 Confusing / Complex Client Maps Some ways to address these issues: Upgrade your server to release Use the server debug flags. Carefully analyze each client specification in your system. Educate your users.

36 Background Processes Other processes can consume resources even on dedicated Perforce servers. Backup utilities Virus scanners Don t run these on db.* metadata files!

37 Backup Utilities Saturate disk and/or network access Compression uses a lot of CPU time Lock db.* which causes server to fail or Can back up inconsistent metadata files.

38 Virus Scanners Can cause problems even when not run on depot metadata. Many sites require scanning of versioned file tree. Monitor CPU utilization.

39 The Perforce Error Log The default error log messages help determine that client connections have failed. Default messages do not show what clients were doing or which clients had problems. Perforce server error: Date 2001/08/27 11:17:32: TCP send failed. write: socket: WSAECONNRESET

40 The Perforce Error Log The server flag server=1 adds: Date and time Process ID of the server process Perforce user ID Client specification name IP address of the client The operation that the client invoked

41 The Perforce Error Log Sample output with server=1 set: Perforce server info: 2001/08/26 16:20:16 pid 'user-opened' Perforce server info: 2001/08/26 16:20:16 pid 'user-resolve -n' Perforce server info: 2001/08/26 16:20:16 pid 'user-resolved'

42 The Perforce Error Log server flag server=2 adds an additional completed message: Perforce server info: 2001/08/26 16:50:16 pid 'user-changes -l -s submitted' Perforce server info: 2001/08/26 16:50:16 pid 1460 completed Perforce server info: 2001/08/26 16:50:25 pid 'user-dirs -C -D //depot/*' Perforce server info: 2001/08/26 16:50:58 pid 'user-files //...' Perforce server info: 2001/08/26 16:51:10 pid 'user-verify //...' Perforce server error: Date 2001/08/26 16:52:34: TCP send failed. write: socket: WSAECONNRESET Can't invoke remote operation 'client-outputdata'. TCP send failed. write: socket: WSAECONNRESET Perforce server info: 2001/08/26 16:58:36 pid 1500 completed

43 The Perforce Error Log Some errors point directly at server problems. Librarian errors indicate that the server can not read/write the versioned file tree. Perforce server error: Date 2001/08/27 08:52:55: Operation: lbr-submitfile Operation 'lbr-submitfile' failed. Librarian checkin depot/file.txt failed. lock on depot/file.txt,v failed open for write: depot/,file.txt,: Access is denied.

44 Gigantic Operations User requests may seem innocuous but have large costs of execution. Cause disk, RAM, I/O strain Block other operations

45 Gigantic Operations Example: p4 submit Sends data to the server. Data is stored in a temporary location. Once data is on server, compute phase begins and appropriate locks are taken. Submit writes to number of tables. During write, a lock will block access to other operations that access these tables.

46 Gigantic Operations Imprecise operations with wildcards can use a lot of resources. Example: p4 files //depot/ /file.txt Causes a full table scan of db.rev Locks other users out of operations using db.rev while this occurs.

47 Gigantic Operations Most large operations can be prevented from overwhelming your server with maxresults. Set on group level. Restricts the maximum number of results returned by a query. Queries have interim steps.

48 Gigantic Operations Requests triggering maxresults show: Request too large (over 10000); see 'p4 help maxresults'. Large operations can usually be easily broken up into smaller ones: p4 sync //depot/... Becomes: p4 sync //depot/dir1/... p4 sync //depot/dir2/... p4 sync //depot/dir3/...

49 Gigantic Operations Use of p4 verify is quite CPU intensive. Checks/generates MD5 checksum of each revision of each file. Degrades performance.

50 Conclusions Perforce runs well when given proper resources Generate a baseline you can use evaluating server performance Use available tools Educate users Call support!