Performance Analysis of a Call Center Telecommunications Interface. CS672 Spring 2004 Mohamed Benalayat Raymond Jordan

Transcription

1 Performance Analysis of a Call Center Telecommunications Interface CS672 Spring 2004 Mohamed Benalayat Raymond Jordan

2 Overview Call center monitoring and recording systems allow corporations to monitor the performance of their agents by providing the ability to record and retrieve telephone conversations in real time. Monitoring systems usually require an interface between the Call Center Monitor/Recorder and the telecommunications infrastructure. The following paper is a study of the performance characteristics of such a call center telecommunications interface.

3 Project Goals Determine the maximum workload that can be placed on the Call Center Telecom Interface. Determine how to improve the Call Center Telecom Interface in order to achieve 100 or more simultaneous input lines.

4 System Description The call center monitoring system under study can be described as being composed of 2 functional components: A front-end telecommunications interface component, which functions as a call-processing unit. It is responsible for real time reception of digitized telephony data, processing the received data, converting to a recognized audio format, and forwarding to recording unit. A back end recording component which functions as a digital recording unit. It is responsible for recording and file management. Call Center PBX Telecom interface CPU/Disk 10BASE-T interface Recording unit

5 System Diagram Call Center Telecommunications Interface In from T-1 card CPU Disk Out to recording device Network

6 Hardware Description Intel GHz CPU 1 GB RAM Seagate Barracuda ATA IV 60 GB Hard Drive Average Write Service Time = 14.3 msec 10/100 Ethernet Card 24-channel telecom interface card(t1)

7 Workload Characterization The system workload is a stream of digitized call conversations. Further monitoring of a production system, reviewing its logged data and discussions with the system architects, revealed the existence of 2 distinct traffic call classes. The first class, which we will call the PCM class, represents the call traffic digitized using the standard pulse code modulation techniques. The second class, which we will call VOIP class, represents the call traffic in the form of voice over IP calls. These calls may contain voice and data elements.

8 Data Collection Method Our performance data collection process involved the use of a telephone call simulation tool. The simulation tool is capable of streaming digitized audio files thru a T1 connection. Provided with a set of audio files, the tool will randomly pick a group of files to streamline. We had available to us a total of 12 audio files. 6 were PCM class and 6 were VOIP class. The Call Center Telecommunications Interface is equipped with a 24 channel T1 card. The audio simulator was configured to concurrently transmit 24 audio streams. Permon, the Microsoft windows performance tools was used to gather the performance statistics during the load testing. Ran 1-hour load test with only a PCM workload and another 1-hour load test with only a VOIP workload.

9 Queuing Network The system can be represented by an Open Queuing Network. The workload intensity is specified by the arrival rate of calls through the T1 interface. The number of calls simulated is limited only by the number of T1 interface cards installed in the unit.

10 Clustering Disk Transfers vs. CPU Time Disk Transfers/sec PCM Disk Transfers/sec vs. %CPU Time VOIP % CPU Time

11 Clustering Network Bytes Sent vs. CPU Time Network Bytes Sent/sec Network Bytes Sent/sec vs. %CPU Time 500, , , , , , , , ,000 50, PCM VOIP % CPU Time

12 System Completions Number of Lines Average Session Length Recordings Generated Class (minutes) PCM VOIP

13 Basic Statistics for the PCM Service Workload PCM % Processor Network Bytes Time % Disk Time Disk Bytes/sec Sent/sec U(cpu) U(disk) X(disk) X(net) Mean , ,432.4 Standard Deviation , ,385.6 Sample Variance ,666, ,970,852.6 Coeff. Of Variance Minimum , First Quartile (Q1) , ,264.7 Median (Q2) , ,520.1 Third Quartile (Q3) , ,268.4 Maximum , ,915.5 Range , ,915.5 Largest , ,915.5 Smallest , Sum 1, ,225, ,543,779.9 Elapsed Time = 1 hour = 60 min. = 3600 sec Q1

14 PCM Box and Whisker Plots Q1 Q2 Q % Processor Time Q1 Q2 Q3 2, , , , ,863.5 Disk Bytes/Sec Q1 Q2 Q , , , ,915.5 Network Bytes Sent/Sec

15 Basic Statistics for the VOIP Service Workload VOIP % Processor Network Bytes Time % Disk Time Disk Bytes/sec Sent/sec U(cpu) U(disk) X(disk) X(net) Mean , ,856.1 Standard Deviation , ,635.0 Sample Variance ,693, ,501,610.0 Coeff. Of Variance Minimum , First Quartile (Q1) , ,367.6 Median (Q2) , ,343.3 Third Quartile (Q3) , ,796.1 Maximum , ,753.4 Range , ,570.8 Largest , ,753.4 Smallest , Sum 9, ,100, ,925,454.6 Elapsed Time = 1 hour = 60 min. = 3600 sec

16 VOIP Box and Whisker Plots Q1 Q2 Q % Processor Time Q1 Q2 Q3 2, , , , ,762.1 Disk Bytes/Sec Q1 Q2 Q , , ,753.4 Network Bytes Sent/Sec 411,796.1

17 Service Demand Calculations Type Description Symbol PCM VOIP Units Formula Comment Time Total Time T 3,600 3,600 sec Performance Monitor Input Lines lines Test Simulator Completed Sessions System Arrivals/Completions A (Co) sessions System Log Average Time Per Session seconds Average Time Per Session minutes Throughput System Throughput Xo sessions/sec = A / T CPU Utilization Ucpu 6.21% 40.32% % Performance Monitor Busy Time Bcpu sec = Ui * T Average Service Time Scpu 7.92E E-10 sec/transaction Ghz processor Service Completions Ccpu 282,199,242,017 1,833,195,030,583 transactions = Bi / Si Throughput Xcpu 78,388, ,220,842 transactions/sec = Ui / Si Utilization Law Visits to CPU Vcpu 1,041,325, ,875,625,098.0 = Ci / Co Service Demand Dcpu sec = Ui / Xo Service Demand Law Disk Utilization Udisk % % % Performance Monitor Busy Time Bdisk sec = Ui * T Average Service Time Sdisk sec/byte = Ui / Xi StorageReview.com Seagate Data Sheet Service Completions Cdisk bytes = Bi / Si Throughput Xdisk 5, , bytes/sec Performance Monitor Visits to Disk Vdisk = Ci / Co Service Demand Ddisk sec = Ui / Xo Network Interface Bytes Sent/sec 356, , bytes/sec Performance Monitor Total Bits/sec 2,851, ,230, Performance Monitor Ethernet Speed 10,000, ,000, Mbps Utilization Unet 28.51% 32.31% % Performance Monitor Busy Time Bnet 1, , sec = Ui * T Average Service Time Snet sec 10 Mbps Network Interface Service Completions Cnet 10,265,253, ,631,054,555.7 bits = Bi / Si Throughput Xnet 2,851, ,230,848.5 bits/sec = Ui / Si Visits to Network Vnet 37,879, ,279,021.0 = Ci / Co Service Demand Dnet sec = Ui / Xo

18 Data Analysis Thread CPU Usage Notice that the CPU utilization for the PCM class is low compared to that of the VOIP class. The service demands show this trend. D(cpu, PCM) = sec. D(cpu, VOIP) = sec. The reason is the additional CPU requirements of the thread called to decode and extract data from VOIP data streams. Thread Processor Usage % CPU Usag Thread 0 Thread 1 Thread 2 Thread 3 Thread 4 Thread 5 Thread 6 Thread 7 Thread 8 Thread 9 Thread 10 Thread 11 Thread 12 Thread 13 Thread 14 Thread 15 Thread 16 Thread 17 Thread 18 Thread 19 Thread 20 Thread 21 Voice Data Thread #

19 Data Analysis PCM Throughput and Resource Utilization vs. Number of Lines PCM Resource Utilization & System Throughput vs. Number of Lines Throughput Utilization PCM Lines PCM Throughput CPU Utilization Disk Utilization Netw ork Utilization

20 Data Analysis PCM Resource Utilization vs. System Throughput PCM Resouce Utilization vs. System Throughput Resource Utilization (%) Throughput (requests/sec) CPU Utilization Disk Utilization Netw ork Utilization

21 Data Analysis VOIP Throughput and Resource Utilization vs. Number of Lines VOIP Resource Utilization & System Throughput vs. Number of Lines Throughput Utilization VOIP Lines VOIP Throughput CPU Utilization Disk Utilization Netw ork Utilization

22 Data Analysis VOIP Resource Utilization vs. System Throughput VOIP Resouce Utilization vs. System Throughput Resource Utilization (%) Throughput (requests/sec) CPU Utilization Disk Utilization Netw ork Utilization

23 Data Analysis PCM Throughput Bounds Throughput Performance Bound (PCM Inputs Only) Throughput Xo N Heavy Load = 1 / max Di Light Load = N/sum(Di) Actual Throughput Xo NOTE: Scale of X-axis N is not linear.

24 Data Analysis PCM Throughput Bounds (close-up) 0.6 Throughput Performance Bound (PCM Inputs Only) Throughput Xo N Heavy Load = 1 / max Di Light Load = N/sum(Di) Throughput Xo

25 Data Analysis Throughput Bounds Throughput Performance Bound (VOIP Inputs Only) Throughput Xo N Heavy Load = 1 / max Di Light Load = N/sum(Di) Actual Throughput Xo NOTE: Scale of X-axis N is not linear.

26 Data Analysis VOIP Throughput Bounds (close-up) Throughput Xo Throughput Performance Bound (VOIP Inputs Only) Heavy Load = 1 / max Di Light Load = N/sum(Di) Throughput Xo N

27 Data Analysis PCM Response Time Bounds Throughput Xo Response Time Bound (PCM Inputs Only) Light Load = sum(di) Heavy Load = N*maxDi Actual Response Time R NOTE: Scale of X-axis N is not linear. N

28 Data Analysis PCM Response Time Bounds (close-up) Throughput Xo Response Time Bound (PCM Inputs Only) N Light Load = sum(di) Heavy Load = N*maxDi Response Time R

29 Data Analysis VOIP Response Time Bounds 600 Response Time Bound (VOIP Inputs Only) 500 Throughput Xo N Light Load = sum(di) Heavy Load = N*maxDi Actual Response Time R NOTE: Scale of X-axis N is not linear.

30 Data Analysis VOIP Response Time Bounds (close-up) Throughput Xo Response Time Bound (VOIP Inputs Only) Light Load = sum(di) Heavy Load = N*maxDi Response Time R N

31 Data Analysis Throughput and Response Time vs. Number of Concurrent PCM Requests Throughput and Response Time vs. Num ber of Concurrent PCM Requests Throughput (request/sec) Number of Concurrent PCM Requests Response Time (seconds) Throughput Response Time

32 Data Analysis Throughput and Response Time vs. Number of Concurrent VOIP Requests Throughput and Response Tim e vs. Num ber of Concurrent VOIP Requests Throughput (request/sec) Response Time (seconds) Number of Concurrent VOIP Requests Throughput Response Time

33 Conclusions PCM and VOIP classes are similar in their service demands for disk and network usage. However, VOIP has a much higher CPU service demand that PCM. System throughput is limited by the network demands of both classes. Additionally, system throughput is limited by the CPU service demand of the VOIP class.

34 Open Queuing Network Solution PCM Lines Average Session Length System Completions System Throughtput Service Demand Utilizaton Busy Time Service Time Resource Throughput Queue Length Residence Time Co Xo Di Ui Bi Si Xi Ni Ri' CPU E-10 78,388, Disk Network , E-07 2,851, Total Response Time VOIP Lines Average Session Length System Completions System Throughtput Service Demand Utilizaton Busy Time Service Time Resource Throughput Queue Length Residence Time Co Xo Di Ui Bi Si Xi Ni Ri' CPU , E ,220, Disk Network , E-07 3,230, Total Response Time Total Lines Average Session Length System Completions System Throughtput Average Service Demand Utilizaton Busy Time Service Time Resource Throughput Queue Length Average Residence Time Co Xo Di Ui Bi Si Xi Ni Ri' CPU , E ,609, Disk Network , E-07 6,082, Total Response Time

35 Maximum Workload Number of Lines for PCM and VOIP at 100% Resource Utilization Number of Lines for PCM and VOIP at 100% Resource Utilization VOIP Lines PCM Lines Number of Lines CPU Utilization Network Utilization

36 Maximum Workload Number of Lines for PCM and VOIP vs. System Throughput Number of Lines for PCM and VOIP vs. System Throughput at 100% Utilization VOIP Lines PCM Lines Number of Lines System Throughput

37 Maximum Workload Number of Lines for PCM and VOIP vs. Response Time CPU Utilization approaching 100% VOIP Lines Number of Lines for PCM and VOIP vs. Response Time at 100% Utilization , , , , , Network Utilization approaching 100% PCM Lines Number of Lines Response Time Trend Line (Response Time) Average Response Time at maximum workload is 1,194 seconds

38 Recommendations for Improvement Both classes will benefit from an upgrade network connection between the Call Center Telecom Interface and the Call Center Monitor/Recorder. Suggest 100 Mbps ethernet. Both classes will benefit from a faster CPU, but it is most important for the response time and throughput of the VOIP class. Today s Intel Pentium 4 processors run at speeds over 3 GHz. Suggest 3 Ghz.

39 Improvement Analysis Improved Service Demand Calculations Type Description Symbol PCM VOIP Units Formula Comment Time Total Time T 3,600 3,600 sec Performance Monitor Input Lines lines Test Simulator Completed Sessions System Arrivals/Completions A (Co) sessions System Log Average Time Per Session seconds Average Time Per Session minutes Throughput System Throughput Xo sessions/sec = A / T CPU Utilization Ucpu 2.61% 16.97% % = Bi / 3600 Busy Time Bcpu sec = Si * Ci Average Service Time Scpu 3.33E E-10 sec/transaction 3.00 Ghz processor Service Completions Ccpu 282,199,242,017 1,833,195,030,583 transactions Throughput Xcpu 78,388, ,220,842 transactions/sec = Ui / Si Utilization Law Visits to CPU Vcpu 1,041,325, ,875,625,098.0 = Ci / Co Service Demand Dcpu sec = Ui / Xo Service Demand Law Disk Utilization Udisk % % % Performance Monitor Busy Time Bdisk sec = Ui * T Average Service Time Sdisk sec/byte = Ui / Xi StorageReview.com Seagate Data Sheet Service Completions Cdisk bytes = Bi / Si Throughput Xdisk 5, , bytes/sec Performance Monitor Visits to Disk Vdisk = Ci / Co Service Demand Ddisk sec = Ui / Xo Network Interface Bytes Sent/sec 356, , bytes/sec Performance Monitor Total Bits/sec 2,851, ,230, Performance Monitor Ethernet Speed 100,000, ,000, Mbps Utilization Unet 2.85% 3.23% % = Total Bits per sec / Ethernet Speed Busy Time Bnet sec = Ui * T Average Service Time Snet sec 100 Mbps Network Interface Service Completions Cnet 10,265,253, ,631,054,555.7 bits = Bi / Si Throughput Xnet 2,851, ,230,848.5 bits/sec = Ui / Si Visits to Network Vnet 37,879, ,279,021.0 = Ci / Co Service Demand Dnet sec = Ui / Xo

40 Improvement Analysis Improved Open Queuing Network Solution PCM Lines Average Session Length System Completions System Throughtput Service Demand Utilizaton Busy Time Service Time Resource Throughput Queue Length Residence Time Co Xo Di Ui Bi Si Xi Ni Ri' CPU E-10 78,388, Disk Network E-08 2,851, Total Response Time VOIP Lines Average Session Length System Completions System Throughtput Service Demand Utilizaton Busy Time Service Time Resource Throughput Queue Length Residence Time Co Xo Di Ui Bi Si Xi Ni Ri' CPU E ,220, Disk Network E-08 3,230, Total Response Time Total Lines Average Session Length System Completions System Throughtput Average Service Demand Utilizaton Busy Time Service Time Resource Throughput Queue Length Average Residence Time Co Xo Di Ui Bi Si Xi Ni Ri' CPU E ,609, Disk Network E-08 6,082, Total Response Time 1.909

41 Improvement Analysis PCM Throughput and Resource Utilization vs. Number of Lines PCM Resource Utilization & System Throughput vs. Number of Lines Throughput Utilization 0.00 PCM Lines PCM Throughput CPU Utilization Disk Utilization Netw ork Utilization

42 Improvement Analysis PCM Resource Utilization vs. System Throughput PCM Resouce Utilization vs. System Throughput Utilization Throughput (request/sec) CPU Utilization Disk Utilization Netw ork Utilization

43 Improvement Analysis VOIP Throughput and Resource Utilization vs. Number of Lines VOIP Resource Utilization & System Throughput vs. Number of Lines Throughput Utilization VOIP Lines VOIP Throughput CPU Utilization Disk Utilization Netw ork Utilization

44 Improvement Analysis VOIP Resource Utilization vs. System Throughput VOIP Resouce Utilization vs. System Throughput Utilization Throughput (request/sec) CPU Utilization Disk Utilization Netw ork Utilization

45 Improvement Analysis Improved Throughput and Response Time vs. Number of Concurrent PCM Requests Throughput and Response Tim e vs. Num ber of Concurrent PCM Requests Throughput (request/sec) Response Time (seconds) Number of Concurrent PCM Requests Throughput Response Time

46 Improvement Analysis Improved Throughput and Response Time vs. Number of Concurrent VOIP Requests Throughput and Response Time vs. Number of Concurrent VOIP Requests Throughput (request/sec) Number of Concurrent VOIP Requests Response Time (seconds) Throughput Response Time

47 Improvement Analysis Improved Maximum Workload Number of Lines for PCM and VOIP at 100% Resource Utilization VOIP Lines PCM Lines CPU Utilization Network Utilization Number of Lines

48 Improvement Analysis Number of Lines for PCM and VOIP vs. System Throughput Number of Lines for PCM and VOIP vs. System Throughput at 100% Resource Utilization VOIP Lines Throughput Number of Lines 400 PCM Lines System Throughtput

49 Improvement Analysis Number of Lines for PCM and VOIP vs. Response Time Number of Lines for PCM and VOIP vs. Response Time at 100% Resource Utilization CPU Utilization approaching 100% VOIP Lines Number of Lines 400 PCM Lines Average Response Time Average Response Time at maximum workload is 446 seconds 4,000 3,500 3,000 2,500 2,000 1,500 1, Response Time

50 Outstanding Issues The theoretical line capacity depends on adding additional 24 channel T1 cards to the system. The overhead associated with each T1 card will slightly reduce the actual maximum line capacity. A good return on investment may not be possible when configuring the interface units at their maximum capacity. Investment costs need to be returned via software license fees, requiring the sale of a predetermined number of units. Response Time when configured at maximum line capacity is beyond acceptable limits. It is recommended to configure units at 50% of maximum capacity. The average response time when configure at 50% CPU utilization is 1.7 seconds.

51 Outstanding Issues Number of PCM vs. VOIP lines at 50% CPU Utilization Number of Lines for PCM and VOIP at 50% Resource Utilization VOIP Lines PCM Lines Number of Lines