KPI Definition and Computation

Transcription

1 KPI Definition and Computation

2 Table of Contents 1 KPI OVERVIEW KPI DEFINITION KPI DIAGRAM CS Service Voice / Video Telephony PS Service FTP / HTTP / WAP MMS Streaming KPI DEFINITION Telephony Telephony Service Non-Accessibility [%] Call Setup Time [s] Telephony Speech Quality on Sample Basis [MOS] Telephony Cut-off Call Ratio [%] VT Service Non-Accessibility [%] VT Setup Failure Ratio [%] VT Service Access Time VT Setup Time [s] VT Cut-off Call Ratio [%] Link Layer mean BLER [%] PEVQ [VMOS] Ping Success Ratio [100%] Ping Round Trip Time [ms] Attach Failure Ratio [%] Attach Setup Time [ms] PDP Context Activation Failure Ratio [%] PDP Context Activation Time [ms] FTP Download Signalling Flow FTP Download KPIs --- Method A FTP Download KPIs --- Method B FTP Download Signalling Flow... 21

3 4.4.9 FTP Upload KPIs --- Method A FTP Upload KPIs --- Method B Attach Failure Ratio [%] Attach Setup Time [ms] PDP Activation Failure Ratio [%] PDP Activation Time [ms] HTTP Signalling Flow HTTP KPIs---Methods A HTTP KPIs---Methods B Attach Failure Ratio [%] Attach Setup Time [ms] PDP Activation Failure Ratio [%] PDP Activation Time [ms] Download Signalling Flow Download KPIs---Method A Download KPIs---Method B Upload Signalling Flow Upload KPIs---Method A Upload KPIs---Method B Attach Failure Ratio [%] Attach Setup Time [ms] PDP Activation Failure Ratio [%] PDP Activation Time [ms] MMS Signalling Flow MMS KPIs Attach Failure Ratio [%] Attach Setup Time [ms] PDP Activation Failure Ratio [%] PDP Activation Time [ms] WAP Signalling Flow WAP KPIs---Method A WAP KPIs ---Method B... 37

4 1 KPI Overview With the introduction of new networks services such as MMS, , HTTP, FTP, Video Streaming, the requirement for RF network testing is increasingly critical. Compared to traditional troubleshooting system, end-to-end KPI testing system is an initiative solution which allows operator inspecting network from the perspective of end users, and finishing troubleshooting before receiving user reports. This document describes the KPI definition and algorithm adopted by Dingli's Pilot Pioneer, the drive test solution, for services including voice services, video telephony, ping, FTP, HTTP, , MMS, WAP. 2 KPI Definition According to ITU specification, three aspects should be measured in order to evaluate network performance from the perspective of the end-user perception. 1. Accessibility Accessibility reflects user tolerance to services under certain conditions. For instance, in order to access data service, user must perform a series of operations, such as attaching network, activating PDP, accessing IP service while PDP is activated, activating WAP session (for WAP and MMS), etc. 2. Retentivity Retentivity is the ability of maintaining the service being accessed within a certain period. The reliability of a service or session means that in a mature network, user does not need to reproduce an operation, such as to re-activate PDP in data network. 3. Integrity Integrity indicates that service is kept in good quality without interference after access to service. Integrity of KPI reveals tremendous changes of network condition after a service is accessed, which might fall from "very good" to "unacceptable". Based on principles above, DingLi product measures end-to-end user perception based on network key indicators (KPI) as follows: Radio network unavailability Attach failure rate Attach setup time

5 Network non-accessibility Setup time Mean data rate Data transfer cut-off ratio IP service failure rate IP service access time

6 3 KPI Diagram 3.1 CS Service Voice / Video Telephony

7 3.2 PS Service FTP / HTTP /

8 3.2.2 WAP

9 3.2.3 MMS

10

11 3.2.4 Streaming 4 KPI Definition Currently two main views about the best way to reflect the user's experience for these services are in place. One preferring the payload throughput philosophy and the other preferring the transaction throughput philosophy: - Method A defines trigger points which are as independent as possible from the service used, therefore representing a more generic view (payload throughput).

12 - Method B defines trigger points on application layer, therefore representing a more service oriented view (transaction throughput). Kindly notice the mutual influence between KPI failure ratio and KPI time. A parameter is not calculated unless the service setup attempt is successful. For instance, KPI non-access ratio and setup time are mutually affected. If the duration for statistics is set to be short, the mean setup time would be low and non-access ratio would be high accordingly. And vice versa. 4.1 Telephony For voice service, KPI is measured for MOC phone, while no KPI for MTC phone Telephony Service Non-Accessibility [%] The probability that the end-user cannot access the mobile telephony service when requested if it is offered by display of the network indicator on the UE. unsuccessful call attempt Telephony Service Non - Accessibility [%] = 100 all call attempts Start: The RRC Connection Request/Channel Request message is sent. It is possible that more than one such message per call attempt is sent. Only the first one should be taken into account for the calculation. Stop: The Connect message is received from MSC. Stop trigger point is not reached for an unsuccessful call Call Setup Time [s] Call setup time describes the time period between sending call attempt and connection established. Call Setup Time[s] = Tconnection established Tcall attempt Start: RRC Connection Request / Channel Request message is sent. It is possible that more than one such message per call attempt is sent. Only the first one should be taken into account for the calculation.

13 Stop: The Connect message is received from MSC. Stop trigger point is not reached for an unsuccessful call Telephony Speech Quality on Sample Basis [MOS] An indicator representing the quantification of the end-to-end speech transmission quality of the mobile telephony service. The score is measured based on PESQ algorithm. Telephony Speech Quality on Sample Basis [MOS] = sum of the PESQ score of all the sampling points total number of PESQ sampling points Start: The Connect message is received Stop: The Disconnect message is sent Telephony Cut-off Call Ratio [%] The probability that a successful call attempt is ended by a cause other than the intentional termination by A- or B-party. Unintentionally terminated telephony calls Telephony Cut-off Call Ratio[%] = 100 All successful telephony call attempts Start: the Connect message is received. Stop: the Disconnect message is sent. 4.2 Video Telephony VT Service Non-Accessibility [%] Probability that the end-user cannot access the service when requested while it is offered by network indication on the mobile equipment. unsuccessful video telephony call access attempts VT Service Non-Accessibility [%] = 100 All video telephony call access attempts

14 Start: RRC Connection Request / Channel Request message is sent. It is possible that more than one such message per call attempt is sent. Only the first one should be taken into account for the calculation. Stop: The Connect message is received from MSC. Stop trigger point is not reached for an unsuccessful call attempt VT Setup Failure Ratio [%] Probability of receiving no video packets after VT call attempt. VT setup failures VT Setup Failure Ratio [%] = 100 All VT call attempts Start: RRC Connection Request / Channel Request message is sent. It is possible that more than one such message per call attempt is sent. Only the first one should be taken into account for the calculation. Stop: The Connect message is received from MSC. Stop trigger point is not reached for an unsuccessful call attempt VT Service Access Time Time between pushing send button after input of MSISDN and receipt of alerting at MO side. VT Service Access Time [s] = Talerting tone- Tpush send button Start: RRC Connection Request / Channel Request message is sent. It is possible that more than one such message per call attempt is sent. Only the first one should be taken into account for the calculation. Stop: The Connect message is received from MSC. Stop trigger point is not reached for an unsuccessful call attempt.

15 4.2.4 VT Setup Time [s] The elapsed time from the MT call acceptance indicated at MO side until audio and video output starts at both sides. VT Setup Time [s] =T audio/video start - T MO sees call acceptance Start: RRC Connection Request / Channel Request message is sent. Stop: Start of reception of audio and video data at both sides VT Cut-off Call Ratio [%] Probability that a successful service access is ended by a cause other than the intentional termination of the user (calling or called party). Video telephony dropped calls VT Cut - off Call Ratio [%] = 100 all successful video telephony service access attempts Start: the Connect message is sent. Stop: the Connect message is received Link Layer mean BLER [%] The mean value of BLER on RLC layer total BLER value (RLC layer) BLER on RLC layer [%] = 100 number of sampling points (RLC layer) PEVQ [VMOS] An objective method for end-to-end speech quality assessment of video telephony. By simulating human visual system, it can detect abnormalities of VT signal and quantify KPI. 4.3 Ping Service

16 4.3.1 Ping Success Ratio [100%] Probability of receiving ICMP echo reply from a destination after ICMP echo request is sent. ICMP echo reply is received by sender from destination Ping Success Rate [%] = 100 ICMP echo request from source Start: ICMP echo request sent. Stop: ICMP echo reply received by the sender Ping Round Trip Time [ms] The round trip time is the time required for a packet to travel from a source to a destination and back. It is used to measure the delay on a network at a given time. For this measurement the service must already be established. Ping Round Trip Time[ms] =T packet received - T packet sent Start: ICMP echo request is sent. Stop: ICMP echo reply is received by the sender 4.4 FTP service Attach Failure Ratio [%] The attach failure ratio describes the probability that a subscriber cannot attach to the PS network unsuccessful attach attempts Attach Failure Ratio [%] = 100 all attach attempts Start: The "ATTACH REQUEST" message is sent by the UE. Stop: The "ATTACH ACCEPT"/ Reject message is received by the UE

17 4.4.2 Attach Setup Time [ms] The attach setup time describes the time period needed to attach to the PS network. Attach Setup Time [ms] = Tattach complete - Tattach request Start: The "ATTACH REQUEST" message is sent by the UE. Stop: The "ATTACH ACCEPT/Reject" message is received by the UE PDP Context Activation Failure Ratio [%] PDP context cannot be activated probability. It is the attempt to the ratio of the total number of unsuccessful attempts PDP context with the PDP context. unsuccessful PDP context activation attempts PDP Context Activation Failure Ratio[%] = 100 all PDP context activation attempts Start: The "PDP Context Activation Request" message is sent by the UE. Stop: The "PDP Context Activation Accept" message is received by the UE PDP Context Activation Time [ms] The PDP context activation time describes the time period needed for activating the PDP context. PDP Context Activation Time [s] = T PDP context activation accept T PDP context activation request Start: The "PDP Context Activation Request" message is sent by UE. Stop: the "PDP Context Activation Accept" message is received by UE.

18 4.4.5 FTP Download Signalling Flow Trigger Point Corresponding Event 1 Attach Request message sent by UE 2 Attach Accept message received by UE 3 PDP Context Activation Request message sent by UE 4 PDP Context Activation Accept message received by UE 9 Connect server 10 Connection setup

19 11 RETR command sent 12 First packet received 13 Last packet received FTP Download KPIs --- Method A Download Mean Data Rate [kbit/s] This parameter describes the average data transfer rate measured throughout the entire connect time to the service. User data transferred[kb] 8 Download Mean Data Rate [kbit/s] = t data transfer complete t data transfer start Start: Reception of the first data packet containing content. Stop: Reception of the last data packet containing content FTP Download Cut-off Ratio [%] The proportion of incomplete data transfers and data transfers that were started successfully. incomplete data transfers FTP Download Cut-off Ratio [%] = 100 successfully started data transfers Start: Reception of the first data packet containing content. Stop: Reception of the last data packet containing content FTP Download KPIs --- Method B Download Mean Data Rate [kbit/s] This parameter describes the average data transfer rate measured throughout the entire connect time to the service. user data transferred[kb] 8 Download Mean Data Rate [kbit/s] = T last packet received T RETR command sent

20 Start: RETR command is sent. Stop: the receipt of the last packet FTP Download Cut-off Ratio [%] The proportion of incomplete data transfers and data transfers that were started successfully. incomplete data transfers FTP Download Cut-off Ratio [%] = 100 successfully started data transfers Start: RETR command is sent Stop: the receipt of the last packet.

21 4.4.8 FTP Download Signalling Flow Trigger Point Corresponding Event 1 Attach Request message sent by UE 2 Attach Accept message received by UE 3 PDP Context Activation Request message sent by UE 4 PDP Context Activation Accept message received by UE 9 Connect server 10 Connection setup

22 11 STOR command sent 12 First packet received 13 Last packet received. Confirmed by server FTP Upload KPIs --- Method A Upload Mean Data Rate [kbit/s] This parameter describes the average data transfer rate measured throughout the entire connect time to the service. Upload Mean Data Rate [kbit/s] = User data transferred [KB] 8 T last packet received T first packet sent Start: the sending of the first packet. Stop: the receipt of the last packet FTP Upload Cut-off Ratio [%] The proportion of incomplete data transfers and data transfers that were started successfully. incomplete data transfers FTP Upload Cut-off Ratio [%] = 100 successfully started data transfers Start: the sending of the first packet; Stop: the receipt of the last packet FTP Upload KPIs --- Method B Upload Mean Data Rate [kbit/s] This parameter describes the average data transfer rate measured throughout the entire connect time to the service. Upload Mean Data Rate [kbit/s] = User data transferred[kb] 8 T last packet received T STOR command sent

23 Start: STOR command is sent. Stop: the receipt of the last packet FTP Upload Cut-off Ratio [%] The proportion of incomplete data transfers and data transfers that were started successfully. incomplete data transfers FTP Upload Cut-off Ratio [%] = 100 successfully started data transfers Start: STOR command is sent Stop: the receipt of the last packet. 4.5 HTTP Attach Failure Ratio [%] Kindly refer to relevant sections of FTP Attach Setup Time [ms] Kindly refer to relevant sections of FTP PDP Activation Failure Ratio [%] Kindly refer to relevant sections of FTP PDP Activation Time [ms] Kindly refer to relevant sections of FTP.

24 4.5.5 HTTP Signalling Flow Trigger Point Corresponding Event 1 Attach Request message sent by UE 2 Attach Accept message received by UE 3 PDP Context Activation Request message sent by UE 4 PDP Context Activation Accept message received by UE 9 Connect to HTTP 10 Start Transfer command issued 11 First packet received

25 13 Last packet received HTTP KPIs---Methods A HTTP Mean Data Rate [kbit/s] service. The average data transfer rate measured throughout the entire connect time to the HTTP file size [KB] 8 HTTP Mean Data Rate [kbit/s] = T last packet received T first packet sent Start: the receipt of the first packet. Stop: the receipt of the last packet HTTP Data Transfer Cut-off Ratio [%] The proportion of incomplete data transfers and data transfers that were started successfully. incomplete data transfers HTTP Data Transfer Cut - off Ratio [%] = 100 successfully started data transfers Start: the receipt of the first packet. Stop: the receipt of the last packet HTTP KPIs---Methods B HTTP Mean Data Rate [kbit/s] service. The average data transfer rate measured throughout the entire connect time to the HTTP file size [KB] 8 HTTP Mean Data Rate [kbit/s] = T last packet received T transfer command issued Start: The transfer command is issued Stop: the receipt of the last packet.

26 HTTP Data Transfer Cut-off Ratio [%] The proportion of incomplete data transfers and data transfers that were started successfully. incomplete data transfers HTTP Data Transfer Cut-off Ratio [%] = 100 successfully started data transfers Start: The transfer command is issued Stop: the receipt of the last packet Attach Failure Ratio [%] Kindly refer to relevant sections of FTP Attach Setup Time [ms] Kindly refer to relevant sections of FTP PDP Activation Failure Ratio [%] Kindly refer to relevant sections of FTP PDP Activation Time [ms] Kindly refer to relevant sections of FTP.

27 Download Signalling Flow Trigger Point Corresponding Event 1 Attach Request message sent by UE 2 Attach Accept message received by UE 3 PDP Context Activation Request message sent by UE 4 PDP Context Activation Accept message received by UE 9 Connect server 12 RETR command sent; Download start 13 First packet received 14 Last packet received

28 Download KPIs---Method A Mean Transfer Rate [kbit/s] The average data transfer rate measured throughout the entire connect time to the service. Mean Transfer Rate [kbit/s] = file size [KB] 8 T last packet received T first packet received Start: The first data packet is received. Stop: The last data packet is received Data Transfer Cut-off Ratio [%] The proportion of unsuccessful data transfers and data transfers that were started successfully. unsuccessful data transfers Download Cut-off Ratio [%] = 100 all successful started data transfers Start: RETR command is sent. download starts Stop: Last packet is received Download KPIs---Method B Mean Transfer Rate [kbit/s] The average data transfer rate measured throughout the entire connect time to the service. Mean Data Rate[kbit/s] = file size [KB] 8 Tlast packet received TRETR command issued Start: RETR command is sent. download starts Stop: Last packet is received Download Cutoff Ratio [%]

29 The proportion of unsuccessful data transfers and data transfers that were started successfully. unsuccessful data transfers Download Cut-off Ratio [%] = 100 all successful started data transfers Start: RETR command is sent. download starts Stop: Last packet is received Upload Signalling Flow Trigger Point Corresponding Event 1 Attach Request message sent by UE 2 Attach Accept message received by UE

30 3 PDP Context Activation Request message sent by UE 4 PDP Context Activation Accept message received by UE 9 Connect to server 12 Mail command issued 13 First packet sent 14 Last packet sent. Acknowledged by server Upload KPIs---Method A Mean Transfer Rate [kbit/s] The average data transfer rate measured throughout the entire connect time to the service. Mean Transfer Rate [kbit/s] = file size [KB] 8 T last packet sent T first packet sent Start: The first data packet is sent. Stop: The last data packet is sent Data Transfer Cut-off Ratio [%] The proportion of unsuccessful data transfers and data transfers that were started successfully. unsuccessful data transfers Upload Cut-off Ratio [%] = 100 all successful started data transfers Start: The first data packet is sent. Stop: Last packet is sent. Acknowledged by server Upload KPIs---Method B Mean Transfer Rate [kbit/s] The average data transfer rate measured throughout the entire connect time to the service.

31 file size [KB] 8 Mean Data Rate[kbit/s] = T last packet sent T RETR command isued Start: Mail command is issued. Stop: Last packet is sent. Acknowledged by server Upload Cut-off Ratio [%] The proportion of unsuccessful data transfers and data transfers that were started successfully. unsuccessful data transfers Upload Cut-off Ratio [%] = 100 all successful started data transfers Start: Mail command is issued. Stop: Last packet is received. Acknowledged by server. 4.7 MMS Attach Failure Ratio [%] Kindly refer to relevant sections of FTP Attach Setup Time [ms] Kindly refer to relevant sections of FTP PDP Activation Failure Ratio [%] Kindly refer to relevant sections of FTP PDP Activation Time [ms] Kindly refer to relevant sections of FTP.

32 4.7.5 MMS Signaling Flow Trigger Point Corresponding Event 1 MMS Send attempt 2 Access WAP gateway attempt 3 Access WAP gateway success 4 MMS sent

33 5 MMS Notification received 6 MMS Receive attempt 7 Access WAP gateway attempt 8 Access WAP gateway success 9 MMS received MMS KPIs MMS Send Failure Ratio (MO) [%] The probability that a MMS-message cannot be send by the subscriber, although he has requested to do so by pushing the "send button" unsuccessful MMS send attempts (failed to access MMS_m-send.conf) MMS Send Failure Ratio [%] = 100 all MMS send attempts Start: Attach Request message is sent. Stop: MMS is sent (MMS m-send confirm) MMS Retrieval Failure Ratio (MT) [%] The probability that the MMS-message cannot be downloaded by the MT mobile, which received a MMS notification before. unsuccessful MMS delivery attempts MMS Delivery Failure Ratio [%] = 100 all MMS delivery attempts Start: WAP Gateways is accessed (WSP/HTTP get request) Stop: MMS is received (MMS_m-Notify Resp.ind) MMS Send Time (MO) [s] The time elapsing from pushing the send button after the editing of a MMS-message to the completion of the data transfer. MMS Send Time[s] = TMMS Sent complete TPDP activation request initiated

34 Start: PDP Context Request message is sent. Stop: MMS is sent (MMS m-send confirm) MMS Retrieval Time (MT) [s] The time elapsing between the WAP Get Request and the completion of the download of the MMS. MMS Delivery Time [s] =TMMS from MMSC complete Tini WAP get request Start: WAP Gateway is accessed (WSP / HTTP get request) Stop: MMS is received completely (MMS_m-Notify Resp.ind) MMS Notification Failure Ratio [%] The probability that the Multimedia Messaging Service (MMS) is not able to deliver the Notification of a MMS-message to the b-parties mobile. failed MMS - notifications MMS Notification Failure Ratio [%] = 100 successfully submitted MMS Start: MMS is sent completely (MMS m-send confirm) Stop: MMS notification message is received (MMS_M_Notification_Ind) MMS Notification Time [s] The time elapsing from the complete submission of the Multimedia-Message to the MMSC to the reception of the Notification (MT). MMS Notification Time [s] = TrecNotif - TMMSsubmit Start: MMS is sent completely. (MMS m-send confirm) Stop: MMS notification message is received. (MMS_M_Notification_Ind) MMS End-to-End Failure Ratio [%] The probability that the Multimedia Messaging Service (MMS) is not able to deliver a MMS-message after the "send button" has been pushed or the MO party has not received an acknowledgement of the successful transmission from the MMSC. unsuccessfully delivered MMS -messages MMS End - to - End Failure Ratio [%] = 100 all MMS send attempts

35 Start: PDP Context Request message is sent by UE. Stop: the MMS is received completely (MMS_M_NotifyResp_Ind) MMS End-to-End Delivery Time [s] The time elapsing from pushing of the "send button" to the reception of the MMS by the b-parties mobile. MMS End-to- End Delivery Time [s] = TMMS rec Tsend attempt Start: PDP Context Request message is sent by UE. Stop: MMS is received completely (MMS_M_NotifyResp_Ind) 4.8 WAP Attach Failure Ratio [%] Kindly refer to relevant sections of FTP Attach Setup Time [ms] Kindly refer to relevant sections of FTP PDP Activation Failure Ratio [%] Kindly refer to relevant sections of FTP PDP Activation Time [ms] Kindly refer to relevant sections of FTP.

36 4.8.5 WAP Signalling Flow Trigger Point Corresponding Event 1 Attach Request message sent by UE 2 Attach Accept message received by UE 3 PDP Context Activation Request message sent by UE 4 PDP Context Activation Accept message received by UE 9 Access gateway attempt 10 Access gateway succeed/failed

37 11 Transfer command issued 12 First packet received 14 Last packet received WAP KPIs---Method A Mean Transfer Rate [kbit/s] The average data transfer rate measured throughout the entire connect time to the WAP service. Mean Transfer Rate [kbit/s] = WAP file size [KB] 8 Start: The first data packet is received. Stop: The last data packet is received. T last packet received T first packet received WAP Data Transfer Cut-off Ratio [%] The proportion of unsuccessful data transfers and data transfers that were started successfully. unsuccessful data transfers WAP Cut-off Ratio [%] = 100 all successful started data transfers Start: The first data packet is received. Stop: Last packet is received. Acknowledged by server WAP KPIs ---Method B Mean Transfer Rate [kbit/s] The average data transfer rate measured throughout the entire connect time to the WAP service. WAP file size [KB] 8 Mean Data Rate[kbit/s] = T last packet received T transfer command issued

38 Start: Transfer command is issued. (HTTP Get or WSP Get) Stop: Last packet is received WAP Data Transfer Cut-off Ratio [%] The proportion of unsuccessful data transfers and data transfers that were started successfully. unsuccessful data transfers WAP Cut-off Ratio [%] = 100 all successful started data transfers Start: Transfer command is issued. (HTTP Get or WSP Get) Stop: Last packet is received.