Performance Measurement of Internet Service

Transcription

1 Performance Measurement of Internet Service Sigit Haryadi Institut Teknologi Bandung March 2018 Paper DOI /OSF.IO/8EXMU Citation: Haryadi, S. (2018, March 5). Performance Measurement of Internet Service. Retrieved from osf.io/preprints/inarxiv/8exmu Abstract This paper summarizes my papers and books, which reports on studies concerning the performance measurement of Internet services, particularly on mobile networks. In detail, Internet service quality measurements are grouped into two, the first is a measurement that aims to measure the provider's customer satisfaction, performed by the provider or by the user, and the second measurement goal is that the provider and/or by the vendor in relation to the quality assurance provided by the vendor to the provider. In detail, the first type of measurement uses a sampling technique and uses only one performance indicator, i.e. the data rate, otherwise, the second type performs the aggregate measurements with references to the signaling ladder diagram of the Internet services and measures many performance indicators. 1. Types of Quality of Experience and Quality of Service Measurement of Internet Service Measurement of internet service quality is grouped into two, the first is a measurement that aims to measure the provider's customer satisfaction and the second measurement goal is that the provider and/or vendor is involved with the quality assurance provided by the vendor to the provider. 1

2 2. Quality Measurement of Internet Service for Customer Satisfaction In relation to the customer satisfaction, internet service quality measurements are carried out in a sampling of customers of the providers, thus the measured is QoE (Quality of Experience) rather than QoS (Quality of Service). In this type of measurement, the measured parameter is the data rate. Unlike the QoE measurements of telephone and chat services, where the Quality of Experience is shown in the form of Mean Opinion Score (MOS), otherwise, the measurement of the mean data rate uses units of kbps or Mbps. The application of statistical science should be accompanied by an in-depth understanding of the customer characteristics of the service provider, where the number of respondents interviewed is generally much more than the amount required for data processing. For example, we only need data to be processed as many as 100 customers, then we should have 100 customers who meet all the criteria of the type of customer provider, assume that is that meet the criteria for comparison of female and male customers, e.g. 55% versus 45%, and also meet the customer age criteria, e.g. 60% of youth and 40% of established age, also meet the criteria of income level, e.g. 65% lower and 35% middle, and should meeting the criteria of call intensity, e.g. 75% rarely call and 25% more than three times a day. There are two ways of reporting the measurement results of the data rate, first reporting the mean data rate with the standard deviation, and secondly reporting the minimum data rate reached by 90% of customers. 3. Quality Measurement of the Internet Services in Relation between Provider and Vendor or OTT Provider. Measurements of network performance and quality of Internet services in relation to vendors should be done based on the aggregate data, not a sampling, and should refer to the Signaling Ladder Diagram of the Internet services. The problem faced by the provider is almost the 2

3 vendor's Signaling Ladder Diagram is not 100% the same as the Signaling Ladder Diagram compiled by International Standard ETSI and 3 GPP. Another problem is related to services provided by over-the-top providers that have no interconnection links with broadband providers, so that providers cannot ask for quality accountability to over-the-top providers. Some important parameters on this type of measurement are: 1) Mean Data Rate or Minimum Data Rate 2) Internet Protocol Service non-accessibility 3) Internet Protocol Service Access Failure Ratio 4) Internet Protocol Service Access Time 5) Session Failure Ratio 6) Internet Protocol Setup times 7) Session times 8) Data Transfer Cut-off Ratio 9) Teardown Failure Ratio 10) Teardown Time 3.1. Measurement of Mean or Minimum Data Rate Mean data rate parameter shows the value of the average speed of the data transmission, where average throughput values calculated from the start to build a data connection until the whole data is received successfully. Accompanying the mean data rate report, it is important to report also the standard deviation. The minimum data rate parameter shows 90% of customers with the fastest data rate, which may also take a percentage larger or smaller than 90%. Pre-condition for this parameter is UE has managed to attach, the connection has been active PDP context, as well as the service has been successfully carried out. 3

4 Abstract Equation of Internet Data rate: Data rate( kbit s ) = User Data Transferred (kbit) t data transfer complete t start of data transfer (1) 3.2. Measurement of Internet Service Non Accessibility This parameter indicates the probability of the UE (User Equipment) can not establish a PDP context and can not access the internet access. Pre condition for this parameter is that the UE has managed to make the process of attachment. Abstract Equation of Internet Protocol Service Non-Accessibility: Internet Service Non Accessiblity(%) = Unsuccessful Attempts to Reach the Point when contents is reveived All Attemptes to reach the Point when contents is received x100 (2) 3.3. Measurement of Internet Service Access Failure Ratio This parameter indicates the probability of an UE cannot successfully establish the TCP/IP connection to the server of the service. Pre condition for this parameter is UE has managed to make the process of attachment and connection PDP context is active. Abstract Equation of Internet Service Access Failure: Internet Service Access Failure(%) = Unsuccessful Attempts to established internet connection to the server All Attemptes to established internet connection to the server 3.4. Measurement of Internet Service Access Time x100 (3) This parameter represents the time required for the UE establish TCP/IP connection to the server from a service successfully. Pre-condition for this parameter is UE has managed to attach and connection PDP context is active, so that the parameter can only be calculated on successful trials. Abstract Equation of Internet Service Access Time: Internet Service Access Time(second) = t Reveption of the Furst of packet data t Service Request (4) 3.5. Measurement of Session Failure Ratio This parameter shows the ratio of a successful session until the normal end of the service. 4

5 Pre condition for this parameter is UE has managed to attach and connection PDP context is active. Abstract Equation of Session Failure Ratio: Session Failure ratio(%) = Uncomplete Sessions Successful Started Sessions x100 (5) 3.6. Measurement of Internet Protocol Setup times This parameter represents the time required to access the full service, from establishing PDP context until receiving the first data packet. Pre-condition for this parameter is UE has managed to make the process of attachment. Abstract Equation of Internet Protocol Service Setup Time: Internet Service Setup Time(second) = t the Content had been sent or Received t the Connection had been Initiated (6) 3.7. Measurement of Session times This parameter shows the time required to process the data sessions to successfully complete. Pre condition for this parameter is UE has managed to attach and connection PDP context is active. Abstract Equation of Internet Protocol Service Setup Time: Session Time(second) = t the End of Session t Session Start (7) 3.8. Measurement of Data Transfer Cut-off ratio This parameter shows the ratio between the incomplete data transfer and data transfer begins with success. Pre-condition for this parameter is UE has managed to attachment and connection PDP context is active. Abstract Equation of Data Transfer Cut-off Ratio (%): Data Transfer Ratio(%) = Incomplete Data Transfer Successful Sarted Sessions x100 (8) 5

6 3.9. Measurement of Teardown Failure Ratio (%) Probability that the RTSP message (Real-Time Streaming Protocol message) or the "Teardown" message is sent from the UE to the server and there is no RTSP response "200 OK" is received from the EU server. Abstract Equation of Teardown Failure Ratio (%): Teardown Failure Ratio(%) = Number of Cases Without the 200 OK Response All Teardown Attempt by UE x100 (9) Measurement of Teardown Time Duration of access to services measures from demand streaming or streaming requests on the portal to the reception of the first stream of data packets to the UE Abstract Equation of Teardown Time: Teardown Time(second) = t Server Response to Teardown Message t UE Sending Teardown Message (10) Teardown Times [s] = t server response to teardown message - t UE sending teardown message [2] (25) 4. Signaling Ladder Diagram The following is a list of Signaling Ladder Diagrams that are essential for measuring the quality of Internet services in an aggregate, most of which are referred to Internet Forum standards, and a small portion of Signaling Ladder Diagrams is referring to ETSI standards. START: Reception of the first data packet which contain the content. STOP: Reception of the last data packet which contain the content. Table 1. Signaling Ladder Diagram of HTTP/Web Browsing Data Rate Signal Name Outgoing HTTP OK 200 HTTP WEB HTTP OK 200 GATEWAY GATEWAY Ack GATEWAY HTTP Continue HTTP WEB Ack HTTP WEB HTTP Continue HTTP WEB 6

7 The process of web browsing mean data rate measurement can be started from the reception of the first data packet containing content, marked by sending a "200 OK" signal from web server to Gateway until the reception of the last data packet containing content, marked by sending a HTTP continue signal from web server to. START: ATD command. Table 2. Signaling Ladder Diagram of Internet Service Non-Accessibility in 3G Signal Name Outgoing Activate PDP Context Create PDP Context GGSN Request Create PDP Context GGSN Response Radio Access Bearer 3G RAN Assignment Request Radio Bearer Setup 3G RAN Radio Bearer Setup 3G RAN Complete Radio Access Bearer 3G RAN Assignment Response Update PDP Context GGSN Request Update PDP Context GGSN Response Activate PDP Context Accept Update PDP Context GGSN Request Update PDP Context GGSN Response Radio Access Bearer 3G RAN Modification Radio Access Bearer 3G RAN Modification Radio Access Bearer 3G RAN Modification Radio Access Bearer 3G RAN Modification Modify PDP Context Request Modify PDP Context Accept Invoke Trace 3G RAN SYN GATEWAY SYN, ACK GATEWAY ACK GATEWAY HTTP GET GATEWAY DNS QUERY GATEWAY DNS DNS REPLY DNS GATEWAY 7

8 STOP: Reception of the first data packet which contain the content. SYN GATEWAY HTTP WEB SYN, ACK HTTP WEB GATEWAY ACK GATEWAY HTTP WEB HTTP GET GATEWAY HTTP WEB Ack HTTP WEB GATEWAY Ack GATEWAY HTTP OK 200 HTTP WEB GATEWAY HTTP OK 200 GATEWAY 8 START: the First [SYN] message sent STOP: Reception of the first data packet which contain the content Table 3 Signaling Ladder Diagram of Internet Service Setup Time Signal Outgoing Name SYN GATEWAY SYN, ACK GATEWAY ACK GATEWAY HTTP GET GATEWAY DNS GATEWAY DNS QUERY DNS DNS GATEWAY REPLY SYN GATEWAY HTTP WEB SYN, ACK POP3 GATEWAY ACK GATEWAY HTTP WEB HTTP GET GATEWAY HTTP WEB Ack HTTP WEB GATEWAY Ack GATEWAY HTTP OK 200 HTTP OK 200 HTTP WEB GATEWAY GATEWAY HTTP services setup time in 3G networks is measured beginning with the transmission of Synchronization signal by to Gateway, until the sending a HTTP 200 OK signal by Gateway to. START: the First [SYN] message sent Table 4. Signaling Ladder Diagram of Internet Service Access Success Rate Log Ch Signal Name Outgoing SYN GATEWAY SYN, ACK GATEWAY ACK GATEWAY

9 STOP: Reception of the first data packet which contain the content HTTP GET GATEWAY DNS QUERY GATEWAY DNS DNS REPLY DNS GATEWAY SYN GATEWAY HTTP WEB SYN, ACK POP3 GATEWAY ACK GATEWAY HTTP WEB HTTP GET GATEWAY HTTP WEB Ack HTTP WEB GATEWAY Ack GATEWAY HTTP OK 200 HTTP WEB GATEWAY HTTP OK 200 GATEWAY HTTP services success access rate in 3G networks, beginning with the transmission of Synchronization signal by to Gateway, until the sending a HTTP 200 OK signal by Gateway to. START: ATD command Table 5. Signaling Ladder Diagram of FTP Download Service Signal Name Outgoing Activate PDP Context Create PDP Context Request GGSN Create PDP Context Response GGSN Radio Access Bearer Assignment 3G RAN Request Radio Bearer Setup 3G RAN Radio Bearer Setup Complete 3G RAN Radio Access Bearer Assignment 3G RAN Response Update PDP Context Request GGSN Update PDP Context Response GGSN Activate PDP Context Accept Update PDP Context Request GGSN Update PDP Context Response GGSN Radio Access Bearer Modification 3G RAN Radio Access Bearer Modification 3G RAN Radio Access Bearer Modification 3G RAN Radio Access Bearer Modification 3G RAN Modify PDP Context Request Modify PDP Context Accept Invoke Trace 3G RAN SYN GATEWAY SYN, ACK GATEWAY ACK GATEWAY Connect GATEWAY 9

10 STOP: Reception of the first data packet which contain the content. DNS QUERY GATEWAY DNS DNS Response DNS GATEWAY SYN GATEWAY FTP SYN, ACK FTP GATEWAY ACK GATEWAY FTP Connect GATEWAY FTP Ack FTP FTP220 FTP Ack FTP FTP331 FTP Ack FTP Send username and password FTP Ack FTP FTP 230 FTP Ack FTP FTP150 FTP SYN FTP SYN, ACK FTP Ack FTP FTP200 FTP Ack FTP LIST FTP Ack FTP FTP 226 FTP Ack FTP FTP GET FTP Ack FTP RETR FTP Ack FTP FTP150 FTP SYN FTP SYN, ACK FTP Ack FTP FTP DATA FTP The process of FTP download measurement can be started from the ATD Command, marked by sending an Activate PDP Context signal from to until the reception of the first data packet containing content, marked by sending a FTP Data signal from FTP server to. The process of FTP download measurement can be started from the ATD Command, marked by sending an Activate PDP Context signal from to until the reception of the first 10

11 data packet containing content, marked by sending a FTP Data signal from FTP server to. Table 6. Signaling Ladder Diagram of FTP Download Signal Name Outgoing START: Reception of the first data packet which contain the content. STOP: Reception of the last data packet which contain the content FTP DATA FTP Ack FTP FTP DATA + FIN FTP The process of FTP download- mean data rate measurement can be started from the reception of the first data packet containing content, marked by sending a FTP Data signal from FTP server to until the reception of the last data packet containing content, marked by sending a FTP Data & FIN signal from FTP server to. START: UE or sends the first "Activate PDP context Request" message (Layer 3). Table 7. Signaling Ladder Diagram of Ping on 3G Signal Name Abbreviation Outgoing Activate PDP Context APC Create PDP Context CPCReq GGSN Request Create PDP Context CPCRes GGSN Response Radio Access Bearer RABAReq 3G RAN Assignment Request Radio Bearer Setup RBS 3G RAN Radio Bearer Setup Complete Radio Access Bearer Assignment Response Update PDP Context Request Update PDP Context Response RBSCom 3G RAN RABARes 3G RAN UPCReq GGSN UPCRes GGSN 11

12 STOP: UE or receives the "Activate PDP context Accept" message (Layer 3) Activate PDP Context Accept APCA The process of Ping - PDP CActFaiRat measurement can be started from by sending a Activate PDP Context signal from server to until sending a activate PDP Context Accept signal from to. START: ICMP echo request sent STOP: ICMP echo reply received by the sender Table 8. Signaling Ladder Diagram of Ping Round Trip Time Measurement Signal Name Outgoing Echo (ping) request GATEWAY DNS QUERY GATEWAY DNS DNS Response DNS GATEWAY Echo (ping) request GATEWAY Echo (ping) Reply GATEWAY Echo (ping) Reply GATEWAY The process of Ping Round trip Time measurement can be started from by sending an Echo (ping) request signal from server to gateway until sending an Echo (ping) Reply signal from Gateway to. START: WAP 1.x, WAP 2.x: WSP Disconnect; WAP 2.x: TCP SYN towards streaming platform Table 9. Signaling Ladder Diagram of Streaming Service on 3G Signal Name Outgoing SYN MEDIA SYN, ACK MEDIA ACK MEDIA RTSP : Setup MEDIA Ack MEDIA 200 OK MEDIA 12

13 STOP: Reception of first data packet. Ack MEDIA Activate Secondary PDP Context Create PDP Context Request GGSN Create PDP Context Response GGSN Radio Access Bearer Assignment Request 3G RAN Radio Bearer Setup 3G RAN Radio Bearer Setup Complete 3G RAN Radio Access Bearer Assignment Response 3G RAN Update PDP Context Request GGSN Update PDP Context Response GGSN Activate Secondary PDP Context Accept RTSP : Play MEDIA Ack MEDIA 200 OK Ack IP/UDP/RTP MEDIA MEDIA MEDIA The process of Streaming Service measurement can be started from by sending a synchronization signal from to Media Server until sending an IP/UDP/RTP signal from Media Server to. START: Streaming player signals the start of the stream reproduction STOP: RTSP Teardown method sent by UE and reception of confirmation "RTSP 200 OK" from media server. Table 10. Signaling Ladder Diagram Streaming Reproduction Signal Name Outgoing IP/UDP/RTP MEDIA IP/UDP/RTP MEDIA RTCP MEDIA RTSP : Teardown MEDIA Ack MEDIA Ok 200 MEDIA 13

14 The process of Streaming Reproduction measurement can be started from by sending an IP/UDP/RTP from Media server to until sending an OK 200 signal from Media Server to. START: TCP: First "SYN" sent by the client STOP: SMTP: Reply "235 Authentication successful" received by the client as an answer to the authentication request. Table 11. Signaling Ladder Diagram of Login (Upload) Signal Name Outgoing SYN GATEWAY SYN, ACK GATEWAY ACK GATEWAY Connect GATEWAY DNS QUERY GATEWAY DNS DNS Response DNS GATEWAY SYN GATEWAY SMTP SYN, ACK SMTP GATEWAY ACK GATEWAY SMTP Ack SMTP 220 SMTP Ack SMTP EHLO SMTP Ack SMTP 250 HELO, SMTP 8BITMIME, SIZE, DSN, AUTH Ack SMTP AUTH SMTP Ack SMTP 334 SMTP authentication challenge 235 Authentication Successful SMTP SMTP The process of login (upload) measurement can be started from by sending a synchronization signal from to Gateway until sending a 235 Authentication Successful signal from SMTP Server to. 14

15 15 START: First TCP "SYN" sent by the client STOP: Termination sequence <CRLF.CRLF> received by the client as an answer to the "RETR" command. Table 11. Signaling Ladder Diagram of Login (Upload) Signal Name Outgoing SYN GATEWAY SYN, ACK GATEWAY ACK GATEWAY Connect GATEWAY DNS QUERY GATEWAY DNS DNS Response DNS GATEWAY SYN GATEWAY POP3 SYN, ACK POP3 GATEWAY ACK GATEWAY POP3 Connect GATEWAY POP3 Ack POP3 +OK POP3 Ack POP3 AUTH POP3 Ack POP3 +OK POP3 Ack POP3 AUTH POP3 Ack POP3 Authentication challenge POP3 +OK message, 1500 octets POP3 Ack POP3 STAT POP3 Ack POP3 +OK POP3 Ack POP3 LIST POP3 Ack POP3 +OK Scan list follows <CRLF>.<CRLF> POP3 Ack POP3 UIDL POP3 Ack POP3 +OK Scan list follows 1 12 <CRLF>.<CRLF> POP3 Ack POP3 RETR 1 POP3 Ack POP3 +OK 1500 octets POP3 <CRLF>.<CRLF> POP3

16 The process of download session measurement can be started from by sending a synchronization signal from to Gateway until sending a <CRLF>.<CRLF> signal from POP3 Server to. Table 12. Signaling Ladder Diagram of (Upload) Data Rate Measurement START: SMTP: "MAIL FROM" sent by the client STOP: SMTP: Reply "250 messages accepted" received by the client. Signal Name Mail from : (name@domain.co m Outgoing SMTP Ack SMTP 250 OK SMTP Ack SMTP RCPT TO SMTP Ack SMTP 250 OK SMTP Ack SMTP DATA SMTP Ack SMTP 354 Start mail input; end with <CRLF>.<CRLF> SMTP Ack SMTP Message SMTP Ack SMTP 250 ok, message accepted SMTP The process of (upload) mean data rate measurement can be started from by sending a Mail from: (name@domain.com) signal from to SMTP Server until sending the 250 OK, message accepted signal from SMTP Server to. Table 13. Signaling Ladder Diagram of (Download) Data Rate measurement Start: POP3: "RETR" command sent by the client STOP: POP3: Termination sequence Signal Name Outgoing network RETR 1 POP3 Ack POP3 +OK 1500 octets POP3 <CRLF>.<CRLF> POP3 16

17 <CRLF.CRLF> received by the client. The process of (Download) mean data rate measurement can be started from by sending a RETR 1 signal from to POP3 Server until sending a <CRLF>.<CRLF> signal from POP3 Server to. Note To the readers of this short paper, if you want to deepen the knowledge of this field, welcome to read the papers and books in the references. References [1] European Telecommunications Standards Institute. (2011). Speech and multimedia Transmission Quality (STQ), QoS aspects for popular services in mobile networks; Part 1. Assessment of Quality of Service, ETSI TS V2.2.1 [2] European Telecommunications Standards Institute. (2011). ETSI Standard TS v2.2.1: Speech and multimedia Transmission Quality (STQ); QoS aspects for popular services in mobile networks; Part 2: Definition of Quality of Service parameters and their computation. [3] ETSI. (2008). ETSI Standard EG V1.3.1: Speech Processing, Transmission and Quality Aspects (STQ). [4] ITU-T. (2014). ITU-T Recommendation G.107: The E-Model, a computational model for use in transmission planning. [5] Sigit Haryadi. (2013). Telecommunication Traffic: Technical and Business Consideration. Lantip Safari Media, Bandung, Indonesia. ISBN: [6] Sigit Haryadi. (2013). Telecommunication Service and Experience Quality. Lantip Safari Media, Bandung, Indonesia. ISBN: [7] Haryadi, Sigit; Limampauw, Ivantius. (2012). QoS Measurement of Telephony Services In 3G s Using Aggregation Method. Conference Proceeding of TSSA Denpasar, Indonesia. [8] Haryadi, Sigit; Nusantara, Sandy. (2012). QoS Measurement of Web Browsing Services In 3G s Using Aggregation Method. Conference Proceeding of TSSA Denpasar, Indonesia. [9] Haryadi, Sigit; Pramudita, Arnold. (2012). QoS Measurement of Video Streaming Services in a 3G s Using Aggregation Method. Conference Proceeding of TSSA Denpasar, Indonesia. [10] Haryadi, Sigit; Andina, Raisha. (2012). QoS Measurement of File Transfer Protocol Services In 3G s Using Aggregation Method. Denpasar, Indonesia [11] Haryadi, S. (2017, November). Probabilitas Statistik Untuk Insinyur. Researchgate. DOI /RG [12] Sigit Haryadi. (2017). the Non-Intercept Linear Regression Method. Researchgate. DOI /RG [13] [37] Sigit Haryadi. (2018, February). Applied Statistics for Assessment of the Regulation and Policy: case study in Telecommunication Industry. DOI /RG [14] Haryadi, S. (2018, February 7). Statistik Terapan: Pengujian Regulasi & Kebijakan Telekomunikasi. Retrieved from osf.io/3pkfw 17

18 18 [15] Haryadi, S. (2018, January 25). Chapter 1. The Concept of Telecommunication Performance and Quality of Service. Retrieved from osf.io/mukqb [16] Haryadi, S. (2018, January 26). Chapter 2 of Performance and Quality of Service: Determination of Key Performance Indicator (KPI). Retrieved from osf.io/preprints/inarxiv/6gtnd [17] Haryadi, S. (2018, January 26). Chapter 3 of Performance and Quality of Service: Technical Measurement of a Mobile Performance and Quality of Service. Retrieved from osf.io/q4wsz [18] Haryadi, S. (2018, January 30). Basic Calculation of the s Availability and Reliability (Chapter 4 of Performance and Quality of Service). Retrieved from osf.io/preprints/inarxiv/z5mwq [19] Haryadi, S. (2018, February 26). Measurement of Cellular s Availability and Reliability based on the Signaling Ladder Diagram. Retrieved from osf.io/preprints/inarxiv/hwvm8 [20] Haryadi, S. (2018, February 27). Performance Measurement of Telephone and Service. Retrieved from osf.io/preprints/inarxiv/5c7qa [21] Haryadi, S. (2018, March 4). Performance Measurement of S and/or Chat Service. Retrieved from osf.io/preprints/inarxiv/h76uc [22] Sigit Haryadi. (2017). Haryadi Index and Its Applications in Science of Law, Sociology, Economics, Statistics, and Telecommunications. Penerbit Elex Media Komputindo. Jakarta. ISBN [23] Sigit Haryadi. (2017). Indeks Haryadi dan Penerapan di Ilmu Hukum, Sosiologi, Ekonomi, Statistik, dan Telekomunikasi. Penerbit Elex Media Komputindo, Jakarta. ISBN [24] Sigit Haryadi. (2016). Haryadi Index for Competition, Equality and Correlation Evaluation. Penerbit Lantip Safari Media, Bandung, Indonesia. ISBN [25] Sigit Haryadi. (2016). Haryadi Index untuk Evaluasi Kompetisi, Kesetaraan dan Korelasi. Penerbit Lantip Safari Media, Bandung, Indonesia. ISBN [26] Sigit Haryadi. (2017). Calculation of the Mobile Communication Competition using Haryadi Index. DOI /RG [27] [97] Sigit Haryadi. (2017). the Equality Correlation Method. Researchgate. DOI /RG [28] Muhammad Rizky Hikmatullah; Sigit Haryadi. (2017). Perceptual evaluation of speech quality over the top call service. 3rd International Conference on Wireless and Telematics (ICWT), Indonesia. [29] Rizna Nofitasari; Sigit Haryadi. (2017). Self-healing mechanism with RSRP measurement in LTE network. 3rd International Conference on Wireless and Telematics (ICWT), Indonesia. [30] Fidyatun Nisa; Sigit Haryadi. (2016). Simulation of the fault management with Self Healing mechanism (case study: LTE in Banda Aceh Area). 10th International Conference on Telecommunication Systems Services and Applications (TSSA), Indonesia. [31] Suci Ramadona; Sigit Haryadi; Dyah Rakhma Aryanti. (2015). Over the top call service key performance indicator. 1st International Conference on Wireless and Telematics (ICWT), Indonesia.

19 19 [32] Maya Rahayu; Sigit Haryadi; Dyah Rakhma Aryanti. (2015). Over the Top chat service key performance indicator. 1st International Conference on Wireless and Telematics (ICWT), Indonesia. [33] Sigit Haryadi. (2016). Telecommunication Quality of Service Concept. Research gate. Available online at _Service_Concept [34] Sigit Haryadi. (2016). Grade of Service & Quality of Service Concept and Analysis. Researchgate. Available online at Service_Concept_and_Analysis [35] Sigit Haryadi. (2016). Performance Indicator of Telecommunication Service. Researchgate. Available online at communication_service.