I. INTRODUCTION. Keywords -- Web Server, Perceived User Latency, HTTP, Local Measuring. interchangeably.

Transcription

1 Evaluaing Web User Perceived Laency Using Server Side Measuremens Marik Marshak 1 and Hanoch Levy School of Compuer Science Tel Aviv Universiy, Tel-Aviv, Israel mmarshak@emc.com, hanoch@pos.au.ac.il 1 Absrac -- The cenral performance problem in he World Wide Web, in recen years, is user perceived laency. This is he ime spen by a user while waiing for a Web page he/she requesed. Impaience wih poor performance is he mos common reason visiors erminae heir visi a Web sies. For e-commerce sies, such abandonmen ranslaes ino los revenue. For his reason, measuring he delay experienced by is cusomers is of high imporance o a Web sie. These measuremens are criical for analyzing he sie behavior and o size is componens. As of oday he main ool for conducing such measuremens are exernal, clien-side ools, whereby agens locaed on he ne reques pages from he sie and measure is laency. In his paper we propose a novel soluion ha conducs he measuremens of he user perceived delay a he Web sie. The major advanage of his measuremen approach, as opposed o clien side approach, is ha is can evaluae he laency experienced by each and every clien (regardless of is nework locaion). Furher, his esimae can be conduced a real ime, hus allowing he server o conrol is operaion and prioriize he requess based on he acual performance observed by he cliens. The soluion does no require any agens o be placed a he ne. Furher, i does no sniff low-level proocols (ha is, IP proocols) and is all based on implemenaion a he HTTP level. As such, i is very efficien and economical. The soluion is based on a novel echnique in which a special iny HTTP objec, called he senry, assiss in measuring he user perceived laency. The algorihm is implemened on he Apache server. The implemenaion was esed hroughou an exensive array of ess and found o provide very accurae measures. Keywords -- Web Server, Perceived User Laency, HTTP, Local Measuring. I. INTRODUCTION The rapid growh of he World Wide Web in recen years has caused a significan shif in he composiion of Inerne raffic. Today, Web raffic forms he dominan componen of he Inerne backbone raffic. Therefore, here is significan value in undersanding he Web performance, and especially ha experienced by he users. The cenral performance problem in he World Wide Web, in recen years is user perceived laency. This is he ime elapses from he momen he user requess a Web page ill he ime he/she receives he requesed page. Impaience wih poor performance is he mos common reason users erminae heir visi a web sies. For commercial sies, such abandonmen ranslaes ino los revenue. A key for reducion of hese delays is proper measuremen of he user perceived laency. A Web sie, once esimaed he perceived laency seen by he users, can use several mehods o reduce i: 1) A mirror sie, 2) Wider conneciviy o he Inerne, 3) Beer Web server or, 4) Load balancing. Furhermore, if he esimaion can be done a real ime, i can be used for online conrol of he server scheduling and prioriizaion mechanism o improve he performance perceived by he cliens. Today, he common approach for measuring his perceived laency, is o conduc he measuremen from ouside of he server using remoe agens. This mehod deploys a limied number of remoe agens placed around he Inerne. These agens fech a specific Web page from he Web server, and hus measure he laency from hose locaions only. The disadvanages of his mehod are 1) The Web sie is depended upon exernal body for conducing hese measuremens, 2) The number of agens is limied; herefore, heir measuremens do no reflec wha real Web users experienced in oher places, 3) The approach canno provide a real-ime measure of he laency as perceived by he individual cliens, 4) The perceived laency measured by he agens does no have a breakdown o he various laency componens, and 5) The agens DNS lookup ime is effeced by prior DNS lookup queries. In his work, we propose a new approach o esimae user perceived laency, based on server side measuremen. The new soluion does no require any agens o be placed a he ne and no addiional hardware. Furher, i does no monior packes a low-level proocols (TCP/IP) and is all based on implemenaion a he HTTP level. The oher properies of he soluion are 1) Low CPU and nework overhead, 2) Minimal nework archiecure dependency. 3) Minimal modificaion o he Web server and, 4) No addiional user perceived laency. Our soluion is based on wo fundamenal observaions. The firs is ha a common Web page consiss of ex, inlined 2 images and scrips. We exploi his srucure, aiming a exracing informaion from he inerreques imes experienced by he server while he user feches he Web page. The second is ha while packe races are normally no colleced, HTTP access logs are kep by mos sies. These logs conain hidden informaion ha can be exraced, and when combined wih he firs observaion, can be used o calculae he user perceived laency. Our soluion is based on a new echnique in which a special iny and pracically unnoiceable HTTP objec, called he senry, is embedded a he HTML file and used in measuring he user perceived laency. Our soluion consiss of wo elemens, one elemen is he measuring elemen and he oher one is he laency esimaion elemen which can be run eiher in online or off-line fashion. The measuring elemen consiss alogeher of 4 componens: 1) The senry - a zero size inline HTTP objec ha is placed a he end of he HTML documen (and hus does no add overhead) which racks he arrival ime o he user, 2) Exended Server Access Log,3) Queue Laency Probe, and 4) Exernal Pinger an exernal program o measure beween 1 M. Marshak is currenly wih EMC, USA. 2 This paper uses he erms of embedded images and inlined images inerchangeably.

2 2 he users and he server (I should be noed ha his is no he only source for compuing, and ha we compue an alernaive measure of from he Web server logs). The esimaion elemen akes as inpu hese daa ses and compue an esimae for he ime ook for he user o fech he exual porion of he Web page and he full Web page (ex and inlines). Our mehod canno measure he DNS lookup ime. However, for mos Web sies he DNS lookup ime [2] in comparison wih he whole ransacion ime is negligible. Noe also, ha oher soluions ha use remoe measuremens face he same problem, since heir DNS queries are affeced by he previous queries, which bias he measuremens. To address he DNS access issue, we will show a scheme o check wheher users have problem accessing he server due o DNS problems. A major advanage of our approach, in comparison o clien (agen) based approaches, is ha i can provide in real-ime a good esimae of he delay experienced by each of he individual cliens visiing he sie. This informaion can be used dynamically by he server, in an online mode, o dynamically conrol is prioriizaion and scheduling algorihms in order o opimize he perceived performance of he sysem; for example, high prioriy can be given o cliens which are subjec o large delays. All his is implemened wihou he need o use low-level proocols or wihou needing an exra expensive piece of hardware. We demonsrae our approach by implemening i on he Apache [3] Web server. The server was running on a PC acing as a dedicaed Web server and we ran Web cliens, ha simulae Web browsers wih exensive monioring capabiliies, under differen Web server loads and from differen locaions wih differen nework characerisics. To examine our approach we conduced an array of experimens. Our experimens show ha he proposed mehod esimaes he laency o fech he exual porion of he Web page and he full Web page wih an average error of 4% under normal server loads, and an average error of 10% under overloaded server loads. The remainder of his work is organized as follows: Secion II provides background and noaions; Secion III analyzes a Web page ransfer; Secions IV describes he observaions led us o he soluion; Secion V describes he measuring archiecure of our soluion; Secion VI describes esimaion algorihm of our soluion; Secion VII evaluaes our soluion; Secion VIII presens relaed problems and heir soluions; Secion IV presens concluding remarks and fuure work. A. Relaed Work The collecion of informaion abou user aciviy on he Web has been he subjec of exensive research in boh academia and indusry. Over he las years, several approaches for collecing informaion abou user Web aciviy were proposed. These are: 1) Modified Web Browsers, 2) Web servers logs, 3) Web proxies logs, and 4) Packe monioring. Each of hese mehods has is own advanages, bu mos suffer from severe limiaions regarding he deail of informaion ha can be logged. Unil now, he only mehod used for collecing informaion for esimaing various laency componens was he packe monioring mehod [5],[7], which used complex algorihm and addiional hardware o produce he HTTP race. I is imporan o emphasis ha, o he bes of our knowledge, no one ried o use any of hose informaion collecion mehods o esimae he laency experienced by Web cliens. Several companies have launched commercial producs o measure Web Sie laencies. These producs esimae he laency by feching a par or whole Web page from several locaions in he Inerne and using remoe agens, companies like Siescope [8] and Keynoe [9]. The remoe agens can measure he laency from heir locaion; heir DNS lookup ime is effeced by previous queries done o he DNS server ha resul in small DNS lookup ime. These measuremens are performed from limied locaions, which are no necessarily he same as he acual clien s locaion. The acual cliens may experience differen nework condiions and differen DNS lookup ime. These soluions canno provide addiional insigh ino he componens of he laency, for example, wheher he laency is high due o server load or due o anoher reason. II. BACKGROUND AND NOTATIONS A. Laencies of Web ransfer The fac ha each Web page consiss of many embedded images implies ha Web browsing sessions ypically consis of many HTTP [4] requess, each for a small size documen. The pracice wih HTTP/1.0 was o use a separae TCP connecion for each HTTP reques and response [10]. This led o incurring connecion-esablishmen and slow-sar laencies on each reques [11]. Persisen connecion addresses his problem by reusing a single long lived TCP connecion for muliple HTTP requess. Persisen connecions became defaul wih HTTP/1.1, which becomes increasingly deployed. Deploymen of HTTP/1.1 reduces he laency incurred in subsequen requess o a server uilizing an exising connecion, bu longer perceived laency is sill incurred when a reques necessiaes esablishmen of a new connecion. The process required for exchanging an HTTP message reques/response beween he clien and he server is as follows: Firs, he IP address of he server is needed in order o esablish a connecion. Therefore, he browser has o map a domain name of he server o he server s IP address. Browsers may send a DNS query o a local name-server or ge i from heir inernal cache. A name-server caches an address for TTL seconds [12]. Second, HTTP messages beween a clien and a Web server are sen using TCP connecions. A new HTTP reques may eiher use an exising persisen TCP connecion, or esablish a new connecion. The evens in he case of a new connecion esablishmen phase of an

3 3 HTTP ransacion are as follows 3 : 1) The clien sends a SYN segmen o he serve, 2) The server s TCP places he reques o a new connecion in he uncompleed connecion queue and sends o he clien he server s SYN segmen wih he ACK flag on, 3) The clien acknowledging he server s SYN and he server s TCP moves he new connecion from he uncompleed connecion queue o he compleed connecion queue, 4) The new connecion is waiing o be acceped by he server. Afer he server esablishes he connecion, he Web server is ready o read he HTTP reques from he clien and sends back an appropriae response. The compleed connecion queue s average lengh depends on how fas he HTTP server process calls accep() and on he reques rae. If a server is operaing a is maximum capaciy, i canno call accep() fas enough o keep up wih he connecion reques rae and he queue grows. When hese queues reach heir limi new connecion requess will be refused. Lasly, once a connecion is esablished a he ranspor layer, he clien sends an HTTP reques o he server over ha connecion and wais o read he server response. The clien reques is processed by he server s HTTP daemon. Before responding o he reques, he server may add laency due o he following: 1) Disk I/O, 2) Generaion of dynamic conens, and 3) Reverse- DNS query which happen rarely. To display a ypical Web page, a browser may need o iniiae several HTTP ransacions o fech he various componens (HTML ex, images) of he page. Wih persisen connecions, suppored by HTTP/1.1 4, subsequen requess (which can be requess for he embedded images or for a new URL) and responses may uilize he same TCP connecion (his feaure is shown in Figure 1). Pipelining is suppored by HTTP/1.1, and allows for several HTTP reques-responses o overlap in ime on he same TCP connecion: A subsequen reques can be issued wihou waiing for he response o he previous one (This feaure is no shown in Figure 1). According o [14], popular browsers implemen persisen connecions, bu do no implemen pipelining. An addiional facor conribuing slighly o he laency facor is he clien program hink ime. This laency can be caused by he following elemens: 1) Waiing for discovering new references of embedded images in he HTML page. 2) The clien s CPU is swiching beween he differen browser hreads. B. Ineracion beween HTTP and TCP a Web Servers The ineracion beween he operaing sysem and he Web server applicaion imposes some separaion beween he HTTP session and he ranspor layer. There is ypically no channel for 'upward' communicaion from he ranspor layer o he HTTP server as o wheher and when ransmission is compleed and acknowledged. 3 For simpliciy, we focus our discussion on a BSD [1] based nework subsysem. The process in many oher implemenaions of TCP/IP, such as hose found in Unix Sysem V and Windows NT is similar. 4 Prior o HTTP/1.1 some browsers and servers used he opion Keep-Alive o keep TCP connecions open and reuse hem. C. Concurren TCP Connecions Modern muli-hreaded browsers ofen faciliae HTTP ransfers by opening muliple concurren TCP connecions. The browser iniiaes muliple parallel image requess as he basic HTML page is received. The maximum number of connecions o one server is browser dependen. The suggesed number is 2 concurren persisen TCP connecions wih HTTP/1.1. According o Wang and Cao [14] and self-measuremens, i seems ha HTTP/1.1 version of Microsof Explorer uses abou 2 concurren persisen TCP connecions o a server and HTTP/1.1 Nescape Navigaor uses abou 4 concurren persisen TCP connecions o a server. Boh browsers open hese concurren persisen TCP connecions regardless of he number of embedded images in he Web page. D. Server Logs As par of processing an HTTP reques, he Web server generaes a log enry wih several fields. The common fields found in mos logs include: 1) IP address or name of he clien, 2) Firs line of he reques including he HTTP mehod and URL, 3) Dae/Time (in whole seconds) samp, 4) HTTP response saus code, and 5) Number of byes in he response, no including headers. The meaning of he dae/ime samp is one of he following: 1) The epoch when he server sars processing he reques, 2) The epoch when he server sars wriing he response conens ino he socke buffer, or 3) The epoch when he server complees wriing he response conens ino he socke. Server Logs may also conain some of he following fields: 1) Cookie informaion and 2) Server processing ime (in whole seconds). Krishnamurhy and Rexford [5] indicae ha currenly sandard log files generaed by Web servers do no include sufficien deail abou he iming of all possible aspecs of daa rerieval. The new sandard HTTP 1.1 [6] inroduces new feaures ha are currenly no logged in he server logs. E. Noaions In he res of his paper we will use he following erminology: Main Page laency - The ime ha elapsed from he momen he user reques he specific URL ill he ime he receive he exual par of he Web page, ha is he HTML porion of he Web page. Web Page laency - The ime ha elapses from he momen he user requess he specific Web page ill he ime he receives all he Web page, ha is he ex and all he embedded images. Connecion laency The ime ha akes o esablish a TCP connecions (3-way handshake). Queuing laency This is ime he clien is waiing in he complee connecion queue. Web page ransacion This includes he all requess he clien issues in he process of requesing he Web page (ex and embedded images) and he corresponding server responses. Iner reques ime This is he ime beween successive HTTP requess on he same persisen TCP connecion.

4 III. ANALYSIS OF A WEB PAGE TRANSFER Our mehod was developed for use wih persisen connecion wih no pipelining. This follows a sudy done by Wang and Cao [14], indicaing ha popular browsers implemen persisen connecions, bu do no implemen pipelining. Exension o our mehod under he assumpion of pipelining will be discussed in a laer secion VIII. For undersanding he principal idea of esimaing he Main Page laency and Web Page laency, we begin wih a deailed analysis of hese laencies. depics he process required for exchanging an HTTP reques and response for feching a whole Web page using wo persisen connecions. In his illusraion he Web page consiss of an HTML documen and wo inlines. The DNS lookup ime is no shown in. The requess on he same connecion are sequenial and no parallel because pipelining is no used. The Main page laency, is he sum of he following imes: 1) DNS lookup ime, 2) TCP connecion esablishmen (, label 4 minus label 1), 3) Queuing ime (label 5 minus label, 4) Server processing ime (label 6 minus label, and 6) Time o ransmi he exual par (label 8 minus label 6). I can be formulaed as follows Main page laency T DNS Lookup 1.5 u HTMLSize TQueuing TServer processing ime 0.5 u Bandwidh (Eq. 1) This equaion depends on he value of bandwidh, which depend on many facors TCP acknowledgmens ha may slow he effecive bandwidh, TCP slow sar and TCP reransmissions. HTML recieved Clien sends HTTP reques for image 2 Web page recieved Clien opens TCP connecion Clien sends HTTP reques for HTML Clien parses HTML while receiving HTML Clien Persisen Connecion 1 TCP SYN TCP SYN + ACK TCP ACK HTTP Reques 1 (HTML) HTTP Response 1 (HTML) HTTP Reques 3 (Image 2) HTTP Response 3 (Image 2) Server Queuing Laency Server Processing Time Server Processing Time Connecion placed in he SYN-RCVD queue Connecion placed in he accep queue Server accep he connecion + read he reques Server wrie he response Server read he reques Server wrie he response The fech ime of he whole Web page (ex and inlines), he Web page laency, is he sum of: 1) DNS lookup ime, 2) TCP connecion esablishmen ime for he connecion ha feches he HTML documen (, label 4 minus label 1), 3) Queuing ime for he connecion ha feches he HTML documen (label 5 minus label 4), and 5) The ime elapses since he server receives he firs reques (reques for he HTML documen) ill he ime clien receives he las response. Several noaions are needed o formulae his ime, we lis hem as follows: Si - The epoch a which he server receives HTTP reques i. Ei - The epoch a which he server wries HTTP response i o he wrie socke. N - Number of HTTP requess needed o fech he whole Web page. ResponseSize- Size of HTTP response i. i Now he Web page laency can be formulaed as follows: Web page laency TDNS Lookup 1.5u T ResponseSizei max{ Ei 0.5u S1} 1did N bandwidh (Eq. 2) Clien opens TCP connecion 1s inline image deeced Clien sends HTTP reques for image 1 Clien Clien wai for recieving HTML and exracing he inlines images Persisen Connecion 2 TCP SYN TCP SYN + ACK TCP ACK HTTP Reques 2 (Image 1) HTTP Response 2 (Image 1) Figure 1: Web Page Transacion for Feching a Web Page Using Persisen Connecions Server Queuing Laency Server Processing Time Connecion placed in he SYN-RCVD queue Connecion placed in he accep queue Server accep he connecion Server read he reques Queuing Server wrie he response 4

5 5 IV. OBSERVATIONS Our objecive is o compue he variables in Eq. 1 and Eq. 2 and we use he following observaions: 1) The server may keep some inernal informaion o he HTTP reques/response imeline, 2) I may be possible using he iner-reques imes o exrac esimaion for he and he bandwidh, 3) There is a dependency beween he epoch a which a clien requess an inline image o he porion of he HTML documen received by ha epoch, 4) Queuing laency canno be measured from inside he web server. So exernal probe will be needed, and 5) The sever logs he clien IP address, so we may ping he clien o ge an esimae for he. In order o esimae he Main page laency and he Web page laency from he server side we will need o know he informaion lised in Table 1. In he following secions, we will show how we evaluae he variables lised in Table 1. 1) The number of persisen connecions he clien uses o fech he Web page. 2) The complee se of HTTP requess/response ha consiue he Web page ransacion. 3) Round Trip Time beween he clien and he server. 4) The bandwidh beween he server and clien (his may change due o nework and server loads). 5) The epoch each reques is received by he server. 6) The epoch he server finishes wriing he response conens ino he socke for each response. 7) Queuing laency. 8) DNS lookup ime. Table 1: Required Informaion for Esimaing Laency. V. DATA COLLECTION ARCHITECTURE We now describe our measuremen archiecure ha combines exended server logging, queue laency measuremen, measuremen and he senry HTTP objec. Figure 2 depics an overview of he archiecure. In he following sub secions, we describe in deail each elemen. Inerne Exernal HTTP Requess and Responses ICMP Echo Reques/Response Message Exernal Pinger Clien Log Clien IP Web Server Compuer Web Server Exended Server Access Log Figure 2: Daa collecion Archiecure Queue Laency Probe Queuing Laency Log HTTP Reques and Response A. Exended Server Access Log We propose o exend he server logs o provide a deailed imeline of he seps involved saisfying a clien reques, wih limied inerference a he server, due o he logging operaion. As was described previously, he ime samp used in curren access log is in 1-second granulariy, we exend his imesamp o have 1-millisecond granulariy. The addiional fields we add are: 1) Flag indicaing wheher his reques is he firs one on is connecion, 2) The clien s por number, 3) Server s accep epoch, which is he epoch he server acceps he connecion, 4) Server s iniial reques processing epoch, which is he epoch when he server sars processing he reques, 5) Server s final reques processing epoch, which is he epoch when he server complees wriing he response conens ino he socke, and 6) Number of byes lef o send when he server logs he reques. The granulariy of all he addiional imesamps is 1- millisecond. B. Queue Laency Probe For shor periods, all new TCP connecions waiing in he complee connecion queue will experience a similar laency. The queue laency probe runs in parallel o he Web server and esimaes a low rae he queuing laency by measuring he ime duraion ha akes o perform an HTTP reques for a small documen on a new TCP connecion. The probe wries in a file a imesamp and he laency measured. I measures he queue laency because for inernal reques, he ime of he TCP connecion, he, he ransmission ime of he reques/response and he server processing ime for a small saic HTML documen are negligible 5. The probes are generaed a a consan rae 6, every 0.5 seconds. The addiional benefi o he probing process is ha he probe will warn us when users connecion requess are declined. C. Exernal Pinger We use wo mehods o esimae he. One using he iner-reques imes (discussed in Secion VI), and he oher using exernal program o esimae he, called he Exernal Pinger.The Exernal Pinger runs in parallel o he web server and a low rae reads he access log. For each new clien IP address i sends a single ICMP reques message and wais for he response, afer receiving he response i will record he for his IP. We are aware ha because we use a single measuremen and we perform he measuremen some ime afer he acual Web page ransacion, i is no accurae, bu i servers as a good approximaion. The exernal pinger has a minimal processing and communicaion overhead o he server. The exernal pinger can also be used o verify ha we do no have rouing problems by pinging a lis of frequen hoss/subnes a low rae and repor us of any problem D. The Senry: HTML Documen Modificaion There are some cases in which we canno exrac addiional informaion from he iner-reques imes (e.g. Web pages wih less hen hree embedded images) or we can 5 In our experimens, we observe an average 1 ms server processing ime. 6 We noe ha he use of Poisson sampling of he queue may add some more precision, due o he Poisson Arrivals See Time Averages propery, bu for simpliciy we avoided i in our implemenaion.

6 6 exrac a poor esimaion for he Main page laency and he Web page laency (e.g. Web page wih large HTML documen wih few small embedded images). Our soluion o hese problems is o add a small inline image o he end of he HTML documen. We shall call his las inline image he Senry. The reques for he Senry will noify us when he clien finishes receiving he HTML documen. The senry allows us o ge an esimae for he Main page laency, and in addiion we will have 2 sequenially dependen requess which will enable us o do some esimaion of he bandwidh and he. The laer will yield a beer esimaion for he laencies. I is imporan o noe ha he Senry imposes negligible overhead on he server, he clien and he communicaions nework. The senry mus be refresh for each documen in order o preven i from being cached by he clien. This can be performed by specifying Time-To-Live (TTL) of 0. VI. LATENCY ESTIMATION ALGORITHM In his secion we presen our algorihm of esimaing he Main page laency and he Web page laency based on he daa colleced by our measuremen archiecure. Our laency esimaion algorihm esimaes hose laencies for each Web page ransacion separaely. Hence, we need o spli he access log o Web page ransacion. Therefore prior o he esimaion here is a spliing phase of he access log per Web page ransacion. The spliing phase consiss of wo seps: 1) Soring he access log according o he HTTP reques imesamp, 2) Spliing he chronicle ordered access log according o he clien IP address and reques for he HTML documen. Afer he spliing phase, we have a complee imeline of he Web page ransacion for each persisen connecion (each persisen connecion has a differen clien por number). The only missing informaion in order o compue he laencies is he and he bandwidh. Nex, we describe our heurisics o esimae hese variables. A. Iner-Reques In his subsecion we presen our mehod of esimaing he from he iner-reques in each persisen connecion. Figure 3 illusraes he imeline of wo consecuive response/reques on he same persisen connecion. The ime elapses beween he epoch a which he server finishes wriing he response and epoch a which i receives he nex reques is he sum of 1) One round-rip ime delay inheren in he communicaion, 2) Transmission ime of he reques, 3) Clien hink ime, and 4) Transmission ime of he response. Assuming ha he response size and he reques size are small, herefore heir ransmission ime is negligible and he clien hink ime is zero, we can esimae he as he iner-reques ime. This migh no be precise in some cases where he ransmission ime of he response is no negligible in comparison o he. Also he clien hink ime can be quie large in cases ha he reference o he inlines in HTML documen are sparsely disribued. Our mehod of esimaing he from he iner-reques imes is o selec he minimum iner-reques ime among all he iner-reques imes in all he persisen connecions used in he Web page ransacion. From now on, min will denoe he measured from he iner-reques imes. I should be clear ha his is no he real and i is larger hen he real. Our encapsulaes in i some bias due o ransmission ime 7. The accuracy of our esimaed depends on he bandwidh. The esimaion will improve as he bandwidh increases. Figure 3: Timeline of Two Consecuive Requess on he Same Persisen Connecion B. Iner-Reques Perceived Packe Rae In his sub-secion we presen our approximaion for he downsream bandwidh. If we knew he for each HTTP response/reques, TCP window size, segmen size, TCP acknowledgmens and TCP reransmissions we could use his informaion o calculae he bandwidh for each HTTP reques/response. Unforunaely, none of hese variables is available and we mus resor o alernaive approaches. From now on, we assume ha he HTTP reques size is small, herefore is ransmission ime is zero. Hence we can esimae he bandwidh using iner-requess using he following equaion: bw Response Sizei /( server _ recv _ ) i 1 serv send i This is however problemaic since we do no know he acual, for his HTTP reques/response couple. We migh ge from some iner-reques imes almos an infinie bandwidh or even a negaive bandwidh. Due o hese problems, we propose he following erms o be used for esimaing he bandwidh: Perceived Packe Transmission Time The ime per packe ha is required o send a collecion of daa packes from he server o he clien ill i is received compleely. Perceived Packe Rae ( Raeiner reques ) Inverse of Perceived Packe Transmission Time. 7 This bias can be reduced by using he server -hroughpu and he inline size (boh can be known a he server size) o compue he response ransmission ime and accouning for i when calculaing min.

7 7 The perceived packe ransmission ime is a leas half he round rip ime. Using Figure 3, we can calculae hese wo values for any iner-reques as follows: Preceived Packe Rae max ^ server ª Response Sizei / Packe Sizeº _ 1 recvi serv _ send i (Eq. 3) 0.5u,0.5u The Perceived Packe Rae esimaes he acual bandwidh only if he dominan par of he ransmission is he ransmission ime and no half of he. Using Eq. 3, we will have a differen value for he Perceived Packe Rae for any wo successive requess on he same persisen connecion. The reasons for he differen Perceived Packe Rae for each couple of iner-reques imes is ha each iner-reques ime is effeced differenly by: high clien hink ime, many TCP acknowledgmens or TCP reransmissions. As he HTTP response size increases he effecive bandwidh decreases due o TCP overheads. Hence, he Perceived Packe Rae is beer esimaion for he ransmission rae for large HTTP response size. Noe ha he acual packe size used does no affec he algorihm. C. Average Iner-Reques Perceived Packe Rae The Web page ransacion imeline has several inerreques imes. The average of all he perceive- packe-rae measures aken over all he iner-requess in all he acive connecions, is a good rae esimaor for deermining when a large HTTP response (e.g. HTML documen) will be received by he cliens. However, for average HTTP response, which is relaively small, his esimae is an under esimae. The reason for his is ha TCP acknowledgmens overhead is low for a small response. Therefore, we define he following rae esimaors: Connecion perceived packe rae ( Rae Conn ) The average perceived packe rae for a paricular connecion. Web ransacion perceived packe rae ( Rae Web _ Trans ) The average of he all RaeConn over all he connecions. Perceived packe line rae ( Rae line ) The average perceived packe rae ( Raeiner reques ) over all he connecions ha he clien uses in he process of feching he Web page, excluding perceived packe rae calculaed from long iner-reques imes. We define shor iner-reques ime if i is less han 6 xmin. I is valid only if here are a leas four samples of shor iner-requess imes. RaeConn is a good rae esimaor for large responses which suffer from TCP overheads, while Raeline is a good rae esimaor for small responses. For connecions wih high, Raeline and RaeConn will be similar. I may happen ha Rae is invalid; his usually happens for line ` Web pages ha have small number of embedded images or for Web pages ha have many large embedded images and few small embedded images. Rarely i may happen ha RaeConn will be invalid for all he connecion; his happens when a Web page has only a few embedded images. Obviously in his case Rae Web _ Trans will be invalid also. In his case by using our esimaion o he epoch a which he clien received he HTML documen (Eq. 5), we can esimae he perceived packe rae for he HTML documen. Using his perceived packe rae, we recalculae RaeConn for he connecion on which he HTML documen reques came on and recalculae Rae _. Web Trans D. Main Page Laency The esimaed Main page laency, excluding he DNS lookup ime, can be summarized in he following formula: Main Page Laency clien _ recv serv _ T HTML recv HTML Queuing 1.5 u (Eq. 4) The queue ime laency is obained from he queue laency log. The used by our algorihm is he minimum beween min and he exernal measured. We esimae he epoch a which he HTML documen was received by he clien, from hree sources: 1) Using inerreques ime, 2) Using Connecion perceived packe rae, and 3) Using he Senry. The acual ime is he minimum calculaed by hese hree sources. Using Figure 3 we derive he esimaion: clien_ recv HTML ª ResponseSizeHTML/ Packe Sizeº server_ send HTML RaeConn min server_rec v 0.5u nex_iner_reques server_rec v 0.5u Senry_image ( a) if RaeConn Valid (b) if a leas 2 reques on HTML s conn (Eq. 5) E. Web Page Laency The esimaed Web page laency, excluding he DNS lookup ime, can be finally calculaed according o he following formula: Web Page clien _ recv Laency Las _ Re sponse T serv _ recv (Eq. 6) Queuing HTML 1.5 u if ( a) if (b)

8 8 clien_recv Las_Response The only unknown ime, is which we nex calculae. We define he following noaions: N - The number of persisen connecion used. Conn j - Connecion number. C - The connecion number on HTML which he reques o he HTML documen came. - The ime he server finished wriing he las serv _ send las, j HTTP reques ino he wrie socke of connecion j. N - j The number of requess on connecion j. N - The number of requess on he connecion on which he reques o HTML he HTML documen came. Response Size las, j -The remaining par of he las response when he server finished wriing o connecion j. - The ime he clien clien _ recvlas, j finished receiving he las HTTP response from connecion j. We esimae his laes ime as follows: max ( Where - clien _ recv Las_Respo nse clien _ recv ) las, j 1d j N clien _ recvlas, j clien _ recvhml serv_sendlas,j serv_sendlas,j ( a) ( b) (c) Conn ª ResponseSizelas, j / PackeSizeº ª ResponseSizelas, j / PackeSizeº N hml no ( N no (N 1and j hml hml Rae Rae line Web_Trans C 1 and j 1 and j (Eq. 7) hml C C hml hml ) and Rae ) and Rae if ( a) if ( b) if ( c) line line Valid no Valid VII. EVALUATION OF THE ALGORITHM This secion describes he experimenal mehodology, benchmarking and analysis ools, he resuls of our experimens, our analysis of he resuls and he measuring overhead on he server. We implemened our algorihm on he Apache version 1.3.9, a public domain HTTP/1.1 server. We conduced he measuremens using hree locaions. The Web server was locaed a Tel-Aviv Universiy (TAU), Israel. The Web cliens were locaed a Tel-Aviv college (MTA), Israel and a he Universiy of California, Irvine (UCI), CA. A. Measuremens Tesbed We wan o evaluae he performance of our esimaion mehod. In order o make our evaluaion valid we need o simulae real cliens wih real-world Web raffic characerisics (bandwidh, and loss rae), feching differen Web pages under various server loads. Also, in order o esimae our performance we need o know he acual laencies he cliens experienced. Figure 6 demonsraes he opology of he esbed. In he following sub-secion we will describe each elemen of he esbed. A.1 Web Server The Web server compuer includes hree elemens: he modified Apache Web server, he queue laency probe and he exernal pinger. The server ran on a dedicaed PC running he Linux operaing sysem version Web Clien Web Clien Inerne (WAN) A.2 Web Cliens Web Loader Web Loader Figure 4: Tesbed Topology Web Server LAN Web Loader Web Loader In order o evaluae our mehod we needed a Web cliens ha will simulae a Web browser and perform measuremens of he Main Page Laency, he Web Page Laency and oher parameers like. Therefore, we wroe a muli-hread Web browser program uilizing BSD-socke inerface. This program suppors HTTP/1.1 persisen connecions and simulaes he dependency beween he reques for he HTML documen and he requess for he embedded images. The program feches whole Web pages using a predefined number of persisen TCP connecions. Afer feching he whole Web page all he hreads close he connecions and he maser program wries in a log file, called he clien log file, he, he Main page laency and he Web page laency. A.3 Web Workload Generaor To generae he workload we wroe a program called Web Loader, based on he Web browser program wih a few modificaions. The Web Loader uses a fixed number of hreads. Each hread runs in infinie loop wih he same ask: opening a TCP connecion o he server, feching one web page from server and closing he connecion. The web page o be feched is seleced randomly from various web pages, which vary in size. Hence, each Web Loader simulaes several cliens running on he same clien machine. We limied he number of hreads per compuer o 10. In order o reach he required workload we ran he Web loader on several machines locaed on our LAN.

9 9 A.4 Represenaive Web Pages Used in he Experimen We waned o es our esimaion on represenaive Web pages of he Web. We waned o deermine ypical HTML documen size, he number of embedded images and heir ypical size for popular Web pages. Several raing sies offer saisics on popular Web sies like Ho100 [16]. Ho100 claims o survey 100,000 users (40% of whom are ouside he USA). Ho100 claims o gaher daa a sraegic poins on he Inerne (no a he browser or server). For hose op 100 sies we measured heir page sizes and found ha he average HTML documen size is 30K, he average number of embedded images is 21 wih an average size of 2.5K per embedded image. As a resul we seleced o use he following Web pages as represenaives: combinaions of HTML documen sizes 10K, 30K and 60K wih 5, 15, 30 and 50 embedded images wih an average size of 2K-3K, his gives us 12 various pages. Two addiional Web pages were used: A Web page consising of a 30K HTML documen and 21 embedded images of average size 6K (a Web page wih very large inlines) and one Web page which includes only 2 inlines. Thus, alogeher we have14 differen Web pages. Lasly, in order o use genuine Web pages, we seleced Web pages of hese characerisics hem from he op 100 Web pages. A.5 Tesing under Real-World Traffic Characerisics We wan o esimae our mehod using cliens wih realworld Web raffic characerisics connecing hrough a WAN. We wan o esimae he performance under WAN effecs: large dispariy, packe loss and various bandwidh characerisics. The PingER projec a Sanford Acceleraor Cener (SLAC) [17] conducs coninuous nework monioring o measure nework laency and packe loss over he Inerne. Their measuremens show average loss rae of 0.5% and of milliseconds in he U.S., Asia and Europe, while beween he coninens he average loss rae is of 1.4% and varies beween 200 o 600 milliseconds. We conduced our experimens wih wo cliens, one locaed a MTA and UCI. Our experimens were performed a various hours of he day and over a one week span. During his ime frame we measured hese cliens nework characerisics in erms of, bandwidh and loss rae. UCI s was milliseconds, bandwidh KB/s and loss rae of 0%. MTA s was milliseconds, bandwidh 5-17 KB/s and loss rae of 2%. UCI s reflec longer nework o he US from he server (locaed in Tel-Aviv). UCI s, bandwidh and Loss Rae showed a minor dispariy. MTA showed large dispariy in he and in he bandwidh. Therefore, hese wo sies are good represenaives for realworld Web raffic characerisics, because hey cover large ranges of, bandwidh and loss rae. Heidemann e al. [15] summarized nework characerisics for several exising neworks, which include ypical bandwidh and. According o Heidemann s erms we can caegorize hem as follows: MTA can be caegorized as somehing beween WAN-Modem o WAN-ISDN due is large dispariy, while UCI can be caegorized as Medium-Inerne conneciviy wih large number of hops. B. Experimens Runs We conduced he es runs as follow: Each web-clien locaed in MTA or UCI feched all he 14 Web pages in serial fashion. For each Web page our browser firs feched i wih 4 persisen connecions for 5 imes and laer he browser feched i wih 2 persisen connecions for 5 imes. Beween each Web page download, i waied for 4 seconds before coninuing. The ess were repeaed under various server loads. We conrolled he server load by number of Web Loader compuers running in our LAN. The number of Web Loader compuers varies beween 0-9, which means beween 0 o 90 cliens, in oal. We used four server loads: Ligh, medium, high and overloaded. Load CPU Usage [%] Requess Per Sec Average Queue Laency [msec] Ligh Medium High Overloaded Table 2: Web Server Load Characerisics C. Performance Evaluaion No. Of Web Loaders Figure 5 depics he accuracy of our Main Page Laency and Web Page Laency esimaion mehod for all he ess runs for boh of he cliens under he various server loads. The figure depics also he effec of he esimaing mehod for he (min or from he exernal pinger). I should be clear ha he exernal pinger is inegral par of our mehod. Therefore, he performance of our mehod should be evaluaed for he case of using he pinger for esimaing he. The measuremens subjec o some errors due o variabiliy, however our resuls are no sensiive grealy o i. Table 3 summarizes he median and average of he esimaion error. The able shows also he median value of he esimaion error because he average values are shifed by he few high errors in he es runs under overloaded server. The average laency esimaion error of our mehod for various Web pages is 4% under normal server loads and 10% under overloaded server. For Web pages wih few embedded images our mehod requires he use of he exernal pinger for he esimaions. In order o beer undersand he performance of he mehod we presen in he following subsecions he resuls of he runs under various condiions. Since he accuracy of he experimens seems o no be affeced by he normal server load (ligh, medium and high) we aggregae he differen load resuls ogeher. The laency esimaion errors for MTA and UCI have similar behavior, so some of he resuls will presen MTA s esimaion errors and oher UCI s esimaion errors. The res of his secion is divided ino he following sub-secions: 1) Evaluaion of perceived delay esimaion for ypical Web pages under normal

10 10 server loads, 2) Evaluaion of perceived delay esimaion for Web pages wih large embedded images under normal server loads, 3) Evaluaion of perceived delay esimaion for Web pages wih a few embedded images under normal Figure 5: Cumulaive Disribuion of Laency Esimaion Error for all Tes Runs Meaning Average Es. Mehod min Pinger Main Page Web Page Main Page Web Page All runs Meaning Median Es. Mehod min Pinger Main Page Web Page Main Page Web Page All runs Table 3 - Average and Median of he Laency Esimaion Error for all Tes Runs server loads, and 4) Evaluaion of perceived delay esimaion for an overloaded server. C.1 Laency Evaluaion wih Typical Web Pages under Normal Server Load Figure 6 depics he accuracy of our Main Page Laency and Web Page Laency esimaion mehod as a funcion of he number of he embedded images for boh of our cliens. We aggregaed he runs of Web pages wih he same number of embedded images. For 15, 30 and 50 inline images, he laency esimaion errors for MTA and UCI s behaved similar so we presen MTA s errors for Web pages wih 15 and 30 inlines and UCI s errors for Web pages wih 50 inlines. Figure 6 depics he effec of esimaing mehod for he. Table 4 summarizes he average laency esimaion error. For ypical Web pages he laency esimaion error for boh of our cliens does no depend on he number of inline images or on he mehod of esimaing he. Hence, he effec of he exernal pinger for ypical Web pages is negligible. For a Web page wih large inline images here is no significan change in he esimaion error. The average esimaion error for MTA was larger han UCI because: 1) UCI has high bandwidh,. 2) MTA has high packe loss. We conclude ha he average laency esimaion error is 4% for a ypical Web page. Clien MTA Es. Mehod min Pinger No. of Inlines Main Page Web Page Main Page C.2 Laency Evaluaion wih Web Pages wih a Few Inlines under Normal Server Load Figure 7 depic he accuracy of our Main Page Laency and Web Page Laency esimaion mehod for Web pages wih a few embedded images (wo and five embedded images). The laency esimaion error for Web pages wih wo and five embedded images behave similar so we presen he error for Web pages wih five embedded images. Each figure also depics he effec of he mehod for esimaing he. Table 5 summarizes he average errors for UCI and MTA. As can be seen from he Table, for a Web page wih a few embedded images i is preferred o use he exernal pinger o esimae he raher han relying on he min. The error decreases in some cases from an average error of 90% o an average error of 6%. We conclude ha for Web pages wih a few embedded images our mehod esimaes he Main Page Laency and he Web Page Laency as good as for Web pages wih many embedded images. This holds when we have esimaion via he exernal pinger. C.3 Laency Evaluaion under Overloaded Server Web Page Clien UCI Es. Mehod min Pinger No. of Inlines Main Page Web Page Main Page Web Page Table 4: Laencies Esimaion Errors for Typical Web Page In his subsecion, we evaluae our mehod under overloaded server condiion. Figure 8 depics he accuracy of our Main Page Laency and Web Page Laency esimaion mehod as a funcion of he number of he embedded images for MTA. Figure 8 depics only he runs wih 5 and 15 inline images because he res runs showed he same behavior. We also do no show UCI s resuls because hey exhibi a similar behavior. We aggregaed he runs of Web pages wih he same number of embedded images. Each figure depics he effec of he mehod of esimaing he. Table 6 summarizes he median and average of he laency esimaion errors for MTA. The behavior of he laency esimaion error is similar for he ess run under normal server load and overload sever as Figure 8, Figure 6 and Table 6 depic, excep o he long ail for he overload case. The reason for ha is ha for an overloaded server here are periods of ime in which he queue laency increases rapidly in shor ime. Our queue laency probe samples he queue laency in low frequency, hence i may sam-

11 11 sample he queue in his rapid increase in queue laency, and herefore i may under-esimae he queue laency. Therefore, in hose runs we ge high error. For his reason, we go higher average error han in he case of he runs under normal server loads. Hence, for overloaded server he median error is more meaningful han he average error. We see, again, ha here is need for he exernal pinger only for Web pages wih a few embedded images. The average laency esimaion error is 10%, in conras he median laency esimaion error is only 4%. C.4 Measuremen Overhead In his secion, we presen he overhead of our measuremen archiecure. The server CPU overhead due o he addiional fields logged is 0.15%. The queue laency probe samples he server every four seconds, which adds an average 0.025% CPU overhead. The exernal pinger is running once every 30 seconds, which adds average 0.04% CPU overhead. Hence, he oal average server CPU overhead due o our measuremens is less hen 0.2%. The esimae of he queue delay can possibly be improved by sudying more sophisicaed sampling and sample averaging echniques. However, since he impac of he queue laency accuracy seems o be small, we avoid i in his conex. Figure 7: Cumulaive Disribuion of Laency Esimaion Error for Web Pages wih 5 Inline Images Figure 6: Cumulaive Disribuion of Laency Esimaion Error for ypical HTML Documens Clien MTA Es. Mehod min Pinger No. of Inlines Main Page Web Page Main Page Web Page Clien UCI Es. Mehod min Pinger No. of Inlines Main Page Web Page Main Page Web Page Table 5: Laencies Esimaion Errors for Web Page wih few Inline Images Figure 8: Cumulaive Disribuion of Laency Esimaion Error under Overloaded Server Meaning Average Es. Mehod min Pinger No. of Inlines Main Page Web Page Main Page Web Page Meaning Median Es. Mehod min Pinger No. of Inlines Main Page Web Page Main Page Web Page Table 6: Average and Median of he Laency Esimaion Error under Overloaded Server

12 12 VIII. RELATED PROBLEMS AND THEIR SOLUTION A. DNS Lookup Time Our mehod canno measure he DNS lookup ime ha he clien may experience. However i seems ha for he curren Web is conribuion on he overall perceived laency is negligible. Keynoe [9] is a commercial company ha measures Web sies laencies using remoe agens. According o heir measuremens, he average DNS lookup ime is 60 milliseconds. Cohen and Kaplan [2] sudied DNS lookup imes under various condiions. Their sudy shows ha DNS lookup imes for 80% of he servers ook under 300 milliseconds, and repeaed DNS query conduced hours laer ook less hen 150 milliseconds for 80% of he servers. Because DNS queries are cached also in Web browsers and ISP s name servers, his ime can be even lower. Their sudy measured he DNS lookup ime compared o he ime o download only he exual porion of home page of some URLs, wihou he embedded conens. They assume ha ransmission ime is negligible, because hey conduced heir measuremens in high bandwidh environmen. In our experimens wih a clien wih average bandwidh and low (MTA), he ypical average ime of downloading HTML documen and whole Web page excluding DNS lookup ime, is 2 sec and 5 sec, respecively. This means ha he DNS lookup ime relaively o he download ime of HTML documen and he download of he whole Web page is 10% and 4%, respecively. This raio can go down o less han 1% if he DNS query is already in he cache of he Web browser or in he ISP s DNS server. Hence for mos Web sies he DNS lookup ime in comparison o he whole ransacion, or even jus he HTML documen download ime, is negligible. Noneheless, we propose a server-side approach for verifying ha he Web access does no have DNS problems: The world is divided ino several zones, where each zone is under he responsibiliy of a well-known Roo DNS server. Therefore, he Web server, by using a program like nslookup, could query each one of he roo DNS servers wih is domain name periodically and repor he adminisraors abou DNS problems in any zone. B. Pipelining If popular web browsers will use pipelining in he fuure, our mehod is sill valid. Some minor modificaions need o be performed. These are: 1) The could no be measured from he iner-reques imes, herefore we will rely only on he exernal measuring program, our pinger, and 2) We need o change he way we calculae he perceived rae in our mehod. In case of pipelining subsequen server responses can be wrien wihou he previous ones being received. Hence, he server will wrie no o an empy wrie buffer as in he case of no using he pipelining. Therefore, we can measure he decrease rae of he wrie buffer each ime he server wries a new response. By monioring he changes along he imeline of he whole ransacion, we will be able o esimae he perceived rae. C. Load Balancing Some Web sies use several Web servers and disribue he cliens requess using a load-balancing device. In case ha all he clien s connecions are sen o he same server our algorihm does no require modificaion. In a case ha each connecion is sen o a differen server here is a need o merge all he sie server s access logs ogeher before running our laency esimaion mehod. D. Web Proxy Our mehod used he clien IP address in order o disinguish beween he differen users. Some Web cliens are forced o go hrough proxies o connec o he Inerne. Therefore, he server will no be able o disinguish beween he differen cliens ha are connecing o he server via he proxy. One can use cookies o disinguish beween hem. E. Disribued Embedded Images Some Web sies disribue heir embedded images in oher servers around he Inerne. Microsof Explorer opens 2 persisen connecions o he server while Nescape Navigaor opens 4 persisen connecions o he server. We noiced ha when Microsof Explorer and Nescape Navigaor deec reference o images a a differen server, hey open addiional 2 or 4 addiional connecions o he new server, respecively. Hence he download of hese images is done simulaneously o he download of he page downloaded from he original sie. These new TCP connecions may slow down he rae of he connecions o he original server if he clien has a limied bandwidh. Thus he new requess on he already esablished connecions o he original server will suffer from rae decrease which will be noiceable from he access log, and hus will be aken ino accoun in our esimaion algorihm. We examined he 100 Web sies ha we used previously and found ha 30% of hese Web pages have no exernal images and 70% have less han 8 exernal images. For 60% of he Web pages he relaive fracion of he exernal images is less hen 20%. In addiion, we can see ha abou 20% of he Web pages have abou 80% of heir images sored in oher servers. The average number of exernal images is 6, while he average number of embedded images per Web page is 21. Hence, on he average mos of he images come from he original Web server. Thus, our algorihm will accuraely esimae he laency of he HTML documen and accuraely esimae he download ime of he images locaed in he original server, which, for mos cases is he major porion of he images. IX. CONCLUDING REMARKS We presened a new approach o esimae user perceived laency, based on server side measuremen. The new solu-