IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX 1

Transcription

1 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX 1 Conex-Aware Proacive Conen Caching wih Service Differeniaion in Wireless Neworks Sabrina Müller, Suden Member, IEEE, Onur Aan, Mihaela van der Schaar, Fellow, IEEE, and Anja Klein, Member, IEEE Absrac Conen caching in small base saions or wireless infosaions is considered o be a suiable approach o improve he efficiency in wireless conen delivery. Placing he opimal conen ino local caches is crucial due o sorage limiaions, bu i requires knowledge abou he conen populariy disribuion, which is ofen no available in advance. Moreover, local conen populariy is subjec o flucuaions since mobile users wih differen ineress connec o he caching eniy over ime. Which conen a user prefers may depend on he user s conex. In his paper, we propose a novel algorihm for conex-aware proacive caching. The algorihm learns conex-specific conen populariy online by regularly observing conex informaion of conneced users, updaing he cache conen and observing cache his subsequenly. We derive a sublinear regre bound, which characerizes he learning speed and proves ha our algorihm converges o he opimal cache conen placemen sraegy in erms of maximizing he number of cache his. Furhermore, our algorihm suppors service differeniaion by allowing operaors of caching eniies o prioriize cusomer groups. Our numerical resuls confirm ha our algorihm ouperforms sae-of-he-ar algorihms in a real world daa se, wih an increase in he number of cache his of a leas 14%. Index Terms Wireless Neworks, Caching a he Edge, Cache Conen Placemen, Online Learning I. INTRODUCTION WIRELESS neworks have been experiencing a seep increase in daa raffic in recen years [2]. Wih he emergence of smar mobile devices wih advanced mulimedia capabiliies and he rend owards high daa rae applicaions, such as video sreaming, especially mobile video raffic is expeced o increase and o accoun for he majoriy of mobile daa raffic wihin he nex few years [2]. However, despie recen advances in cellular mobile radio neworks, hese neworks canno keep up wih he massive growh of mobile daa raffic [3]. As already invesigaed for wired neworks [4], conen caching is envisioned o improve he Manuscrip received May 19, 2016; revised Sepember 30, 2016; acceped November 20, A preliminary version of his work has been presened in par a he IEEE Inernaional Conference on Communicaions (ICC), 2016 [1]. The work by S. Müller and A. Klein has been funded by he German Research Foundaion (DFG) as par of projecs B3 and C1 wihin he Collaboraive Research Cener (CRC) 1053 MAKI. The work by O. Aan and M. van der Schaar is suppored by NSF CCF and NSF ECCS gran. S. Müller and A. Klein are wih he Communicaions Engineering Lab, Technische Universiä Darmsad, Darmsad, Germany ( s.mueller@n.u-darmsad.de, a.klein@n.u-darmsad.de). O. Aan and M. van der Schaar are wih he Deparmen of Elecrical Engineering, Universiy of California, Los Angeles, CA, USA ( oaan@ucla.edu, mihaela@ee.ucla.edu). efficiency in wireless conen delivery. This is no only due o decreasing disk sorage prices, bu also due o he fac ha ypically only a small number of very popular conens accoun for he majoriy of daa raffic [5]. Wihin wireless neworks, caching a he edge has been exensively sudied [1], [6] [19]. A he radio access nework level, curren approaches comprise wo ypes of wireless local caching eniies. The firs ype are macro base saions (MBSs) and small base saions (SBSs) ha are implemened in wireless small cell neworks, dispose of limied sorage capaciies and are ypically owned by he mobile nework operaor (MNO). The second ype are wireless infosaions wih limied sorage capaciies ha provide high bandwidh local daa communicaion [16], [17], [20], [21]. Wireless infosaions could be insalled in public or commercial areas and could use Wi-Fi for local daa communicaion. They could be owned by conen providers (CPs) aiming a increasing heir users qualiy of experience. Alernaively, hird paries (e.g., he owner of a commercial area) could offer caching a infosaions as a service o CPs or o he users [17]. Boh ypes of caching eniies sore a fracion of available popular conen in a placemen phase and serve local users requess via localized communicaion in a delivery phase. Due o he vas amoun of conen available in mulimedia plaforms, no all available conen can be sored in local caches. Hence, inelligen algorihms for cache conen placemen are required. Many challenges of cache conen placemen concern conen populariy. Firsly, opimal cache conen placemen primarily depends on he conen populariy disribuion, however, when caching conen a a paricular poin in ime, i is unclear which conen will be requesed in fuure. No even an esimae of he conen populariy disribuion migh be a hand. I herefore mus be compued by he caching eniy iself [1], [13] [19], which is no only legiimae from an overhead poin of view, since else a periodic coordinaion wih he global mulimedia plaform would be required. More imporanly, local conen populariy in a caching eniy migh no even replicae global conen populariy as moniored by he global mulimedia plaform [22] [24]. Hence, caching eniies should learn local conen populariy for a proacive cache conen placemen. Secondly, differen conen can be favored by differen users. Consequenly, local conen populariy may change according o he differen preferences of flucuaing mobile users in he viciniy of a caching eniy. Therefore, proacive cache conen placemen should ake ino accoun he diversiy in conen populariy across he local user populaion. Thirdly, he users preferences

2 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. 2 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX in erms of consumed conen may differ based on heir conexs, such as heir locaion [24], personal characerisics (e.g., age [25], gender [26], personaliy [27], mood [28]), or heir devices characerisics [29]. Hence, cache conen placemen should be conex-aware by aking ino accoun ha conen populariy depends on a user s conex. Thereby, a caching eniy can learn he preferences of users wih differen conexs. Fourhly, while is ypical goal is o maximize he number of cache his, cache conen placemen should also ake ino accoun he cache operaor s specific objecive. In paricular, appropriae caching algorihms should be capable of incorporaing business models of operaors o offer service differeniaion o heir cusomers, e.g., by opimizing cache conen according o differen prioriizaion levels [30], [31]. For example, if users wih differen preferences are conneced o a caching eniy, he operaor could prioriize cerain users by caching conen favored by hese users. Moreover, cerain CPs conen could be prioriized in caching decisions. In his paper, we propose a novel conex-aware proacive caching algorihm, which for he firs ime joinly considers he above four aspecs. Firsly, insead of assuming a priori knowledge abou conen populariy, which migh be exernally given or esimaed in a separae raining phase, our algorihm learns he conen populariy online by observing he users requess for cache conen. Secondly, by explicily allowing differen conen o be favored by differen users, our algorihm is especially suiable for mobile scenarios, in which users wih differen preferences arrive a he wireless caching eniy over ime. Thirdly, we explicily model ha he conen populariy depends on a user s conex, such as his/her personal characerisics, equipmen, or exernal facors, and propose an algorihm for conen caching ha learns his conex-specific conen populariy. Using our algorihm, a caching eniy can proacively cache conen for he currenly conneced users based on wha i has previously learned, insead of simply caching he files ha are popular on average, across he enire populaion of users. The learned cache conen placemen sraegy is proven o converge o he opimal cache conen placemen sraegy which maximizes he expeced number of cache his. Fourhly, he algorihm allows for service differeniaion by cusomer prioriizaion. The conribuions of his paper are as follows: We presen a conex-aware proacive caching algorihm based on conexual muli-armed bandi opimizaion. Our algorihm incorporaes diversiy in conen populariy across he user populaion and akes ino accoun he dependence of users preferences on heir conex. Addiionally, i suppors service differeniaion by prioriizaion. We analyically bound he loss of he algorihm compared o an oracle, which assumes a priori knowledge abou conen populariy. We derive a sublinear regre bound, which characerizes he learning speed and proves ha our algorihm converges o he opimal cache conen placemen sraegy which maximizes he expeced number of cache his. We presen addiional exensions of our approach, such as is combinaion wih mulicas ransmissions and he incorporaion of caching decisions based on user raings. We numerically evaluae our caching algorihm based on a real world daa se. A comparison shows ha by exploiing conex informaion in order o proacively cache conen for currenly conneced users, our algorihm ouperforms reference algorihms. The remainder of he paper is organized as follows. Secion II gives an overview of relaed works. In Secion III, we describe he sysem model, including an archiecure and a formal problem formulaion. In Secion IV, we propose a conex-aware proacive caching algorihm. Theoreical analysis of regre and memory requiremens are provided in Secions V and VI, respecively. In Secion VII, we propose some exensions of he algorihm. Numerical resuls are presened in Secion VIII. Secion IX concludes he paper. II. RELATED WORK Pracical caching sysems ofen use simple cache replacemen algorihms ha updae he cache coninuously during he delivery phase. Common examples of cache replacemen algorihms are Leas Recenly Used (LRU) or Leas Frequenly Used (LFU) (see [32]). While hese simple algorihms do no consider fuure conen populariy, recen work has been devoed o developing sophisicaed cache replacemen algorihms by learning conen populariy rends [33], [34]. In his paper, however, we focus on cache conen placemen for wireless caching problems wih a placemen phase and a delivery phase. We sar by discussing relaed work ha assumes a priori knowledge abou conen populariy. Informaion-heoreic gains achieved by combining caching a user devices wih a coded mulicas ransmission in he delivery phase are calculaed in [7]. The proposed coded caching approach is opimal up o a consan facor. Conen caching a user devices and collaboraive device-o-device communicaion are combined in [8] o increase he efficiency of conen delivery. In [9], an approximaion algorihm for uncoded caching among SBSs equipped wih caches is given, which minimizes he average delay experienced by users ha can be conneced o several SBSs simulaneously. Building upon he same caching archiecure, in [10], an approximaion algorihm for disribued coded caching is presened for minimizing he probabiliy ha moving users have o reques pars of conen from he MBS insead of he SBSs. In [11], a mulicas-aware caching scheme is proposed for minimizing he energy consumpion in a small cell nework, in which he MBS and he SBSs can perform mulicas ransmissions. The ouage probabiliy and average conen delivery rae in a nework of SBSs equipped wih caches are analyically calculaed in [12]. Nex, we discuss relaed work on cache conen placemen wihou prior knowledge abou conen populariy. A comparison of he characerisics of our proposed algorihm wih relaed work of his ype is given in Table I. Driven by a proacive caching paradigm, [13], [14] propose a caching algorihm for small cell neworks based on collaboraive filering. Fixed global conen populariy is esimaed using a raining se and hen exploied for caching decisions o

3 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. MÜLLER e al.: CONTEXT-AWARE PROACTIVE CONTENT CACHING WITH SERVICE DIFFERENTIATION IN WIRELESS NETWORKS 3 TABLE I COMPARISON WITH RELATED WORK ON LEARNING-BASED CACHING WITH PLACEMENT AND DELIVERY PHASE. [13], [14] [15] [17] [18] [19] This work Model-Free Yes Yes No Yes Yes Online/Offline-Learning Offline Online Online Online Online Free of Training Phase No Yes Yes No Yes Performance Guaranees No Yes No No Yes Diversiy in Conen Populariy No No No Yes Yes User Conex-Aware No No No No Yes Service Differeniaion No No No No Yes maximize he average user reques saisfacion raio based on heir required delivery raes. While heir approach requires a raining se of known conen populariies and only learns during a raining phase, our proposed algorihm does no need a raining phase, bu learns he conen populariy online, hus also adaping o varying conen populariies. In [15], using a muli-armed bandi algorihm, an SBS learns a fixed conen populariy disribuion online by refreshing is cache conen and observing insananeous demands for cached files. In his way, cache conen placemen is opimized over ime o maximize he raffic served by he SBS. The auhors exend heir framework for a wireless infosaion in [16], [17] by addiionally aking ino accoun he coss for adding files o he cache. Moreover, hey provide heoreical sublinear regre bounds for heir algorihms. A differen exension of he muli-armed bandi framework is given in [18], which explois he opology of users connecions o he SBSs by incorporaing coded caching. The approach in [18] assumes a specific ype of conen populariy disribuion. Since in pracice he ype of disribuion is unknown a priori, such an assumpion is resricive. In conras, our proposed algorihm is model-free since i does no assume a specific ype of conen populariy disribuion. Moreover, in [15] [18], he opimal cache conen placemen sraegy is learned over ime based only on observaions of insananeous demands. In conras, our proposed algorihm addiionally akes diversiy of conen populariy across he user populaion ino accoun and explois users conex informaion. Diversiy in conen populariy across he user populaion is for example aken ino accoun in [19], bu again wihou considering he users conexs. Users are clusered ino groups of similar ineress by a specral clusering algorihm based on heir requess in a raining phase. Each user group is hen assigned o an SBS which learns he conen populariy of is fixed user group over ime. Hence, in [19], each SBS learns a fixed conen populariy disribuion under he assumpion of a sable user populaion, whereas our approach allows reacing o arbirary arrivals of users preferring differen conen. In summary, compared o relaed work on cache conen placemen (see Table I), our proposed algorihm for he firs ime joinly learns he conen populariy online, allows for diversiy in conen populariy across he user populaion, akes ino accoun he dependence of users preferences on heir conex and suppors service differeniaion. Compared o our previous work [1], we now ake ino accoun conex informaion a a single user level, insead of averaging conex informaion over he currenly conneced users. This enables more fine-grained learning. Addiionally, we incorporae service differeniaion and presen exensions, e.g., o mulicas ransmission and caching decisions based on user raings. We model he caching problem as a muli-armed bandi problem. Muli-armed bandi problems [35] have been applied o various scenarios in wireless communicaions before [36], such as cogniive jamming [37] or mobiliy managemen [38]. Our algorihm is based on conexual muli-armed bandi algorihms [39] [42]. The closes relaed work is [42], in which several learners observe a single conex arrival in each ime slo and selec a subse of acions o maximize he sum of expeced rewards. While [42] considers muliple learners, our sysem has only one learner he caching eniy selecing a subse of files o cache in each ime slo. Compared o [42], we exended he algorihm in he following direcions: We allow muliple conex arrivals in each ime slo, and selec a subse of acions which maximize he sum of expeced rewards given he conex arrivals. In he caching scenario, his ranslaes o observing he conexs of all currenly conneced users and caching a subse of files which maximize he sum of expeced numbers of cache his given he users conexs. In addiion, we enable each arriving conex o be annoaed wih a weigh, so ha if differen conexs arrive wihin he same ime slo, differeniaed services can be provided per conex, by selecing a subse of acions which maximize he sum of expeced weighed rewards. In he caching scenario, his enables he caching eniy o prioriize cerain users when selecing he cache conen, by placing more weigh on files ha are favored by prioriized users. Moreover, we enable each acion o be annoaed wih a weigh, such ha acions can be prioriized for selecion. In he caching scenario, his enables he caching eniy o prioriize cerain files when selecing he cache conen. III. SYSTEM MODEL A. Wireless Local Caching Eniy We consider a wireless local caching eniy ha can eiher be an SBS equipped wih a cache in a small cell nework or a wireless infosaion. The caching eniy is characerized by a limied sorage capaciy and a reliable backhaul link o he core nework. In is cache memory, he caching eniy can sore up o m files from a finie file library F conaining F N files, where we assume for simpliciy ha all files are of he same size. Users locaed in he coverage area can connec o he caching eniy. The se of currenly conneced

4 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. 4 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX MBS Mobile device requess file from MBS since file is no cached a caching eniy caching eniy Mobile device requess cached file from caching eniy TABLE II EXAMPLES OF CONTEXT DIMENSIONS. Class personal characerisics user equipmen exernal facors Conex Dimension demographic facors personaliy mood ype of device device capabiliies baery saus locaion ime of day, day of he week evens Fig. 1. Sysem model. users may change dynamically over ime due o he users mobiliy. A mos U max N users can be simulaneously conneced o he caching eniy. To inform conneced users abou available files, he caching eniy periodically broadcass he informaion abou he curren cache conen [15] [17]. If a user is ineresed in a file ha he caching eniy sored in is cache, he user s device requess he file from he caching eniy and is served via localized communicaion. In his case, no addiional load is pu on neiher he macro cellular nework nor he backhaul nework. If he file is no sored in he caching eniy, he user s device does no reques he file from he caching eniy. Insead, i requess he file from he macro cellular nework by connecing o an MBS. The MBS downloads he file from he core nework via is backhaul connecion, such ha in his case, load is pu on boh he macro cellular as well as he backhaul nework. Hence, he caching eniy can only observe requess for cached files, i.e., cache his, bu i canno observe requess for non-cached files, i.e., cache misses. Noe ha his resricion is specific o wireless caching and is usually no used in wired caching scenarios. In his way, he caching eniy is no congesed by cache misses [15] [17], bu learning conen populariy is more difficul. Fig. 1 shows an illusraion of he considered sysem model. In order o reduce he load on he macro cellular nework and he backhaul nework, a caching eniy migh aim a opimizing he cache conen such ha he raffic i can serve is maximized, which corresponds o maximizing he number of cache his. For his purpose, he caching eniy should learn which files are mos popular over ime. B. Service Differeniaion Maximizing he number of cache his migh be an adequae goal of cache conen placemen in case of an MNO operaing an SBS, one reason being ne neuraliy resricions. However, he operaor of an infosaion, e.g., a CP or hird pary operaor, may wan o provide differeniaed services o is cusomers (hose can be boh users and CPs). For example, if users wih differen preferences are conneced o an infosaion, he operaor can prioriize cerain users by caching conen favored by hese users. In his case, a cache hi by a prioriized user is associaed wih a higher value han a cache hi by a regular user. For his purpose, we consider a finie se S of service ypes. For service ype s S, le v s 1 denoe a fixed and known weigh associaed wih receiving one cache hi by a user of service ype s. Le v max := max s S v s. The weighs migh be seleced based on a pricing policy, e.g., by paying a monhly fee, a user can buy a higher weigh. Alernaively, he weighs migh be seleced based on a subscripion policy, e.g., subscribers migh obain prioriy compared o one-ime users. Ye anoher prioriizaion migh be based on he imporance of users in erms of adverisemen or heir influence on he operaor s repuaion. Finally, prioriizaion could be based on usage paerns, e.g., users migh indicae heir degree of openness in exploring oher han heir mos preferred conen. Taking ino accoun he service weighs, he caching eniy s goal becomes o maximize he number of weighed cache his. Clearly, he above service differeniaion only akes effec if users wih differen preferences are presen, i.e., if conen populariy is heerogeneous across he user populaion. Anoher service differeniaion can be applied in case of a hird pary operaor whose cusomers are differen CPs. The operaor may wan o prioriize cerain CPs by caching heir conen. In his case, each conen is associaed wih a weigh. Here, we consider a fixed and known prioriizaion weigh w f 1 for each file f F and le w max := max f F w f. The prioriizaion weighs can eiher be chosen individually for each file or per CP. The case wihou service differeniaion, where he goal is o maximize he number of (non-weighed) cache his, is a special case, in which here is only one service ype s wih weigh v s = 1 and he prioriizaion weighs saisfy w f = 1 for all f F. While we refer o he more general case in he subsequen secions, his special case is naurally conained in our analysis. C. Conex-Specific Conen Populariy Conen populariy may vary across a user populaion since differen users may prefer differen conen. A user s preferences migh be linked o various facors. We refer o such facors as conex dimensions and give some examples in Table II. Relevan personal characerisics may, for example, be demographic facors (e.g., age, gender), personaliy, or mood. In addiion, a user s preferences may be influenced by user equipmen, such as he ype of device used o access and consume he conen (e.g., smar phone, able), as well as is capabiliies, or is baery saus. Besides, exernal facors may have an impac on a user s preferences, such as he user s

5 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. MÜLLER e al.: CONTEXT-AWARE PROACTIVE CONTENT CACHING WITH SERVICE DIFFERENTIATION IN WIRELESS NETWORKS 5 Sorage Servers Upon Reques Periodically 8 8 Sorage Inerface Users Cache Managemen 9 8 Cache Conroller Reques Handler User Inerface Caching Eniy Local Cache Fig. 2. Conex-aware proacive caching archiecure Learning Module Learning Daabase Decision Engine 1 Conex Daabase 3 Conex Monior 2 6 Exernal Informaion locaion, he ime of day, he day of he week, and he aking place of evens (e.g., soccer mach, concer). Clearly, his caegorizaion is no exhausive and he impac of each single conex dimension on conen populariy is unknown a priori. Moreover, a caching eniy may only have access o some of he conex dimensions, e.g., due o privacy reasons. However, our model does no rely on specific conex dimensions; i can use he informaion ha is colleced from he user. If he caching eniy does have access o some relevan conex dimensions, hese can be exploied o learn conex-specific conen populariy. D. Conex-Aware Proacive Caching Archiecure Nex, we describe he archiecure for conex-aware proacive caching, which is designed similarly o an archiecure presened in [33]. An illusraion of he conex-aware proacive caching archiecure is given in Fig. 2. Is main building blocks are he Local Cache, a Cache Managemen eniy, a Learning Module, a Sorage Inerface, a User Inerface, and a Conex Monior. The Cache Managemen consiss of a Cache Conroller and a Reques Handler. The Learning Module conains a Decision Engine, a Learning Daabase, and a Conex Daabase. The workflow consiss of several phases as enumeraed in Fig. 2 and is described below. Iniializaion (1) The Learning Module is provided wih he goal of caching (i.e., maximize number of cache his or achieve operaor-specific goal). I fixes he appropriae periodiciy of conex monioring and cache refreshmen. Then, i informs he Cache Managemen and he Conex Monior abou he periodiciy. Periodic Conex Monioring and Cache Refreshmen (2) The Conex Monior periodically gahers conex informaion by accessing informaion abou currenly conneced users available a he User Inerface and opionally by collecing addiional informaion from exernal sources (e.g., social media plaforms). If differen service ypes exis, he Conex Monior also rerieves he service ypes of conneced users. (3) The Conex 4 Fig. 3. slo. observe user conexs, service ypes selec cache conen observe cache his +1 ime slo Sequence of operaions of conex-aware proacive caching in ime Monior delivers he gahered informaion o he Conex Daabase in he Learning Module. (4) The Decision Engine periodically exracs he newly moniored conex informaion from he Conex Daabase. (5) Upon comparison wih resuls from previous ime slos as sored in he Learning Daabase, (6) he Decision Engine decides which files o cache in he coming ime slo. (7) The Decision Engine insrucs he Cache Conroller o refresh he cache conen accordingly. (8) The Cache Conroller compares he curren and he required cache conen and removes non-required conen from he cache. If some required conen is missing, he Cache Conroller direcs he Sorage Inerface o fech he conen from sorage servers and o sore i ino he local cache. (9) Then, he Cache Conroller informs he User Inerface abou he new cache conen. (10) The User Inerface pushes he informaion abou new cache conen o currenly conneced users. User Requess (11) When a user requess a cached file, he User Inerface forwards he reques o he Reques Handler. The Reques Handler sores he reques informaion, rerieves he file from he local cache and serves he reques. Periodic Learning (12) Upon compleion of a ime slo, he Reques Handler hands he informaion abou all requess from ha ime slo o he Learning Module. The Learning Module updaes he Learning Daabase wih he conex informaion from he beginning of he ime slo and he number of requess for cached files in ha ime slo. E. Formal Problem Formulaion Nex, we give a formal problem formulaion for conexaware proacive caching. The caching sysem operaes in discree ime slos = 1, 2,..., T, where T denoes he finie ime horizon. As illusraed in Fig. 3, each ime slo consiss of he following sequence of operaions: (i) The conex of currenly conneced users and heir service ypes are moniored. Le U be he number of currenly conneced users. We assume ha 1 U U max and we specifically allow he se of currenly conneced users o change in beween he ime slos of he algorihm, so ha user mobiliy is aken ino accoun. Le D be he number of moniored conex dimensions per user. We denoe he D-dimensional conex space by X. I is assumed o be bounded and can hence be se o X := [0, 1] D wihou loss of generaliy. Le x,i X be he conex vecor of user i observed in ime slo. Le x = (x,i ) i=1,...,u be he collecion of conexs of all users in ime slo. Le s,i S be he service ype of user i

6 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. 6 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX in ime slo and le s = (s,i ) i=1,...,u be he collecion of service ypes of all users in ime slo. (ii) The cache conen is refreshed based on he conexs x, he service ypes s and heir service weighs, he file prioriizaion weighs w f, f F, and knowledge from previous ime slos. Then, conneced users are informed abou he curren cache conen, which is denoed by C = {c,1,..., c,m }. (iii) Unil he end of he ime slo, users can reques currenly cached files. Their requess are served. The demand d c,j (x,i, ) of each user i = 1,..., U for each cached file c,j C, j = 1,..., m, in his ime slo is observed, i.e., he number of cache his for each cached file is moniored. The number of imes a user wih conex vecor x X requess a file f F wihin one ime slo is a random variable wih unknown disribuion. We denoe his random demand by d f (x) and is expeced value by µ f (x) := E(d f (x)). The random demand is assumed o ake values in [0, R max ], where R max N is he maximum possible number of requess a user can submi wihin one ime slo. This explicily incorporaes ha a user may reques he same file repeaedly wihin one ime slo. In ime slo, he random variables (d f (x,i )) i=1,..,u,f F, are assumed o be independen, i.e., he requess of currenly conneced users and beween differen files are independen of each oher. Moreover, each random variable d f (x,i ) is assumed o be independen of pas caching decisions and previous demands. The goal of he caching eniy is o selec he cache conen in order o maximize he expeced cumulaive number of (weighed) cache his up o he finie ime horizon T. We inroduce a binary variable y,f, which describes if file f is cached in ime slo, where y,f = 1, if f C, and 0 oherwise. Then, he problem of cache conen placemen can be formally wrien as max T U y,f w f v s,i µ f (x,i ) (1) =1 f F i=1 s.. y,f m, f F = 1,..., T, y,f {0, 1}, f F, = 1,..., T. Le us now firs assume ha he caching eniy had a priori knowledge abou conex-specific conen populariy like an omniscien oracle, i.e., suppose ha for each conex vecor x X and for each file f F, he caching eniy would know he expeced demand µ f (x) = E(d f (x)). In his case, problem (1) corresponds o an ineger linear programming problem. The problem can be decoupled ino T independen sub-problems, one for each ime slo. Each sub-problem is a special case of he knapsack problem [43] wih a knapsack of capaciy m and wih iems of non-negaive profi and uni weighs. Hence, is opimal soluion can be easily compued in a running ime of O( F log( F )) as follows. In ime slo, given he conexs x and he service ypes s, he opimal soluion is given by ranking he files according o heir (weighed) expeced demands and by selecing he m highes ranked files. We denoe hese op-m files for pair (x, s ) by f 1 (x, s ), f 2 (x, s ),..., f m(x, s ) F. Formally, for j = 1,..., m, hey saisfy 1 f j (x, s ) argmax f F \( j 1 k=1 {f k (x,s)}) w f U i=1 v s,i µ f (x,i ), (2) where 0 k=1 {f k (x, s )} :=. We denoe by C (x, s ) := m k=1 {f k (x, s )} an opimal choice of files o cache in ime slo. Consequenly, he collecion (C (x, s )) =1,...,T (3) is an opimal soluion o problem (1). Since his soluion can be achieved by an omniscien oracle under a priori knowledge abou conen populariy, we call i he oracle soluion. However, in his paper we assume ha he caching eniy does no have a priori knowledge abou conen populariy. In his case, he caching eniy canno simply solve problem (1) as described above, since he expeced demands µ f (x) = E(d f (x)) are unknown. Hence, he caching eniy has o learn hese expeced demands over ime by observing he users demands for cached files given he users conexs. For his purpose, over ime, he caching eniy has o find a radeoff beween caching files abou which lile informaion is available (exploraion) and files of which i believes ha hey will yield he highes demands (exploiaion). In each ime slo, he choice of files o be cached depends on he hisory of choices in he pas and he corresponding observed demands. An algorihm which maps he hisory o he choices of files o cache is called a learning algorihm. The oracle soluion given in (3) can be used as a benchmark o evaluae he loss of learning. Formally, he regre of learning wih respec o he oracle soluion is given by R(T ) = T m U ( ) v s,i (w f j (x,s )E d f j (x,s )(x,i ) =1 j=1 i=1 E ( w c,j d c,j (x,i, ) )), where d c,j (x,i, ) denoes he random demand for he cached file c,j C of user i wih conex vecor x,i a ime. Here, he expecaion is aken wih respec o he choices made by he learning algorihm and he disribuions of he demands. IV. A CONTEXT-AWARE PROACTIVE CACHING ALGORITHM In order o proacively cache he mos suiable files given he conex informaion abou currenly conneced users, he caching eniy should learn conex-specific conen populariy. Due o he above formal problem formulaion, his problem corresponds o a conexual muli-armed bandi problem and we can adap and exend a conexual learning algorihm [41], [42] o our seing. Our algorihm is based on he assumpion ha users wih similar conex informaion will reques similar files. If his naural assumpion holds rue, he users conex 1 Several files may have he same expeced demands, i.e., he opimal se of files may no be unique. This is also capured here. (4)

7 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. MÜLLER e al.: CONTEXT-AWARE PROACTIVE CONTENT CACHING WITH SERVICE DIFFERENTIATION IN WIRELESS NETWORKS 7 informaion ogeher wih heir requess for cached files can be exploied o learn for fuure caching decisions. For his purpose, our algorihm sars by pariioning he conex space uniformly ino smaller ses, i.e., i splis he conex space ino pars of similar conexs. Then, he caching eniy learns he conen populariy independenly in each of hese ses of similar conexs. The algorihm operaes in discree ime slos. In each ime slo, he algorihm firs observes he conexs of currenly conneced users. Then, he algorihm selecs which files o cache in his ime slo. Based on a cerain conrol funcion, he algorihm is eiher in an exploraion phase, in which i chooses a random se of files o cache. Theses phases are needed o learn he populariy of files which have no been cached ofen before. Oherwise, he algorihm is in an exploiaion phase, in which i caches files which on average were requesed mos when cached in previous ime slos wih similar user conexs. Afer caching he new se of files, he algorihm observes he users requess for hese files. In his way, over ime, he algorihm learns conex-specific conen populariy. The algorihm for selecing m files is called Conex- Aware Proacive Caching wih Cache Size m (m-cac) and is pseudocode is given in Fig. 4. Nex, we describe he algorihm in more deail. In is iniializaion phase, m-cac creaes a pariion P T of he conex space X = [0, 1] D ino (h T ) D ses, ha are given by D-dimensional hypercubes of idenical size 1 h T... 1 h T. Here, h T is an inpu parameer which deermines he number of ses in he pariion. Addiionally, m- CAC keeps a couner N f,p () for each pair consising of a file f F and a se p P T. The couner N f,p () is he number of imes in which file f F was cached afer a user wih conex from se p was conneced o he caching eniy up o ime slo (i.e., if 2 users wih conex from se p are conneced in one ime slo and file f is cached, his couner is increased by 2). Moreover, m-cac keeps he esimaed demand ˆd f,p () up o ime slo of each pair consising of a file f F and a se p P T. This esimaed demand is calculaed as follows: Le E f,p () be he se of observed demands of users wih conex from se p when file f was cached up o ime slo. Then, he esimaed demand of file f in se p is given by he sample mean ˆd f,p () := 1 E f,p () d E f,p () d.2,3 In each ime slo, m-cac firs observes he number of currenly conneced users U, heir conexs x = (x,i ) i=1,...,u and he service ypes s = (s,i ) i=1,...,u. For each conex vecor x,i, m-cac deermines he se p,i P T, o which he conex vecor belongs, i.e., such ha x,i p,i holds. The collecion of hese ses is given by p = (p,i ) i=1,...,u. Then, he algorihm can eiher be in an exploraion phase or in an exploiaion phase. In order o deermine he correc phase in he curren ime slo, he algorihm checks if here are files ha have no been explored sufficienly ofen. For his purpose, 2 The se E f,p () does no have o be sored since he esimaed demand ˆd f,p () can be updaed based on ˆd f,p ( 1), N f,p ( 1) and on he observed demands a ime. 3 Noe ha in he pseudocode in Fig. 4, he argumen is dropped from couners N f,p () and ˆd f,p () since previous values of hese couners do no have o be sored. m-cac: Conex-Aware Proacive Caching Algorihm 1: Inpu: T, h T, K() 2: Iniialize conex pariion: Creae pariion P T of conex space [0, 1] D ino (h T ) D hypercubes of idenical size 3: Iniialize couners: For all f F and all p P T, se N f,p = 0 4: Iniialize esimaed demands: For all f F and all p P T, se ˆd f,p = 0 5: for each = 1,..., T do 6: Observe number U of currenly conneced users 7: Observe user conexs x = (x,i ) i=1,...,u and service ypes s = (s,i ) i=1,...,u 8: Find p = (p,i ) i=1,...,u such ha x,i p,i P T, i = 1,..., U 9: Compue he se of under-explored files Fp ue () in (5) 10: if Fp ue () hen Exploraion 11: u = size(fp ue ()) 12: if u m hen 13: Selec c,1,..., c,m randomly from Fp ue () 14: else 15: Selec c,1,..., c,u as he u files from Fp ue () 16: Selec c,u+1,..., c,m as he (m u) files ˆf 1,p,s (),..., ˆf m u,p,s () from (6) 17: end if 18: else Exploiaion 19: Selec c,1,..., c,m as he m files ˆf 1,p,s (),..., ˆf m,p,s () from (7) 20: end if 21: Observe demand (d j,i ) of each user i = 1,..., U for each file c,j, j = 1,..., m 22: for i = 1,..., U do 23: for j = 1,..., m do 24: ˆdc,j,p,i = N c,j,p,i = N c,j,p,i : end for 26: end for 27: end for Fig. 4. Pseudocode of m-cac. ˆd c,j,p,i N c,j,p,i +d j,i N c,j,p,i +1 he se of under-explored files F ue p () is calculaed based on F ue p () := U i=1 F ue p,i () and := U i=1 {f F : N f,p,i () K()}, (5) where K() is a deerminisic, monoonically increasing conrol funcion, which is an inpu o he algorihm. The conrol funcion has o be se adequaely o balance he rade-off beween exploraion and exploiaion. In Secion V, we will selec a conrol funcion ha guaranees a good balance in erms of his rade-off. If he se of under-explored files is non-empy, m-cac eners he exploraion phase. Le u() be he size of he se of under-explored files. If he se of under-explored files conains a leas m elemens, i.e., u() m, he algorihm randomly selecs m files from Fp ue () o cache. If he se of under-explored files conains less han m elemens, i.e., u() < m, i selecs all u() files from Fp ue () o cache. Since he cache is no fully

8 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. 8 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX filled by u() < m files, (m u()) addiional files can be cached. In order o exploi knowledge obained so far, m-cac selecs (m u()) files from F \Fp ue () based on a file ranking according o he esimaed weighed demands, as defined by he files ˆf 1,p,s (),..., ˆf m u(),p,s () F \ Fp ue (), which saisfy for j = 1,..., m u(): ˆf j,p,s () U argmax w f j 1 f F \(Fp ue () { ˆf k,p,s ()}) i=1 k=1 v s,i ˆdf,p,i (). If he se of files defined by (6) is no unique, ies are broken arbirarily. Noe ha by his procedure, even in exploraion phases, he algorihm addiionally explois, whenever he number of under-explored files is smaller han he cache size. If he se of under-explored files Fp ue () is empy, m-cac eners he exploiaion phase. I selecs m files from F based on a file ranking according o he esimaed weighed demands, as defined by he files ˆf 1,p,s (),..., ˆf m,p,s () F, which saisfy for j = 1,..., m: ˆf j,p,s () argmax w f f F \( j 1 k=1 { ˆf k,p,s ()}) U i=1 v s,i ˆdf,p,i (). If he se of files defined by (7) is no unique, ies are again broken arbirarily. Afer selecing he subse of files o cache, he algorihm observes he users requess for hese files in his ime slo. Then, i updaes he esimaed demands and he couners of cached files. V. ANALYSIS OF THE REGRET In his secion, we give an upper bound on he regre R(T ) of m-cac in (4). The regre bound is based on he naural assumpion ha expeced demands for files are similar in similar conexs, i.e., ha users wih similar characerisics are likely o consume similar conen. This assumpion is realisic since he users preferences in erms of consumed conen differ based on he users conexs, so ha i is plausible o divide he user populaion ino segmens of users wih similar conex and similar preferences. Formally, he similariy assumpion is capured by he following Hölder condiion. Assumpion 1. There exiss L > 0, α > 0 such ha for all f F and for all x, y X, i holds ha µ f (x) µ f (y) L x y α, where denoes he Euclidean norm in R D. Assumpion 1 is needed for he analysis of he regre, bu i should be noed ha m-cac can also be applied if his assumpion does no hold rue. However, a regre bound migh no be guaraneed in his case. The following heorem shows ha he regre of m-cac is sublinear in he ime horizon T, i.e., R(T ) = O(T γ ) wih γ < 1. This bound on he regre guaranees ha he (6) (7) algorihm has an asympoically opimal performance, since R(T ) hen lim T T = 0 holds. This means, ha m-cac converges o he oracle soluion sraegy. In oher words, m-cac converges o he opimal cache conen placemen sraegy, which maximizes he expeced number of cache his. In deail, he regre of m-cac can be bounded as follows for any finie ime horizon T. Theorem 1 (Bound for R(T )). Le K() = 2α 3α+D log() and h T = T 1 3α+D. If m-cac is run wih hese parameers and Assumpion 1 holds rue, he leading order of he regre R(T ) is O (v max w max mu max R max F T 2α+D 3α+D log(t ) ). The proof can be found in our online appendix [44]. The regre bound given in Theorem 1 is sublinear in he ime horizon T, proving ha m-cac converges o he opimal cache conen placemen sraegy. Addiionally, Theorem 1 is applicable for any finie ime horizon T, such ha i provides a bound on he loss incurred by m-cac for any finie number of cache placemen phases. Thus, Theorem 1 characerizes m-cac s speed of convergence Furhermore, Theorem 1 shows ha he regre bound is a consan muliple of he regre bound in he special case wihou service differeniaion, in which v max = 1 and w max ) = 1. Hence, he order of he regre is O (T 2α+D 3α+D log(t ) in he special case as well. VI. MEMORY REQUIREMENTS The memory requiremens of m-cac are mainly deermined by he couners kep by he algorihm during is runime (see also [41]). For each se p in he pariion P T and each file f F, he algorihm keeps he couners N f,p and ˆd f,p. The number of files is F. If m-cac runs wih he parameers from Theorem 1, he number of ses in P T is upper bounded by (h T ) D = T 1 3α+D D 2 D T D 3α+D. Hence, he required memory is upper bounded by F 2 D T D 3α+D and is hus sublinear in he ime horizon T. This means, ha for T, he algorihm would require infinie memory. However, for pracical approaches, only he couners of such ses p have o be kep o which a leas one of he conneced users conex vecors has already belonged o. Hence, depending on he heerogeneiy in he conneced users conex vecors, he required number of couners ha have o be kep can be much smaller han given by he upper bound. VII. EXTENSIONS A. Exploiing he Mulicas Gain So far, we assumed ha each reques for a cached file is immediaely served by a unicas ransmission. However, our algorihm can be exended o mulicasing, which has been shown o be beneficial in combinaion wih caching [7], [11]. For his purpose, o exend our algorihm, each ime slo is divided ino a fixed number of inervals. In each inerval, incoming requess are moniored and accumulaed. A he end of he inerval, requess for he same file are served by a mulicas ransmission. In order o exploi knowledge abou conen populariy learned so far, a reques for a file wih low esimaed demand could, however, sill be served

9 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. MÜLLER e al.: CONTEXT-AWARE PROACTIVE CONTENT CACHING WITH SERVICE DIFFERENTIATION IN WIRELESS NETWORKS 9 by a unicas ransmission. In his way, unnecessary delays are prevened in cases in which anoher reques and hus a mulicas ransmission are no expeced. Moreover, service differeniaion could be aken ino accoun. For example, highprioriy users could be served by unicas ransmissions, such ha heir delay is no increased due o waiing imes for mulicas ransmissions. B. Raing-Based Conex-Aware Proacive Caching So far, we considered cache conen placemen wih respec o he demands d f (x) in order o maximize he number of (weighed) cache his. However, a CP operaing an infosaion migh wan o cache no only conen ha is requesed ofen, bu which also receives high raings from he users. Consider he case ha afer consumpion users rae conen in a range [r min, r max ] R +. For a conex x, le r f (x) be he random variable describing he raing of a user wih conex x if he requess file f and makes a raing hereafer. Then, we define he random variable d f (x) := r f (x)d f (x), (8) which combines he demand and he raing for file f of a user wih conex x. By carefully designing he range of raings, he CP chooses he rade-off beween raings and cache his. Now, we can apply m-cac wih respec o d f (x). In his case, m-cac addiionally needs o observe he users raings in order o learn conen populariy in erms of raings. If he users raings are always available, Theorem 1 applies and provides a regre bound of O (v max w max r max mu max R max F T 2α+D 3α+D log(t ) ). However, users migh no always reveal a raing afer consuming a conen. When a user s raing is missing, we assume ha m-cac does no updae he couners based on his user s reques. This may resul in a higher required number of exploraion phases. Hence, he regre of he learning algorihm is influenced by he users willingness o reveal raings of requesed conen. Le q (0, 1) be he probabiliy ha a user reveals his raing afer requesing a file. Then, he regre of he learning algorihm is bounded as given below. Theorem 2 (Bound for R(T ) for raing-based caching wih missing raings). Le K() = 2α 3α+D log() and h T = T 1 3α+D. If m-cac is run wih hese parameers wih respec o d f (x), Assumpion 1 holds rue for df (x) and a user reveals his raing wih probabiliy ( q, he leading order of he regre R(T ) is 1 O q v maxw max r max mu max R max F T 2α+D 3α+D log(t ) ). The proof can be found in our online appendix [44]. Comparing Theorem 2 wih Theorem 1, he regre of m-cac is scaled up by a facor 1 q > 1 in case of raing-based caching wih missing raings. This facor corresponds o he expeced number of requess unil he caching eniy receives one raing. However, he ime order of he regre remains he same. Hence, m-cac is robus under missing raings in he sense ha if some users refuse o rae requesed conen, he algorihm sill converges o he opimal cache conen placemen sraegy. C. Asynchronous User Arrival So far, we assumed ha he se of currenly conneced users only changes in beween he ime slos of our algorihm. This means, ha only hose users conneced o he caching eniy a he beginning of a ime slo, will reques files wihin ha ime slo. However, if users connec o he caching eniy asynchronously, m-cac should be adaped. If a user direcly disconnecs afer he conex monioring wihou requesing any file, he should be excluded from learning. Hence, in m- CAC, he couners are no updaed for disconnecing users. If a user connecs o he caching eniy afer cache conen placemen, his conex was no considered in he caching decision. However, his requess can be used o learn faser. Hence, in m-cac, he couners are updaed based on his user s requess. D. Muliple Wireless Local Caching Eniies So far, we considered online learning for cache conen placemen in a single caching eniy. However, real caching sysems conain muliple caching eniies, each of which should learn local conen populariy. In a nework of muliple caching eniies, m-cac could be applied separaely and independenly by each caching eniy. For he case ha coverage areas of caching eniies overlap, in his subsecion, we presen m-cacao, an exension of m-cac o Conex- Aware Proacive Caching wih Area Overlap. The idea of m- CACao is ha caching eniies can learn conen populariy faser by no only relying on heir own cache his, bu also on cache his a neighboring caching eniies wih overlapping coverage area. For his purpose, he caching eniies overhear cache his produced by users in he inersecion o neighboring coverage areas. In deail, m-cac is exended o m-cacao as follows: The conex monioring and he selecion of cache conen works as in m-cac. However, he caching eniy no only observes is own cache his (line 21 in Fig. 4), bu i overhears cache his a neighboring caching eniies of users in he inersecion. Then, he caching eniy no only updaes he couners of is own cached files (lines in Fig. 4), bu i addiionally updaes he couners of files of which i overheard cache his a neighboring caches. This helps he caching eniy o learn faser. VIII. NUMERICAL RESULTS In his secion, we numerically evaluae he proposed learning algorihm m-cac by comparing is performance o several reference algorihms based on a real world daa se. A. Descripion of he Daa Se We use a daa se from MovieLens [45] o evaluae our proposed algorihm. MovieLens is an online movie recommender operaed by he research group GroupLens from he Universiy of Minnesoa. The MovieLens 1M DaaSe [46] conains raings of 3952 movies. These raings were made by 6040 users of MovieLens wihin he years 2000 o Each daa se enry consiss of an anonymous user ID,

10 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. 10 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX number of conen requess ime slo Fig. 5. Number of conen requess in used daa se as a funcion of ime slos. Time slos a an hourly basis. a movie ID, a raing (in whole numbers beween 1 and 5) and a imesamp. Addiionally, demographic informaion abou he users is given: Their gender, age (in 7 caegories), occupaion (in 20 caegories) as well as heir Zip-code. For our numerical evaluaions, we assume ha he movie raing process in he daa se corresponds o a conen reques process of users conneced o a wireless local caching eniy (see [33], [34] for a similar approach). Hence, a user raing a movie a a cerain ime in he daa se for us corresponds o a reques o eiher he caching eniy (in case he movie is cached in he caching eniy) or o he macro cellular nework (in case he movie is no cached in he caching eniy). This approach is reasonable since users ypically rae movies afer waching hem. In our simulaions, we only use he daa gahered wihin he firs year of he daa se, since around 94% of he raings were provided wihin his ime frame. Then, we divide a year s ime ino 8760 ime slos of one hour each (T = 8760), assuming ha he caching eniy updaes is cache conen a an hourly basis. Then, we assign he requess and corresponding user conexs o he ime slos according o heir imesamps and we inerpre each reques as if i was coming from a separae user. A he beginning of a ime slo, we assume o have access o he conex of each user responsible for a reques in he coming ime slo. Fig. 5 shows ha he corresponding conen reques process is bursy and flaens ou owards he end. As conex dimensions, we selec he dimensions gender and age. 4 B. Reference Algorihms We compare m-cac wih five reference algorihms. The firs algorihm is he omniscien Oracle, which has complee knowledge abou he exac fuure demands. In each ime slo, he oracle selecs he opimal m files ha will maximize he number of cache his in his ime slo. 5 4 We neglec he occupaion as conex dimension since by mapping hem o a [0,1] variable, we would have o classify which occupaions are more and which are less similar o each oher. 5 Noe ha his oracle yields even beer resuls han he oracle used as a benchmark o define he regre in (4). In he definiion of regre, he oracle only explois knowledge abou expeced demands, insead of exac fuure demands. The second reference algorihm is called m-ucb, which consiss of a varian of he UCB algorihm. UCB is a classical learning algorihm for muli-armed bandi problems [35], which has logarihmic regre order. However, i does no ake ino accoun conex informaion, i.e., he logarihmic regre is wih respec o he average expeced demand over he whole conex space. While in classical UCB, one acion is aken in each ime slo, we modify UCB o ake m acions a a ime, which corresponds o selecing m files. The hird reference algorihm is he m-ɛ-greedy. This is a varian of he simple ɛ-greedy [35] algorihm, which does no consider conex informaion. The m-ɛ-greedy caches a random se of m files wih probabiliy ɛ (0, 1). Wih probabiliy (1 ɛ), he algorihm caches he m files wih highes o m-h highes esimaed demands. These esimaed demands are calculaed based on previous demands for cached files. The fourh reference algorihm is called m-myopic. This is an algorihm aken from [15], which is invesigaed since i is comparable o he well-known Leas Recenly Used algorihm (LRU) for caching. m-myopic only learns from one ime slo in he pas. I sars wih a random se of files and in each of he following ime slos discards all files ha have no been requesed in he previous ime slo. Then, i randomly replaces he discarded files by oher files. The fifh reference algorihm, called Random, caches a random se of files in each ime slo. C. Performance Measures The following performance measures are used in our analysis. The evoluion of per-ime slo or cumulaive number of cache his allows comparing he absolue performance of he algorihms. A relaive performance measure is given by he cache efficiency, which is defined as he raio of cache his compared o he overall demand, i.e., cache his cache efficiency in % = cache his + cache misses 100. The cache efficiency describes he percenage of requess which can be served by cached files. D. Resuls In our simulaions, we se ɛ = 0.09 in m-ɛ-greedy, which is he value a which heurisically he algorihm on average performed bes. In m-cac, we se he conrol funcion o K() = c 2α 3α+D log() wih c = 1/( F D). 6 The simulaion resuls are obained by averaging over 100 runs of he algorihms. Firs, we consider he case wihou service differeniaion. The long-erm behavior of m-cac is invesigaed wih he following scenario. We assume ha he caching eniy can sore m = 200 movies ou of he F = 3952 available movies. Hence, he cache size corresponds o abou 5% of he file library size. We run all algorihms on he daa se and sudy heir resuls as a funcion of ime, i.e., over he ime slos = 1,..., T. Fig. 6(a) and 6(b) show he per-ime slo and he 6 Compared o he conrol funcion in Theorem 1, he addiional facor reduces he number of exploraion phases which allows for beer performance.

11 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. MÜLLER e al.: CONTEXT-AWARE PROACTIVE CONTENT CACHING WITH SERVICE DIFFERENTIATION IN WIRELESS NETWORKS 11 number of cache his Oracle m-cac m-ucb ǫ-m-greedy m-myopic Random overall cache efficiency in % Oracle m-cac m-ucb ǫ-m-greedy m-myopic Random ime slo (a) Number of cache his per ime slo cache size m Fig. 7. Overall cache efficiency a T as a funcion of cache size m. cumulaive number of cache his Oracle 6 m-cac m-ucb 5 ǫ-m-greedy m-myopic 4 Random ime slo (b) Cumulaive number of cache his. Fig. 6. Time evoluion of algorihms for m = 200. cumulaive numbers of cache his up o ime slo as a funcion of ime, respecively. Due o he bursy conen reques process (compare Fig. 5), also he number of cache his achieved by he differen algorihms is bursy over ime. As expeced, he Oracle gives an upper bound o he oher algorihms. Among he oher algorihms, m-cac, m-ɛ-greedy and m-ucb clearly ouperform m-myopic and Random. This is due o he fac ha hese hree algorihms learn from he hisory of observed demands, while m-myopic only learns from one ime slo in he pas and Random does no learn a all. I can be observed ha m-ɛ-greedy shows a beer performance han m-ucb, even hough i uses a simpler learning sraegy. Overall, m- CAC ouperforms he oher algorihms by addiionally learning from conex informaion. A he ime horizon, he cumulaive number of cache his achieved by m-cac is 1.146, 1.377, and imes higher han he ones achieved by m-ɛ- Greedy, m-ucb, m-myopic and Random, respecively. Nex, we invesigae he impac of he cache size m by varying i beween 50 and 400 files, which corresponds o beween 1.3% and 10.1% of he file library size, which is a realisic assumpion. All remaining parameers are kep as before. Fig. 7 shows he overall cache efficiency achieved a he ime horizon T as a funcion of cache size, i.e., he cumulaive number of cache his up o T is normalized by he cumulaive number of requess up o T. The overall cache efficiency of all algorihms is increasing wih increasing cache size. Moreover, he resuls indicae ha again m-cac and m- ɛ-greedy slighly ouperform m-ucb and clearly ouperform m-myopic and Random. Averaged over he range of cache sizes, he cache efficiency of m-cac is 28.4%, compared o an average cache efficiency of 25.3%, 21.4%, 7.76% and 5.69% achieved by m-ɛ-greedy, m-ucb, m-myopic and Random, respecively. Now, we consider a case of service differeniaion, in which wo differen service ypes 1 and 2 wih weighs v 1 = 5 and v 2 = 1 exis. Hence, service ype 1 should be prioriized due o he higher value i represens. We randomly assign 10% of he users o service ype 1 and classify all remaining users as service ype 2. Then, we adjus each algorihm o ake ino accoun service differeniaion by incorporaing he weighs according o he service ypes. Fig. 8 shows he cumulaive number of weighed cache his up o ime slo as a funcion of ime. A he ime horizon, he cumulaive number of weighed cache his achieved by m-cac is 1.156, 1.219, and imes higher han he ones achieved by m-ɛ-greedy, m-ucb, m-myopic and Random, respecively. A comparison wih Fig. 6(b) shows ha he behavior of he algorihms is similar o he case wihou service differeniaion. Finally, we invesigae he exension o muliple caching eniies and compare he performance of he proposed algorihms m-cac and m-cacao. We consider a scenario wih wo caching eniies and divide he daa se as follows: A fracion o [0, 0.3] of randomly seleced requess is considered o be made in he inersecion of he wo coverage areas. We use he parameer o as a measure of he overlap beween he caching eniies. The remaining requess are randomly assigned o eiher one of he caching eniies. These requess are considered o be made by users solely conneced o one caching eniy. Then, on he one hand we run m-cac separaely on each caching eniy and on he oher hand we run m-cacao on boh caching eniies. Fig. 9 shows he cumulaive number of cache his achieved in sum by he wo caching eniies a

12 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. 12 IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX cumulaive number of weighed cache his Oracle 12 m-cac m-ucb 10 ǫ-m-greedy m-myopic 8 Random ime slo Fig. 8. Cumulaive number of weighed cache his for m = 200 as a funcion of ime. cumulaive number of cache his a T m-cacao 2.75 m-cac overlap parameer o Fig. 9. Cumulaive number of cache his a T as a funcion of he overlap parameer o. he ime horizon T as a funcion of he overlap parameer o. As expeced, m-cac and m-cacao perform idenically for non-overlapping coverage areas. Wih increasing overlap, he number of cache his achieved by boh m-cac and m-cacao increases. The reason is ha users in he inersecion can more likely be served since hey have access o boh caches. Hence, even hough he caching eniies do no coordinae heir cache conen, more cache his occur. For up o 25% of overlap (o 0.25), m-cacao ouperforms m-cac. Clearly, m- CACao performs beer since by overhearing cache his a he neighboring caching eniy, boh caching eniies learn conen populariy faser. For very large overlap (o > 0.25), m-cac yields higher numbers of cache his. The reason is ha when applying m-cacao in case of a large overlap, neighboring caching eniies overhear such a large number of cache his, ha hey learn very similar conen populariy disribuions. Hence, over ime i is likely ha heir caches conain he same files. In conras, applying m-cac, a higher diversiy in cache conen is mainained over ime. Clearly, furher gains in cache his could be achieved by joinly opimizing he cache conen of all caching eniies. However, his would eiher require coordinaion among he caching eniies or a cenral planner deciding on he cache conen of all caching eniies, which resuls in a high communicaion overhead. In conras, our heurisic algorihm m-cacao does no require addiional coordinaion or communicaion and yields good resuls for small overlaps. IX. CONCLUSION In his paper, we presened a conex-aware proacive caching algorihm for wireless caching eniies based on conexual muli-armed bandis. To cope wih unknown and flucuaing conen populariy among he dynamically arriving and leaving users, he algorihm regularly observes conex informaion of conneced users, updaes he cache conen and subsequenly observes cache his. In his way, he algorihm learns conex-specific conen populariy online, which allows for a proacive adapaion of cache conen according o flucuaing local conen populariy. We derived a sublinear regre bound, which characerizes he learning speed and proves ha our proposed algorihm converges o he opimal cache conen placemen sraegy, which maximizes he expeced number of cache his. Moreover, he algorihm suppors cusomer prioriizaion and can be combined wih mulicas ransmissions and raing-based caching decisions. Numerical sudies showed ha by exploiing conex informaion, our algorihm ouperforms sae-of-he-ar algorihms in a real world daa se. REFERENCES [1] S. Müller, O. Aan, M. van der Schaar, and A. Klein, Smar caching in wireless small cell neworks via conexual muli-armed bandis, in Proc. IEEE Inernaional Conference on Communicaions (ICC), 2016, pp [2] [Online]. Available: hp:// soluions/collaeral/service-provider/visual-neworking-index-vni/ mobile-whie-paper-c hml [3] X. Wang, M. Chen, T. Taleb, A. Ksenini, and V. Leung, Cache in he air: Exploiing conen caching and delivery echniques for 5G sysems, IEEE Communicaions Magazine, vol. 52, no. 2, pp , Feb [4] S. Bors, V. Gupa, and A. Walid, Disribued caching algorihms for conen disribuion neworks, in Proc. IEEE Inernaional Conference on Compuer Communicaions (INFOCOM), 2010, pp [5] L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker, Web caching and Zipf-like disribuions: evidence and implicaions, in Proc. IEEE Inernaional Conference on Compuer Communicaions (INFOCOM), vol. 1, 1999, pp [6] J. Erman, A. Gerber, M. Hajiaghayi, D. Pei, S. Sen, and O. Spascheck, To cache or no o cache: The 3G case, IEEE Inerne Compuing, vol. 15, no. 2, pp , Mar [7] M. Maddah-Ali and U. Niesen, Fundamenal limis of caching, IEEE Transacions on Informaion Theory, vol. 60, no. 5, pp , May [8] N. Golrezaei, A. Molisch, A. Dimakis, and G. Caire, Femocaching and device-o-device collaboraion: A new archiecure for wireless video disribuion, IEEE Communicaions Magazine, vol. 51, no. 4, pp , Apr [9] K. Shanmugam, N. Golrezaei, A. Dimakis, A. Molisch, and G. Caire, Femocaching: Wireless conen delivery hrough disribued caching helpers, IEEE Transacions on Informaion Theory, vol. 59, no. 12, pp , Dec [10] K. Poularakis and L. Tassiulas, Exploiing user mobiliy for wireless conen delivery, in Proc. IEEE Inernaional Symposium on Informaion Theory (ISIT), 2013, pp [11] K. Poularakis, G. Iosifidis, V. Sourlas, and L. Tassiulas, Exploiing caching and mulicas for 5G wireless neworks, IEEE Transacions on Wireless Communicaions, vol. 15, no. 4, pp , Apr

13 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. The final version of record is available a hp://dx.doi.org/ /twc MU LLER e al.: CONTEXT-AWARE PROACTIVE CONTENT CACHING WITH SERVICE DIFFERENTIATION IN WIRELESS NETWORKS [12] E. Basug, M. Bennis, and M. Debbah, Cache-enabled small cell neworks: Modeling and radeoffs, in Proc. Inernaional Symposium on Wireless Communicaions Sysems (ISWCS), 2014, pp [13], Living on he edge: The role of proacive caching in 5G wireless neworks, IEEE Communicaions Magazine, vol. 52, no. 8, pp , Aug [14] E. Basug, M. Bennis, E. Zeydan, M. A. Kader, A. Karaepe, A. S. Er, and M. Debbah, Big daa mees elcos: A proacive caching perspecive, Journal of Communicaions and Neworks, Special Issue on Big Daa Neworking Challenges and Applicaions, vol. 17, no. 6, pp , Dec [15] P. Blasco and D. Gu ndu z, Learning-based opimizaion of cache conen in a small cell base saion, in Proc. IEEE Inernaional Conference on Communicaions (ICC), 2014, pp [16], Muli-armed bandi opimizaion of cache conen in wireless infosaion neworks, in Proc. IEEE Inernaional Symposium on Informaion Theory (ISIT), 2014, pp [17], Conen-level selecive offloading in heerogeneous neworks: Muli-armed bandi opimizaion and regre bounds, arxiv preprin, arxiv: , [18] A. Sengupa, S. Amuru, R. Tandon, R. Buehrer, and T. Clancy, Learning disribued caching sraegies in small cell neworks, in Proc. IEEE Inernaional Symposium on Wireless Communicaions Sysems (ISWCS), 2014, pp [19] M. ElBamby, M. Bennis, W. Saad, and M. Lava-aho, Conen-aware user clusering and caching in wireless small cell neworks, in Proc. IEEE Inernaional Symposium on Wireless Communicaions Sysems (ISWCS), 2014, pp [20] D. Goodman, J. Borras, N. B. Mandayam, and R. Yaes, Infosaions: A new sysem model for daa and messaging services, in Proc. IEEE Vehicular Technology Conference (VTC), vol. 2, 1997, pp [21] A. L. Iacono and C. Rose, Infosaions: New perspecives on wireless daa neworks, in Nex Generaion Wireless Neworks, S. Tekinay, Ed. Boson, MA: Springer US, 2002, ch. 1, pp [22] P. Gill, M. Arli, Z. Li, and A. Mahani, YouTube raffic characerizaion: A view from he edge, in Proc. ACM Conference on Inerne Measuremen (IMC), 2007, pp [23] M. Zink, K. Suh, Y. Gu, and J. Kurose, Characerisics of YouTube nework raffic a a campus nework measuremens, models, and implicaions, Compuer Neworks, vol. 53, no. 4, pp , Mar [24] A. Brodersen, S. Scellao, and M. Waenhofer, YouTube around he world: Geographic populariy of videos, in Proc. ACM Inernaional Conference on World Wide Web (WWW), 2012, pp [25] M.-L. Mares and Y. Sun, The muliple meanings of age for elevision conen preferences, Human Communicaion Research, vol. 36, no. 3, pp , July [26] C. A. Hoffner and K. J. Levine, Enjoymen of mediaed frigh and violence: A mea-analysis, Media Psychology, vol. 7, no. 2, pp , Nov [27] P. J. Renfrow, L. R. Goldberg, and R. Zilca, Lisening, waching, and reading: The srucure and correlaes of enerainmen preferences, Journal of Personaliy, vol. 79, no. 2, pp , Apr [28] D. Zillmann, Mood managemen hrough communicaion choices, American Behavioral Scienis, vol. 31, no. 3, pp , Jan [29] C. Zhou, Y. Guo, Y. Chen, X. Nie, and W. Zhu, Characerizing user waching behavior and video qualiy in mobile devices, in Proc. IEEE Inernaional Conference on Compuer Communicaion and Neworks (ICCCN), 2014, pp [30] B.-J. Ko, K.-W. Lee, K. Amiri, and S. Calo, Scalable service differeniaion in a shared sorage cache, in Proc. IEEE Inernaional Conference on Disribued Compuing Sysems (ICDCS), 2003, pp [31] Y. Lu, T. F. Abdelzaher, and A. Saxena, Design, implemenaion, and evaluaion of differeniaed caching services, IEEE Transacions on Parallel and Disribued Sysems, vol. 15, no. 5, pp , May [32] P. Cao and S. Irani, Cos-aware WWW proxy caching algorihms, in Proc. USENIX Symposium on Inerne Technologies and Sysems, 1997, pp [33] S. Li, J. Xu, M. van der Schaar, and W. Li, Populariy-driven conen caching, in Proc. IEEE Inernaional Conference on Compuer Communicaions (INFOCOM), 2016, pp [34], Trend-aware video caching hrough online learning, IEEE Transacions on Mulimedia, vol. 18, no. 12, pp , Dec [35] P. Auer, N. Cesa-Bianchi, and P. Fischer, Finie-ime analysis of he muliarmed bandi problem, Machine Learning, vol. 47, no. 2-3, pp , May [36] S. Maghsudi and E. Hossain, Muli-armed bandis wih applicaion o 5G small cells, IEEE Wireless Communicaions, vol. 23, no. 3, pp , Jun [37] S. Amuru, C. Tekin, M. v. der Schaar, and R. M. Buehrer, Jamming bandis a novel learning mehod for opimal jamming, IEEE Transacions on Wireless Communicaions, vol. 15, no. 4, pp , Apr [38] C. Shen, C. Tekin, and M. van der Schaar, A non-sochasic learning approach o energy efficien mobiliy managemen, IEEE Journal on Seleced Areas in Communicaions, Series on Green Communicaions and Neworking, o be published. [39] T. Lu, D. Pal, and M. Pal, Conexual muli-armed bandis, in Proc. Inernaional Conference on Arificial Inelligence and Saisics (AISTATS), 2010, pp [40] A. Slivkins, Conexual bandis wih similariy informaion, Journal of Machine Learning Research, vol. 15, no. 1, pp , Jan [41] C. Tekin and M. van der Schaar, Disribued online learning via cooperaive conexual bandis, IEEE Transacions on Signal Processing, vol. 63, no. 14, pp , Mar [42] C. Tekin, S. Zhang, and M. van der Schaar, Disribued online learning in social recommender sysems, IEEE Journal of Seleced Topics in Signal Processing, vol. 8, no. 4, pp , Aug [43] H. Kellerer, U. Pferschy, and D. Pisinger, Knapsack Problems. Springer Berlin Heidelberg, [44] [Online]. Available: hp://kang.n.e-echnik.u-darmsad.de/n/ fileadmin/k/publikaionen PDFs/2016/TWC/TWC2016 Mueller App. pdf [45] F. M. Harper and J. A. Konsan, The MovieLens daases: Hisory and conex, ACM Transacions on Ineracive Inelligen Sysems, vol. 5, no. 4, pp. 1 19, Dec [46] [Online]. Available: hp://grouplens.org/daases/movielens/1m/ [47] W. Hoeffding, Probabiliy inequaliies for sums of bounded random variables, Journal of he American Saisical Associaion, vol. 58, no. 301, pp , Mar [48] E. Chlebus, An approximae formula for a parial sum of he divergen p-series, Applied Mahemaics Leers, vol. 22, no. 5, pp , May Sabrina Mu ller (S 15) received he B.Sc. and he M.Sc. degrees in mahemaics from he Technische Universia Darmsad, Germany, in 2012 and 2014, respecively. She is currenly pursuing he Ph.D. degree in elecrical engineering as member of he Communicaions Engineering Laboraory a he Technische Universia Darmsad, Germany. Her research ineress include machine learning and opimizaion mehods and heir applicaions o wireless neworks. Onur Aan received he B.Sc. degree in elecrical engineering from Bilken Universiy, Ankara, Turkey, in 2013 and he M.Sc. degree in elecrical engineering from he Universiy of California, Los Angeles, USA, in He is currenly pursuing he Ph.D. degree in elecrical engineering a he Universiy of California, Los Angeles, USA. He received he bes M.Sc. hesis award in elecrical engineering a he Universiy of California, Los Angeles, USA. His research ineress include online learning and muli-armed bandi problems.

14 This is he auhor's version of an aricle ha has been published in his journal. Changes were made o his version by he publisher prior o publicaion. The final version of record is available a hp://dx.doi.org/ /twc IEEE TRANSACTIONS ON WIRELESS COMMUNICATIONS, VOL. XX, NO. XX, XX XXXX Mihaela van der Schaar (M99 SM04 F10) is Chancellor s Professor of Elecrical Engineering a Universiy of California, Los Angeles, USA. Her research ineress include machine learning for decision making, medical informaics and educaion, online learning, real-ime sream mining, nework science, engineering economics, social neworks, game heory, wireless neworks and mulimedia. Prof. van der Schaar was a Disinguished Lecurer of he Communicaions Sociey from 2011 o 2012, he Edior in Chief of he IEEE Transacions on Mulimedia from 2011 o 2013, and a Member of he Ediorial Board of he IEEE Journal on Seleced Topics in Signal Processing in She was a recipien of he NSF CAREER Award (2004), he Bes Paper Award from he IEEE Transacions on Circuis and Sysems for Video Technology (2005), he Okawa Foundaion Award (2006), he IBM Faculy Award (2005, 2007, 2008), he Mos Cied Paper Award from he EURASIP Journal on Image Communicaions (2006), he Gamenes Conference Bes Paper Award (2011), and he IEEE Circuis and Sysems Sociey Darlingon Award Bes Paper Award (2011). She received hree ISO Awards for her conribuions o he MPEG video compression and sreaming inernaional sandardizaion aciviies, and holds 33 graned U.S. paens. Anja Klein (M96) received he Diploma and Dr.Ing. (Ph.D.) degrees in elecrical engineering from he Universiy of Kaiserslauern, Germany, in 1991 and 1996, respecively. In 1996, she joined Siemens AG, Mobile Neworks Division, Munich and Berlin. She was acive in he sandardizaion of hird generaion mobile radio in ETSI and in 3GPP, for insance leading he TDD group in RAN1 of 3GPP. She was vice presiden, heading a developmen deparmen and a sysems engineering deparmen. In 2004, she joined he Technische Universi Darmsad, Germany, as full professor, heading he Communicaions Engineering Laboraory. Her main research ineress are in mobile radio, including inerference managemen, cross-layer design, relaying and muli-hop, compuaion offloading, smar caching and energy harvesing. Dr. Klein has auhored over 290 refereed papers and has conribued o 12 books. She is invenor and co-invenor of more han 45 paens in he field of mobile radio. In 1999, she was named he Invenor of he Year by Siemens AG. She is a member of Verband Deuscher Elekroechniker - Informaionsechnische Gesellschaf (VDE-ITG).