Cellular Network Traffic Scheduling using Deep Reinforcement Learning

Transcription

1 Cellular Network Traffic Scheduling using Deep Reinforcement Learning Sandeep Chinchali, et. al. Marco Pavone, Sachin Katti Stanford University AAAI 2018

2 Can we learn to optimally manage cellular networks? Delay Tolerant (DT) Traffic Pre-fetched content IoT: Map/SW updates Internet Real-time Mobile Traffic Delay Sensitive 2

3 Why is IoT/DT traffic scheduling hard? Utilization Acceptable Limit IoT Contending goals Max IoT/DT data IoT Loss to mobile traffic Network limits Optimal Control 3

4 Why is IoT/DT traffic scheduling hard? Melbourne Central Business District, Rolling Average = 1 min Shopping center O ce building Southern cross station Melbourne central station Diverse city-wide cell patterns Congestion C :00 11:00 13:00 15:00 17:00 19:00 21:00 Local time csandeep@stanford.edu 4

5 Our contributions 1. Identify inefficiencies in real cellular networks 4 weeks, 10 diverse cells in Downtown Melbourne, Australia 2. Data Driven, Deep Learning Network Model Our live network experiments match MDP dynamics 3. Adaptive RL scheduler Flexibly responds to operator reward functions Network State IoT Scheduler IoT rate csandeep@stanford.edu 5

6 Why Deep Learning? Congestion C Melbourne Central Business District, Rolling Average = 1 min Shopping center O ce building Southern cross station Melbourne central station 1. Learn time-variant network dynamics 2. Adapt to high-level network operation goals 3. Generalize to diverse cells 0 4. Abundance of network data 09:00 11:00 13:00 15:00 17:00 19:00 21:00 Local time csandeep@stanford.edu 6

7 Related Work 1. Dynamic Resource Allocation Electricity grid (Reddy 2011), call admission (Marbach 1998), traffic control (Chu 2016) 2. Data-driven Optimal Control + Forecasting Deep RL (Mnih 2013, Silver 2014, Lillicrap 2015) LSTM networks (Hochreiter 1997, Laptev 2017, Shi 2015) 3. Machine Learning for Computer Networks Cluster Resource Management (Mao 2016) Mobile Video Streaming (Mao 2017, Yin 2015) csandeep@stanford.edu 7

8 Data-driven problem formulation 1. Network State Space 2. IoT Scheduler Actions 3. Time-variant dynamics 4. Network operator policies Congestion IoT Scheduler Cell efficiency IoT rate Num Users Network state + forecasts 8

9 Primer on Cell Networks (Link Quality) Goal: Max safe IoT traffic V t over day csandeep@stanford.edu 9

10 RL setup (1): State Space Reward Current Network State Agent Action Environment Full State with Temporal Features Network state Stochastic Forecast (LSTM) Horizon: Day of T mins csandeep@stanford.edu 10

11 RL setup (2): Action Space Reward IoT Traffic Rate: Agent Action Environment IoT Volume per minute: Network state Utilization gain: 11

12 RL setup (3): Transition Dynamics 1.6 Controlled tra c Reward 1.5 Agent Action Environment Congestion C Background dynamics Network state :10 20:15 20:20 Local time csandeep@stanford.edu 12

13 RL setup (4): Operator Rewards Reward Overall weighted reward Agent Action Environment 1. IoT traffic volume What-if model Network state 2. Loss to regular users Goal: Find Optimal Operator Policy 3. Traffic below network limit 13

14 Evaluation 14

15 Evaluation Criteria 1. Robust performance on diverse cell-day pairs 2. Ability to exploit better forecasts 3. Interpretability Congestion IoT Scheduler Cell efficiency IoT rate Num Users Network state + forecasts 15

16 1. RL generalizes to several cell-day pairs 100 Respond to operator priorities α 1 8tilization gain V IoT /V 0 (%) Significant gains: FCC Spectrum Auction (Reardon 2016): $4.5B for 10 MHz of spectrum 14.7% median gain for α = 2 Significant cost savings [simulated] 0 TUain Test csandeep@stanford.edu 16

17 2. RL effectively leverages forecasts RL Benchmark Richer LSTM forecasts 17

18 3a. RL exploits transient dips in utilization Controlled Congestion Utilization gain Congestion C Original Heuristic control DDPG control Transient Dip 0 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 Local time Utilization gain VIoT /V0 (%) Heuristic control DDPG control 0 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 Local time 18

19 3b. RL smooths network throughput Controlled Congestion Resulting Throughput Congestion C Original Heuristic control DDPG control 0 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 Local time Throughput B (MBps) Original Heuristic control DDPG control Throughput limit 0.0 9:00 10:00 11:00 12:00 13:00 14:00 15:00 16:00 Local time csandeep@stanford.edu 19

20 Conclusion Modern networks are evolving Delay tolerant traffic (IoT updates, pre-fetched content) Data-driven optimal control LSTM forecasts + RL controller 14.7% simulated gain -> significant savings Future work: Operational network tests Decouple prediction and control Questions: csandeep@stanford.edu csandeep@stanford.edu

21 Extra slides 21

22 2. RL effectively leverages forecasts Better forecasts enhance performance Discretized MDP for offline optimal Reward R Ā =5 Ā =20 Ā =40 Ā = S Richer LSTM forecasts Approach Cts MDP csandeep@stanford.edu 22