A PERFORMANCE ANALYSIS FRAMEWORK FOR MOBILE-AGENT SYSTEMS Marios D. Dikaiakos Department of Computer Science University of Cyprus George Samaras Speaker: Marios D. Dikaiakos mdd@ucy.ac.cy http://www.cs.ucy.ac.cy/mdd Workshop on Infrastructure for Scalable Multi-Agent Systems The Fourth International Conference on Autonomous Agents 2000
Summary A conceptual framework to analyze the performance of MA systems quantitatively Materializing this framework as a hierarchy of benchmarks Benchmark implementation and experimentation Evaluation and restructuring of benchmarks and experiments Marios Dikaiakos, Univ. of Cyprus 2 http://www.cs.ucy.ac.cy/mdd/
Acknowledgements Melinos Kyriakou Constantine Spyrou Dimitris Zeinalipour-Yiazti Marios Dikaiakos, Univ. of Cyprus 3 http://www.cs.ucy.ac.cy/mdd/
Overview Motivation. A Performance Analysis Framework. Benchmarks and Experimentation. Conclusions and Future Work. Marios Dikaiakos, Univ. of Cyprus 4 http://www.cs.ucy.ac.cy/mdd/
New Computing Paradigms Towards integrated services covering many dimensions: Different levels of user interaction (push vs. pull, ) Spectrum of "user experience" (rich vs. poor) Alternative Connection modalities (wireless-fixed) All different kinds of clients (thin, thick, portable, wearable, home) A paradigm shift from Client-Server computing towards more flexible schemes that adapt dynamically to the various dimensions of future integrated Internet services. Mobile Agents. Marios Dikaiakos, Univ. of Cyprus 5 http://www.cs.ucy.ac.cy/mdd/
A framework is required to: Objectives study & argue about performance issues of mobile-agentbased systems compare mobile-agent platforms quantitatively discover potential performance bottlenecks monitor MA-based systems performance Marios Dikaiakos, Univ. of Cyprus 6 http://www.cs.ucy.ac.cy/mdd/
The Need for Performance Evaluation Quantitative performance evaluation is the foundation for: performance debugging and optimization comparison of systems extrapolation of properties of future systems. The more complex a system/application is, the harder its evaluation becomes. E.g., in multiprocessor systems: What is a representative workload? Software models not stabilized. Many degrees of freedom in system/application configuration. What are the appropriate metrics? Marios Dikaiakos, Univ. of Cyprus 7 http://www.cs.ucy.ac.cy/mdd/
Evaluation of Mobile-Agent Systems Quantitative evaluation of mobile-agent-based distributed systems is even harder: The absence of global time, control and state information. The heterogeneity/complexity of platforms: difficult to describe performance properties via small sets of metrics. The variety of distributed computing (software) models. The diversity of operations found in distributed applications: hard to construct simple and portable benchmarks. The flexibility of system configuration: hard to provide concise representation of system resources. Issues affecting performance of JAVA. Marios Dikaiakos, Univ. of Cyprus 8 http://www.cs.ucy.ac.cy/mdd/
Overview Motivation. A Performance Analysis Framework. Benchmarks and Experimentation. Conclusions and Future Work. Marios Dikaiakos, Univ. of Cyprus 9 http://www.cs.ucy.ac.cy/mdd/
A Performance Analysis Spectrum Full Applications Application Frameworks Patterns of Interaction Basic Elements Repr. Workloads Repr. Resources Proper Metrics Establish causality relationships Marios Dikaiakos, Univ. of Cyprus 10 http://www.cs.ucy.ac.cy/mdd/
A Performance Analysis Framework Identify & benchmark basic elements of mobile-agent systems. Agents, Places, Behaviors Identify & benchmark patterns of interaction appropriate for mobile-agent applications. Software models for Distr. Computing Formulate application frameworks that instantiate relevant software models and can be used in anticipated mobile-agent applications. Database access over the Web Marios Dikaiakos, Univ. of Cyprus 11 http://www.cs.ucy.ac.cy/mdd/
Basic Elements of M.A. Platforms Agents: State, Implementation (code), Interface, Identifier, etc. Places (environment where agents execute): Engine, Resources, Location Behaviors (within and between places): Creation, Transfer, Arrival, Communication via messages and agents, Multicasting, Synchronization. Marios Dikaiakos, Univ. of Cyprus 12 http://www.cs.ucy.ac.cy/mdd/
Software Models Patterns of Interaction or Agent Design Patterns: Represent the synthesis of basic MA behaviors into more complex frameworks of MA behavior and interaction, which are common to many MA-based systems. Encoded as Software (Distributed-Computing ) Models. We focus on: Distr. Computing Models: the Client-Server model and extensions: C/S, C/A/S, C/I/S Agent Design Patterns: Proxy, Router, Meeting Marios Dikaiakos, Univ. of Cyprus 13 http://www.cs.ucy.ac.cy/mdd/
Client-Agent-Server Model Client-Agent-Server (C/A/S) Fixed Network Application Client Agent Application Server Mobile Host Wireless Link Marios Dikaiakos, Univ. of Cyprus 14 http://www.cs.ucy.ac.cy/mdd/
M.A. Application Frameworks Mobile-Agent Application Frameworks: implementation of software models, using MA, for: Particular applications Under characteristic workloads Application Frameworks are libraries of mobile-agent routines, materializing some software model and implementing core sets of services for a particular application. We examine application frameworks for Database-access provision over the Web. Generate characteristic workloads according to TPC-W benchmark suite. Marios Dikaiakos, Univ. of Cyprus 15 http://www.cs.ucy.ac.cy/mdd/
Overview Motivation. A Performance Analysis Framework. Benchmarks and Experimentation. Conclusions and Future Work. Marios Dikaiakos, Univ. of Cyprus 16 http://www.cs.ucy.ac.cy/mdd/
Benchmarking MA Systems Micro-benchmarks: short codes designed to isolate and measure performance properties of basic behaviors of mobile-agentbased systems for typical system configurations. Micro-kernels: short, synthetic codes designed to measure and investigate performance properties of software-model implementations, for typical applications and system configurations. Application kernels: instantiations of micro-kernels for particular application domains and for typical workloads derived from the TPC-W (Web Commerce) specification. Marios Dikaiakos, Univ. of Cyprus 17 http://www.cs.ucy.ac.cy/mdd/
Micro-benchmarks Key software components: Mobile Agents to materialize modules of C/S, C/A/S, etc. Messenger Agents for flexible communication. Messaging for efficient communication and synchronization. [AC-L]: captures the overhead of local agent-creation. [AC-R]: captures the overhead of remote agent-creation. [AL]: captures the overhead of agent-launching. [AR]: captures the overhead of receiving an incoming agent. [MSG]: captures point-to-point messaging overhead. [MULT]: captures multicasting overhead. [SYNCH]: captures synchronization overhead. [ROAM]: captures agent-travelling overhead. Marios Dikaiakos, Univ. of Cyprus 18 http://www.cs.ucy.ac.cy/mdd/
Tentative List: Micro-benchmark Parameters Configuration of places where agents reside, roam and perform basic behaviors. Configuration of channels used by agents in their movements from place to place. Number of iterations executed. Mobile Agent-size. Marios Dikaiakos, Univ. of Cyprus 19 http://www.cs.ucy.ac.cy/mdd/
Places and Channels Agent-Execution Environment MA MA Agent-enabled Applet Places Modem Ethernet Channels Radio Tower Wide Area Network Marios Dikaiakos, Univ. of Cyprus 20 http://www.cs.ucy.ac.cy/mdd/
Metrics Aggregate time to completion: Raw performance measurements. Performance scaling under various load-conditions. Identification of bottlenecks & performance problems. Examination of platform-robustness. Peak Rates: Sustained performance under "ideal" conditions. Quantitative comparisons. Marios Dikaiakos, Univ. of Cyprus 21 http://www.cs.ucy.ac.cy/mdd/
Micro-benchmark Experiments We have implemented the benchmarks with four Java-based platforms: IBM s Aglets, Mitsubishi s Concordia, Voyager and Grasshopper. We are currently running tests on a LAN; in the near future we shall repeat them across different LANs of our WAN, and on top of GSM connections. We are testing two scenarios: 1. Full agent-execution environment installed on client. 2. Client with minimal resources-configuration downloads agent-aware applet Marios Dikaiakos, Univ. of Cyprus 22 http://www.cs.ucy.ac.cy/mdd/
Micro-benchmark Experiments [SYNCH] [MULT] [MSG] [AR] Behavior Mitsubishi s Concordia Voyager IBM Aglets Grasshopper [AL] [AC-R] LAN WAN GSM-LAN GSM-GSM [AC-L] A-A Win95 A-W Win95 A-A WinNT A-W WinNT Place Resources Marios Dikaiakos, Univ. of Cyprus 23 http://www.cs.ucy.ac.cy/mdd/
Agent Creation-Local (Win 95) 0.4 1.1 2.8 28.3 93.7 0.71 1.54 0.3 0.2 0.17 0.22 1 2 10 50 100 0.1 0.08 500 1000 0 Aglet Voyager Concordia Marios Dikaiakos, Univ. of Cyprus 24 http://www.cs.ucy.ac.cy/mdd/
AC-L Benchmark: Average Timings CONCORDIA VOYAGER AGLETS 1000 Average Creation Time (msec) 800 600 400 200 0 1 2 10 50 100 500 1000 5000 Number Of Created Agents Marios Dikaiakos, Univ. of Cyprus 25 http://www.cs.ucy.ac.cy/mdd/
AC-L: Rates of Agent Creation CONCORDIA VOYAGER AGLETS Agents / second 4000 3000 2000 1000 0 3571.4 704.2 100 1 2 10 50 100 500 1000 5000 Number of Agents created Locally Marios Dikaiakos, Univ. of Cyprus 26 http://www.cs.ucy.ac.cy/mdd/
AC-L Benchmark (Win NT) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0 0.8 1.5 WinNT WinNT WinNT 1 2 10 50 100 500 1000 Aglet Voyager Concordia Marios Dikaiakos, Univ. of Cyprus 27 http://www.cs.ucy.ac.cy/mdd/
Agent Launching Benchmark (Win95) 30 42.7 240.6 550.3 51.3 110.2 65.42 20 10 1 2 10 50 100 500 1000 0 Aglets Voyager Concordia Marios Dikaiakos, Univ. of Cyprus 28 http://www.cs.ucy.ac.cy/mdd/
AL Benchmark: Average Timings CONCORDIA VOYAGER AGLETS 1200 Average Launch Time (msec) 1000 800 600 400 200 0 1 2 10 50 100 500 1000 Number of Launched Agents Marios Dikaiakos, Univ. of Cyprus 29 http://www.cs.ucy.ac.cy/mdd/
AL: Rates of Agent Launching CONCORDIA VOYAGER AGLETS Launched Agents / Sec 25 20 15 10 5 0 1 2 10 50 100 500 1000 Number Of Launched Agents Marios Dikaiakos, Univ. of Cyprus 30 http://www.cs.ucy.ac.cy/mdd/
Agent Launch Benchmark (WinNT) 5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 6.06 34.11 69.82 15 30.934 Aglets Voyager Concordia 1 2 10 50 100 500 1000 Marios Dikaiakos, Univ. of Cyprus 31 http://www.cs.ucy.ac.cy/mdd/
50 Agent Launching: Summary Aglet Win95 Aglet WinNT Voyager Win95 Voyager WinNT Concordia Win95 Concordia WinNT 42.7 240.6 51.3 550.3 110.2 69.82 65.42 Seconds 25 23.79 0.9 1.4 7.8 18.4 0 0.33 0.17 1.1 1 1.1 0.21 3.6 2.1 1.54 0.44 4.2 1 2 10 50 100 500 1000 Number of Agents 3.13 2.163 5.28 3.695 Marios Dikaiakos, Univ. of Cyprus 32 http://www.cs.ucy.ac.cy/mdd/
MSG: One-way Messaging 24 21 18 Aglet Win95 Aglet WinNT Voyager Win95 Voyager WinNT Concordia Win95 Concordia WinNT 23.9 48.3 24.44 15 Seconds 12 9 6 3 0 1 2 10 50 100 500 1000 Number of One-Way Messages Marios Dikaiakos, Univ. of Cyprus 33 http://www.cs.ucy.ac.cy/mdd/
MSG: Rate of Message Dispatch CONCORDIA VOYAGER AGLETS One-Way Messages/Second 250 200 150 100 50 0 1 2 10 50 100 500 1000 Number of One-Way Messages Marios Dikaiakos, Univ. of Cyprus 34 http://www.cs.ucy.ac.cy/mdd/
An Assessment of Micro-benchmarks Micro-benchmarks provide useful insights into: The behavior of Mobile-Agent Systems. The factors that determine Mobile-Agent performance. Initial micro-benchmark results guide the redesign of old or the deployment of new micro-benchmarks. Expand our understanding on MA systems and their capabilities. Help us understand and explain the performance of microkernels. Marios Dikaiakos, Univ. of Cyprus 35 http://www.cs.ucy.ac.cy/mdd/
More Conclusions Performance of "light" MA depends a lot on: Loading and caching classes to main and remote memories. Robustness and performance of MA servers under heavy load. Concordia shows a performance advantage over Aglets and Voyager in agent creation and launching. Voyager is a clear winner when it comes to messaging. Applets cannot sustain efficient and robust mobile-agent activity. WinNT provide more stable performance measurements than Win95. Marios Dikaiakos, Univ. of Cyprus 36 http://www.cs.ucy.ac.cy/mdd/