Data Transfers in the Grid: Workload Analysis of Globus GridFTP

Data Transfers in the Grid: Workload Analysis of Globus GridFTP Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Adriana Iamnitchi University of South Florida Dan Fraser Argonne National Laboratory

Objective : Quantify volume of transfers What is the transfer size distribution? What is the volume of activity for the most active hosts? Objective : Understand how tuning capabilities are used What are the buffer sizes used during the transfers? What is the average bandwidth? What is the utilization of functionalities like streams and stripes? Objective 3: 3 Quantify user base and predict usage trends How does the user base evolve over time? What are the geographical characteristics of the GridFTP data transfers? 3

Outline Metrics dataset Surprises and zoom in (TeraGrid( TeraGrid) Lessons and discussions

GridFTP Metrics Dataset Field Range of Values Comment Source hostname/host IP String/IPnet Anonymized Start time of the transfer Timestamp Accuracy: ms End time of the transfer Timestamp Accuracy: ms TCP Buffer Size Integer (Bytes) Total Number of Bytes Integer (Bytes) Number of Streams Integer Number of Stripes Integer Store or Retrieve Integer (,,) STOR, RETR, LIST 5

Metrics Dataset Started with ~37.5 million records (Jul 5 - Mar 7) Cleaning: transfer size : ~. million records buffer size <: ~ records directory listings: ~3.9 million records invalid hostnames (e.g., /[B@97e): ~,6 records ANL-TeraGrid testing: ~. million records duplicate reports: ~6. million records self transfers (source=destination): ~5.75 million records Clean database: ~77. million records (~56.%) 6

Surprise #: Transfer Size Distribution % of total 5 6 3 6 56 5 6 3 6 56 5 6 3 6 56 5 Bytes KB MB Transfer size Objective : Quantify volume of transfers 7

Zoom-in: TeraGrid Are these results representative for production grids? GridFTP testing for deployment and learning Identify transfers from TeraGRid and analyze dataset.

Transfer Size Distribution (TG) 5 % of total 5 5 6 3 6 56 5 6 3 6 56 5 6 3 6 56 5 Bytes KΒ MΒ Transfer Size Objective : Quantify volume of transfers 9

5 56 5 6 3 6 56 5 6 3 6 56 5 6 3 6 5 5 5 Bytes KB MB Transfer size 56 5 % of total 6 3 6 56 5 6 3 6 56 5 6 3 6 Bytes KΒ MΒ Transfer Size % of total All TG

Why So Small Transfers? There are still many old versions (i.e., before v3.9.5) of GridFTP in use. These versions do not include trace reporting capabilities. Other data transfer protocols and implementations are used Users have turned off the reporting capability Some of the logs are inevitably lost due to the UDP-based reporting mechanism The low transfer volumes could suggest a shift towards data-aware aware job scheduling (?)

Server to Server Transfers % 7% 6% 5% % 7.% # Transfers Volume 39.5% 3.% 3% % %.7% 9.7%.% % InterDomain InterIP SelfTransfers High reporting of Self Transfers (more than /3) Objective : Quantify volume of transfers

Top 6 Active Hosts (all) 5 Volume Transferred Number of Transfers.E+ Volume Transferred (TB) 5 5.E+7.E+6.E+5.E+.E+3.E+.E+ Number of Transfers 3 Host 5 6.E+ Top 6 hosts traffic adds up to ~% of total volume Next hosts (IPs( IPs) ) transferred s of TB Objective : Quantify volume of transfers 3

Number of Transfers & Volume (TG) 3 5 5 5 Number of Transfers per Month Total Volume per Month 7 6 5 3 Aug-5 Sep-5 Oct-5 Nov-5 Dec-5 Jan-6 Feb-6 Mar-6 Apr-6 May-6 Jun-6 Jul-6 Aug-6 Sep-6 Oct-6 Nov-6 Dec-6 Jan-7 Feb-7 Mar-7 Number of Transfers (thousands) Volume (TB) Month-Year Objective : Quantify volume of transfers

Average Transfer Size & Total Volume (TG) 6 Total Volume (TB) Average Transfer Size (GB)..7 Total Volume (TB) 6.6.5..3.. Average Transfer Size (GB) 3 5 6 7 TERAGRID SITES Objective : Quantify volume of transfers 5

Daily Workload (TG) Volume (TB).....6... Average Volume per Day Average Number of Transfers per Day Monday Tuesday WednesdayThursday Friday Saturday Sunday Average volume transferred per day: ~.6TB GridFTP doesn t t get weekends free! Objective : Quantify volume of transfers 5 5 35 3 5 5 5 Number of Transfers 6

Monthly Workload (TG).5 Average Volume Transferred per day Average Number of Transfers per day Volume (TB)..5..5 6 Number of Transfers. 3 5 7 9 3 5 7 9 3 5 7 9 3 Day of the month ~5, transfers per day ~TB per day of total volume Lowest around.5tb per day Peaks due to particular days Objective : Quantify volume of transfers 7

Objective : Quantify volume of transfers What is the transfer size distribution? What is the volume of activity for the most active hosts? Objective : Understand how tuning capabilities are used What are the buffer sizes used during the transfers? What is the average bandwidth? What is the utilization of functionalities like streams and stripes? Objective 3: 3 Quantify user base and predict usage trends How does the user base evolve over time? What are the geographical characteristics of the GridFTP data transfers?

Surprise #: Usage of Streams and Stripes Unreliably reported (from Globus team). Reliable observations: At least ~% of the transfers used streams (suggested number by ANL s website) At least ~% of the transfers used a different value, larger than one. Maximum number of streams reported: (!!) Objective : Understand how tuning capabilities are used 9

Buffer Size Distribution 9 7 % of total 6 5 3 6 3 6 56 5 6 KB MB Buffer Size 6% from the original table: : OS-controlled ( bytes) Most commonly used: 6 KB KB Largest buffer size: -GB (9 records) Objective : Understand how tuning capabilities are used

Average Bandwidth Distribution % of total 6 6 3 6 56 5 6 3 6 56 5 6 3 6 56 5 bps Kbps Mbps Average Bandwidth Peak: -56Mbps, 56Mbps, ~7.7 million records Most common (5%): Mbps Gbps

Average Bandwidth Distribution (TG) % of total 6 6 6 3 6 56 5 6 3 6 56 5 Kbps Mbps Gbps Average Bandwidth Compared to the total dataset: => The region of Mbps to Gbps includes more than 5% of the transfers (5% for the whole dataset)

Objective : Quantify volume of transfers What is the transfer size distribution? What is the volume of activity for the most active hosts? Objective : Understand how tuning capabilities are used What are the buffer sizes used during the transfers? What is the average bandwidth? What is the utilization of functionalities like streams and stripes? Objective 3: 3 Quantify user base and predict usage trends How does the user base evolve over time? What are the geographical characteristics of the GridFTP data transfers? 3

Surprise #3: Geographic Distribution AT GB ES CH AU DE IT JP TW CA # transfers volume % % % 3% % 5% 6% 7% % 9% % USA: 7.% or ~5. million transfers and.9% or ~.7 PB Canada+Taiwan+Japan+Spain:~M transfers and 36TB 9 different countries and 6 different cities (7 cities from USA) Objective 3: 3 Quantify user base and predict usage trends

User and Domain Evolution (all) # IPs # Domains Linear Fit (# Domains) IPs 6 y =.3x + 67.7 R =.97 Jul-5 Aug-5 Sep-5 Oct-5 Nov-5 Dec-5 Jan-6 Feb-6 Mar-6 Apr-6 May-6 Jun-6 Jul-6 Aug-6 Sep-6 Oct-6 Nov-6 Dec-6 Jan-7 Feb-7 Mar-7 35 3 5 5 5 Domains Continuing increase of user and organization population Forecasts: 67 new IPs and new domains per month Objective 3: 3 Quantify user base and predict usage trends 5

Summary of Results Many transfers in the range of KBs to s MB (peak in 6MB-3MB). relevant for setting up realistic simulations. previous work assume different, larger file sizes. Bandwidth measured in previous work is confirmed by our workload analysis. Tuning parameters: Users tend not to set the buffer size explicitly (6%), leaving it to the OS The unexpectedly small transfers do not justify tuning GridFTP parameters (stripes and streams) The usage of Globus GridFTP is growing over time in terms of IPs (users), domains (organizations), and volume transferred. Missing some of the big players 6