file:///h:/temp/htmlö/analyse+dfsio+write+test.html

Size: px

Start display at page:

Download "file:///h:/temp/htmlö/analyse+dfsio+write+test.html"

Tyler Parrish
6 years ago
Views:

1 nalyse DFSIO Write Test of :45 Testreihe Ergebnisse: Probleme: Die Ergebnisse können der Inputspezifikation nicht mehr zugeordnet werden ("hier wollte ich so und so viele Dateien mit so und so vielen MB schreiben"), da die Tests nicht mehr vollständig durchlaufen: Zeiten überlappen, da es zu lange dauert, Dateien zu gross sind. Die Jobs beschreiben Sektoren wiederholt. Es Daten von der vielfachen Grösse des Dateisystems beschrieben Schreiben bricht oft ab unklar im Moment ob aufgrund überlappender Jobs (Jobs in der Queue ) Alle files wurden nie geschrieben Die im Design beschriebenen 600GB wurden nie geschrienen, das maximal Erreichte: gute 400GB ### Fragen: Bisher noch nicht mit buffersize gespielt. Bisher immer den Default von 1MB getestet, any ideas? In [ ]: shellscript: jarname="/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar" testspec="testdfsio -write -nrfiles ${1} -filesize ${2}MB" now=$(date '+%Y%0m%d-%Hh%M') outfile="/tmp/"${now}.txt hdfsloc="/user/hive/warehouse/dfsiostaged/" cmd="hadoop jar ${jarname} ${testspec} -resfile ${outfile}" echo ${cmd} ${cmd} echo "ran"${cmd} echo "outfile to "${outfile}", move it to "${hdfsloc} hdfs dfs -copyfromlocal ${outfile} ${hdfsloc}

2 nalyse DFSIO Write Test Driven by cront ("at") full-factorial: files MB IM schimmsten Fall also werden 60 Files a 10GB geschrieben - sollte Kapazität überschreiten können - s. cloudera vm beschreibung at now + 5 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 1 1" at now + 10 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 1 10" at now + 15 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 1 100" at now + 20 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 25 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 30 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 10 1" at now + 35 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 10 10" at now + 40 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 45 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 50 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 55 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 20 1" at now + 60 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 20 10" at now + 65 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 70 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 75 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 80 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 30 1" at now + 85 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 30 10" at now + 90 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now + 95 minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 40 1" at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 40 10" at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 50 1" at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 50 10" at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 60 1" at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh 60 10" at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " at now minutes <<< "sh /home/cloudera/bdcollation/dfsio/runtest.sh " Die Cloudera vm Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_quickstart-lv_root 55G 49G 3.1G 95% / tmpfs 1.9G 88K 1.9G 1% /dev/shm /dev/sda1 485M 37M 423M 8% /boot /dev/sr0 57M 57M 0 100% /media/vboxadditions_5.1.14_ RAM: 4GB (4096MB) of :45

3 3 of :45 In [33]: import pandas as pd #infilename = "data/dfsio.csv" infilename = "data/dfsio/part-00000" columnnames = ["date","nf","mbtot","thru","iomean","iosd","dur"] dateparse = lambda x: pd.datetime.strptime(x, '%Y-%m-%d %H:%M:%S') #parse_dates=['datetime'], date_parser=dateparse df = pd.read_csv(infilename, header=0, sep=',', names=columnnames, infer_datetime_format=true, parse_dates=['date'], date_parser = dateparse ) # dies ist die bevorzugte methode "from_csv" print(df.dtypes) mask = (df['date'] >= ' ') mask = (df['date'] > ' ') & (df['date'] <= ' ') mask = (df['date'] > ' :50') & (df['date'] <= ' ') df["gbwritten"]=df['mbtot']*df['nf']/1000 df=df.loc[mask] df.head(40)

4 4 of :45 date datetime64[ns] nf int64 MBtot float64 thru float64 IOmean float64 IOsd float64 dur float64 dtype: object Out[33]: date nf MBtot thru IOmean IOsd dur GBwritten :53: :58: :04: :09: :14: :19: :27: :38: :41: :44: :47: :02: :06: :12: :25: :30: :35: :41: :56: :58: :17: :20: :31:

5 of 8 29.03.2017 12:45 In [3]: df.describe() Out[3]: nf MBtot thru IOmean IOsd dur count 372.000000 372.000000 372.000000 372.000000 372.000000 372.000000 mean 2.701613 346.129032 44.945398 44.

5 5 of :45 In [3]: df.describe() Out[3]: nf MBtot thru IOmean IOsd dur count mean std min % % % max In [34]: df.hist(bins=50) Out[34]: array([[<matplotlib.axes._subplots.axessubplot object at 0x E2A6EF0>, <matplotlib.axes._subplots.axessubplot object at 0x E9B3B38>, <matplotlib.axes._subplots.axessubplot object at 0x E9D45F8>], [<matplotlib.axes._subplots.axessubplot object at 0x EA40EF0>, <matplotlib.axes._subplots.axessubplot object at 0x EAA1978>, <matplotlib.axes._subplots.axessubplot object at 0x EAA19B0>], [<matplotlib.axes._subplots.axessubplot object at 0x EB66828>, <matplotlib.axes._subplots.axessubplot object at 0x EBBD550>, <matplotlib.axes._subplots.axessubplot object at 0x EC20F60>]], dtype=object) Duration Je mehr Dateien geschrieben, desto länger dauert es, unabhängig von der Dateigrösse (unabhängig von der insgesamt geschriebenen Anzahl MB)

6 6 of :45 In [39]: %matplotlib inline import seaborn as sns sns.set(color_codes=true) df.plot.scatter(x="nf", y="dur",s=df['mbtot']/50) df.plot.scatter(x="nf", y="dur",c='mbtot') Out[39]: <matplotlib.axes._subplots.axessubplot at 0xf6b7f28>

7 of 8 29.03.2017 12:45 In [43]: df.plot.scatter(x="nf", y="iomean", s=df['mbtot']/50) #date nf thru IOmean IOsd dur df.plot.scatter(x="gbwritten",y="thru",s=df['mbtot']/50) MBtot Out[43]: <matplotlib.

7 7 of :45 In [43]: df.plot.scatter(x="nf", y="iomean", s=df['mbtot']/50) #date nf thru IOmean IOsd dur df.plot.scatter(x="gbwritten",y="thru",s=df['mbtot']/50) MBtot Out[43]: <matplotlib.axes._subplots.axessubplot at 0x10c08fd0> Helpfile hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-tests.jar DF SCIOTest [-nrfiles N] [-filesize MB] [-resfile resultfilename] [-buffersize Bytes] -nrfiles Number of files to generate within HDFS. This is also equivialvent to the number of map tasks that will get executed. -filesize Genterate a file for each map task ( -nrfiles ) with this size. Examp le -nrfiles 10 -filesize 250GB would generate 2500GB of data in HDFS cluster -resfile Path to local file system ( not hdfs ) where TestDFSIO will store the resutls for the test -buffersize Buffer size in bytes for each map tasks when reading and writing IO. Defaults to bytes ( about 1MB )

8 of 8 29.03.2017 12:45 In [44]: pd.tools.plotting.scatter_matrix(df[["nf","mbtot","thru","iomean","dur"]], alpha =0.2, diagonal='hist') Out[44]: array([[<matplotlib.axes._subplots.

8 8 of :45 In [44]: pd.tools.plotting.scatter_matrix(df[["nf","mbtot","thru","iomean","dur"]], alpha =0.2, diagonal='hist') Out[44]: array([[<matplotlib.axes._subplots.axessubplot object at 0x C70320>, <matplotlib.axes._subplots.axessubplot object at 0x D13400>, <matplotlib.axes._subplots.axessubplot object at 0x D77F28>, <matplotlib.axes._subplots.axessubplot object at 0x DD0C88>, <matplotlib.axes._subplots.axessubplot object at 0x E3D5F8>], [<matplotlib.axes._subplots.axessubplot object at 0x E3D630>, <matplotlib.axes._subplots.axessubplot object at 0x EFDDD8>, <matplotlib.axes._subplots.axessubplot object at 0x F5ADA0>, <matplotlib.axes._subplots.axessubplot object at 0x FC8828>, <matplotlib.axes._subplots.axessubplot object at 0x A128>], [<matplotlib.axes._subplots.axessubplot object at 0x ED68>, <matplotlib.axes._subplots.axessubplot object at 0x E9BA8>, <matplotlib.axes._subplots.axessubplot object at 0x D8>, <matplotlib.axes._subplots.axessubplot object at 0x AF668>, <matplotlib.axes._subplots.axessubplot object at 0x C0B8>], [<matplotlib.axes._subplots.axessubplot object at 0x >, <matplotlib.axes._subplots.axessubplot object at 0x D6B38>, <matplotlib.axes._subplots.axessubplot object at 0x A8>, <matplotlib.axes._subplots.axessubplot object at 0x C91518>, <matplotlib.axes._subplots.axessubplot object at 0x FAED2E8>], [<matplotlib.axes._subplots.axessubplot object at 0x FBA0E48>, <matplotlib.axes._subplots.axessubplot object at 0x F94CF60>, <matplotlib.axes._subplots.axessubplot object at 0x F880CC0>, <matplotlib.axes._subplots.axessubplot object at 0x F2903C8>, <matplotlib.axes._subplots.axessubplot object at 0x F4BDCF8>]], dtype=object)

Ulrich Stärk

< Ulrich Stärk ulrich.staerk@fu-berlin.de http://www.plat-forms.org 2 Einige Vorurteile Everything in the box is kind of weird and quirky, but maybe not enough to make it completely worthless. PHP: a fractal