Data Manipulation with AWK. Evangelos Pournaras, Izabela Moise

Transcription

1 Data Manipulation with AWK Evangelos Pournaras, Izabela Moise Evangelos Pournaras, Izabela Moise 1

2 AWK A "Swiss knife" for data manipulation, retrieval, formatting, processing, transformation, prototyping and more... Evangelos Pournaras, Izabela Moise 2

3 About AWK Check A pattern scanning and processing language. AWK name: Alfred V. Aho, Peter J. Wein-berger and Brian W. Kernighan (creators) An evolving yet, stable, cross-platform language. Written in 1977 at AT&T Bell Laboratories. Data-driven language. Posix standard for AWK: Various Implementations: gawk, nawk, mawk, spawk, etc. "AWK is a convenient and expressive programming language that can be applied to a wide variety of computing and data-manipulation tasks." Evangelos Pournaras, Izabela Moise 3

4 What you can do with AWK Manage small databases Validate data Produce indexes & perform document preparation tasks Experiment with algorithms you can adapt later to other programming languages Evangelos Pournaras, Izabela Moise 4

5 Implementations GAWK Extract bits and pieces of data for processing Sort bits Perform simple network communications MAWK JAWK Efficiency, byte code interpreter Java support NAWK, XGAWK, SPAWK, QTAWK, RunAWK, etc. Evangelos Pournaras, Izabela Moise 5

6 AWK Advantages Very simple Easy learning curve Standardized On-the-fly calculations No need to open/close files Interpreted, not compiled Avoiding the edit-compile-test-debug lifecycle Evangelos Pournaras, Izabela Moise 6

7 Programming Philosophy Programming in AWK: Building a list of rules Rules consist of a pattern and an action (pattern-1){action} (pattern-2){action}... Linear scans, handling one data element per time Resembling Hadoop philosophy Random access seek times vs. hard drives sizes Manipulating delimited text files in a single pass By design, division of a file in records & fields Each line is a record Fields are delimited by a special character Every clause is a potential action performed on the current record! Evangelos Pournaras, Izabela Moise 7

8 Comparison with other Languages A case study with converting triplets to sparse matrices: Source: Evangelos Pournaras, Izabela Moise 8

9 Running an AWK program Three ways to run an AWK program from command line: 1. >awk program input-file1 input-file >awk -f program-file input-file1 input-file Unix script: my-awk-script.sh #!/usr/bin/awk -f #awk rules go here Evangelos Pournaras, Izabela Moise 9

10 Program Structure # Initialization body BEGIN{ # initialization actions } #Main execution body { # main program actions } # Finalization body END{ # Final actions } Evangelos Pournaras, Izabela Moise 10

11 AWK Demonstration example-01.awk, example-02.awk Evangelos Pournaras, Izabela Moise 11

12 AWK Regular Expressions A pattern enclosed in slashes ( / ) checked if it matches each input record. letters, numbers, both. /foo/ matches! does not match alternation expression ˆ matches the beginning of a string $ matches the end of a string. matches any single character Evangelos Pournaras, Izabela Moise 12

13 AWK Demonstration Evangelos Pournaras, Izabela Moise 13

14 Scripts >awk /.edu/ {print $0} mail-list.txt >awk $1 ~ /J/ inventory-shipped.txt >awk $3 ~ /edu$ be$/ mail-list.txt >awk {if (length($0)>max) max=length($0)} END{print max} mail-list.txt >awk NF>0 inventory-shipped.txt >awk END{print NR} >awk NR%2==0 mail-list.txt >awk $1=="Jan" {sum+=$5} END{print sum} inventory-shipped.txt Evangelos Pournaras, Izabela Moise 14

15 Variables No variable declaration is needed. No type declaration is needed. Built-in variables: NF: number of fields NR: current record number FS: field separator Evangelos Pournaras, Izabela Moise 15

16 Functions Specified as follows: function awkfunction(a,b,c,d){ return a+b+c+d } Built-in functions: Numeric: String: sqrt, log, sin, cos, rand, log, etc. index, length, match, split, substr, etc. Evangelos Pournaras, Izabela Moise 16

17 Arrays Associative arrays: String for indices rather than numbers arrayname[string]=value Multi-dimensional arrays: Supported by concatenation of indices into one string foo[5,12]="value" Evangelos Pournaras, Izabela Moise 17

18 AWK Demonstration example-03.awk, example-04.awk Evangelos Pournaras, Izabela Moise 18

19 AWK Example - Arrays BEGIN{} { letters[$4]++; } END{ for(var in letters) print var, "exists", letters[var], if("a" in letters) print "A exists" else print "A does not exist" } "times." Evangelos Pournaras, Izabela Moise 19

20 Proposed Literature AWK scripts: A. D. Robbins. Gawk: Effective AWK Programming. Free Software Foundation, Inc., 4.1 edition, April How to read the user guide: Fast reading: Chapters 1-10 Practical examples: Chapters 11 Evangelos Pournaras, Izabela Moise 20

21 What is next? SQL and relational databases Plotting and visualizing data Evangelos Pournaras, Izabela Moise 21