Mineração de Dados Aplicada

Size: px

Start display at page:

Download "Mineração de Dados Aplicada"

Brice Thompson
5 years ago
Views:

1 Simple but Powerful Text-Processing Commands August, 29 th 2018 DCC ICEx UFMG

2 Unix philosophy Unix philosophy Doug McIlroy (inventor of Unix pipes). In A Quarter-Century of Unix (1994): Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface. 2 / 22

3 Utilities from the 70s The Unix operating system came with several text-processing commands that are still very useful today. Specific, these commands are very efficient. The GNU project, started in 1984, has improved them a great deal (e. g., adding options). The original commands are part of some POSIX standards. 3 / 22

4 Utilities from the 70s The Unix operating system came with several text-processing commands that are still very useful today. Specific, these commands are very efficient. The GNU project, started in 1984, has improved them a great deal (e. g., adding options). The original commands are part of some POSIX standards. Most Free operating systems are POSIX-compliant: GNU/Linux, BSD, illumos, Haiku, etc. Mac OS X is too. Windows is not but Cygwin is a good compatibility layer. 3 / 22

5 The shell The shell interprets every command fired in a terminal or in an executable file, whose first line indicates the shell to use: #!/bin/sh (any POSIX-compliant shell) or #!/bin/bash or #!/bin/dash or #!/bin/zsh, etc. A 2-line shell script #!/bin/sh # Get started with the MDA exercises wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj mv data data-$(date +%m%d) 2> /dev/null 4 / 22

6 Geeks and repetitive tasks 5 / 22

7 Standard I/O POSIX text processing commands: process the input line by line; read, by default, the standard input (the keyboard if not redirected); write, by default, on the standard output (the terminal if not redirected). 6 / 22

8 Standard I/O POSIX text processing commands: process the input line by line; read, by default, the standard input (the keyboard if not redirected); write, by default, on the standard output (the terminal if not redirected). </> redirects the standard I/O from/to a file. A pipe binds an output stream to an input stream. It can bear a name (in argument of mkfifo) but most workflows only need the unnamed pipe,. redirects the standard output of the command on the left to the standard input of the command on the right. 6 / 22

9 Getting the data $ wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj 7 / 22

10 Getting the data $ wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj wget and tar are two GNU commands. Like all GNU commands: the man command (e. g., man wget) gives their specifications; 7 / 22

11 Getting the data $ wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj wget and tar are two GNU commands. Like all GNU commands: the man command (e. g., man wget) gives their specifications; the info command (e. g., info wget) often provides more detailed explanations, examples of use, etc.; 7 / 22

12 Getting the data $ wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj wget and tar are two GNU commands. Like all GNU commands: the man command (e. g., man wget) gives their specifications; the info command (e. g., info wget) often provides more detailed explanations, examples of use, etc.; long options are prefixed with --, short (i. e., one letter) options with - and can be grouped (e. g., -xj); 7 / 22

13 Getting the data $ wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj wget and tar are two GNU commands. Like all GNU commands: the man command (e. g., man wget) gives their specifications; the info command (e. g., info wget) often provides more detailed explanations, examples of use, etc.; long options are prefixed with --, short (i. e., one letter) options with - and can be grouped (e. g., -xj); options can take (right after) an argument; - means the standard input (/dev/stdin) or the standard output (/dev/stdout); 7 / 22

14 Getting the data $ wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj wget and tar are two GNU commands. Like all GNU commands: the man command (e. g., man wget) gives their specifications; the info command (e. g., info wget) often provides more detailed explanations, examples of use, etc.; long options are prefixed with --, short (i. e., one letter) options with - and can be grouped (e. g., -xj); options can take (right after) an argument; - means the standard input (/dev/stdin) or the standard output (/dev/stdout); the unnamed pipe,, redirects the standard output of the command on the left to the standard input of the command on the right. 7 / 22

15 Reading a large text file Your favorite text editor (Vim or Emacs?) loads the entire file in main memory, a problem if it weights gigabytes or more. 8 / 22

16 Reading a large text file Your favorite text editor (Vim or Emacs?) loads the entire file in main memory, a problem if it weights gigabytes or more. The solution is named less. It is the viewer for man pages. A few commands inside less: Page-up/down, R (repaint), F (follow), [0-9]+ (scroll that many lines), / (search forwards for a regexp),? (search backwards for a regexp), q (quit). 8 / 22

17 Reading a large text file Your favorite text editor (Vim or Emacs?) loads the entire file in main memory, a problem if it weights gigabytes or more. The solution is named less. It is the viewer for man pages. A few commands inside less: Page-up/down, R (repaint), F (follow), [0-9]+ (scroll that many lines), / (search forwards for a regexp),? (search backwards for a regexp), q (quit). Exercise Find, with less, the IP address of the first Brazilian visitor after the 100 th line of DistroWatch/ /debian. 8 / 22

18 Printing the first/last lines Do not test your scripts on the whole dataset! head outputs the head of a file; tail its tail. A few options: -[0-9]+ tunes the number of lines (10 by default), -n too but a -/+ prefix asks head/tail to display all lines except the provided number of last/first lines, -f follows the appended data (tail only). 9 / 22

19 Printing the first/last lines Do not test your scripts on the whole dataset! head outputs the head of a file; tail its tail. A few options: -[0-9]+ tunes the number of lines (10 by default), -n too but a -/+ prefix asks head/tail to display all lines except the provided number of last/first lines, -f follows the appended data (tail only). Exercise Print the lines 5 to 15 of one file in DistroWatch.com s logs. 9 / 22

20 Shuffling lines The shuf command shuffles the lines of a file. A few options: -n [0-9]+ outputs only the number of lines in argument (uniform random sampling without replacement), -r allows repetitions (uniform random sampling with replacement), -e specifies a line (rather than taking those in a file). 10 / 22

21 Shuffling lines The shuf command shuffles the lines of a file. A few options: -n [0-9]+ outputs only the number of lines in argument (uniform random sampling without replacement), -r allows repetitions (uniform random sampling with replacement), -e specifies a line (rather than taking those in a file). Exercise Sample, in a uniformly random way, 100 visits to the Ubuntu page on April, 28 th / 22

22 Concatenating files The cat command concatenates files and prints them. 11 / 22

23 Concatenating files The cat command concatenates files and prints them. Exercise Concatenate the files related to the visits to the Ubuntu page. 11 / 22

24 Replacing characters Given, in argument, two lists of characters, the tr command outputs its standard input after replacing (aka translating) every character in the first list with the one at the same position in the second list. The last character in the second list is considered repeated up to the size of the first list. - between two characters defines an interval. A few options: -c specifies the first list as the complement of the provided characters, -d deletes (rather than replaces) the characters in the first list, given consecutive characters found in the first list, -s substitutes the first one and deletes the rest (squeeze). 12 / 22

25 Exercise Basic reformatting with tr Choose one of the files in DistroWatch.com s logs and: 1 change its delimiters into spaces; 2 make its country codes lower cased. Notice that the shell processes the command line before executing it: the command line is broken w.r.t. whitespaces, $var is replaced by the content of the shell variable var, * is replaced by any string to match existing file paths, etc. To preserve the literal meaning: a backslash can precede every special character; a (portion of an) argument can be enclosed with single quotes (with double quotes to still let the shell interpret $ and ). 13 / 22

26 Counting lines, words and/or characters The wc command counts the number of lines (option -l), of words (option -w) or of characters (option -m). 14 / 22

27 Counting lines, words and/or characters The wc command counts the number of lines (option -l), of words (option -w) or of characters (option -m). Exercise In DistroWatch.com s logs, what are the numbers of visits per day to the Ubuntu page. Is there something special (compare to other distributions)? 14 / 22

28 Selecting fields The cut command selects fields. cut considers that there is an empty field between two subsequent delimiters. -d specifies the delimiter, -f specifies the fields cut must keep (comma-separated numbers or intervals, using -). 15 / 22

29 Selecting fields The cut command selects fields. cut considers that there is an empty field between two subsequent delimiters. -d specifies the delimiter, -f specifies the fields cut must keep (comma-separated numbers or intervals, using -). Exercise From one of the file in DistroWatch.com s logs, get rid of the IP addresses. 15 / 22

30 Pasting The paste command concatenates the lines of the input files in the order they are given. -d specifies the delimiter. 16 / 22

31 Comparing comm compares the lines of two sorted files. It outputs the lines only in the first file (first column), only in the second file (second column), and those in both files (third column). -1, -2 and -3 respectively remove the first, second and third column from the output. 17 / 22

32 Joining join joins the lines of two files. The join fields of both files must be sorted. A few options: -1 NUM sets the join field of the first file, -2 NUM for the second file, -j for both, -a {1, 2} prints unpairable lines from the specified file too, -v {1, 2} only prints unpairable lines from the specified file, -i ignores case, -t CHAR sets the delimiter. 18 / 22

33 Sorting The sort command sorts a text file. The locale settings affect the ordering (you may want LC ALL=C). A few options: -r reverses the ordering, -f ignores case, -n numerically sorts, -c checks if sorted, -k POS1[,POS2] sorts according to the fields in the interval, -t sets the field delimiter, -u removes duplicates, -m merges already sorted files. 19 / 22

34 Sorting The sort command sorts a text file. The locale settings affect the ordering (you may want LC ALL=C). A few options: -r reverses the ordering, -f ignores case, -n numerically sorts, -c checks if sorted, -k POS1[,POS2] sorts according to the fields in the interval, -t sets the field delimiter, -u removes duplicates, -m merges already sorted files. Exercise In DistroWatch.com s logs: 1 How many different countries are there? 2 On the 29th, what are the countries who originated visits to the site but no visit to Ubuntu s page? 19 / 22

35 Reporting or omitting repeated lines The uniq command removes adjacent repeated lines. A few options: -c reports the counts of repetitions, -d only prints repeated lines, -u only prints unique lines, -i ignores case. 20 / 22

36 Reporting or omitting repeated lines The uniq command removes adjacent repeated lines. A few options: -c reports the counts of repetitions, -d only prints repeated lines, -u only prints unique lines, -i ignores case. Exercise In DistroWatch.com s logs, what are the ten countries that originated the highest numbers of accesses to the index page, during the three days. 20 / 22

37 Homework Exercise Exercise for next Wednesday Given a list of keywords and a text, use the commands presented so far to: 1 list each keyword occurring in the text; 2 count the occurrences of each keyword in the text. Reading for next Wednesday Basic Regular Expressions in Wikipedia. 21 / 22

38 License c These slides are licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. 22 / 22

Mineração de Dados Aplicada

Mineração de Dados Aplicada sed September, 10 th 2018 DCC ICEx UFMG Outline for the next sessions Today sed; 12/09 awk s basics; 17/09 awk s array; 19/09 A few words about efficiency; 24/09 Oral presentation of your data. 2 / 17