Mineração de Dados Aplicada

Size: px

Start display at page:

Download "Mineração de Dados Aplicada"

Matilda McLaughlin
5 years ago
Views:

1 sed September, 10 th 2018 DCC ICEx UFMG

2 Outline for the next sessions Today sed; 12/09 awk s basics; 17/09 awk s array; 19/09 A few words about efficiency; 24/09 Oral presentation of your data. 2 / 17

3 Outline for the next sessions Today sed; 12/09 awk s basics; 17/09 awk s array; 19/09 A few words about efficiency; 24/09 Oral presentation of your data. 2 / 17

4 Outline for the next sessions Today sed; 12/09 awk s basics; 17/09 awk s array; 19/09 A few words about efficiency; 24/09 Oral presentation of your data. Date for the exam on the POSIX text-processing commands? 2 / 17

5 Outline for this session 1 Correction of last week exercises 2 Non-interactive editing 3 / 17

6 Correction of last week exercises Outline 1 Correction of last week exercises 2 Non-interactive editing 4 / 17

7 Correction of last week exercises Repeated user names Repeated user names tr -s < retweets.txt cut -d -f 5 sort -u sort -f uniq -id 5 / 17

8 Correction of last week exercises Repeated user names Repeated user names (all versions) tr -s < retweets.txt cut -d -f 5 sort -u sort -f uniq -id --all-repeated=separate 5 / 17

9 Correction of last week exercises Repeated user names Repeated user names tr -s < retweets.txt cut -d -f 5 sort -u sort -f uniq -id Without cut (whole lines are output) sort -uk 4 retweets.txt sort -fk 4 uniq -idf 3 5 / 17

10 Correction of last week exercises Count keywords With a Shell loop tr -s [:space:][:punct:] \n < text > words while read keyword do grep -ixc $keyword words done < keywords paste -d keywords - 6 / 17

11 Correction of last week exercises Count keywords With a Shell loop tr -s [:space:][:punct:] \n < text > words while read keyword do grep -ixc $keyword words done < keywords paste -d keywords - Without a Shell loop tr -s [:space:][:punct:] \n < text grep -ixf keywords sort -f uniq -ci 6 / 17

12 Correction of last week exercises Count keywords With a Shell loop tr -s [:space:][:punct:] \n < text > words while read keyword do grep -ixc $keyword words done < keywords paste -d keywords - Without a Shell loop tr -s [:space:][:punct:] \n < text grep -ixf keywords sort -f uniq -ci To only list the keywords occuring in the text tr -s [:space:][:punct:] \n < text grep -ixf keywords sort -uf 6 / 17

13 Outline 1 Correction of last week exercises 2 Non-interactive editing 7 / 17

14 Adding transformational capabilities grep only selects lines based on a regexp. sed additionally allows to transform those lines. 8 / 17

15 Adding transformational capabilities grep only selects lines based on a regexp. sed additionally allows to transform those lines. sed is Turing-complete. A complex sed script is quite hard to read (unless you are a computer). 8 / 17

16 Learning sed 1 The following slides; 2 Tutorials on the Web; 3 info sed. 9 / 17

17 Learning sed 1 The following slides; 2 Tutorials on the Web; 3 info sed. And cheat sheets on the Web... 9 / 17

18 A few options Generalizing grep, the -e option introduces a new script and -f a script file. By default, every line that is read is output (edited or not). The -n option avoids that. sed edits in place with the -i option. 10 / 17

19 sed s program structure In its simple form, a sed script is a condition (to select lines) followed by an action (on these lines). 11 / 17

20 sed s program structure In its simple form, a sed script is a condition (to select lines) followed by an action (on these lines). The most useful conditions are: a line number or $ for the last line, e. g., 8; 11 / 17

21 sed s program structure In its simple form, a sed script is a condition (to select lines) followed by an action (on these lines). The most useful conditions are: a line number or $ for the last line, e. g., 8; a regexp, e. g., /b.*a/; 11 / 17

22 sed s program structure In its simple form, a sed script is a condition (to select lines) followed by an action (on these lines). The most useful conditions are: a line number or $ for the last line, e. g., 8; a regexp, e. g., /b.*a/; ranges of lines whose boundaries are specified through line numbers or regexps, e. g., 8,$ or 1,/banana/; 11 / 17

23 sed s program structure In its simple form, a sed script is a condition (to select lines) followed by an action (on these lines). The most useful conditions are: a line number or $ for the last line, e. g., 8; a regexp, e. g., /b.*a/; ranges of lines whose boundaries are specified through line numbers or regexps, e. g., 8,$ or 1,/banana/; the negation of a condition by appending!, e. g., 6,/[xyz]/!. 11 / 17

24 sed s program structure In its simple form, a sed script is a condition (to select lines) followed by an action (on these lines). The most useful conditions are: a line number or $ for the last line, e. g., 8; a regexp, e. g., /b.*a/; ranges of lines whose boundaries are specified through line numbers or regexps, e. g., 8,$ or 1,/banana/; the negation of a condition by appending!, e. g., 6,/[xyz]/!. If no condition is given, every line is selected. 11 / 17

25 sed s commands sed s commands are one letter long: p prints the current line; 12 / 17

26 sed s commands sed s commands are one letter long: p prints the current line; d deletes the current line; 12 / 17

27 sed s commands sed s commands are one letter long: p prints the current line; d deletes the current line; i inserts a line (before the current line); 12 / 17

28 sed s commands sed s commands are one letter long: p prints the current line; d deletes the current line; i inserts a line (before the current line); a appends a line (after the current line); 12 / 17

29 sed s commands sed s commands are one letter long: p prints the current line; d deletes the current line; i inserts a line (before the current line); a appends a line (after the current line); / 17

30 sed s commands sed s commands are one letter long: p prints the current line; d deletes the current line; i inserts a line (before the current line); a appends a line (after the current line);... s substitutes part(s) of the current line. 12 / 17

31 Substituting with sed s s command The usual syntax is s/regexp/replacement text/{flags} but any character can replace the / delimiter. 13 / 17

32 Substituting with sed s s command The usual syntax is s/regexp/replacement text/{flags} but any character can replace the / delimiter. Like grep, sed matches the largest string satisfying the regexp. In the replacement text, & stands for the whole match and \n for the string matching the part of the regexp between the n th . 13 / 17

33 Substituting with sed s s command The usual syntax is s/regexp/replacement text/{flags} but any character can replace the / delimiter. Like grep, sed matches the largest string satisfying the regexp. In the replacement text, & stands for the whole match and \n for the string matching the part of the regexp between the n th . The most useful flags are a number N to transform the N th match per line (the first by default), g to transform all matches and p, in conjunction with the -n option, to only print the transformed lines. 13 / 17

34 Getting the data $ wget dcc.ufmg.br/~lcerf/data.tar.xz -O - tar -xj 14 / 17

35 Exercises Exercises 1 In teams.txt, replace Atlético with Atl.. 2 In teams.txt, place an exclamation mark at the beginning of the lines containing Atlético. 3 Execute the script teams.sed on retweets.txt. 4 If s:200:atlético-mg: would be the first line of teams.sed, this script would undesirably edit some retweet numbers and some user names that include the sequence of digits 200. Explain how the actual script avoids those issues. 5 How to obtain teams.sed from teams.txt? Notice that underscores replace the spaces in the team names. 15 / 17

36 Turning teams.txt into teams.sed From: 213 Gr^emio Prudente To: s:$[0-9] $213 :\1Gr^emio Prudente : 16 / 17

37 Turning teams.txt into teams.sed From: 213 Gr^emio Prudente To: s:$[0-9] $213 :\1Gr^emio Prudente : With one barely readable substitution tr < teams.txt sed s/$.*$ $.*$/s:\$[0-9] \$\1 :\\1\2 :/ > teams.sed 16 / 17

38 Turning teams.txt into teams.sed From: 213 Gr^emio Prudente To: s:$[0-9] $213 :\1Gr^emio Prudente : With one barely readable substitution tr < teams.txt sed s/$.*$ $.*$/s:\$[0-9] \$\1 :\\1\2 :/ > teams.sed With three substitutions tr < teams.txt sed -e s/^/s:\$[0-9] \$/ -e s/ / :\\1/ -e s/$/ :/ > teams.sed 16 / 17

39 License c These slides are licensed under the Creative Commons Attribution-ShareAlike 4.0 International License. 17 / 17

Mineração de Dados Aplicada

Mineração de Dados Aplicada Simple but Powerful Text-Processing Commands August, 29 th 2018 DCC ICEx UFMG Unix philosophy Unix philosophy Doug McIlroy (inventor of Unix pipes). In A Quarter-Century of Unix (1994): Write programs