T-( )-MALV, Natural Language Processing The programming language Perl

Transcription

1 T-( )-MALV, Natural Language Processing The programming language Perl Hrafn Loftsson 1 Hannes Högni Vilhjálmsson 1 1 School of Computer Science, Reykjavik University September 2010

2 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

3 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

4 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

5 Perl Background PERL = "Practical Extraction and Report Language" First version appeared in 1987: Larry Wall General purpose tool for Unix-like systems More powerful than scripting tools and simpler than C Emphasis on fast text processing An interpreted language Obtaining Perl: Part of most Linux installations. Part of Cygwin.

6 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

7 Hello world The code: hello.pl use s t r i c t ; # F o r c e s d e c l a r a t i o n o f v a r i a b l e s use w a r n i n g s ; # Compile and run time w a r n i n g s p r i n t "What i s your name? " ; my $name = <STDIN>; p r i n t " P l e a s e d to meet you ", $name ; Running it perl hello.pl

8 Perl The format of a program An ordinary text file (*.pl) with a sequence of Perl statements White spaces are ignored (except in strings) Each statements ends with a semicolon (";") Everything after "#" is ignored (comments) A short program #P r i n t s t e x t p r i n t " This i s a t e x t \n", " which o c c u p i e s two l i n e s \n" ;

9 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

10 Working with scalars Scalars The most basic data type in Perl Can be strings of characters or numbers Perl numbers are stored internally as floating-point values Scalar variables always start with "$" (dollar) Perl does automatic conversion between numbers and strings Perl does scalar interpolation Scalars are dynamically typed

11 Working with scalars Example $a =2; $b=6; $c=$a. $b ; $d=$c / 2 ; p r i n t "$d\n" ; # S c a l a r i n t e r p o l a t i o n Example $name= john ; print "My name is:\t$name\n";

12 Working with scalars Undefined variables use s t r i c t ; use w a r n i n g s ; my $hasvalue = " H e l l o " ; my $hasnovalue ; p r i n t " $hasvalue $hasnovalue \n" ;

13 Numerical operators use s t r i c t ; my $x = ; # $x i s 13 my $y = 2 $x / 4 ; # $y i s 6. 5 my $z = (2 6) 2 ; # $z i s 4096 my $a = ( $z 96) 2 ; # $a i s 8000 my $b = $x % 5 ; # 3, 13 modulo 5

14 Comparison operators Operation Numeric String version version less than < lt less than or equal to <= le greater than > gt greater than or equal to >= ge equal to == eq not equal to!= ne

15 Other data types Arrays A list is an ordered collection of scalars An array is a variable that contains a list Array variables start with "@" (at) Each element of a list is a separate scalar variable with an independent scalar value john,25); print "$g[0] is $g[1] @c=(@a,@b);

16 Other data types Arrays Perl arrays grow and shrink dynamically as needed You can easily mix different types of scalars within the same array A scalar variable associated with arrays $#array This variable contains the subscript of the last element in the array $array[$#array] # is the last element of the array The length of the array is always $#array + 1

17 Array scalar variable use s t r i c t ; = qw / H e l l o and welcome / ; : an a r r a y o f 3 e l e m e n t s p r i n t \n" ; $#somestuff = 0 ; now i s s i m p l y (" H e l l o ") p r i n t \n" ; $somestuff [ 1 ] = " Joe " ; i s (" H e l l o ", " Joe ") p r i n t \n" ; $#somestuff = 1; i s now = ( ) ; p r i n t \n" ;

18 Other data types Associative Array Same as a hash table Like an array. Instead of indexing the values by number, values are looked up by strings. Strings are keys. Hashes start with % Example %e n g l i s h =( e p l i => a p p l e, pera => pear ) ; p r i n t $ e n g l i s h { e p l i }, "\n" ; p r i n t keys(% e n g l i s h ), "\n" ; p r i n t values(% e n g l i s h ), "\n" ;

19 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

20 Control structures if statement while/until statement for statement If statement p r i n t "What i s your name? " ; my $name = <STDIN>; chomp( $name ) ; # remove the n e w l i n e c h a r a c t e r i f ( $name gt Fred ) { p r i n t "$name" comes a f t e r Fred \n" ; } e l s e { p r i n t "$name does not come a f t e r Fred \n" ; p r i n t "Maybe i t s the same s t r i n g, i n f a c t \n " ; }

21 Control structures foreach statement use s t r i c t ; c o l l e c t i o n = qw/ hat s h o e s s h i r t s s h o r t s / ; foreach my $item c o l l e c t i o n ) { p r i n t " $item \n" ; }

22 Control structures foreach statement use s t r i c t ; my %e n g l i s h =( e p l i => a p p l e, pera => pear ) ; foreach my $key ( keys %e n g l i s h ) { p r i n t " $ e n g l i s h { $key }\n" ; }

23 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

24 File handling A file handle A string (in upper case) which points to a file First associated with a file name and then used for reading and writing Perl has already associated STDIN, STDOUT and STDERR Example open (INPUT, " t e s t. t x t " ) ; open (OUTPUT, "> r e s u l t s. t x t " ) ; while (<INPUT>) { # Each l i n e saved i n the v a r i a b l e $_ p r i n t OUTPUT $_ ; } c l o s e (INPUT ) ; c l o s e (OUTPUT) ;

25 Command line arguments # I n p u t : A f i l e n a m e and 2 numbers ( x, y ) as p a r a m e t e r s # C o p ies l i n e s number x to y # Output i s w r i t t e n to s t a n d a r d output $ f i l e = s h i f t (@ARGV ) ; # get the f i l e name $ s t a r t = s h i f t (@ARGV ) ; # f i r s t l i n e $ l a s t = s h i f t (@ARGV ) ; # l a s t l i n e open ( INFILE, " $ f i l e " ) ; # open the f i l e $num = 1 ; while (<INFILE >) { i f ($num >= $ s t a r t && $num <= $ l a s t ) { p r i n t $_ ; } $num++; } c l o s e ( INFILE ) ;

26 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

27 Regular expressions Text processing Regular expressions are the core of Perl s text processing construct We use them to do pattern matching on text documents Three main usages: Matching: m/regex/modifiers Substitution: s/regex/replacement/modifiers Translation: tr/ search_list /replacement_list/ modifiers

28 Examples Simple match while(<stdin>) { p r i n t i f (m/ab c / ) ; } Simple match case insensitive while(<stdin>) { p r i n t i f (m/ab c/ i ) ; }

29 Examples Substitution and translation while ( $ l i n e = <STDIN>) { i f ( $ l i n e =~ m/ab c/ i ) { $ l i n e =~ s /ab c/abc/g ; } e l s e { $ l i n e =~ t r /a z /A Z / ; } p r i n t $ l i n e ; }

30 Grouping and parsing Refering to a matched string One often needs to refer to a matched string, for example, to see if it repeats itself or to print it A matched string can be cached by using () and then refered to by a number within or outside the expression Example while ( $ l i n e = <STDIN>) { while ( $ l i n e=~ m/\$ ([0 9]+)\.?([0 9] )/ g ) { p r i n t " D o l l a r s : ", $1, " Cents : ", $2, "\n" ; } }

31 The split function An extremely useful function Splits up a string and places it into an array The function uses a regular expression See

32 Outline 1 Perl Background Format of a program Data types and operators Control structures File handling and command line arguments Regular expressions 2 When to use Perl?

33 When should one use Perl? Quick and dirty When you need to write a quick and dirty program in a hurry! In most cases, Perl is used to develop a program which takes less than 1 hour to design, write and test. When processing text Great support for regular expression Simple file handling Useful data structures

34 Concordance (í. Orðstöðulykill) Concordances consist of text excerpts centered on a specific word and surrounded by a limited number of words before and after it. The most common concordances are so-called KWIC (Key Word In Context), in which each example of a key word stands in the middle of a line together with context words preceeding and following it in the text.

35 Implementing concordances in Perl concordance.pl use warnings; ($file_name, $pattern, $width) open(file, "<:utf8", "$file_name"); # Read UTF-8 data while ($line = <FILE>) { $text.= $line; } # new lines are replaced by spaces $text =~ s/\n/ /g; binmode STDOUT, ":utf8"; # matches the pattern with 0..width to the right and left while ($text =~ m/(.{0,$width}$pattern.{0,$width})/g) { print "$1\n"; # $1 contains the match }

36 Implementing concordances in Perl concordance.pl perl concordance.pl visindavefur2.txt vegna 15 > concordance.out perl concordance.pl visindavefur2.txt einnig 20 > concordance.out