Pro Perl Parsing. Christopher M. Frenz

Transcription

1 Pro Perl Parsing Christopher M. Frenz

2 Pro Perl Parsing Copyright 2005 by Christopher M. Frenz Lead Editors: Jason Gilmore and Matthew Moodie Technical Reviewer: Teodor Zlatanov Editorial Board: Steve Anglin, Dan Appleman, Ewan Buckingham, Gary Cornell, Tony Davis, Jason Gilmore, Jonathan Hassell, Chris Mills, Dominic Shakeshaft, Jim Sumser Associate Publisher: Grace Wong Project Manager: Beth Christmas Copy Edit Manager: Nicole LeClerc Copy Editor: Kim Wimpsett Assistant Production Director: Kari Brooks-Copony Production Editor: Laura Cheu Compositor: Linda Weidemann, Wolf Creek Press Proofreader: Nancy Sixsmith Indexer: Tim Tate Artist: Wordstop Technologies Pvt. Ltd., Chennai Cover Designer: Kurt Krames Manufacturing Manager: Tom Debolski Library of Congress Cataloging-in-Publication Data Frenz, Christopher. Pro Perl parsing / Christopher M. Frenz. p. cm. Includes index. ISBN (hardcover : alk. paper) 1. Perl (Computer program language) 2. Natural language processing (Computer science) I. Title. QA76.73.P22F '3--dc All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. Printed and bound in the United States of America Trademarked names may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, we use the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. Distributed to the book trade worldwide by Springer-Verlag New York, Inc., 233 Spring Street, 6th Floor, New York, NY Phone SPRINGER, fax , orders-ny@springer-sbm.com, or visit For information on translations, please contact Apress directly at 2560 Ninth Street, Suite 219, Berkeley, CA Phone , fax , info@apress.com, or visit The information in this book is distributed on an as is basis, without warranty. Although every precaution has been taken in the preparation of this work, neither the author(s) nor Apress shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in this work. The source code for this book is available to readers at in the Downloads section.

3 For Jonathan! You are the greatest son any father could ask for.

4 Contents at a Glance About the Author xiii About the Technical Reviewer xv Acknowledgments xvii Introduction xix CHAPTER 1 Parsing and Regular Expression Basics CHAPTER 2 Grammars CHAPTER 3 Parsing Basics CHAPTER 4 Using Parse::Yapp CHAPTER 5 Performing Recursive-Descent Parsing with Parse::RecDescent CHAPTER 6 Accessing Web Data with HTML::TreeBuilder CHAPTER 7 Parsing XML Documents with XML::LibXML and XML::SAX CHAPTER 8 Introducing Miscellaneous Parsing Modules CHAPTER 9 Finding Solutions to Miscellaneous Parsing Problems CHAPTER 10 Performing Text and Data Mining INDEX v

5 Contents About the Author xiii About the Technical Reviewer xv Acknowledgments xvii Introduction xix CHAPTER 1 Parsing and Regular Expression Basics Parsing and Lexing Parse::Lex Using Regular Expressions A State Machine Pattern Matching Quantifiers Predefined Subpatterns Posix Character Classes Modifiers Assertions Capturing Substrings Substitution Troubleshooting Regexes GraphViz::Regex Using Regexp::Common Regexp::Common::Balanced Regexp::Common::Comments Regexp::Common::Delimited Regexp::Common::List Regexp::Common::Net Regexp::Common::Number Universal Flags Standard Usage Subroutine-Based Usage In-Line Matching and Substitution Creating Your Own Expressions Summary vii

6 viii CONTENTS CHAPTER 2 Grammars Introducing Generative Grammars Grammar Recipes Sentence Construction Introducing the Chomsky Method Type 1 Grammars (Context-Sensitive Grammars) Type 2 Grammars (Context-Free Grammars) Type 3 Grammars (Regular Grammars) Using Perl to Generate Sentences Perl-Based Sentence Generation Avoiding Common Grammar Errors Generation vs. Parsing Summary CHAPTER 3 Parsing Basics Exploring Common Parser Characteristics Introducing Bottom-Up Parsers Coding a Bottom-Up Parser in Perl Introducing Top-Down Parsers Coding a Top-Down Parser in Perl Using Parser Applications Programming a Math Parser Summary CHAPTER 4 Using Parse::Yapp Creating the Grammar File The Header Section The Rule Section The Footer Section Using yapp The -v Flag The -m Flag The -s Flag Using the Generated Parser Module Evaluating Dynamic Content Summary

7 CONTENTS ix CHAPTER 5 Performing Recursive-Descent Parsing with Parse::RecDescent Examining the Module s Basic Functionality Constructing Rules Subrules Introducing Actions and %item and %arg $return $text $thisline and $prevline $thiscolumn and $prevcolumn $thisoffset and $prevoffset $thisparser $thisrule and $thisprod $score Introducing Startup Actions Introducing Autoactions Introducing Autotrees Introducing Autostubbing Introducing Directives <commit> and <uncommit> <reject> <skip> <resync> <error> <defer> <perl...> <score> and <autoscore> Precompiling the Parser Summary

8 x CONTENTS CHAPTER 6 Accessing Web Data with HTML::TreeBuilder Introducing HTML Basics Specifying Titles Specifying Headings Specifying Paragraphs Specifying Lists Embedding Links Understanding the Nested Nature of HTML Accessing Web Content with LWP Using LWP::Simple Using LWP Using HTML::TreeBuilder Controlling TreeBuilder Parser Attributes Searching Through the Parse Tree Understanding the Fair Use of Information Extraction Scripts Summary CHAPTER 7 Parsing XML Documents with XML::LibXML and XML::SAX Understanding the Nature and Structure of XML Documents The Document Prolog Elements and the Document Body Introducing Web Services XML-RPC RPC::XML Simple Object Access Protocol (SOAP) SOAP::Lite Parsing with XML::LibXML Using DOM to Parse XML Parsing with XML::SAX::ParserFactory Summary CHAPTER 8 Introducing Miscellaneous Parsing Modules Using Text::Balanced Using extract_delimited Using extract_bracketed Using extract_codeblock

9 CONTENTS xi Using extract_quotelike Using extract_variable Using extract_multiple Using Date::Parse Using XML::RSS::Parser Using Math::Expression Summary CHAPTER 9 Finding Solutions to Miscellaneous Parsing Problems Parsing Command-Line Arguments Parsing Configuration Files Refining Searches Formatting Output Summary CHAPTER 10 Performing Text and Data Mining Introducing Data Mining Basics Introducing Descriptive Modeling Clustering Summarization Association Rules Sequence Discovery Introducing Predictive Modeling Classification Regression Time Series Analysis Prediction Summary INDEX

10 About the Author CHRISTOPHER M. FRENZ is currently a bioinformaticist at New York Medical College. His research interests include applying artificial neural networks to protein engineering as well using molecular modeling techniques to determine the role that protein structures have on protein function. Frenz uses the Perl programming language to conduct much of his research. Additionally, he is the author of Visual Basic and Visual Basic.NET for Scientists and Engineers (Apress, 2002) as well as numerous scientific and computer articles. Frenz has more than ten years of programming experience and, in addition to Perl and VB, is also proficient in the Fortran and C++ languages. Frenz can be contacted at cfrenz@gmail.com. xiii

11 About the Technical Reviewer TEODOR ZLATANOV earned his master s degree in computer engineering from Boston University in 1999 and has been happily hacking ever since. He always wonders how it is possible to get paid for something as fun as programming, but tries not to make too much noise about it. Zlatanov lives with his wife, 15-month-old daughter, and two dogs, Thor and Maple, in lovely Braintree, Massachusetts. He wants to thank his family for their support and for the inspiration they always provide. xv

12 Acknowledgments Bringing this book from a set of ideas to the finished product that you see before you today would not have been possible without the help of others. Jason Gilmore was a great source of ideas for refining the content of the early chapters in this book, and Matthew Moodie provided equally insightful commentary for the later chapters and assisted in ensuring that the final page layouts of the book looked just right. I am also appreciative of Teodor Zlatanov s work as a technical reviewer, since he went beyond the role of simply finding technical inaccuracies and made many valuable suggestions that helped improve the clarity of the points made in the book. Beth Christmas also played a key role as the project manager for the entire process; without her friendly prompting, this book would probably still be in draft form. I would also like to express my appreciation of the work done by Kim Wimpsett and Laura Cheu, who did an excellent job preparing the manuscript and the page layouts, respectively, for publication. Last, but not least, I would like to thank my family for their support on this project, especially my wife, Thao, and son, Jonathan. xvii

13 Introduction Over the course of the past decade, we have all been witnesses to an explosion of information, in terms of both the amounts of knowledge that exists within the world and the availability of such information, with the proliferation of the World Wide Web being a prime example. Although these advancements of knowledge have undoubtedly been beneficial, they have also created new challenges in information retrieval, in information processing, and in the extraction of relevant information. This is in part due to a diversity of file formats as well as the proliferation of loosely structured formats, such as HTML. The solution to such information retrieval and extraction problems has been to develop specialized parsers to conduct these tasks. This book will address these tasks, starting with the most basic principles of data parsing. The book will begin with an introduction to parsing basics using Perl s regular expression engine. Once these regex basic are mastered, the book will introduce the concept of generative grammars and the Chomsky hierarchy of grammars. Such grammars form the base set of rules that parsers will use to try to successfully parse content of interest, such as text or XML files. Once grammars are covered, the book proceeds to explain the two basic types of parsers those that use a top-down approach and those that use a bottom-up approach to parsing. Coverage of these parser types is designed to facilitate the understanding of more powerful parsing modules such as Yapp (bottom-up) and RecDescent (top-down). Once these powerful and flexible generalized parsing modules are covered, the book begins to delve into more specialized parsing modules such as parsing modules designed to work with HTML. Within Chapter 6, the book also provides an overview of the LWP modules, which facilitate access to documents posted on the Web. The parsing examples within this chapter will use the LWP modules to parse data that is directly accessed from the Web. Next the book examines the parsing of XML data, which is a markup language that is increasingly growing in popularity. The XML coverage also discusses SOAP and XML-RPC, which are two of the most popular methods for accessing remote XML-formatted data. The book then covers several smaller parsing modules, such as an RSS parser and a date/time parser, as well as some useful parsing tasks, such as the parsing of configuration files. Lastly, the book introduces data mining. Data mining provides a means for individuals to work with extracted data (as well as other types of data) so that the data can be used to learn more about a given area or to make predictions about future directions that area of interest may take. This content aims to demonstrate that although parsing is often a critical data extraction and retrieval task, it may just be a component of a larger data mining system. xix

14 xx INTRODUCTION This book examines all these problems from the perspective of the Perl programming language, which, since its inception in 1987, has always been heralded for its parsing and text processing capabilities. The book takes a practical approach to parsing and is rich in examples that are relevant to real-world parsing tasks. While covering all the basics of parser design to instill understanding in readers, the book highlights numerous CPAN modules that will allow programmers to produce working parser code in an efficient manner.