REFACTORING FRAMENET FOR EFFICIENT RELATIONAL QUERIES. Zeeshan Asim Ahmad, B.S. Problem in Lieu of Thesis Prepared for the Degree of MASTER OF SCIENCE

Transcription

1 REFACTORING FRAMENET FOR EFFICIENT RELATIONAL QUERIES Zeeshan Asim Ahmad, B.S. Problem in Lieu of Thesis Prepared for the Degree of MASTER OF SCIENCE UNIVERSITY OF NORTH TEXAS December 2003 APPROVED: Paul Tarau, Major Professor Robert Brazile, Graduate Advisor Krishna Kavi, Chair of the Department of Computer Science Sandra L. Terrell, Interim Dean of the Robert B. Toulouse School of Graduate Studies

2 Ahmad, Zeeshan Asim, Refactoring FrameNet for Efficient Relational Queries. Master of Science (Computer Science), December 2003, 43 pp., 2 tables, 3 illustrations, 3 references. The FrameNet database is being used in a variety of NLP research and applications such as word sense disambiguation, machine translation, information extraction and question answering. The database is currently available in XML format. The XML database though a wholesome way of distributing data in its entireness, is not practical for use unless converted to a more application friendly database. In light of this we have successfully converted the XML database to a relational MySQL database. This conversion reduced the amount of data storage amount to less than half. Most importantly the new database enables us to perform fast complex querying and facilitates use by applications and research. We show the steps taken to ensure relational integrity of the data during the refactoring process and a simple demo application demonstrating ease of use.

3 ACKNOWLEDGMENTS I would like to thank two important people, without whom this thesis would not have been possible: my thesis advisor and my fiancée. First I thank my thesis advisor, Dr. Paul Tarau, a talented teacher and passionate scientist. I am indebted and thankful for the fresh new ideas and opportunities he offered. Secondly I would like to thank Jun my fiancée for the encouragement and support she provided throughout my work. ii

4 TABLE OF CONTENTS Page ACKNOWLEDGMENTS..ii LIST OF TABLES.iv LIST OF ILLUSTRATIONS.v Chapter 1. INTRODUCTION TO FRAMENET DATABASE 1 Overview Uses of the FrameNet Database 2. DESCRIPTION OF THE FRAMENET DATABASE... 3 Content of the Database Structure of the XML Database Limitations of the XML Database 3. REFACTORING PROCESS TO MYSQL TM DATABASE....6 Choice of Database Process of Creating the New Database 4. APPLICATION USING JDBC TM QUERY...12 Application Demo Future Work Conclusion APPENDIXES...14 REFERENCES...43 iii

5 LIST OF TABLES 1 MySQL tables for the Frames MySQL tables for the Lexical Units iv

6 LIST OF ILLUSTRATIONS 1 Entity-Relation Diagram for Frames Entity-Relation Diagram for Lexical Units Process diagram v

7 CHAPTER 1 INTRODUCTION TO FRAMENET DATABASE Overview The FrameNet database is the result of the FrameNet project [Baker et al. 1998; Fillmore & Baker 2001]. This database is an online lexical resource that provides a significant portion of the vocabulary of contemporary English. It contains a collection of semantically and syntactically annotated sentences from which structured information and combinatorial possibilities of each item can be retrieved. The project used a descriptive model based on semantic frames [Fillmore 1977, 1982, 1985; Fillmore & Atkins 1988] to create this database. The project enters observations into the database by taking careful annotated attestations taken from corpora. Every sentence is annotated with relation to a given word and is incorporated of phrases that are grammatically labeled to reflect the grammatical relation to the word, the semantic role played within the frame to which the word belongs, and its phrase type. Persons trained in frame semantic theory annotate the semantic roles manually. The syntactic information however is added automatically, and the full structured descriptions are produced automatically. 1

8 Uses of the FrameNet Database The FrameNet database may be used in a variety of NLP research and applications. The goal within the FrameNet is to show that the information recorded and made available in the database is usable for various NLP applications like word sense disambiguation, machine translation, information extraction and question answering. Currently FrameNet work is also taking place for other languages like Japanese, German and Chinese. There is some research in progress at the University of Colorado focusing on creating a semantic parser that produces FrameNet roles for input sentences and have also began work on semantic parsing in Chinese. At ICSI and UC Berkeley, they are investigating the use of the FrameNet semantic resource in Information Extraction (IE) and Natural Language Understanding (NLU). Also at San Diego State University a research team is studying the relation between the frames developed for English and the frames needed to represent Japanese texts. 2

9 CHAPTER 2 DESCRIPTION OF THE FRAMENET DATABASE Content of the Database The FrameNet data is broken into two sets of the data, the frames data and the lexical units data. The released data consists of information on 376 frames, and more than 6,800 lexical units. The data contained in them is available in both human-readable format and XML, designed for use by researchers in NLP, IE, etc. We are using the data provided in XML format for data extraction and programming purposes. The XML data can be queried using XPath Query, however this brings about some problems discussed later in this chapter. Structure of the XML Database The XML database provided for FrameNet database is comprised of one large file "frames.xml" which gives full information on all the frames. There is also one file for each lexical unit. If the lexical unit is annotated, each sentence is given with its text and labels for each of the layers. The file names for the lexical units are of the form lunnnn.xml. The DTD s for the two types of XML files can be found at the FrameNet website and are also provided with the data. I will give an abridge description of the two DTD s. 3

10 This interpretation of the DTD s excludes some elements which are part of the DTD solely for the purpose of formatting the data in XML but do no contain any tangible information. The frames.xml file contains a root element, which contains the 376 frames. Each frame then contains a definition, one or more FE s (Frame Elements), zero or more lemmas and zero or more notes regarding the frame and attributes: ID; name; semantictype; cby; and Cdate. Each FE in turn has a definition and attributes: ID; name; abbrev; cby; cdate; core; semanticrolerank; fgcolors; bgcolors; fgcolorp; and bgcolorp. Every lemma is comprised of zero or more notes, the number of times annotated, total, one or more lexeme and attributes: ID; Type; cby; and cdate. Element lexeme contains it s name and an a breakbefore field. Notes incorporate ID, type, cby and cdate. Lexunit is comprised of a definition, zero or more subcorpus s and attributes: ID; name; frame; and pos. The subcorpus consists of a name and zero or more sentences. Each sentence consists of a text, zero or more layers for the text, zero or more parents and attributes: ID; cby; cdate; and apos. Every layer consists of zero or more labels for that layer of the grammatical interpretation of the sentence text and attributes: ID; name; cby; cdate; lexunitref; frame; and lemma. The labels contain attributes with information: ID; name; cby; cdate; start; end; itype; fgcolors; bgcolors; fgcolorp; and bgcolorp. 4

11 Limitations of the XML Database The XML database though a wholesome way of distributing data in its entireness, is not practical for use unless converted to a more application friendly database. Querying the XML database is very slow and cannot be used for applications and research. Also being in text format, the uncompressed files total approximately 1 GB. Loading such large amount of data would take considerable time. To make the data more useable, it has to be transferred to a more application friendly database. 5

12 CHAPTER 3 REFACTORING PROCESS TO MYSQL TM DATABASE Choice of Database Three desirable features needed for a FrameNet database are data integrity, ease of use and query speed. Integrity of data is important for obvious reasons. Ease of use is important when using the database in an application as well as research. The FrameNet database is very limited when compared to the English language corpora, suggesting that the data is going to grow to many times it current size. We therefore desire a database with fast querying with very large data sets and such a database requires indexing. MySQL database was chosen because it has all three of the desired features. Also MySQL database is available for many common platforms and is free. Process of Creating the New Database The process of creating a new databases required first analyzing the XML database and creating an entity-relation diagram for the new database. Using the entity-relation diagram deriving the required tables becomes and elementary task. Figure1 and Figure2 show the entity-relation diagram for the frames and the lexical units respectively, followed by Table1 and Table2 showing the derived database tables. 6

13 Figure1: Entity-Relation Diagram for Frames 7

14 Table 1: MySQL database tables for the Frames Tables Col number frames fes fnotes lemmas Lexemes lnotes 1 fid feid fnid lid Lexname lnid 2 fname fename type lname breakbefore 3 femantictype abbrev cby pos Lid cby 4 fby cby cdate cby Lname date 5 fdate cdate note cdate note 6 definition core fid annotated lid 7 sementicrolerank fname total lname 8 fgcolors fid 9 bgcolors fname 10 fgcolorp 11 bgcolorp 12 definition 13 fid 14 fname yype 8

15 Figure 2: Entity-Relation Diagram for Lexical Units 9

16 Table 2: MySQL database tables for the Lexical Units Tables Col Number lus subcorpus sentences layers lables parents 1 luid subcorpusname sid laid lid parent 2 luname luid cby laname lname laid 3 frame luname cdate cby cby laname 4 pos apos cdate cdate 5 definition text lexunitref start 6 subcorpusname frame end 7 luid lemma itype 8 luname sid fgcolors 9 bgcolors 10 fgcolorp 11 bgcolorp 12 laid 13 laname 10

17 Figure 3 below shows the diagram for the process employed in converting the XML database to MySQL relational database. The SAX parser used to parse the XML files was downloaded from the Sun Microsystems website and customized to create insert statements in a MySQL batch file. The Java TM programming language code for the SAX parser is in Appendix A. Sample of the MySQL batch file created is in Appendix B. Script to create the tables in the new database is in Appendix C. Figure 3: Process diagram With the above process in place, more data can readily be added to the MySQL database when released by the FrameNet staff. The converted database now consumes less than half the storage memory space. Most importantly data querying is very fast and acceptable for application and research use. A nested query takes 0.01 second. 11

18 CHAPTER 4 APPLICATION USING JDBC TM QUERY Application Demo A demo application program for querying the database for test and performance is in Appendix D. This Java program connects to the MySQL database, which now contains the FrameNet data, to query the database and print the results. The program is given a sentence or a set of words. It parses the set of words and puts each word in a string array. It then takes each word and looks for it in the database and prints the result to screen. Below is an example of the set of words given to the program, the query and the results printed out. Example 1 Input sentence in the demo is There is an abundance of oil in the middle-east. The sql query executed by JDBC is rs = stmt.executequery( SELECT fname, lname FROM lemmas WHERE lname = + words[i] + ); Where word[j] iterates through all the words in the input sentence. Output to the screen is: Trying to connect Conncted! Frame : quantity Lemma : abundance Frame : food Lemma : oil Frame : locative_relation Lemma : in 12

19 Future Work More MySQL framenet database specific work on the JDBC connection is required for it to be more helpful for future research. Such work would include the development of a complete easy to use Java API which will perform faster and complex querying. This would make FrameNet specfic applications and research easier and broaden the use of this database. Conclusion The refactoring of the XML database into a MySQL relational database though a long process requiring careful planning has its rewards. The storage space required has been reduced to less than half. There is no data loading time because the database is ready and waiting to execute queries. Most of all fast and complex queries can easily be performed. 13

20 APPENDIX A SAX PARSER CODE 14

21 // File SAXSample.java import java.io.file; import java.io.inputstream; // Main class for SAX Sample public class SAXSample { // Main method args - String[] arguments public static void main(string[] args) { SAXSample jfs = new SAXSample(); // Create Order's Handler LuHandler ohandler = new LuHandler(); // Create the parser CreateParser parser = new CreateParser(oHandler); String curfilename = ""; String tempstring = ""; File f1; // Parse the XML file, handler generates the output // for loop is used to iterate the file names for the Lexical Unit // as the file names are of the form lunnnn.xml for(int i=1; i<=8000; i++) { curfilename = new String("lu" + i + ".xml"); f1 = new File(curfilename); // the file names are not in sequence so check if the file exists if (f1.exists()) { // prints the name of the current file underprocess System.out.println("Current file : " + curfilename); parser.parse(curfilename); // end if // end for 15

22 // File CreateParser.java import java.io.file; import java.io.inputstream; import javax.xml.parsers.saxparser; import javax.xml.parsers.saxparserfactory; import org.xml.sax.helpers.defaulthandler; import org.xml.sax.xmlreader; // Class to create a parser and parse an XML file public class CreateParser { private DefaultHandler handler; private SAXParser saxparser; private File f1; /** * Constructor handler - DefaultHandler for the SAX parser */ public CreateParser(DefaultHandler handler) { this.handler = handler; create(); /** * Create the SAX parser */ private void create() { try { // Obtain a new instance of a SAXParserFactory. SAXParserFactory factory = SAXParserFactory.newInstance(); // Specifies that the parser produced by this code will provide support for XML // namespaces. factory.setnamespaceaware(true); // Specifies that the parser produced by this code will validate documents as // they are parsed. factory.setvalidating(true); // Creates a new instance of a SAXParser using the currently configured //factory parameters. saxparser = factory.newsaxparser(); catch (Throwable t) { 16

23 t.printstacktrace(); /** * Parse a File file - File */ public void parse(file file){ try{ saxparser.parse(file,handler); // end try catch (Throwable t) { t.printstacktrace(); /** * Parse a URI uri - String */ public void parse(string uri){ try{ saxparser.parse(uri,handler); catch (Throwable t) { t.printstacktrace(); /** * Parse a Stream stream - InputStream */ public void parse(inputstream stream){ try{ saxparser.parse(stream,handler); catch (Throwable t) { t.printstacktrace(); 17

24 // File FrameHander.java import java.io.*; import java.text.numberformat; import org.xml.sax.helpers.defaulthandler; import org.xml.sax.saxexception; import org.xml.sax.helpers.defaulthandler; import org.xml.sax.attributes; // FrameHandler extends DefaultHandler, public class FrameHandler extends DefaultHandler { static private FileWriter out; private int inlemma = 0; private int infe = 0; private int startflag = 0; private String s = ""; private String frameelement = ""; // strings to store the names of the elements in the file "frame.xml" private String ROOT = "root"; private String FRAME = "frame"; private String FES = "fes"; private String FE = "fe"; private String DEFINITION = "definition"; private String LEMMAS = "lemmas"; private String LEMMA = "lemma"; private String NOTES = "notes"; private String NOTE = "note"; private String ANNOTATION = "annotation"; private String ANNOTATED = "annotated"; private String TOTAL = "total"; private String LEXEME = "lexeme"; // strings store the data for be put in the MySQL insert statement private String fid = ""; private String fname =""; private String semantictype = ""; private String fcby = ""; private String fcdate = ""; private String fdefinition = ""; private String feid = ""; 18

25 private String fename = ""; private String abbrev = ""; private String fecby = ""; private String fecdate = ""; private String core =""; private String semanticrolerank = ""; private String fgcolors = ""; private String bgcolors = ""; private String fgcolorp = ""; private String bgcolorp = ""; private String fedefinition = ""; private String fnid = ""; private String fntype = ""; private String fncby = ""; private String fncdate = ""; private String fnnote = ""; private String lid = ""; private String lname = ""; private String pos = ""; private String lcby = ""; private String lcdate = ""; private String annotated = ""; private String total = ""; private String lexname = ""; private String breakbefore = ""; private String lnid = ""; private String lntype = ""; private String lncby = ""; private String lncdate = ""; private String lnnote = ""; /** * Receive notification of the start of an element. namespaceuri - The Namespace URI, or the empty string if the element has no Namespace URI or if Namespace processing is not being performed. localname - The local name (without prefix), or the empty string if Namespace processing is not being performed. qname - The qualified name (with prefix), or the empty string if qualified names are not available. 19

26 atts - The attributes attached to the element. If there are no attributes, it shall be an empty Attributes object. SAXException - Any SAX exception, possibly wrapping another exception. */ public void startelement(string namespaceuri, String localname, String qname, Attributes atts) throws SAXException { if (startflag==0) { try { // Set up output stream out = new FileWriter("createframetables.txt" ); catch (Throwable t) { startflag = 1; frameelement = localname; // collect the attributes of the element 'layer' if (FRAME.equals(localName)) { fid = atts.getvalue("id"); fname = atts.getvalue("name"); semantictype = atts.getvalue("semantictype"); fcby = atts.getvalue("cby"); fcdate = atts.getvalue ("cdate"); // collect the attributes of the element 'fe' if (FE.equals(localName)) { feid = atts.getvalue("id"); fename = atts.getvalue("name"); fename = fename.replace('\'', '@'); abbrev = atts.getvalue("abbrev"); fecby = atts.getvalue("cby"); fecdate = atts.getvalue("cdate"); core = atts.getvalue("core"); semanticrolerank = atts.getvalue("semanticrolerank"); fgcolors = atts.getvalue("fgcolors"); bgcolors = atts.getvalue("bgcolors"); 20

27 fgcolorp = atts.getvalue("fgcolorp"); bgcolorp = atts.getvalue("bgcolorp"); infe = 1; // collect the attributes of the element 'note' if (NOTE.equals(localName)) { if (inlemma == 0) { fnid = atts.getvalue("id"); fntype = atts.getvalue("type"); fncby = atts.getvalue("cby"); fncdate = atts.getvalue("cdate"); else { lnid = atts.getvalue("id"); lntype = atts.getvalue("type"); lncby = atts.getvalue("cby"); lncdate = atts.getvalue("cdate"); // collect the attributes of the element 'lemma' if (LEMMA.equals(localName)) { lid = atts.getvalue("id"); lname = atts.getvalue("name"); lname = lname.replace('\'', '@'); pos = atts.getvalue("pos"); lcby = atts.getvalue("cby"); inlemma = 1; // collect the attributes of the element 'lexeme' if (LEXEME.equals(localName)) { lexname = atts.getvalue("name"); lexname = lexname.replace('\'', '@'); breakbefore = atts.getvalue("breakbefore"); // end startelement /** * Receive notification of character data inside an element. ch - The characters. start - The start position in the character array. 21

28 length - The number of characters to use from the character array. SAXException - Any SAX exception, possibly wrapping another exception. */ public void characters(char[] ch, int start, int length) throws SAXException { // read in the character data inside the element 'definition' if (DEFINITION.equals(frameelement)){ if (infe == 0) { fdefinition = new String(ch, start, length); fdefinition = fdefinition.replace('\'', '@'); else { fedefinition = new String(ch, start, length); fedefinition = fedefinition.replace('\'', '@'); // read in the character data inside the element 'note' if (NOTE.equals(frameelement)) { if (inlemma == 0) { fnnote = new String(ch, start, length); fnnote = fnnote.replace('\'', '@'); else { lnnote = new String(ch, start, length); lnnote = lnnote.replace('\'', '@'); // read in the character data inside the element 'annotated' if (ANNOTATED.equals(frameelement)) { annotated = new String(ch, start, length); // read in the character data inside the element 'total' if (TOTAL.equals(frameelement)) { total = new String(ch, start, length); /* Receive notification of the end of an namespaceuri - The Namespace URI, or the empty string if the element has no Namespace URI or if Namespace processing is not being 22

29 localname - The local name (without prefix), or the empty string if Namespace processing is not being qname - The qualified name (with prefix), or the empty string if qualified names are not SAXException - Any SAX exception, possibly wrapping another exception. */ public void endelement(string namespaceuri, String localname, String qname) throws SAXException { // change all data to lowercase to be stored in the database so that // queries will be stricly use lowercase changetolower(); // write the insert statement for the table 'frame' if (FRAME.equals(localName)) { emit("insert into frames values('" + fid + "','" + fname + "','" + semantictype + "','" + fcby + "','"+ fcdate + "','" + fdefinition + "');"); // write the insert statement for the table 'fe' if (FE.equals(localName)) { emit("insert into fes values('" + feid + "','" + fename + "','" + abbrev + "','" + fecby + "','"+ fecdate + "','" + core + "','" + semanticrolerank + "','" + fgcolors + "','" + bgcolors + "','"+ fgcolorp + "','" + bgcolorp + "','" + fdefinition + "','" + fid + "','" + fname + "');"); infe = 0; // write the insert statement for the table 'note' if (NOTE.equals(localName)) { if (inlemma == 0) { emit("insert into fnotes values('" + fnid + "','" + fntype + "','" + fncby + "','" + fncdate + "','"+ fnnote+ "','" + fid + "','" + fname + "');"); else { emit("insert into lnotes values('" + lnid + "','" + lntype + "','" + lncby + "','" + lncdate + "','"+ lnnote+ "','" + lid + "','" + lname + "');"); // write the insert statement for the table 'lemma' if (LEMMA.equals(localName)) { 23

30 emit("insert into lemmas values('" + lid + "','" + lname + "','" + pos + "','" + lcby + "','"+ lcdate + "','" + annotated + "','" + total + "','" + fid + "','" + fname + "');"); inlemma = 0; // write the insert statement for the table 'lexeme' if (LEXEME.equals(localName)) { emit("insert into lexemes values('" + lexname + "','" + breakbefore + "','" + lid + "','" + lname + "');"); // end of endelement // function changes all the required string data to lowercase private void changetolower() { fid = fid.tolowercase(); fname = fname.tolowercase(); semantictype = semantictype.tolowercase(); fcby = fcby.tolowercase(); fcdate = fcdate.tolowercase(); fdefinition = fdefinition.tolowercase(); feid = feid.tolowercase(); fename = fename.tolowercase(); abbrev = abbrev.tolowercase(); fecby = fecby.tolowercase(); fecdate = fecdate.tolowercase(); core = core.tolowercase(); semanticrolerank = semanticrolerank.tolowercase(); fgcolors = fgcolors.tolowercase(); bgcolors = bgcolors.tolowercase(); fgcolorp = fgcolorp.tolowercase(); bgcolorp = bgcolorp.tolowercase(); fedefinition = fedefinition.tolowercase(); fnid = fnid.tolowercase(); fntype = fntype.tolowercase(); fncby = fncby.tolowercase(); fncdate = fncdate.tolowercase(); fnnote = fnnote.tolowercase(); lid = lid.tolowercase(); lname = lname.tolowercase(); pos = pos.tolowercase(); lcby = lcby.tolowercase(); lcdate = lcdate.tolowercase(); annotated = annotated.tolowercase(); 24

31 total = total.tolowercase(); lexname = lexname.tolowercase(); breakbefore = breakbefore.tolowercase(); lnid = lnid.tolowercase(); lntype = lntype.tolowercase(); lncby = lncby.tolowercase(); lncdate = lncdate.tolowercase(); lnnote = lnnote.tolowercase(); // used to write the MySQL insert statements to file private void emit(string s) throws SAXException { try { out.write(s); nl(); out.flush(); catch (IOException e) { throw new SAXException("I/O error", e); // used to write an end of line to the file. private void nl() throws SAXException { String lineend = System.getProperty("line.separator"); try { out.write(lineend); catch (IOException e) { throw new SAXException("I/O error", e); 25

32 // File LuHander.java import java.io.*; import java.text.numberformat; import org.xml.sax.helpers.defaulthandler; import org.xml.sax.saxexception; import org.xml.sax.helpers.defaulthandler; import org.xml.sax.attributes; // LuHandler extends DefaultHandler public class LuHandler extends DefaultHandler { static private FileWriter out; private int inlemma = 0; private int infe = 0; // flag used to set the debugger on=1 / off=0; private int debugerror = 0; private int startflag = 0; private String s = ""; private String luelement = ""; // names of the element in the XML lexical unit files private String ROOT = "root"; private String DEFINITION = "definition"; private String NOTES = "notes"; private String NOTE = "note"; private String LEXUNIT = "lexunit"; private String SUBCORPUS = "subcorpus"; private String SENTENCE = "sentence"; private String TEXT = "text"; private String LAYER = "layer"; private String LABEL = "label"; private String PARENT = "parent"; // lexunit processing strings private String luid = ""; private String luname = ""; private String luframe = ""; private String lupos = ""; private String ludefinition = ""; private String sid = ""; 26

33 private String scby = ""; private String scdate = ""; private String sapos = ""; private String stext = ""; private String subcorpusname = ""; private String laid = ""; private String laname = ""; private String lacby = ""; private String lacdate = ""; private String lalexunitref = ""; private String laframe = ""; private String lalemma = ""; private String labid = ""; private String labname = ""; private String labcby = ""; private String labcdate = ""; private String labstart = ""; private String labend = ""; private String labitype = ""; private String labfgcolors = ""; private String labbgcolors = ""; private String labfgcolorp = ""; private String labbgcolorp = ""; private String parent = ""; /** * Receive notification of the start of an element. namespaceuri - The Namespace URI, or the empty string if the element has no Namespace URI or if Namespace processing is not being performed. localname - The local name (without prefix), or the empty string if Namespace processing is not being performed. qname - The qualified name (with prefix), or the empty string if qualified names are not available. atts - The attributes attached to the element. If there are no attributes, it shall be an empty Attributes object. SAXException - Any SAX exception, possibly wrapping another exception. */ 27

34 public void startelement(string namespaceuri, String localname, String qname, Attributes atts) throws SAXException { if (startflag==0) { try { // Set up output file stream when the start of the first element is detected out = new FileWriter("createlutables.txt" ); catch (Throwable t) { startflag = 1; if (debugerror==1) System.out.println("In the startelement : \n"); luelement = localname; // collect the attributes of the element 'layer' if (LAYER.equals(localName)) { laid = atts.getvalue("id"); laname = atts.getvalue("name"); laname = laname.replace('\'', '@'); lacby = atts.getvalue("cby"); lacdate = atts.getvalue ("cdate"); lalexunitref = atts.getvalue ("lexunitref"); laframe = atts.getvalue ("frame"); lalemma = atts.getvalue ("lemma"); lalemma = lalemma.replace('\'', '@'); if (debugerror == 1) System.out.println("laid =" + laid + "\n"); // collect the attributes of the element 'label' if (LABEL.equals(localName)) { labid = atts.getvalue("id"); labname = atts.getvalue("name"); labname = labname.replace('\'', '@'); labcby = atts.getvalue("cby"); labcdate = atts.getvalue("cdate"); labstart = atts.getvalue("start"); labend = atts.getvalue("end"); labitype = atts.getvalue("itype"); 28

35 labfgcolors = atts.getvalue("fgcolors"); labbgcolors = atts.getvalue("bgcolors"); labfgcolorp = atts.getvalue("fgcolorp"); labbgcolorp = atts.getvalue("bgcolorp"); if (debugerror == 1) System.out.println("labid =" + labid + "\n"); // collect the attributes of the element 'lexunit' if (LEXUNIT.equals(localName)) { luid = atts.getvalue("id"); luname = atts.getvalue("name"); luname = luname.replace('\'', '@'); luframe = atts.getvalue("frame"); lupos = atts.getvalue("pos"); if (debugerror == 1) System.out.println("luid =" + luid + "\n"); // collect the attributes of the element 'sentence' if (SENTENCE.equals(localName)) { sid = atts.getvalue("id"); scby = atts.getvalue("cby"); scdate = atts.getvalue ("cdate"); sapos = atts.getvalue ("apos"); if (debugerror == 1) System.out.println("sid =" + sid + "\n"); // collect the attributes of the element 'subcorpus' if (SUBCORPUS.equals(localName)) { subcorpusname = atts.getvalue("name"); if (debugerror == 1) System.out.println("subcorpusname =" + subcorpusname + "\n"); // end startelement /** 29

36 * Receive notification of character data inside an element. ch - The characters. start - The start position in the character array. length - The number of characters to use from the character array. SAXException - Any SAX exception, possibly wrapping another exception. */ public void characters(char[] ch, int start, int length) throws SAXException { // read in the character data inside the element 'definition' if (DEFINITION.equals(luelement)){ ludefinition = new String(ch, start, length); ludefinition = ludefinition.replace('\'', '@'); // read in the character data inside the element 'text' if (TEXT.equals(luelement)) { stext = new String(ch, start, length); stext = stext.replace('\'', '@'); // read in the character data inside the element 'parent' if (PARENT.equals(luelement)) { parent = new String(ch, start, length); /** * Receive notification of the end of an element. namespaceuri - The Namespace URI, or the empty string if the element has no Namespace URI or if Namespace processing is not being performed. localname - The local name (without prefix), or the empty string if Namespace processing is not being performed. qname - The qualified name (with prefix), or the empty string if qualified names are not available. SAXException - Any SAX exception, possibly wrapping another exception. */ public void endelement(string namespaceuri, String localname, String qname) throws SAXException { if (debugerror == 1) 30

37 System.out.println("endElement =" + luelement + "\n"); // change all data to lowercase to be stored in the database so that //queries will be stricly use lowercase changetolower(); // write the insert statement for the table 'lu' if (LEXUNIT.equals(localName)) { emit("insert into lus values('" + luid + "','" + luname + "','" + luframe + "','" + lupos + "','"+ ludefinition + "');"); // write the insert statement for the table 'sentences' if (SENTENCE.equals(localName)) { emit("insert into sentences values('" + sid + "','" + scby + "','" + scdate + "','"+ sapos + "','" + stext + "','" + subcorpusname + "','" + luid + "','" + luname + "');"); // write the insert statement for the table 'subcorpus' if (SUBCORPUS.equals(localName)) { emit("insert into subcorpus values('" + subcorpusname + "','" + luid + "','" + luname + "');"); // write the insert statement for the table 'layers' if (LAYER.equals(localName)) { emit("insert into layers values('" + laid + "','" + laname + "','" + lacby + "','" + lacdate + "','"+ lalexunitref+ "','" + laframe + "','" + lalemma + "','" + sid + "');"); // write the insert statement for the table 'labels' if (LABEL.equals(localName)) { if (debugerror == 1) System.out.println("Inside endelement LABEL :" + labid + "\n"); emit("insert into labels values('" + labid + "','" + labname + "','" + labcby + "','"+ labcdate + "','" + labstart + "','" + labend + "','" + labitype + "','" + labfgcolors + "','"+ labbgcolors + "','" + labfgcolorp + "','" + labbgcolorp + "','" + laid + "','" + laname + "');"); if (debugerror == 1) System.out.println("after emitting label =" + labid + "\n"); 31

38 // write the insert statement for the table 'parents' if (PARENT.equals(localName)) { if (debugerror == 1) System.out.println("parent =" + parent); emit("insert into parents values('" + parent + "','" + laid + "','" + laname + "');"); // end of endelement // function changes all the required string data to lowercase private void changetolower() { luid = luid.tolowercase(); luname = luname.tolowercase(); luframe = luframe.tolowercase(); lupos = lupos.tolowercase(); ludefinition = ludefinition.tolowercase(); sid = sid.tolowercase(); scby = scby.tolowercase(); scdate = scdate.tolowercase(); stext = stext.tolowercase(); subcorpusname = subcorpusname.tolowercase(); laid = laid.tolowercase(); laname = laname.tolowercase(); lacby = lacby.tolowercase(); lacdate = lacdate.tolowercase(); lalexunitref = lalexunitref.tolowercase(); laframe = laframe.tolowercase(); lalemma = lalemma.tolowercase(); labid = labid.tolowercase(); labname = labname.tolowercase(); labcby = labcby.tolowercase(); labcdate = labcdate.tolowercase(); labfgcolors = labfgcolors.tolowercase(); labbgcolors = labbgcolors.tolowercase(); labfgcolorp = labfgcolorp.tolowercase(); labbgcolorp = labbgcolorp.tolowercase(); parent = parent.tolowercase(); // used to write the MySQL insert statements to file 32

39 private void emit(string s) throws SAXException { try { out.write(s); nl(); out.flush(); catch (IOException e) { throw new SAXException("I/O error", e); // used to write an end of line to the file. private void nl() throws SAXException { String lineend = System.getProperty("line.separator"); try { out.write(lineend); catch (IOException e) { throw new SAXException("I/O error", e); 33

40 APPENDIX B SAMPLE MYSQL BATCH FILE 34

41 insert into labels values(' ','pun','bcronin',' ','157','157','null','black','white','black','white',' ','pos'); insert into layers values(' ','pos','bcronin',' ','7001','abundance','surfaced',' '); insert into labels values(' ','location','hmcarvey',' ','66','69','null','white','red','white','red',' ','fe'); insert into parents values(' ',' ','fe'); insert into layers values(' ','fe','bcronin',' ','7001','abundance','surfaced',' '); insert into labels values(' ','head','hmcarvey',' ','66','69','null','white','red','white','red',' ','gf'); 35

42 APPENDIX C SCRIPT CREATING THE MYSQL TABLES 36

43 CREATE TABLE frames (fid VARCHAR(30),fname VARCHAR(30),semantictype VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),definition TEXT,PRIMARY KEY(fname)); CREATE TABLE fes (feid VARCHAR(30),fename VARCHAR(30),abbrev VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),core VARCHAR(30),sementicrolerank VARCHAR(30),fgcolors VARCHAR(30),bgcolors VARCHAR(30), fgcolorp VARCHAR(30), bgcolorp VARCHAR(30), definition TEXT, fid VARCHAR(30), fname VARCHAR(30),PRIMARY KEY(feid(30),fename(30))); CREATE TABLE fnotes (fnid VARCHAR(30),type VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),note VARCHAR(30), fid VARCHAR(30), fname VARCHAR(30), PRIMARY KEY(fnid)); CREATE TABLE lemmas (lid VARCHAR(30),lname VARCHAR(30),pos VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),annotated VARCHAR(30),total VARCHAR(30), fid VARCHAR(30), fname VARCHAR(30),PRIMARY KEY(lid(30),lname(30))); CREATE TABLE lexemes (lexname VARCHAR(30),breakbefore VARCHAR(30),lid VARCHAR(30),lname VARCHAR(30), PRIMARY KEY(lexeme(30,lid(30),lname(30))); CREATE TABLE lnotes (lnid VARCHAR(30),type VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),note VARCHAR(30),lid VARCHAR(30), lname VARCHAR(30),PRIMARY KEY(lnid(30))); CREATE TABLE lus (luid VARCHAR(30),luname VARCHAR(30),frame VARCHAR(30),pos VARCHAR(30),definition TEXT,PRIMARY KEY(luid(30),luname(30))); CREATE TABLE subcorpus (subcorpus VARCHAR(30),luid VARCHAR(30),luname VARCHAR(30), PRIMARY KEY (subcorpus(30),luid(30),luname(30))); CREATE TABLE sentences (sid VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),apos VARCHAR(30),sentence_text TEXT,subcorpusname VARCHAR(30),luid VARCHAR(30),luname VARCHAR(30),PRIMARY KEY(sid(30))); CREATE TABLE layers (laid VARCHAR(30),laname VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),lexunitref VARCHAR(30),frame 37

44 VARCHAR(30),lemma VARCHAR(30),sid VARCHAR(30),PRIMARY KEY(laid(30),laname(30))); CREATE TABLE labels (lid VARCHAR(30),lname VARCHAR(30),cby VARCHAR(30),cdate VARCHAR(30),start VARCHAR(30),end VARCHAR(30),itype VARCHAR(30),fgcolors VARCHAR(30),bgcolors VARCHAR(30),fgcolorp VARCHAR(30),bgcolorp VARCHAR(30),laid VARCHAR(30),laname VARCHAR(30),PRIMARY KEY(lid(30),lname(30))); CREATE TABLE parents (parent VARCHAR(30),laid VARCHAR(30),laname VARCHAR(30), PRIMARY KEY(parent(30),laid(30),laname(30))); 38

45 APPENDIX D JDBC CONNECTION CODE 39

46 // File "SqlConn.java" import java.sql.*; import java.sql.connection; import java.sql.drivermanager; import java.sql.sqlexception; public class SqlConn { public static String inputsentence="there is an abundance of oil in the middle-east."; public static void main(string[] args) { try { // The newinstance() call is a work around for some // broken Java implementations Class.forName("com.mysql.jdbc.Driver").newInstance(); catch (Exception ex) { // handle the error try { System.out.println("Trying to connect..."); Connection conn = DriverManager.getConnection("jdbc:mysql://localhost/framenet"); System.out.println("connected!"); // now that conn has created JDBC connection Statement stmt = null; ResultSet rs = null; int strlength = inputsentence.length(); int index=0; int wordindex=0; int spaceindex=0; int newspaceindex=0; String words []= new String[30]; int lastindex=inputsentence.lastindexof(" "); int numwords=0; 40

47 while (newspaceindex < strlength) { if (newspaceindex==lastindex) { words[index]=inputsentence.substring(lastindex+1,strlength-1); newspaceindex=strlength; else { newspaceindex=inputsentence.indexof(' ',spaceindex+1); if (spaceindex!=0) spaceindex++; words[index] = inputsentence.substring(spaceindex,newspaceindex); numwords++; spaceindex=newspaceindex; index++; // end while try { stmt = conn.createstatement(); for(int i=0; i<numwords; i++) { // looking in lemmas for the frames rs = stmt.executequery("select fname, lname FROM lemmas WHERE lname = '" + words[i] + "'" ); //rs = stmt.executequery("select luname, frame FROM lus WHERE luname = '" + words[i] + "'" ); int x=0; while (rs.next()) { String name = ("Frame : " + rs.getstring(1) + " Lemma : " + rs.getstring(2)); //String name = ("Lexical Unit : " + rs.getstring(1) + " Frame : " + rs.getstring(2)); System.out.println(name); 41

48 x++; // end for // Now do something with the ResultSet... finally { // it is a good idea to release // resources in a finally{ block // in reverse-order of their creation // if they are no-longer needed if (rs!= null) { try { rs.close(); catch (SQLException sqlex) { // ignore rs = null; if (stmt!= null) { try { stmt.close(); catch (SQLException sqlex) { // ignore stmt = null; // End of doing something catch (SQLException ex) { // handle any errors System.out.println("SQLException: " + ex.getmessage()); System.out.println("SQLState: " + ex.getsqlstate()); System.out.println("VendorError: " + ex.geterrorcode()); // end main // end SqlConn 42

49 REFERENCES Baker, Collin. F., Charles J. Fillmore, and John B Lowe. (1998) The Berkeley FrameNet Project. In COLING-ACL 98: Proceedings of the Conference, 1998, in Montreal, Canada, pp Baker, Collin. F. and Charles J. Fillmore. (2001). Frame semantics for text understanding. NAACL Annual Meeting. In Proceedings of WordNet and Other Lexical Resources Workshop. Johnson, Christopher R. and Charles J. Fillmore (2000): The FrameNet tagset for frame-semantic and syntactic coding of predicate-argument structure. In Proceedings of the 1st Meeting of the North American Chapter of the Association for Computational Linguistics (ANLP-NAACL 2000), April 29-May 4, 2000, Seattle WA, pp