Web-based File Upload and Download System

Transcription

1 COMP4905 Honor Project Web-based File Upload and Download System Author: Yongmei Liu Student number: Supervisor: Dr. Tony White 1

2 Abstract This project gives solutions of how to upload documents to the webserver as well as how to perform a fast search using Lucene search engine. The user needs to be authenticated to be able to use the service. The search can be done by using the document s contents as well as keywords and category assigned when it was uploaded. The supported file formats are: html, PDF, rtf, doc, xml and txt. The web container used in the project is Tomcat. 2

3 Acknowledgement I would like to thank my supervisor, Dr. Tony White and my friend: Mr. Kevin Yamamoto for their guidance, help and support during my project. 3

4 Table of Contents COMP4905 Honor Project...1 Web-based File Upload and Download System...1 Abstract...2 Acknowledgement...3 Table of Contents...4 Introduction...5 Design Decision...7 Developing Environment and Tools...8 Developing Platform...8 Environment overview...9 APIs...10 Developing Application Tools...11 Implementation...12 File upload...13 Document indexing...14 Document search...17 Run time environment setup and deployment...19 User guide...20 User authentication...20 File upload and indexing...20 Document Search and Download...20 Test scenarios...22 Reference...24 Glossary...25 Appendix...26 File list

5 Introduction The purpose of the project is to build a web site that allows students to submit papers to a central repository. The Users must provide keywords to describe the paper submitted, along with a small number of categories. The papers received are automatically indexed and inserted into a database. The web site provides three types of interfaces: 1) User login/register interface: Users need to be authenticated to be able to access the service. 2) Upload (build index) interface: Users are allowed to provide keywords to describe the paper submitted. The paper received will be automatically indexed and inserted into a database. 3) Search interface: Users can retrieve papers from the database. Lucene is used to index the files in the database. Papers submitted in PDF, DOC, RTF, text, HTML and XML formats are supported in both interfaces. Lucene is used to index the files in the database. Only authenticated users are able to submit papers to the database. Accessible through a web browser and based on a thin client, the Central Repository provides a single environment to store the papers submitted by users. Lucene also is used as the core of any search functionality. This report covers the Design Decisions, Developing Environment and Tools, Implementation, Run Time Environment Setup and Deployment, and User Guide. 5

6 Figure 1 charts the process. Figure 1 The flow chart User login Fail Authenticate Pass Upload Search Database Index 1 Index 2 Index 3 File1 File2 File3 Webserver 6

7 Design Decision 1. Tomcat was chosen to be web server container because it is open source, easy to install (embedded in JWSDP) and easy to use. It works on most all popular operating system and independent of platform. 2. In order to achieve the ease for change, a file: configuration.jsp being created to specify the paths. 3. Lucene was selected as it is fast indexing, independent index database platform that adapts from one platform to another. Lucene also integrates directly with the Web application. Lucene works with any kind of plain text data. 4. Upload Bean is a Java component that allows the uploading of files. This bean can be integrated in any JSP/Servlets application. The application is available as shareware as a low cost solution. 5. DOC is the most popular document format. In this project, we provide the solution for DOC format indexing. POI (an API is for converting word format to.txt format) was selected as it supports Microsoft's OLE 2 Compound Document format. Also it will support Java Excel solution as well as Java Word solution. 7

8 Developing Environment and Tools This project is developed using Java WSDP1.2 with Apache Tomcat container under Windows environment. The following Open Source APIs are also used to implement the required functionalities: UploadBean1.5 File upload Java Bean Lucene 1.3 Document keywords index and search PDFBox Convert PDF document to text format for index building POI - Convert MS Word document to text format for index building Javax.swing.text.rtf Convert rtf document to text format for index building Developing Platform Eclipse is used to develop Java, JSP, and Servlet. Eclipse is an open platform for tool integration built by an open community of tool providers. Operating under an open source paradigm, with a common public license that provides royalty free source code and worldwide redistribution rights, the eclipse platform provides tool developers with ultimate flexibility and control over their software technology. Lomboz plug-in is added mainly for Servlet development on top of Eclipse as well as compressing all the executable files and libraries in this project. Lomboz is an open-source plug-in for Eclipse Development Platform. Lomboz for 8

9 Eclipse essentially enables Java developers to build, test and deploy using J2EE based 100% Java Application Servers. Lomboz integrates various J2EE component and web application development tasks with Eclipse and some of the most popular application servers. Environment overview Operating System: Windows 2000/XP JDK 1.4 with runtime environment (JRE) JWSDP The Java Web Services Developer Pack (Java WSDP) is a free integrated toolkit that allows Java developers to build and test XML applications, Web services, and Web applications with the latest Web services technologies and standards implementations. It has Apache Tomcat built-in. Apache Tomcat implements the Servlet and JavaServer Pages specifications from Java Software, and includes many additional features that make it a useful platform for developing and deploying web applications and web services. 9

10 APIs Lucent gets involved in most part of the project. It is a high-performance, full-featured text search engine library written entirely in Java. It is a technology suitable for nearly any application that requires full-text search, especially cross-platform. Upload Bean provides a simple API to read and store uploaded files sent from a browser. It is a "technical" component that could be integrated in any JAVA/JSP/Servlets application. PDFBox is a Java API providing conversion from PDF format to text. POI is a Java API used to convert word type file to text. The POI project consists of APIs for manipulating various file formats based upon Microsoft's OLE 2 Compound Document format using pure Java. In short, you can read and write MS Excel files using Java. Soon, you'll be able to read and write Word files using Java. POI is your Java Excel solution as well as your Java Word solution. Javax.swing.text.rtf is a Java API used to convert rtf document to text format for index building. 10

11 Developing Application Tools Servlet Servlets are the Java platform technology of choice for extending and enhancing Web servers. Servlets provide a component-based, platform-independent method for building Web-based applications, without the performance limitations of CGI programs. And unlike proprietary server extension mechanisms (such as the Netscape Server API or Apache modules), servlets are server- and platform-independent. This leaves you free to select a "best of breed" strategy for your servers, platforms, and tools. JSP JSP technology is an extension of the servlet technology created to support authoring of HTML and XML pages. It makes it easier to combine fixed or static template data with dynamic content. Even if you're comfortable writing servlets, there are several compelling reasons to investigate JSP technology as a complement to your existing work. 11

12 Implementation Figure 2 The process diagram Client User login/ Upload/ Search/ register Server index result User data user search upload index database file index 12

13 User authentication The system requires user login, where the user s identity can be checked at the client side. In the login page, the user will be asked to supply his/her user ID and password. The length of the password is set to be at least 6 characters long and must include at least one number. When a user has done typing his/her password and clicked on any area in the window other than password area, the format of the password will be checked by the JavaScript at the client side. An alert pop-up window will be prompted if the format of the password is wrong. The file: users.dat contains the entries of the user name, password and . After the user hit the login button, the servlet: UserAuth is called to authenticate the user-supplied information at the server side. Here the program goes through the user.dat file to see if a match can be found. If not, the user will be given an error message and asked to re-login. For the first time user, he/she has to register first. register.html will bring up a registration interface. The user will be asked to input his/her user name, password and address. The format of the password and address will be checked by the JavaScript. If the required data are valid, they will be written into the file: users.dat. File upload Once the user is successfully logged in, he/she will be directed to the file upload and search interface. In this page, the user needs to select a file to be uploaded from the web browser. It is optional to the user to supply keywords or specify a category to go 13

14 with the file or not. The default value a category is ALL if it is not specified. If keywords or category are provided, later the user can use them to search for the file. No matter the user supplies his/her own keywords or not, Lucene will specify keywords by using a standard analyzer. Once the Upload and Index button is hit, the file will be uploaded to the server (the directory specified in the configuraton.jsp). This task is executed by Java Bean: UploadBean. UploadBean is a JAVA component (JavaBean ) that allows uploading files. This bean could be integrated in any JSP/Servlets application. It provides a simple API to read and store uploaded files sent from a browser. Document indexing This is most important part of the project. After the file has been uploaded to the server, Lucene is used to index the files in the database. The index is a special database that contains a compiled version of the documents and is optimize for quick lookup for a list of documents that contain certain words (or 'terms'). Lucene API provides elaborate control over the information stored in the index for each document and how this information is used during indexing and searching. On one extreme, you can store for each document just its location (e.g. URL) and index the content of the document as a monolithic piece of text. On the other extreme, you can you can store the entire document as well as various attributes such as Category, Keywords, or something else and perform searches that consider these attribute for matching and ranking. 14

15 These various attributes are considered as fields. (When performing a search you can either specify a field, or use the default field. The field names and default field is implementation specific.) The default field is content. Each field has a name, a value associated and three attributes. The three attributes are: 1) isindexed : set to true if this field will be used during the search. 2) isstored : set to true if the content of the field needs to be stored in the index. 3) istokenized : set to true if the content of this field needs to be tokenized into terms. The fields defined in this project are as follows: Name: Uid Path Modified Contents HTML Keywords Category Value: File name File Last The file Summary Keywords Category location modified and title supplied by chosen by time the user the user The attribute: isindexed is set to true for fields: Contents, Keywords and Category in order to find the pointed file during search. And the attribute: istokenized is set to be true for fields: Contents and Keywords so that values can be broken into terms. The above fields construct a Document, and then Lucene uses an Indexwriter to add this document to create index files. So, each field with isindex attribute true is indexed to the file. In this project, the user has four options to index a file: 1) Supply keywords: The file is indexed by the keywords supplied by the user as well as the contents of the file. 2) Not supply keywords: The file is indexed only by the contents of the file. 15

16 3) Select a category: The file is indexed by the category specified by the user as well as the contents of the file. 4) Supply keywords, contents and select a category: The file is indexed by all three fields: Contents, Keywords and Category. When the file in indexed by the contents of the file, a standard Analyzer provided by Lucene is used to break the contents into terms, so they can be used for search later. Since Lucene 1.3 only support the text format document index building and searching, those documents with special format, such as MS WORD DOC, RTF and PDF, should be converted into text format before being indexed. My project supplied three of most popular documents format indexing and searching by using Open Source API to convert them into text format. 1) Convert PDF to.txt A function PDF2txt in AllTypeDocument.java is called to convert PDF file into text format when building the index of PDF files. It is implemented by using PDFBox API. The PDFBox represents a high level API for creating and manipulating PDF documents. 2) Convert DOC to.txt A function DOC2txt is called to convert DOC file into text format when building the index of DOC files. It is implemented by using POI API. It is for manipulating various file formats based upon Microsoft's OLE 2 Compound Document format using pure Java. 16

17 3) Convert RTF to.txt A Java API: Javax.swing.text.rtf is called to convert rtf document to text format for index building. For HTML files, HTML parser supplied by Lucene has to be used to extract content from HTML pages. xml files are treated as text files. As last resort, Re-buildAllIndex function is supplied to re-index all the documents uploaded in case something is wrong. Document search JSP fileresult will search the keyword and list the matched files. There are several ways to do a search: 1) If the user doesn t supply the keywords, which were specified by the user when the file was uploaded to the server, the search will be done according to the contents. 2) If the user supplies the keywords, which were specified by the user when the file was uploaded to the server, Lucene will locate the file through the mapping between the keywords and the file. 3) The user can search by the category. 4) The combination of all of the above. Note: the relationship between category and either contents or keywords is AND. Although Lucene provides the ability to create your own queries through its API, it 17

18 also provides a rich query language through the Query Parser. Query Parser is used to turn query string into a query object. The query string is in the form of contents + keywords: xxx. Then the IndexSearcher provided by Lucene is called to do the job. 18

19 Run time environment setup and deployment 1) Download and Install a Java Development Kit: version or higher of the Java TM 2 SDK 2) Install the JDK according to the instructions included with the release. 3) Set an environment variable JAVA_HOME to the pathname of the directory into which you installed the JDK release. 4) Download JWSDP (with Tomcat servlet container) 5) Setup (unzip) JWSDP 6) Copy log4j.jar =>$TOMCATHOME /share/lib and $TOMCATHOME /jwsdp-shared/lib 7) Create a user account directory:/byproj under root. 8) Copy user account file: users.dat into /byproj. 9) Unzip bydoclibrary.war and copy it to $TOMCATHOME/WEBAPPS 10) Open up configuration.jsp in bydoclibrary.war, change the directory of webroot according to your setup. Note: All the documents uploaded are stored at: \webapps\root\byproj\lucene\doc 19

20 User guide Start Tomcat. Open up a browser. Access the service through the browser: User authentication 1) For the first login, click on register Now. 2) Login. 3) Click on Document Repository Main Menu. File upload and indexing 1) Click on Upload and index 2) Select a file by clicking on browse. Here you can specify the keywords or a category. If no category is selected, by default, the category is all. 3) Click on Upload and index button. After it is done, it would return the location of the file uploaded in the server. Document Search and Download 1) Go back the main menu. 2) In the search area, specify the search criteria. The user has several options to look for a file: 20

21 Any keywords in the file. The keywords specified by the user when the file was uploaded and indexed. To do this, the user needs to type keywords: and then the keywords after it. Select a category. Note: the space between words represent OR, if you need to find this AND that, you need type AND in between search words. 3) Download the file by clicking on it. 21

22 Test scenarios 1.Test all categories Uploaded File Name Category Keywords Contents Expected search results by category andrew-fell-2003.pdf PtP andrew-fell-2003.pdf Asst1(part2_q2).htm MaS Asst1(part2_q2).htm course outline winter 2003.doc IS course outline winter 2003.doc course outline winter 2003.rtf AIS course outline winter 2003.rtf derivative.txt BiPS derivative.txt index.html EC index.html james-helferty-2003.pdf NSM james-helferty-2003.pdf Lect2.pdf AI Lect2.pdf web.xml WT web.xml 2. Test by supplying keywords Uploaded File Name Category Keywords Contents Expected search results by keywords web.xml web web.xml studentlist.txt school stdentlist.txt james-helferty-2003.pdf, james-helferty-2003.pdf honour Proposal of honour project.rtf Proposal of honour project.rtf honour index.html index index.html 22

23 3. Searching by contents: Contents examination algorithm Expected search results by keywords course outline winter 2003.doc, course outline winter 2003.rtf james-helferty-2003.pdf, andrew-fell-2003.pdf 4. Searching by one term "AND" with another Contents Expected search results by keywords algorithm AND xxxx no match 5. Searching by one term "OR" with another Contents Expected search results by keywords james-helferty-2003.pdf, algorithm xxxx andrew-fell-2003.pdf 6. Test all combinations Contents proposal keywords: school Expected search results by keywords studentlist.txt Proposal of Honour Project.rtf 23

24 Reference [1] [2] [3] [4] [5] [6] [7] [8] [9] 24

25 Glossary OLE 2 Compound Document Format based files include most Microsoft Office files such as XLS and DOC as well as MFC serialization API based file formats. 25

26 Appendix File list Servlets: UserAuth.java BuildAllIndex.java AddUser.java //user authentication //re-build all indices for the uploaded documents //register new users JSP: bylibrary.jsp UploadIndex.jsp fileresults.jsp configuration.jsp header.jsp footer.jsp //main menu of document operations //upload & index one document //document search results //run-time environment configuration //header information of web pages //footerinformation of web pages Utililies: FileConvert.java ByUtil.java //API for Converting DOC(RTF) to text, and PDF to text //some tools AllTypeDocument.java //Constructors for document's objects to be used by indexing 26

27 buildindexexamples.java //indexing specific HTML: index.html help.html register.html //welcome & user login page //help page //user register page Data file: /byproj/users.dat //user information data file Libraries: uploadbean.jar poi-2.5-final jar PDFBox jar lucene-demos-1.3-final.jar lecene-1.3-final.jar //API of upload bean //API of converting doc(rtf) to text //API of converting PDF to text //lucene demo API (including HTML parser) //lucene API 27