IDEAL Climate Change Michael Steele, Somn Kafley, Samyak Singh CS 4624, Virginia Tech, Blacksburg, VA /27/2016

Transcription

1 1 IDEAL Climate Change Michael Steele, Somn Kafley, Samyak Singh CS 4624, Virginia Tech, Blacksburg, VA /27/2016 Instructor: Edward A. Fox Client: Mohammed M. Gharib Farag

2 Table Of Contents 2 Table of Tables 4 Table of Figures 5 Executive Summary 6 Chapter 1: User s Manual 7 I. Objectives II. Communication Method III. User Roles IV. Interactions With Other Systems V. Production Rollout Considerations VI. Glossary VII. Interface Chapter 2: Design & Requirements 11 I. Architectural Design A. Decomposition Description B. Design Rationale II. Data Description III. Screen Objects & Actions IV. Functional Requirements A. Functionality Statement B. Scope C. Performance D. Usability Chapter 3: Developer s Manual 14 I. Management Overview A. Implementation Description B. Major Tasks C. Timeline D. Security & Privacy II. Implementation Support A. Hardware, Software, Facilities, & Materials B. Staffing Requirements C. Implementation Impact

3 3 D. Performance Monitoring III. Implementation Details A. System Requirements B. System Implementation C. Acceptance Criteria Chapter 4: Prototype & Refinement 18 I. Components II. Data Flow & Functionality A. Data Extraction Prototype B. Data Indexing Prototype C. User Interaction / Search Query Prototype Chapter 5: Testing 21 I. Summary II. Testing Scope III. Types Of Testing Performed A. Black-Box Testing B. Functional Testing: Search C. Functional Testing: Page Ranking IV. Exit Criteria V. Conclusion Chapter 6: Future Work 23 I. Summary II. Necessary Changes Chapter 7: Lessons Learned 24 I. Summary II. Lessons III. Problems & Solutions IV. Conclusion Bibliography 27 Acknowledgements 28

4 Table of Tables 4 Table No. Title Page 1 Glossary 7 2 Point of contacts 14 3 Timeline 14 4 Future Work 23 5 Problems Encountered & Lessons Learned 25

5 Table of Figures 5 Figure No. Title Page 1 An example of search display 8 2 Search Result Display 9 3 Display detail of each webpage 9 4 Search history 10 5 Bookmarks 10 6 Data flow map 11 7 Data Flow 1: Data Extraction 18 8 Data Flow 2: Data Indexing 19 9 Data Flow 3: User Interface & Data Query The landing page of the interface 20

6 Executive Summary 6 IDEAL Climate Change is a digital library and search engine project. It aims to provide an efficient way for graduate students, researchers, and other stakeholders to search and access archived tweets and web pages related to climate change. It will allow them to utilize the enormous collection of climate change data. The application consists of data containing tweets and webpages that have been extracted and indexed to SOLR. The results of user search is organized and displayed on the interface provided by Blacklight. This report aims to highlight the design and software requirements of the application to demonstrate the scope, functional requirements, data decomposition, design rationale, and usability; a developer s manual to provide implementation descriptions, major tasks, timeline, hardware and software requirements, staff requirements, and acceptance criteria to future developers who want to expand on the current progress; a user s manual to inform stakeholders about user roles, communication methods, rollout considerations, and application glossary; prototype and refinement to explain various components and prototypes of the application; and finally a summary of tests performed and lessons learned. It can be concluded that the user has been provided with an efficient tool, which can help them search a bulk of archived data. There are a lot of prospective features that can be implemented to enhance the application. For example, a personalized user experience with search recommendations based on search-history. Enhancements like this will be a great project for future CS-4624 students to learn about searching, indexing, Artificial Intelligence, and Machine Learning.

7 Chapter 1: User s Manual 7 I. Objective The primary goal of the IDEAL climate change project is to provide an efficient way for technical and non-technical users to search and access archived tweets and websites related to climate change. The project serves as a tool for researchers who want to utilize the large pool of data collected. II. III. IV. Communication Method The mode of communication between the engineers and the client is currently via , due to ease-of-access. Any functional or design requirements are sent to the engineers Virginia Tech . So far, this method of communication has been effective due to the high correspondence-rate between the parties. User Roles Our primary clients/users are researchers/scientists/engineers who want to utilize the large pool of data access through these websites. The main role of the user is to search tweets and webpages by typing in a keyword on the given user-interface. The user is then responsible for using data-extracted from the underlying SOLR search engine for research related to climate-control. Users can also report any search error or usability issues to the design team via . Interaction With Other Systems Currently, the IDEAL climate change system interacts with an interface built on an Apache search platform called SOLR (SOLR Quick Start). SOLR is an excellent system because of its reliability, scalability, tolerance to faulty queries, distributed indexing, load-balanced querying, automated failover and recovery, and centralized configuration (Apache). SOLR controls the search and navigation, but it sits below a user-friendly framework called Blacklight (Beer). V. Production Rollout Considerations The final-product will be a web-application that allows users to successfully search climate-change related data, bookmark search results, and keep track of search history. It will be hosted on the Virginia Tech server at for use by end users. VI. Glossary Apache: Popular web server software that Ruby: A dynamic, object-oriented,

8 8 includes SOLR. Blacklight: A Ruby on Rails engine plugin, which provides an application that runs on top of SOLR. A user-friendly framework that helps interaction between SOLR and the user. Climate Change: The change in global and regional climate patterns due largely to the increased levels of atmospheric carbon dioxide produced by fossil fuels. Java: A general-purpose programming language well known for class-based and object-oriented approach to problem solving. Python: A general-purpose, high-level programming language focused on readability and light syntax. general-purpose programming language. Ruby on Rails: A web application framework written in Ruby, which provides data structures for a database, a web service, and web pages. SOLR: An open source enterprise search platform written in Java from Apache. Tweet: A status update from a user limited to 140 characters on Twitter. Twitter: A free social networking platform, which allows users to follow, tweet, and reach out to other users. URL: Acronym for Uniform Resource Locator, which is a reference to a resource on the WWW. VII. Interface Table 1: Glossary User Interface: Part of software with which a human being interacts. Figure 1: An example of search result display In Figure 1, the screenshot shows the search field where users can type climate change related keywords to search application.

9 9 Figure 2: Search Result Display In Figure 2, the screenshot shows the search results organized and displayed. Here, the user can bookmark certain search results and browse results. Figure 3: Display detail of each webpage In Figure 3, the screenshot shows the detailed view of a search result, which consists of Title, ID, URL, Type, and Content.

10 10 Figure 4: Search History. In Figure 4, the screenshot shows the search history view that shows the user s search keywords with the most recent on top. The history can be deleted by clicking the Clear Search History button. Figure 5: Bookmarks In Figure 5, the screenshot shows the Bookmarks view which stores all of the user s favorite search results. These can also be deleted by clicking the Clear Bookmarks button.

11 11 Chapter 2: Design & Requirements I. Architectural Design A. Decomposition Description The data was originally three JSON files composed of 1000 tweets each. The links from the tweets were then extracted and text files representing webpages were created using a Python script. These files (tweets and webpages) were then indexed into the SOLR server where index tables are created that list where keywords are stored in each document. The SOLR server relays the search engine requests and returns the search results. Figure 6: Data flow map In Figure 6, the pink file represents the 3000 tweets in JSON format and the blue file represents the hundreds of webpages in.txt format. The Extract URLs and Index processes were both executed using Python Scripts. From SOLR, the data is then organized and displayed on Blacklight. B. Design Rationale SOLR was used instead of a Relational Database because of several crucial reasons. SOLR is fast and sophisticated because the query speed is incredibly fast and predictable; a relational database speed depends on the type of search query and is not predictable. SOLR is targeted towards text search, which is what the IDEAL Climate Change project focused on; SOLR also allows access to internal data structures if needed by this project.

12 II. Data Description The data used by the IDEAL Climate Change project will be tweets (initial JSON file) and information from websites (represented as text files after URLs extracted using a Python script). The initial sources of the data will mainly be organizations (e.g., Oxfam, WWF, Friends of the Earth, Greenpeace, Global Action Plan), politicians and government (e.g., Al Gore, Bernie Sanders, Ed Miliband, Department of Energy and Climate Change), news agencies (e.g. The Ecologist, Digg Environment, James Murray, The Climate Desk), bloggers (e.g., Grist, TreeHugger, Kate Sheppard, Julian Wong), and campaign groups (e.g., Stop Climate Chaos, 350.org, Plane Stupid, One Climate, Climate Camp). III. Screen Objects & Actions A user can click the search button after typing keywords after which screen objects related to climate change will display in the interface. A user with admin privilege can add content to the website according to their needs. In addition, a user changes their search entry anytime and look for different results. By looking into the result a user can correlate trends in climate change webpages & tweets. It will be easier for users to see the change in the frequency of climate change. Thus, through our web application, users will benefit by being able to search by keyword and various facets, and will use this information for research. IV. Functional Requirements A. Functionality Statement The software s top priority is to address every query with pinpoint accuracy. As a search based application, it is important that the user is presented with accurate and relevant information. The targeted users are individuals involved in research who may not have a lot of time to dwell on non-relevant search results. The software requires a user-interface that is efficient and fast; a clean and organized way to present the search results, and an accessible platform to be hosted on. B. Scope The scope of this project consists of several phases with various tasks due at the end of every phase, which will also be tested incrementally. Phase 1: Environment setup. This is when the SOLR server is set up, and the Blacklight framework is installed. It also involves downloading dependencies like Ruby, Rails, and Java. After this phase, the environment will be tested so that data extraction and indexing can begin. Phase 2: Data extraction from collection. The data-extraction phase involves running a Python script to parse, scan, and extract URLs from relevant tweets related to climate change. The collection of tweets is given to us by the client and resides in a database in the research lab, in Torgersen. 12

13 Phase 3: Data indexing to SOLR. After the tweets have been processed and the URLs have been extracted from them, the text files that represent websites are indexed to SOLR along with the tweets. Once indexed to SOLR, the user can search by keyword. Phase 4: User interface design and development. The user-interface phase is the last phase whose goal is to create an easy-to-use interface for the user. This is will be done only after Phases 1-3, because the interface provided by Blacklight is usable, but it is possible to have a better one. C. Performance The software needs to be consistently running so that busy researchers are not helplessly wasting precious time. The software doesn t need to render large images or files, which allows it to be swift and secure. Performance should be assessed by how fast the software responds to queries and how fast it can display a large number of results. So, the main factors of performance are time and correct results. D. Usability The user-interface provided by Blacklight is good. However, the interface can be simplified by removing redundant and unnecessary features. The design goal is to provide a simple experience to achieve good performance. The elements required and interface-features required will be updated and refined as the team approaches Phase 4: User-Interface Design & Development. 13

14 Chapter 3: Developer s Manual 14 I. Management Overview A. Implementation Description Our goal is to design and implement a user interface where a non-technical user can search tweets and webpages by typing a keyword related to climate change. We have two major phases of implementation: tweet extraction, archiving and indexing. Role Work Description Name Instructor and Client Instructor Project assignment and supervision. Edward Fox fox@vt.edu Client Research lead and supervision Mohammed Farag mmagdy@vt.edu Development Team Developer File extraction and design Samyak Singh ssingh94@vt.edu Developer File indexing and code analysis Michael Steele derek22@vt.edu Developer File extraction and documentation Somn Kafley somnkk13@vt.edu B. Major Tasks Table 2: Point of contacts The main tasks in the implementation of our project involve receiving Python scripts and archived tweet files from the research lead; running Python scripts on archived JSON files to extract targeted information from tweets related to climate change; using Python scripts to index.txt files containing extracted webpages to SOLR and Blacklight; changing the SOLR web interface to properly index tweets into SOLR; and demonstrating the newly built climate change search engine to clients for feedback and testing. C. Timeline Date February 25, 2016 March 1, 2016 March 4, 2016 Description Meet with research-lead to gather archived tweet files. Extract all data from archived file of tweets into text files. Evaluate inconsistencies in Python scripts and finalize text files

15 15 March 31, 2016 April 18, 2016 April 25, 2016 Index files to SOLR, make it searchable and test Set up Blacklight and have a search engine up and running. Make changes in the interface, deliver to client Table 3: Timeline D. Security & Privacy The server where our extracted data is hosted runs on the Virginia Tech server network, which is secured by the university. There is no risk of data invasion, or breach. Users will be able to safely access data once granted access to the system. II. Implementation Support This section describes the support systems for our project. Implementation support includes the hardware, software, facilities, and materials required for the project. This section also includes documentation, people involved, outstanding issues and impact, and monitoring of performance. A. Hardware, Software, Facilities, & Materials Hardware The research-lead and developers all work on laptops for implementing the system. The SOLR server is installed on a desktop in the IDEAL research lab in Torgersen The rationale behind that was to maintain security of the server. Software On the MacBook and PC, terminal is used to run scripts and organize files. Sublime Text and Notepad++ are used to edit and comment Python scripts. On the web, the Blacklight framework is used over SOLR. Additionally, Microsoft Word, Google Chrome, and Mozilla Firefox are used for research, documentation, and reports. Facilities Virginia Tech s research facilities in Torgersen have been very useful for the research, developer meetings, and client presentations. However, McBryde 106 is where most of the implementation takes place. B. Staffing Requirements The staff required to implement this project must have the following skills: Code (Python, Java, etc.) Analyze written scripts Good sense of system design Experience with servers, database, or search-engines Excellent communication abilities

16 C. Implementation Impact 16 The implementation of this IDEAL climate-change search engine will have a positive impact on the research community. Because of the current system that allows keyword search functionality with the data collected, anybody interested in climate-change will now have access to the plethora of information indexed in this system. There will be a considerable amount of traffic once the system is up and running, however, it will not be big enough to affect the network infrastructure of Virginia Tech. The data made available for users will be regularly backed-up in case of network crash, which has a very minute possibility. D. Performance Monitoring The performance of the system will be measured and monitored by the research-lead and developers. A system/implementation/test plan has been created. Once the data extracted is indexed to the server, the team will start searching and accessing data via keywords. The speed of the search, the accuracy of the results, and the functionality of the interface will measure performance. The monitoring will be simultaneous to the lifetime of the system with issue reporting, debugging, and enhancements done as the system lives on. For monitoring purposes, a log should be created in the future to track bugs and changes. III. Implementation Details This section contains information regarding any requirements of the system implementation. This involves software and hardware requirements, risks, and acceptance criteria. A. System Requirements Software Ruby Version 1.9 or higher Rails 3.2 or 4.x Java Version 1.7 or higher Apache SOLR Blacklight Terminal or Command Prompt Hardware Dedicated server with the following minimum specifications: Intel Core i5 8GB RAM 500GB Hard Drive Internet Access B. System Implementation Procedures Install Blacklight using Quick Start

17 17 Write/amend Python scripts to extract webpages and index information to SOLR. Script descriptions provided below. Test and confirm that SOLR is searchable with default configurations. Update scripts if test is not successful. Index tweets to SOLR by adding an additional core. Test and confirm that SOLR can search tweets and webpages. Enhance appearance and user interface for ease of use. Demo SOLR to social scientists, and gather their feedback. Re-configure SOLR based on the feedback from the clients. Script Descriptions downloadwebpagesfromtweets_txts_titles.py: This script extracts information from tweets specifically URLs provided in the raw tweet data. The input of the script includes a JSON file that includes thousands of tweets related to climate change. It outputs text files containing URLs for related webpages. indextosolr.py: This script takes all the text files generated by the first script and archives them to the SOLR database. The first line of each webpage contains its URL. Specifically, the script connects to the SOLR s Blacklight URL that accepts indexed files. It then sends a dictionary object containing each tag as the key and the value of each tag as the value. All of the scripts can be found at: Implementation Verification & Validation To ensure that the implementation was executed correctly, the research lead along with the developers will conduct an end-user test. This will happen after all the steps described in system implementation have been completed. C. Acceptance Criteria The system will be labeled as approved or accepted for production after it has been thoroughly tested, verified, and validated as described in the system implementation and performance monitoring sections. The primary exit criteria of the system are a qualitative measure of the user-friendliness of the system and a quantitative monitoring of the system performance. It is based on the display result of the search results.

18 Chapter 4: Prototype & Refinement 18 The purpose of our prototype was to have a system ready to be used such that users could start searching Climate Change related subjects via keywords. To do this, we created a high-level navigational flow of the system to map the flow of data. Since running the extraction and indexing scripts didn t require any prototype, our primary goal was to have well-organized search functionality. I. Components Apart from the server there are no hardware components in the prototype. The prototype consists of software components. The software controls the interface, queries, data-retrieval, and display. The primary software components of the prototype involved a raw tweet data (JSON), two Python scripts, text files extracted from the JSON file containing webpage data, SOLR search platform, and Blacklight. II. Data Flow & Prototype Functionality A. Data Extraction Prototype Figure 7: Data Flow 1: Data Extraction The data flow for the first process involved an initial JSON file containing raw tweet data. This file was then taken by the Python script (1) to extract webpages from individual tweet, and then store in a local directory. The webpages were of text format ready to be archived to SOLR. More information about the specifics about the Python script can be found in Chapter 3, Section B. B. Data Indexing Prototype The indexed data is then archived inside SOLR with the help of the Python script (2). SOLR is configured to use ID tags and content tags. The content tags contain full text of each search result. This is used for 2 reasons: first to populate the index files with keywords and secondly to save and display the content of the search result to the user.

19 19 Figure 8: Data Flow 2: Data Indexing We create a unique ID tag for each web page by hashing the URL of that webpage. As a result of running this script, the text of all archived web pages will be indexed and saved on the SOLR server and will be searchable. This includes all working URLs mentioned in all archived tweets. C. User Interaction / Search Query Prototype User interaction with SOLR directly is incredibly inefficient so we added a mid-layer component, Blacklight. Now, the user has a friendly interface where they can execute search queries to the platform. The prototype for this involves a two-way data-flow unlike the prototype for extraction and indexing due to the search query and results display. Figure 9: Data Flow 3: User Interaction & Data Query

20 Blacklight Interface 20 Figure 10: The landing page of the interface. For other screenshots of user interface views such as search-fields, bookmarks, search-history, and detailed view, refer to figures in Chapter 1: User s Manual.

21 Chapter 5: Testing 21 I. Summary This section of the report provides a summary of the results of the tests performed on our project. Testing phases were used to: evaluate the correctness of the project, solicit feedback from the user testers, and design improvements to use during the next build phase of the project. We did three kinds of tests: Black Box Testing and two types of Functional Testing. The result from each type of test is outlined below. II. Testing Scope The testing scope of the IDEAL climate project was to efficiently test the user interface and the search functionality of the application. The user interface testing involved how well an end-user could execute a search and read the results. Navigating the application is fairly simple but testing was required to identify any possible errors. The functional testing of the application involved confirming that the search queries returned correct results. III. Types Of Testing Performed A. Black-Box Testing Three users (undergraduate students) who have no knowledge of the internal workings of the project tested the user-interface. They were told about the overall goal of the project and were given the link to the application ( and asked to use the GUI to perform several search queries. They were also asked to browse the returned results of the query. These tasks would test the usability of the application by checking if users can perform tasks efficiently. B. Functional Testing: Search The developers tested the search functionality in the following way: 1. Go to the application at 2. Open one of the webpage text files from a local directory. 3. Choose a unique string that appears in only one webpage, and search it. 4. Confirm that there is only one result for the search query. C. Functional Testing: Page Ranking The developers tested the page ranking functionality in the following way: 1. Created a text file with a certain keyword present more than 10 times. 2. Created another text file with the same keyword present 5 to 10 times. 3. Created another text file with the same keyword present less than 5 times. 4. Then, those files were indexed to SOLR. 5. When that keyword was searched, the files with the highest number of that keyword present will appear on top.

22 IV. Exit Criteria The exit criteria include: 100% positive feedback from the black-box testers so that they can continue to efficiently perform search tasks and feel comfortable with the user-interface. Continue to pass all functional testing done on the search and page ranking, as the application is refined to match the black-box feedbacks. V. Conclusion The primary types of testing done on the IDEAL Climate Change project were the black box testing for usability and functional testing for search results and page ranking. These tests exposed some user interface design problems that are currently being fixed. From these errors, we learned that although our goal was to provide a simple and efficient interface, it is important to plan the interface design from the beginning instead of attempting to build on top of the Blacklight interface. 22

23 Chapter 6: Future Work 23 I. Summary Unlike many other projects, the climate change project is an ongoing process. Someone will take charge from the point we left off. So, it is important for us to show some of the possible future works. Our system currently addresses the scope of the project. However, like any other system, there are various aspects, which can be refined and updated to increase overall efficiency and productivity of the system. Some refinements can be implemented but others require more time than what is available to us. Cumulatively, these refinements are a future upgrade to the system. II. Necessary Changes There are many prospective extensions that can be added to this project such as a mobile-ready application, more filter options, and a customized user interface that represents the Virginia Tech colors and themes. However, the following table (Table 4) shows the most prioritized changed necessary to the project by future developers. Present Scenario Manually running scripts to download webpages from tweets and then indexing them to SOLR is not only time consuming, but also tedious. The scripts written only support Python 2.7.x, which is a problem because it is outdated. The latest version of Python is 3.5.x and offers new, updated features. The downloadtowebpagesfromtweets script operates slowly because it must locally save each webpage in an individual text file. Making HTTP request to webpages is the slowest part of the script because some webpages are slow to response and some 404. (Possible solution #1) Future work Automate this process so that these scripts can be run without the need of any developers. Maybe schedule the script execution so that it happens daily. Discuss with the client and decide to upgrade the version of Python used. This will require updating both scripts entirely. The updates will enhance in search results and performance. Don t bother saving webpage contents locally on the machine. In one Python script: 1. Save the contents of webpages in an array. 2. Index those contents to SOLR. Experiment with using Google Cache ( Download the cached version of a webpage instead of the live version for faster speed. Making HTTP request to webpages is the slowest part of the script because some webpages are slow to response and some 404. (Possible solution #2) Experiment with multithreading this part of the Python script to increase speed. Table 4: Future work

24 Chapter 7: Lessons Learned 24 I. Summary This section outlines the various lessons learned, problems encountered, and solutions designed while working on the IDEAL Climate Change Project. Working on a project of such a large scale and efficiently synchronizing the team s progress was our biggest challenge. From the SOLR instance to version control and usability testing, every challenge allowed us to learn a new aspect of cooperative development. II. Lessons Initially, the SOLR server was intended to run individually on the developers machines and a separate one on the client s machine. We quickly realized that this would be a tremendous problem because of synchronization issues. If files were archived from one machine to its relative installation of SOLR, other developers and the client would not be able to access it. For this reason, we hosted SOLR on a universally accessible server that belongs to Virginia Tech. Synchronization was not only a problem with SOLR however. The Python codes written and webpages extracted were all done in individual machines and uploaded to Google Drive. This was extremely inefficient because there was no version control and it wasn t possible to go back to previous editions of codes. To solve this, we created a repository on GitHub ( where all the scripts, reports, data files, and a readme file was pushed. This increased our productivity significantly. Every development project needs to be done in an iterative fashion to efficiently keep track of progress and boost productivity. Another lesson learned was that it would have been very helpful and complimentary if the team used Agile methodology especially scrums and sprints with the developers and the client in order to organize the milestones. Although all requirements were met, this added workflow-technique would be a great addition. During the Usability Testing (via Black Box Testing), we realized that the interface design was not personalized to our project. It has the Blacklight logo and title that confused the users about the interface. Removing the Blacklight logo and changing the color schemes can mend that. Users were curious if the search results could be sorted by date, title, etc. and if there were available filters that could help filter the results. This helped us learn that simply displaying results is not enough and that allowing users to interact with the organization is crucial. III. Problems and Solutions We have encountered many problems from the initial phase of the project. Many of these problems are during testing phase. Below we have included a table encountering some of the problems uncovered during testing, and the solutions we designed to solve those problems.

25 25 Problems The script given by the client attempted to index both tweets and webpages and was confusing. Invalid characters on the webpage crash the scripts when they try to read the text of the webpage. Script uploads one file at a time to SOLR and is too slow. Some of the search fields such as content and URL and from user were not included by default in SOLR. Script only accepts one tweet file at a time and then stops. Very time consuming to keep restarting script after each file. Script only saves the contents of each webpage to files but does not save the URL or the Title, which we need as search fields for SOLR. The JSON files given to me used single quotes, which the Python JSON processing library sees as invalid. Some webpages do not have any content: it is pointless to add them to the SOLR search library. Solutions Created three separate scripts, each with a division of responsibilities: 1. A script to download the contents of webpages mentioned in tweets & save them. 2. A script to index saved webpages to SOLR. A script to index tweets in the JSON file. 1. Convert the text to Unicode when possible. 2. When reading file, use the option to ignore encoding errors. For tweets: upload them in batches of 10,000. For webpages: upload them in batches of 100. Inserted these fields as new search fields into the SOLR configuration XML file. Script now accepts a folder as input, and runs on all JSON files in that folder. Script now puts the URL on the first line, title on the second line, and then adds the webpage contents. These lines are parsed and then indexed to SOLR. Before processing a JSON file, first go through the file and replace all single quotes with double quotes. In the webpage indexing script: if the webpage content is smaller than 1000 characters, then do not index it into SOLR. The original Python JSON processing library the scripts tried to use would fail on very large JSON files with an Out Of Memory Error. Switched to another Python JSON processing library: JSON. This library does not try to load the entire JSON file into memory all at once. Instead, it loads it in one object at a time. Table 5: Problems Encountered & Lessons Learned

26 IV. Conclusion The entire IDEAL climate change project has been a great learning experience. We came across many unexplored areas of digital library search engine. The background work that has to be done before a user gets into the interface was found to be immense. Although lots of work still needs to be done, we have to conclude our project due to the time frame available to us. We hope great success to whoever decides to take on this project further. 26

27 Bibliography Beer, Chris. "ProjectBlacklight/Blacklight." GitHub. Feb Web. Mar < 2. "SOLR Quick Start." Apache SOLR. Apache SOLR Foundation. Web. 14 Mar < 3. "SOLR Admin." SOLR Admin. Ed. Mohammed M. Garib Farag. Open Source. Web. Mar < 4. Beer, Chris. "Blacklight." GitHub. Web. 01 Apr <

28 Acknowledgements 28 Special thanks to National Science Foundation grant IIS , III: Small: Integrated Digital Event Archiving and Library (IDEAL). We would like to express our sincere appreciation to Dr. Edward Fox for providing us the climate changes project and guiding us with valuable and constructive suggestions. His willingness to give his time and walk us through the project is much more appreciated. In addition, we would like to thanks Mr. Mohammed M. Gharib Farag for providing us the valuable information we needed.