What is NovelTorpedo?

Transcription

1 NovelTorpedo

2 What is NovelTorpedo? A website designed to index online literature. Enables users to read all of their favorite fanfiction in one place.

3 Who will use NovelTorpedo? Avid readers of fanfiction Many sources accessible from one website Full text search Filters Content Creators Link new content directly More exposure

4 Assumptions We can make web scrapers that get literature People want this service We can avoid copyright infringement by only providing snippets of text

5 Web Scraper Constraints Fanfiction sites differ so each will have its own webscraper. Some sites have API's while others do not. When sites change we will need to update the webscraper.

6 Fanfiction Constraints Detecting Multiple copies Writers post to different sites Sometimes these posts are not identical

7 References Comic Rocket is the website created by our sponsors. Our project is like Comic Rocket in that it: Uses web scrapers to scrape content suggested by users Stores links to that contents in a database. Has a website for people to search that database for content.

8 Features and Deliverables: Overview Literature Websites End Users Web Scrapers Databases Website (Search Engine)

9 Features and Deliverables: Overview Literature Websites End Users Web Scrapers Databases Website (Search Engine)

10 Features and Deliverables: Databases Implemented in Elasticsearch and PostgreSQL Accommodates: The full text contents of each story Relationships between stories, such as: Sequel of another story Fanfiction of another story Metadata about each story, such as: The sources of each story, including authors and hosts

11 Features and Deliverables: Scrapers Written in Scrapy (Python web scraping framework) Separate scripts to consume: Each API (like Tumblr) Each HTTP website (parsing HTML) Injects scraped content into PostgreSQL and/or Elasticsearch

12 Features and Deliverables: Website Search engine Implemented in Django (Python MTV framework) Paginated results with: Text snippets Links to source (author/host) websites

13 Scheduling & Responsibilities: Major Docs Document Completion Target Current Status Responsible Requirements Specification Rough draft ~1 week after sponsor meeting. Draft complete, refinement in progress. Matthew & Rubin Risk Assessment End of fall quarter. Finished. Brook & Jacob Schedule Continuous; rough draft end of fall quarter. Draft completed, needs details. Finn & Sage Design Documents for Project Subparts End of fall quarter. Pushed back pending spec completion. Everyone

14 Scheduling & Responsibilities: Project Parts Part Description Responsible Web Crawlers Databases Scrapy crawlers and Python API calls to get data from source websites. MySQL and Elasticsearch databases for storing and searching works and metadata. Finn & Matthew Rubin & Sage Website Django work search engine (and stretch goals). Brook & Jacob Integration Making all these parts talk to each other! TBD

15 Risks and Mitigations Social Conflict MIAs Fall Behind Schedule Technical Issues Lose Google Docs Codebase Breaks Takedown Notice

16 But not everything is as well-defined as is ideal.

17 Elasticsearch Elasticsearch has many useful features, such as: match highlighting MoreLikeThis producing text snippets and more that we don't yet know about But we have to: instantiate it verse ourselves with using these features determine how we are integrating this with Postgres

18 Tumblr The good: Tumblr has an API! The bad: We can't use that API to tell if something is fanfiction The API only lets us fetch the most recent 20 posts So that work, then, falls on us!

19 supports a basic admin interface we need to decide whether we'll use that feature as well as concretely define our user system, if any

20