linkgrabber Documentation

Transcription

1 linkgrabber Documentation Release Eric Bower Jun 08, 2017

2

3 Contents 1 Install 3 2 Tutorial Quickie Documentation API Reference linkgrabber Upcoming 9 5 Changelog v (7/09/2015) v (7/09/ v0.2.9 (1/24/2014) v0.2.8 (10/23/2014) v0.2.7 (06/25/2014) v0.2.6 (06/25/2014) v0.2.5 (06/23/2014) v0.2.4 (06/10/2014) v0.2.3 (05/22/2014) v0.2.2 (05/19/2014) v0.2.1 (05/18/2014) v0.2.0 (05/17/2014) v0.1.9 (05/17/2014) v0.1.8 (05/16/2014) v0.1.7 (05/16/2014) v0.1.6 (05/16/2014) v0.1.5 (05/16/2014) v0.1.0 (05/16/2014) Authors 15 7 Indices and tables 17 Python Module Index 19 i

4 ii

5 linkgrabber Documentation, Release Link Grabber provides a quick and easy way to grab links from a single web page. This python package is a simple wrapper around BeautifulSoup, focusing on grabbing HTML s hyperlink tag, a. Github: Issues: Contents: Contents 1

6 linkgrabber Documentation, Release Contents

7 CHAPTER 1 Install How to install linkgrabber: $ python setup.py install OR on pypi: $ pip install linkgrabber 3

8 linkgrabber Documentation, Release Chapter 1. Install

9 CHAPTER 2 Tutorial Quickie import re import linkgrabber links = linkgrabber.links(" links.find() # limit the number of "a" tags to 5 links.find(limit=5) # filter the "a" tag href attribute links.find(href=re.compile("plus.google.com")) Documentation find Parameters: filters (dict): Beautiful Soup s filters as a dictionary limit (int): Limit the number of links in sequential order reverse (bool): Reverses how the list of <a> tags are sorted sort (function): Accepts a function that accepts which key to sort upon within the List class Find all links that have a style containing 11px import re from linkgrabber import Links 5

10 linkgrabber Documentation, Release links = Links(" links.find(style=re.compile("11px"), limit=5) Reverse the sort before limiting links: from linkgrabber import Links links = Links(" links.find(limit=2, reverse=true) Sort by a link s attribute: from linkgrabber import Links links = Links(" links.find(limit=3, sort=lambda key: key['text']) Exclude text: import re from linkgrabber import Links links = Links(" links.find(exclude=[{ "text": re.compile("read More") }]) Remove duplicate URLs and make the output pretty: from linkgrabber import Links links = Links(" links.find(duplicates=false, pretty=true) Link Dictionary All attrs from BeautifulSoup s Tag object are available in the dictionary as well as a few extras: text (text inbetween the <a></a> tag) seo (parse all text after last / in URL and attempt to make it human readable) 6 Chapter 2. Tutorial

11 CHAPTER 3 API Reference linkgrabber Module that scrapes a web page for hyperlinks class linkgrabber.links(href=none, text=none, parser= lxml ) Grabs links from a web page based upon a URL and filters find(limit=none, reverse=false, sort=none, exclude=none, duplicates=true, pretty=false, **filters) Using filters and sorts, this finds all hyperlinks on a web page Parameters limit Crop results down to limit specified reverse Reverse the list of links, useful for before limiting exclude Remove links from list duplicates Determines if identical URLs should be displayed pretty Quick and pretty formatting using pprint filters All the links to search for linkgrabber.exclude_match(exclude, link_value) Check excluded value against the link s current value linkgrabber.seoify_hyperlink(hyperlink) Modify a hyperlink to make it SEO-friendly by replacing hyphens with spaces and trimming multiple spaces. Parameters hyperlink URL to attempt to grab SEO from 7

12 linkgrabber Documentation, Release Chapter 3. API Reference

13 CHAPTER 4 Upcoming More unit tests Take over the world 9

14 linkgrabber Documentation, Release Chapter 4. Upcoming

15 CHAPTER 5 Changelog v (7/09/2015) Added parser parameter to Links class Default parser set to lxml v (7/09/2015 Added six as a dependency v0.2.9 (1/24/2014) Updated documentation v0.2.8 (10/23/2014) Added better documentation v0.2.7 (06/25/2014) Fixed exclude for non-iterable strings 11

16 linkgrabber Documentation, Release v0.2.6 (06/25/2014) Exclude parameter is now a list of dictionaries Added pretty property Added duplicates property which will remove any identical URLs Added more tests Added better docs v0.2.5 (06/23/2014) Added exclude parameter to Links.find() which removes links that match certain criteria v0.2.4 (06/10/2014) Updated documentation to be better read on pypi Removed scrape.py and moved it to init.py Now using nose for unit testing v0.2.3 (05/22/2014) Updated setup py file and some verbage v0.2.2 (05/19/2014) linkgrabber.links.find() now responds with all Tag.attrs from BeautifulSoup4 as well as text and seo keys v0.2.1 (05/18/2014) Added more tests v0.2.0 (05/17/2014) Modified naming convention, reduced codebase, more readable structure 12 Chapter 5. Changelog

17 linkgrabber Documentation, Release v0.1.9 (05/17/2014) Python 3.4 compatability v0.1.8 (05/16/2014) Changed paramerter names to better reflect functionality v0.1.7 (05/16/2014) Update README v0.1.6 (05/16/2014) Update README with more examples v0.1.5 (05/16/2014) Updated find_links to accept link_reverse=(bool) and link_sort=(function) v0.1.0 (05/16/2014) Initial release v0.1.9 (05/17/2014) 13

18 linkgrabber Documentation, Release Chapter 5. Changelog

19 CHAPTER 6 Authors Eric Bower <neurosnap@gmail.com> 15

20 linkgrabber Documentation, Release Chapter 6. Authors

21 CHAPTER 7 Indices and tables genindex modindex search 17

22 linkgrabber Documentation, Release Chapter 7. Indices and tables

23 Python Module Index l linkgrabber, 7 19

24 linkgrabber Documentation, Release Python Module Index

25 Index E exclude_match() (in module linkgrabber), 7 F find() (linkgrabber.links method), 7 L linkgrabber (module), 7 Links (class in linkgrabber), 7 S seoify_hyperlink() (in module linkgrabber), 7 21