picrawler Documentation

Size: px

Start display at page:

Download "picrawler Documentation"

Coleen Peters
5 years ago
Views:

1 picrawler Documentation Release Ikuya Yamada October 07, 2013

3 CONTENTS 1 Installation 3 2 Getting Started PiCloud Setup Basic Usage Using Real-time Cores Customizing Requests Defining Callbacks API Reference 7 4 Indices and tables 9 i

4 ii

5 picrawler Documentation, Release PiCrawler is a distributed web crawler using PiCloud. Using PiCrawler, you can easily implement a distributed web crawler within a few lines of code. from picrawler import PiCloudConnection with PiCloudConnection() as conn:... response = conn.send([ ])... print status code:, response[0].status_code... print content:, response[0].content[:15] status code: 200 content: <!DOCTYPE html> Contents: CONTENTS 1

6 picrawler Documentation, Release CONTENTS

7 CHAPTER ONE INSTALLATION To install PiCrawler, simply: $ pip install picrawler Alternatively, $ easy_install picrawler 3

8 picrawler Documentation, Release Chapter 1. Installation

9 CHAPTER TWO GETTING STARTED 2.1 PiCloud Setup Before using PiCrawler, it is neccessary to configure API key of PiCloud. import cloud cloud.setkey(api_key, API_SECRETKEY) You can obtain an API key by signing up on PiCloud. 2.2 Basic Usage from picrawler import PiCloudConnection with PiCloudConnection() as conn:... response = conn.send([ ])... print status code:, response[0].status_code... print content:, response[0].content[:15] status code: 200 content: <!DOCTYPE html> 2.3 Using Real-time Cores PiCloud enables you to reserve your exclusive computational resources by requesting real-time cores. PiCrawler provides a thin wrapper class for this. from picrawler import RTCoreRequest with RTCoreRequest(core_type= s1, num_cores=10):... pass Note: s1 core is the most suitable for crawling tasks, because PiCloud ensures that each s1 core has a unique IP address. 5

10 picrawler Documentation, Release Customizing Requests You can easily customize the request headers and other internal behaviors by using Request instances instead of raw URL strings. Since PiCrawler internally uses Python requests, it supports all arguments that are supported in Python requests. from picrawler import PiCloudConnection from picrawler.request import Request req = Request( GET,... headers={ User-Agent : MyCrawler },... args={ timeout : 5}) with PiCloudConnection() as conn:... response = conn.send([req]) 2.5 Defining Callbacks You can also define callbacks to the request. import logging from picrawler import PiCloudConnection from picrawler.request import Request req = Request( GET,... success_callback=lambda resp: logging.info(resp.content),... error_callback=lambda resp: logging.exception(resp.exception)) with PiCloudConnection() as conn:... response = conn.send([req]) 6 Chapter 2. Getting Started

11 CHAPTER THREE API REFERENCE class picrawler.picloudconnection(max_parallel_jobs=10, core_type= s1 ) Class that represents a connection to PiCloud. Usage: from picrawler import PiCloudConnection with PiCloudConnection() as conn:... response = conn.send([ ])... print status code:, response[0].status_code... print content:, response[0].content[:15] status code: 200 content: <!DOCTYPE html> Parameters max_parallel_jobs (int) (optional) The number of parallel jobs to run. core_type (str) (optional) PiCloud core type. close() Closes the connection. connect() Establishes a connection to PiCloud. send(req) Sends the requests to PiCloud. Parameters req Requests to be sended to PiCloud. Must be one of the following: A string that contains a URL A list or a tuple or an iteratable that consists of URL strings A Request instance An list or a tuple or an iteratable that consists of Request instance Returns List of BaseResponse instances. class picrawler.rtcorerequest(core_type, num_cores, max_duration=none) A wrapper class that requests real-time cores to PiCloud. Usage: from picrawler.rt_cores import RTCoreRequest with RTCoreRequest(core_type= s1, num_cores=10):... pass 7

12 picrawler Documentation, Release Parameters core_type (str) The PiCloud core type. num_cores (int) The number of cores. max_duration (int) (optional) The lifetime of the request, in hours. release(req_id) Releases PiCloud s realtime cores. Parameters req_id (int) The request ID. request() Requests PiCloud s realtime cores. Returns Request ID class picrawler.request.request(url, method= get, headers={}, args={}, success_callback=none, error_callback=none) Class that represents a request to PiCloud. Parameters url (str) A URL to be fetched. method (str) Target HTTP method. ( GET or POST or PUT or DELETE or HEAD or OPTIONS ) headers (dict) (optional) HTTP headers to send. args (dict) (optional) Additional arguments, that are directly passed to requests. Any arguments that supported in requests are supported. success_callback (function) (optional) A function called if the request is successfully completed. A Response instance is passed as the first argument. error_callback (function) (optional) A function called if the request is failed. ErrorResponse instance is passed as the first argument. class picrawler.response.baseresponse(request) class picrawler.response.response(request, status_code, content, headers) Class that represents a response from PiCloud. content HTTP content. Type str headers HTTP headers. Type dict status_code HTTP status code. Type int class picrawler.response.errorresponse(request, exception) Class that represents a runtime error occurred on running the job. exception Exception raised in the runtime. Type Exception An 8 Chapter 3. API Reference

13 CHAPTER FOUR INDICES AND TABLES genindex modindex search 9

mprpc Documentation Release Studio Ousia

mprpc Documentation Release Studio Ousia mprpc Documentation Release 0.1.13 Studio Ousia Apr 05, 2017 Contents 1 Introduction 3 1.1 Installation................................................ 3 1.2 Examples.................................................