ABOUT THE AUTHOR ABOUT THE TECHNICAL REVIEWER ACKNOWLEDGMENTS INTRODUCTION 1

Transcription

1

2 CONTENTS IN DETAIL ABOUT THE AUTHOR xxiii ABOUT THE TECHNICAL REVIEWER xxiii ACKNOWLEDGMENTS xxv INTRODUCTION 1 Old-School Client-Server Technology... 2 The Problem with Browsers... 2 What to Expect from This Book... 2 Learn from My Mistakes... 3 Master Webbot Techniques... 3 Leverage Existing Scripts... 3 About the Website... 3 About the Code... 4 Requirements... 5 Hardware... 5 Software... 6 Internet Access... 6 A Disclaimer (This Is Important)... 6 PART I: FUNDAMENTAL CONCEPTS AND TECHNIQUES 7 1 WHAT S IN IT FOR YOU? 9 Uncovering the Internet s True Potential... 9 What s in It for Developers? Webbot Developers Are in Demand Webbots Are Fun to Write Webbots Facilitate Constructive Hacking What s in It for Business Leaders? Customize the Internet for Your Business Capitalize on the Public s Inexperience with Webbots Accomplish a Lot with a Small Investment Final Thoughts... 12

3 2 IDEAS FOR WEBBOT PROJECTS 15 Inspiration from Browser Limitations Webbots That Aggregate and Filter Information for Relevance Webbots That Interpret What They Find Online Webbots That Act on Your Behalf A Few Crazy Ideas to Get You Started Help Out a Busy Executive Save Money by Automating Tasks Protect Intellectual Property Monitor Opportunities Verify Access Rights on a Website Create an Online Clipping Service Plot Unauthorized Wi-Fi Networks Track Web Technologies Allow Incompatible Systems to Communicate Final Thoughts DOWNLOADING WEB PAGES 23 Think About Files, Not Web Pages Downloading Files with PHP s Built-in Functions Downloading Files with fopen() and fgets() Downloading Files with file() Introducing PHP/CURL Multiple Transfer Protocols Form Submission Basic Authentication Cookies Redirection Agent Name Spoofing Referer Management Socket Management Installing PHP/CURL LIB_http Familiarizing Yourself with the Default Values Using LIB_http Learning More About HTTP Headers Examining LIB_http s Source Code Final Thoughts BASIC PARSING TECHNIQUES 37 Content Is Mixed with Markup Parsing Poorly Written HTML Standard Parse Routines Using LIB_parse Splitting a String at a Delimiter: split_string() Parsing Text Between Delimiters: return_between() xii

4 Parsing a Data Set into an Array: parse_array() Parsing Attribute Values: get_attribute() Removing Unwanted Text: remove() Useful PHP Functions Detecting Whether a String Is Within Another String Replacing a Portion of a String with Another String Parsing Unformatted Text Measuring the Similarity of Strings Final Thoughts Don t Trust a Poorly Coded Web Page Parse in Small Steps Don t Render Parsed Text While Debugging Use Regular Expressions Sparingly ADVANCED PARSING WITH REGULAR EXPRESSIONS 49 Pattern Matching, the Key to Regular Expressions PHP Regular Expression Types PHP Regular Expressions Functions Resemblance to PHP Built-In Functions Learning Patterns Through Examples Parsing Numbers Detecting a Series of Characters Matching Alpha Characters Matching on Wildcards Specifying Alternate Matches Regular Expressions Groupings and Ranges Regular Expressions of Particular Interest to Webbot Developers Parsing Phone Numbers Where to Go from Here When Regular Expressions Are (or Aren t) the Right Parsing Tool Strengths of Regular Expressions Disadvantages of Pattern Matching While Parsing Web Pages Which Are Faster: Regular Expressions or PHP s Built-In Functions? Final Thoughts AUTOMATING FORM SUBMISSION 63 Reverse Engineering Form Interfaces Form Handlers, Data Fields, Methods, and Event Triggers Form Handlers Data Fields Methods Multipart Encoding Event Triggers Unpredictable Forms JavaScript Can Change a Form Just Before Submission Form HTML Is Often Unreadable by Humans Cookies Aren t Included in the Form, but Can Affect Operation Analyzing a Form xiii

5 Final Thoughts Don t Blow Your Cover Correctly Emulate Browsers Avoid Form Errors MANAGING LARGE AMOUNTS OF DATA 77 Organizing Data Naming Conventions Storing Data in Structured Files Storing Text in a Database Storing Images in a Database Database or File? Making Data Smaller Storing References to Image Files Compressing Data Removing Formatting Thumbnailing Images Final Thoughts PART II: PROJECTS 91 8 PRICE-MONITORING WEBBOTS 93 The Target Designing the Parsing Script Initialization and Downloading the Target Further Exploration IMAGE-CAPTURING WEBBOTS 101 Example Image-Capturing Webbot Creating the Image-Capturing Webbot Binary-Safe Download Routine Directory Structure The Main Script Further Exploration Final Thoughts LINK-VERIFICATION WEBBOTS 109 Creating the Link-Verification Webbot Initializing the Webbot and Downloading the Target Setting the Page Base Parsing the Links Running a Verification Loop Generating Fully Resolved URLs xiv

6 Downloading the Linked Page Displaying the Page Status Running the Webbot LIB_http_codes LIB_resolve_addresses Further Exploration SEARCH-RANKING WEBBOTS 117 Description of a Search Result Page What the Search-Ranking Webbot Does Running the Search-Ranking Webbot How the Search-Ranking Webbot Works The Search-Ranking Webbot Script Initializing Variables Starting the Loop Fetching the Search Results Parsing the Search Results Final Thoughts Be Kind to Your Sources Search Sites May Treat Webbots Differently Than Browsers Spidering Search Engines Is a Bad Idea Familiarize Yourself with the Google API Further Exploration AGGREGATION WEBBOTS 129 Choosing Data Sources for Webbots Example Aggregation Webbot Familiarizing Yourself with RSS Feeds Writing the Aggregation Webbot Adding Filtering to Your Aggregation Webbot Further Exploration FTP WEBBOTS 139 Example FTP Webbot PHP and FTP Further Exploration WEBBOTS THAT READ 145 The POP3 Protocol Logging into a POP3 Mail Server Reading Mail from a POP3 Mail Server Executing POP3 Commands with a Webbot Further Exploration Controlled Webbots Interfaces xv

7 15 WEBBOTS THAT SEND , Webbots, and Spam Sending Mail with SMTP and PHP Configuring PHP to Send Mail Sending an with mail() Writing a Webbot That Sends Notifications Keeping Legitimate Mail out of Spam Filters Sending HTML-Formatted Further Exploration Using Returned s to Prune Access Lists Using as Notification That Your Webbot Ran Leveraging Wireless Technologies Writing Webbots That Send Text Messages CONVERTING A WEBSITE INTO A FUNCTION 163 Writing a Function Interface Defining the Interface Analyzing the Target Web Page Using describe_zipcode() Final Thoughts Distributing Resources Using Standard Interfaces Designing a Custom Lightweight Web Service PART III: ADVANCED TECHNICAL CONSIDERATIONS SPIDERS 173 How Spiders Work Example Spider LIB_simple_spider harvest_links() archive_links() get_domain() exclude_link() Experimenting with the Spider Adding the Payload Further Exploration Save Links in a Database Separate the Harvest and Payload Distribute Tasks Across Multiple Computers Regulate Page Requests xvi

8 18 PROCUREMENT WEBBOTS AND SNIPERS 185 Procurement Webbot Theory Get Purchase Criteria Authenticate Buyer Verify Item Evaluate Purchase Triggers Make Purchase Evaluate Results Sniper Theory Get Purchase Criteria Authenticate Buyer Verify Item Synchronize Clocks Time to Bid? Submit Bid Evaluate Results Testing Your Own Webbots and Snipers Further Exploration Final Thoughts WEBBOTS AND CRYPTOGRAPHY 193 Designing Webbots That Use Encryption SSL and PHP Built-in Functions Encryption and PHP/CURL A Quick Overview of Web Encryption Final Thoughts AUTHENTICATION 197 What Is Authentication? Types of Online Authentication Strengthening Authentication by Combining Techniques Authentication and Webbots Example Scripts and Practice Pages Basic Authentication Session Authentication Authentication with Cookie Sessions Authentication with Query Sessions Final Thoughts ADVANCED COOKIE MANAGEMENT 209 How Cookies Work PHP/CURL and Cookies xvii

9 How Cookies Challenge Webbot Design Purging Temporary Cookies Managing Multiple Users Cookies Further Exploration SCHEDULING WEBBOTS AND SPIDERS 215 Preparing Your Webbots to Run as Scheduled Tasks The Windows XP Task Scheduler Scheduling a Webbot to Run Daily Complex Schedules The Windows 7 Task Scheduler Non-calendar-based Triggers Final Thoughts Determine the Webbot s Best Periodicity Avoid Single Points of Failure Add Variety to Your Schedule SCRAPING DIFFICULT WEBSITES WITH BROWSER MACROS 227 Barriers to Effective Web Scraping AJAX Bizarre JavaScript and Cookie Behavior Flash Overcoming Webscraping Barriers with Browser Macros What Is a Browser Macro? The Ultimate Browser-Like Webbot Installing and Using imacros Creating Your First Macro Final Thoughts Are Macros Really Necessary? Other Uses HACKING IMACROS 239 Hacking imacros for Added Functionality Reasons for Not Using the imacros Scripting Engine Creating a Dynamic Macro Launching imacros Automatically Further Exploration DEPLOYMENT AND SCALING 249 One-to-Many Environment One-to-One Environment xviii

10 Many-to-Many Environment Many-to-One Environment Scaling and Denial-of-Service Attacks Even Simple Webbots Can Generate a Lot of Traffic Inefficiencies at the Target The Problems with Scaling Too Well Creating Multiple Instances of a Webbot Forking Processes Leveraging the Operating System Distributing the Task over Multiple Computers Managing a Botnet Botnet Communication Methods Further Exploration PART IV: LARGER CONSIDERATIONS DESIGNING STEALTHY WEBBOTS AND SPIDERS 265 Why Design a Stealthy Webbot? Log Files Log-Monitoring Software Stealth Means Simulating Human Patterns Be Kind to Your Resources Run Your Webbot During Busy Hours Don t Run Your Webbot at the Same Time Each Day Don t Run Your Webbot on Holidays and Weekends Use Random, Intra-fetch Delays Final Thoughts PROXIES 273 What Is a Proxy? Proxies in the Virtual World Why Webbot Developers Use Proxies Using Proxies to Become Anonymous Using a Proxy to Be Somewhere Else Using a Proxy Server Using a Proxy in a Browser Using a Proxy with PHP/CURL Types of Proxy Servers Open Proxies Tor Commercial Proxies Final Thoughts Anonymity Is a Process, Not a Feature Creating Your Own Proxy Service xix

11 28 WRITING FAULT-TOLERANT WEBBOTS 285 Types of Webbot Fault Tolerance Adapting to Changes in URLs Adapting to Changes in Page Content Adapting to Changes in Forms Adapting to Changes in Cookie Management Adapting to Network Outages and Network Congestion Error Handlers Further Exploration DESIGNING WEBBOT-FRIENDLY WEBSITES 297 Optimizing Web Pages for Search Engine Spiders Well-Defined Links Google Bombs and Spam Indexing Title Tags Meta Tags Header Tags Image alt Attributes Web Design Techniques That Hinder Search Engine Spiders JavaScript Non-ASCII Content Designing Data-Only Interfaces XML Lightweight Data Exchange SOAP REST Final Thoughts KILLING SPIDERS 309 Asking Nicely Create a Terms of Service Agreement Use the robots.txt File Use the Robots Meta Tag Building Speed Bumps Selectively Allow Access to Specific Web Agents Use Obfuscation Use Cookies, Encryption, JavaScript, and Redirection Authenticate Users Update Your Site Often Embed Text in Other Media Setting Traps Create a Spider Trap Fun Things to Do with Unwanted Spiders Final Thoughts xx

12 31 KEEPING WEBBOTS OUT OF TROUBLE 317 It s All About Respect Copyright Do Consult Resources Don t Be an Armchair Lawyer Trespass to Chattels Internet Law Final Thoughts A PHP/CURL REFERENCE 327 Creating a Minimal PHP/CURL Session Initiating PHP/CURL Sessions Setting PHP/CURL Options CURLOPT_URL CURLOPT_RETURNTRANSFER CURLOPT_REFERER CURLOPT_FOLLOWLOCATION and CURLOPT_MAXREDIRS CURLOPT_USERAGENT CURLOPT_NOBODY and CURLOPT_HEADER CURLOPT_TIMEOUT CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR CURLOPT_HTTPHEADER CURLOPT_SSL_VERIFYPEER CURLOPT_USERPWD and CURLOPT_UNRESTRICTED_AUTH CURLOPT_POST and CURLOPT_POSTFIELDS CURLOPT_VERBOSE CURLOPT_PORT Executing the PHP/CURL Command Retrieving PHP/CURL Session Information Viewing PHP/CURL Errors Closing PHP/CURL Sessions B STATUS CODES 337 HTTP Codes NNTP Codes C SMS GATEWAYS 341 Sending Text Messages Reading Text Messages A Sampling of Text Message Addresses INDEX 345 xxi