RESTful API. Documentation

Transcription

1 RESTful API Documentation

2 Copyright 2014, 2015, 2016 by cybertoolbelt.com All Rights Reserved Revision: 1.0 9/3/ /27/ /16/ /27/ /6/ /11/ /18/ /19/ /22/ /17/ /15/ /26/ /29/ /8/2016 CyberTOOLBELT is a registered trademark of ICG, Inc.

3 CD INTRODUCTION This document describes the use of the RESTful API to the cybertoolbelt.com database. It is assumed that the reader has familiarity with both API usage and the protocol. In order to use the CTB API you must have an API key which is procured from CyberTOOLBELT. This key is passed to the API on every call. CONVENTIONS Anything that appears in mono-spaced text with a gray background is either a data description or program code. Example: <?php // // This is an example bit of code // We will primarily use the PHP programming language in our examples although there is a sprinkling of curl command line examples also. Appendix D is a customer-supplied Python interface. Note that the CTB API is language agnostic. USING THE API The API provides access to CyberTOOLBELT s data. The data provided and the methods of access are outlined in this section. The API supports various data queries on different datasets. You can use either GET or POST verbs depending on the query type you are performing. The various endpoints of the API are documented in the individual dataset descriptions. Substitute your actual CTB-issued API Key for the term API_KEY In the following documentation as part of the URL s. Errors are returned in one of two ways as follows: 1. The HTTP status code. 200 is success. Anything starting with a 4 or a 5 is a problem. 2. The response will contain a non-blank error field as in: { error : Daily quota limit exceeded Most errors will return a HTTP status and an error field. For example, an invalid API key will return status 401 Not Authorized as well as { error : Not Authorized. pg. 1

4 Some of the calls will also return a warning field in the JSON. The format will be something like this: { warning : domain not found. These types of warnings are usually given when the data requested in not in our database yet. SUBSCRIPTION INFORMATION CTB API access is fee based. There are a number of toolsets left column which all require individual subscriptions in addition to the base API access. Pricing is available from CTB or our resellers. Note that a number of the toolsets contain more than one tool. For example, the Whois toolset contains three individual tools. Toolset Description Includes Tools Whois Provides access to the Historic Whois Dataset Search by Whois ID Search by domain name Translate Whois IDs to domain names and IDs IP Whois Provides access to Historic IP address Whois Search by IP address Dataset Domain Whois Provides ability to mine the Whois Dataset using Whois Data Mining data mining powerful full text queries IP Operations Various IP address operations Reverse IP lookup ASN Lookup by IP address TOR node check Proxy check Domain Ops Various domain-related operations Domain Zone File History Find Domain Names Find Subdomains Domains Related to Issues Domain/IP issues dataset Domain Issues IP Whois data Provides ability to mine IP dataset using IP Whois Data Mining mining Person Search powerful full-text queries. Various searches for person data based on data such as , Twitter handles, telephone number and Facebook user names. Search by data item pg. 2

5 DOMAIN WHOIS DATASET You can query the CTB Whois dataset in a number of different ways as follows: 1. Return the historical Whois information for a domain whose Whois unique ID is passed to the call. This ID is returned by search and data mining calls. 2. Return the historical Whois information for a domain whose domain name is passed as a query parameter. 3. Perform a data mine operation wherein you pass a query string to the call and the system returns you a list of matching domains. You must be subscribed to the Domain Whois data mining tool. 4. Return historical Whois information about an IP address. You must be subscribed to the IP Whois tool. All Whois API endpoints start with /whois. Examples: The above examples cover both the primary Whois tools: domain and IP. RETRIEVE HISTORICAL WHOIS RECORD BY WHOIS ID OPERATION: /GET FORMAT: /whois/{id This operation returns the historical Whois information for a domain whose unique Whois ID is passed in the command. This ID is returned as the result of a DATAMINE operation described below. There are two optional parameters: nodiffs=y CTB normally returns the most recent record in its database when a historical Whois lookup is performed. This includes all the changes made to the record from the first time CTB fetched the record. These changes are returned as an array which denotes the field that changed, when it changed, the old value and the new value. If you DON T want this information returned in the call add the NODIFFS parameter to the query string. Example: raw=y The default behavior of CTB is to return the parsed record when available without returning the original fetched record from the Whois server who initially responded to the Whois query CTB submitted. Using the RAW parameter will cause the raw record to be inserted into the response. Example: pg. 3

6 Note that some or all of these parameters can be used at the same time as in the following example: Example Program to retrieve the historical Whois record whose ID is : <?php $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="whois/123456"; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response is described in detail in Appendix A. pg. 4

7 RETRIEVE HISTORICAL WHOIS RECORD BY DOMAIN NAME OPERATION: /GET FORMAT: /whois/domain/{domain_name This operation returns the historical Whois information for a domain whose name is passed as DOMAIN_NAME in the URL. There are two optional parameters: nodiffs=y CTB normally returns the most recent record in its database when a historical Whois lookup is performed. This includes all the changes made to the record from the first time CTB fetched the record. These changes are returned as an array which denotes the field that changed, when it changed, the old value and the new value. If you DON T want this information returned in the call add the NODIFFS parameter to the query string. Example: raw=y The default behavior of CTB is to return the parsed record when available without returning the original fetched record from the Whois server who initially responded to the Whois query CTB submitted. Using the RAW parameter will cause the raw record to be inserted into the response. Example: lookup=y Normally when CTB does an operation through the REST API it just returns what s in our database. Using this parameter will cause CTB to perform a lookup on the domain over the Internet. While our database call is very fast, specifying lookup puts you at the mercy of Internet connection overhead and the speed of the responding server which can be considerable. Example:: Note that some or all of these parameters can be used at the same time. Example code used to retrieve the historical Whois record for cybertoolbelt.com: pg. 5

8 <?php $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="whois/domain/cybertoolbelt.com"; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response is the same as the Retrieve Whois record by ID call which is described in detail in the Appendixes. pg. 6

9 RETRIEVE HISTORICAL IP WHOIS RECORD BY IP ADDRESS OPERATION: /GET FORMAT: /whois/ip/{ipv4_address This operation returns the historical Whois information for an IPv4 address passed to the API as IPv4_ADDRESS in the URL. There is a single optional parameter: nodiffs=y CTB normally returns the most recent record in its database when a historical IP Whois lookup is performed. This includes all the changes made to the record from the first time CTB fetched the record. These changes are returned as an array which denotes the field that changed, when it changed, the old value and the new value. If you DON T want this information returned in the call add the NODIFFS parameter to the query string. Example: This operation requires a subscription to the IP Whois dataset. Example code used to retrieve the historical IP Whois record for : <?php $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="whois/ip/ "; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response is detailed in Appendix C. pg. 7

10 RETRIEVE DOMAIN NAMES FROM WHOIS ID NUMBERS OPERATION: /POST FORMAT: /whois/getdomainnames POST BODY: ids=whois_record_list This operation returns the domain name for each Whois record ID passed in a comma-separated list. Note that since this is a POST operation the API is expecting the IDS parameter in the body. The format of the returned data is: domain_recid:domain_name{,domain_recid:domain_name Example call: ids=123,245,576 This call is primarily a batch processor for converting Whois search results which are Whois IDs into their respective domain names when the actual Whois detail information isn t required. Each Whois ID you submit will count against your tool usage quota. pg. 8

11 IP ADDRESS DATASET CTB has a large amount of data on IP addresses, both historic and current. This tool requires a subscription to the IP ASN tool. All IP API endpoints start with /IP. Example: RETRIEVE ASN INFORMATION FOR AN IPV4 ADDRESS OPERATION: /GET FORMAT: /ip/asn/{ipv4 ADDRESS This operation is used to return information about the ASN that the IP address supplied in the call is owned by. Example Program to retrieve the ASN information for IP address : <?php $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="ip/asn/ ; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The call returns the below information. Note that we are only discussing the highlights here. Print the results of the JSON response to see the results in detail. 1. Information about the ASN such as the owner, the IP range, contact, etc. 2. A list of one or more contacts such as Admin or Technical. 3. The history of the ASN which defines the IP range change, activity status and the date of the change. 4. Active IP blocks. 5. Inactive IP blocks. pg. 9

12 RETRIEVE DOMAINS RELATED TO AN IPV4 ADDRESS OPERATION: /GET FORMAT: /ip/domains/{ipv4 ADDRESS This call is used to retrieve all the domains that CTB knows about are that related to the IP address specified in the call. This is essentially a reverse DNS call. Example: <?php $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="ip/domains/ ; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The call returns an array of related domains that would look something like the following: pg. 10

13 Array [0] => Array [domain] => nepadodgeball.com [domain_id] => [when_related] => [1] => Array [domain] => mickisticki.com [domain_id] => [when_related] => [2] => Array [domain] => hss4.us [domain_id] => [when_related] => Each element in the array consists of three fields: 1. The name of the related domain. 2. The internal ID of the domain. 3. A Unix format timestamp of when CTB related the domain to the IP address. pg. 11

14 RETRIEVE TOR NODE INFORMATION RELATED TO AN IPV4 ADDRESS OPERATION: /GET FORMAT: /ip/is_tor/{ipv4 ADDRESS This call is used to check of an IPv4 address is a Tor exit node and retrieve all information that CTB knows about that is related to the IP address specified in the call. Results are only returned if the supplied IP address is recorded as a TOR exit node in our database. Example: <?php $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="ip/is_tor/ ; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The call returns an array of related domains that would look something like the following: pg. 12

15 Array [0] => Array [name] => omrphuim [router_port] => 9001 [directory_port] => 9030 [roles] => Array [0] => Fast [1] => Running [2] => V2 Dir [3] => Valid [uptime] => [version] => Tor [contact] => Jay FUCKSPAMMERS.chickenkiller.com> [last_seen] => [active] => inactive The fields are pretty self-explanatory. There may be a diffs array which will represent changes to the record that CTB has recorded. It is in the standard format of when changed, what changed, old value and new value. pg. 13

16 RETRIEVE PROXY INFORMATION RELATED TO AN IPV4 ADDRESS OPERATION: /GET FORMAT: /ip/is_proxy/{ipv4 ADDRESS This call is used to check of an IPv4 address acts as a proxy and retrieve all information that CTB knows about that is related to the IP address specified in the call. Results are only returned if the supplied IP address is recorded in our database as a proxy. Example: <?php $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="ip/is_proxy/ ; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The call returns an array of related domains that would look something like the following: pg. 14

17 Array [0] => Array [country] => ID [port] => 3128 [first_seen] => [last_seen] => [type] => [protocol] => https [anonymity] => anon [score] => 1 [1] => Array [country] => ID [port] => 3128 [first_seen] => [last_seen] => [type] => [protocol] => http [anonymity] => anon [score] => 1 The fields should be self-explanatory. Only four fields are guaranteed to have data: first_seen, last_seen, port and score. score is set to 1 if the IP address being queried has been on a proxy list or otherwise determined to be a proxy. Values less than 1 can be dynamically computed and returned. At this time there is limited support for such dynamic determination. pg. 15

18 DATAMINING OPERATIONS CTB has a large amount of data that can be mined. All data mining endpoints begin with /datamine. Example: PERFORM HISTORICAL WHOIS RECORDS SEARCH OPERATION: /POST FORMAT: /datamine/whois/?key=api_key POST BODY: terms=query This operation allows you assuming your API_KEY is authorized for this operation to search the historical Whois records and return records that match your query terms. Note that since this is a POST operation the API is expecting the TERMS parameter in the body. This operation requires a subscription to the Domain Whois data mining tool. Example which will return all records which contain the string mlewis anywhere within the record: <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="datamine/whois/"; $base = " $req="terms=".urlencode'"mlewis"'; $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_POST, 1; curl_setopt$ch, CURLOPT_POSTFIELDS, $req; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; pg. 16

19 The format of the response object is: Array [total_hits] => 390 [returned_hits] => 390 [hits] => Array [0] => Array [whois_id] => [domain_id] => [score] => [1] => Array [whois_id] => [domain_id] => [score] => [2] => Array [whois_id] => [domain_id] => [score] => This call will return up to 500 results in descending order of relevance. The total_hits field contains the total number of matching records. The returned_hits field returns the actual number of results returned in the hits array. Each element in the hits array has three elements as follows: 1. whois_id The is the ID of the matching Whois record. The record itself can be retrieved using the /whois/{id endpoint described above. 2. domain_id This is the internal domain ID. 3. score This is the relevance score of the match. It is predicated on a number of factors such as number of times the matching terms appeared in the document vs. all documents. Without getting into the weeds the important thing to keep in mind is that higher scores mean greater relevance of the hit for that record to your query. pg. 17

20 The example search was the simplest available. The API supports a number of logical operations as well as phrase matching, specific field searches, date operations, etc. For example, your terms could be any of the following: terms=mlewis and pixel terms=mlewis and pixel not network solutions terms=joeblow@gmail.com since last Friday terms= contains mlewis terms= contains mlewis and registrar=netowl or registrar=godaddy As you can see parentheses are supported. The full list of operators are: 1. AND. The two terms must both be present in the record for the condition to be true. Example: joe AND blow. Both Joe and Blow must exist in a record for it to be selected. Note that the capitalization of the operators is strictly to improve readability for the example. They are case-insensitive in your actual queries. 2. OR. Either of the terms must be true. Note that in the absence of any specific operator between terms that OR is the default. Example: mike bill sam is identical to mike OR bill OR sam. 3. NOT. The following term CAN not appear in the record for a record to be selected. Example: joe AND blow NOT godady. 4. CONTAINS. This is a string operator that requires a term and an operand. For example: registrar contains godaddy. This is a full-text operator. 5. IS or =. This is a string operator that requires an exact match to be true. Example: state=nj. 6. Date operations. There are three date fields associated with Whois records: Created, Updated and Expires. You can use the following date operations: a. on or =. Date must match exactly. Examples: updated on 5/1/2015, created=1/1/2015 b. on or after or >= or since. Examples: created on or after last Monday, expires>=march 5,2014, since yesterday. note that this is equivalent to: created >=3/5/2014 OR updated >=3/5/2014 if today is 3/6/2014. c. on or before or <=. Examples: created before last month, expires<=2/8/2012 d. after or >. Example: expires after 1/1/2020 e. before or <. Example: created after 3/1/2015 AND created before 3/3/2015. You can specify dates in almost any manner such as: today, last Friday, last year 5/1/2015, March 8, 2011, , etc. The following are the individual fields that you can search on can be used as terms with an operator and and operand: pg. 18

21 pg created Date field representing the date when the Whois record was created. Valid for use with all date operators. 2. updated Date field representing the date when the Whois record was last updated at the time the record was last fetched by us. Valid for use with all date operators. 3. expires Date field representing the date when the Whois record is due to expire. Valid for use with all date operators. 4. server String, non-full text field, that represents the Whois server that was queried for the data. The only valid operator is is or its corresponding sugar =. 5. String, non-full text field, that represents any address associated with the record. This includes any address found in a contact record for the Whois rcord. The only valid operator is is or its corresponding sugar =. 6. status String full-text field that represents the Whois record s status. This is a full-text searchable field that supports the contains and phrase match syntax. 7. registrar String full-text field that represents the Whois registrar name. This is a full-text searchable field that supports the contains and phrase match syntax. 8. name String full-text field that represents any of the registrant s names. These names can appear in the main record as the registrant contact or any of the other contacts. This is a full-text searchable field that supports the contains and phrase match syntax. 9. organization String full-text field that represents any of the registrant s organization names. These organization names can appear in the main record as the registrant contact or any of the other contacts. This is a full-text searchable field that supports the contains and phrase match syntax. 10. address String full-text field that represents any of the registrant s street addresses. These addresses can appear in the main record as the registrant contact or any of the other contacts. This is a full-text searchable field that supports the contains and phrase match syntax. 11. city String full-text field that represents any of the registrant s city names. These city names can appear in the main record as the registrant contact or any of the other contacts. This is a full-text searchable field that supports the contains and phrase match syntax. 12. state String, non-full text field, that represents any of the registrant s state names. These state names can appear in the main record as the registrant contact or any of the other contacts. The only valid operator is is or its corresponding sugar =.

22 13. country String, non-full text field, that represents any of the registrant s country names. These country names can appear in the main record as the registrant contact or any of the other contacts. The only valid operator is is or its corresponding sugar =. 14. telephone String, non-full text field, that represents any of the registrant s telephone number fields. These telephone numbers can appear in the main record as the registrant contact or any of the other contacts. The only valid operator is is or its corresponding sugar =. 15. fax String, non-full text field, that represents any of the registrant s fax number fields. These fax numbers can appear in the main record as the registrant contact or any of the other contacts. The only valid operator is is or its corresponding sugar =. 16. Postal_code String, non-full text field, that represents any of the registrant s postal code zip code in the US fields. These postal codes can appear in the main record as the registrant contact or any of the other contacts. The only valid operator is is or its corresponding sugar =. In the above field descriptions there are references to full-text fields and non-full-text fields. The only difference is what operators can be used to search them. For example, the field is a non-full text field. That means you can only search for the example address as in: =jblow@gmail.com If the field was a full-text field you could search three different ways: 1. contains jblow 2. contains jblow@gmail.com 3. contains gmail.com These are only restrictions when you are using individual field operations as part of your query. The entire Whois record and all of its diffs are stored in one large full-text field. This means that any data is searchable using full-text operators. When this becomes a problem is where you submit a search such as the following: jblow@cyberertoolbelt.com Expecting to get back records that match jblow@cybertoolbelt.com exactly. What you will, in fact, get back are all the records that match jblow or cybertoolbelt.com. This is because the query processor breaks on character so your query becomes instead jblow OR cybertoolbelt OR com. pg. 20

23 PERFORM IP WHOIS RECORDS SEARCH OPERATION: /POST FORMAT: /datamine/ip_whois/?key=api_key POST BODY: terms=query This operation allows you assuming your API_KEY is authorized for this operation to search the IP Whois records and return records that match your query terms. Note that since this is a POST operation the API is expecting the TERMS parameter in the body. This operation requires a subscription to the IP Whois data mining tool. Example which will return all records which contain the string peer1 anywhere within the record: <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="datamine/ip_whois/"; $base = " $req="terms=".urlencode'"peer1"'; $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_POST, 1; curl_setopt$ch, CURLOPT_POSTFIELDS, $req; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response object is: pg. 21

24 Array [total_hits] => 199 [returned_hits] => 199 [keywords] => ["peer1"] [hits] => Array [0] => Array [whois_id] => [ip] => 0 [score] => [1] => Array [whois_id] => [ip] => 0 [score] => This call will return up to 500 results in descending order of relevance. The total_hits field contains the total number of matching records. The returned_hits field returns the actual number of results returned in the hits array. Each element in the hits array has three elements as follows: 1. whois_id The is the ID of the matching Whois record. The record itself can be retrieved using the /whois/{id endpoint described above. 2. ip Reserved for later expansion. 3. score This is the relevance score of the match. It is predicated on a number of factors such as number of times the matching terms appeared in the document vs. all documents. Without getting into the weeds the important thing to keep in mind is that higher scores mean greater relevance of the hit for that record to your query. pg. 22

25 PERFORM TO DOMAIN SEARCH OPERATION: /GET FORMAT: /datamine/ /{ _address?key=api_key This operation allows you assuming your API_KEY is authorized for this operation to return all domains related to the supplied _ADDRESS. This operation requires a subscription to the data mining tool. This example returns all domains related to mlewis@xelent.net : <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="datamine/ /mlewis@xelent.net"; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response object is an array of domain names and their IDs as shown here: Array [0] => Array { [name]=> xelent.net [domain_id] => [1] => Array { [name]=> xample.com [domain_id] => pg. 23

26 DOMAIN OPERATIONS These operations are related to domains and all begin with the /domains endpoint. Example: RETURN ZONE FILE HISTORY FOR DOMAIN OPERATION: /GET FORMAT: /domains/zone_history/domain_name ID/?key=API_KEY This operation returns the zone file history of a domain assuming your API_KEY is authorized for this operation. Usually a domain will have an ADD only record if it is still active. It will have a DELETE record when it expires and is removed from the zone file. Note if there is an ADD record with a later time stamp than the DELETE record it usually means that the domain was reregistered. This tool requires a subscription to the domain operations toolset. Example <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="domains/zone_history/xelent.net/"; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response object is: pg. 24

27 Array [history] => Array [0] => Array [action] => add [when] => [1] => Array [action] => delete [when] => pg. 25

28 FIND MATCHING DOMAINS OPERATION: /POST FORMAT: /domains/find/?key=api_key POST BODY: terms=query {&age=n This operation returns the domain names that match the passed QUERY. The optional AGE field is the number of days from today to restrict the search operation. For example, to find all matching days in the last 2 weeks specify age=14. The format of the query string is comprised of terms which are either a full domain name with a TLD ie: example.com or a partial name without a dot to find all domains containing the partial name ie: herbal, or you can enter the characters that a domain must start with followed by a "%" character to get all domains back that start with the characters before the "%" character. For example, pixel% will return all domains starting with the substring "pixel". You can also specify AND and NOT functions as follows: xelent +income xelent -ment The first would match all domains with XELENT anywhere within the name and the substring INCOME appearing anywhere in the domain name. The second example would select all domains with the string XELENT anywhere within the name but NOT also containing the substring MENT. You can string more than one condition together: xelent +income -ment xelent +pixel +enter The first example would return all matches that contain the substrings XELENT and INCOME but do not contain the substring MENT. The second example would return domains that contain all three substrings: XELENT, PIXEL and ENTER. Note that you cannot specify terms that will match the TLD part of the domain name. For example, xelent -.com is not valid. The response is a JSON object comprised of zero or more matching array elements. The first element of each matching element is the domain ID of the domain. The second element is the domain name. Example used to get the names of all domains that start with the string xelent : pg. 26

29 <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="domains/find/"; $base = " $req="terms=".urlencode'xelent"'; $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_POST, 1; curl_setopt$ch, CURLOPT_POSTFIELDS, $req; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response object is: Array [0] => Array [id] => [domain] => xelent-design.de [1] => Array [id] => [domain] => xelent-it.com [2] => Array [id] => [domain] => xelent-it.de pg. 27

30 RETURN SUBDOMAINS RELATED TO DOMAIN OPERATION: /GET FORMAT: /domains/subdomains/{domain?key=api_key This operation allows you assuming your API_KEY is authorized for this operation to return all subdomains related to the supplied DOMAIN. This operation requires a subscription to the Domain Operations tool. This example returns all subdomains related to cybertoolbelt.com : <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="domains/subdomains/cybertoolbelt.com"; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The format of the response object is an array as follows: pg. 28

31 Array [badhost] => [hosts] => Array [0] => Array [when_related] => [name] => api [first] => [last] => [ip] => Array [0] => Array [id] => [when_related] => [ip] => [1] => Array [when_related] => [name] => aspmx.l [first] => [last] => [ip] => Array The only unusual field in the above list is badhost. If this field has a non-zero value it will contain the count of the number of subdomains related to the domain and only return the first 200 subdomains related to the domain. In that case the field will contain the actual number of subdomains that are related to the domain. The cut-off is 2000 subdomains to determine if this field receives a value. By definition, if this field is defined it will have a value of at least 2000.,. Typically domains that meet this criteria are spam-related domains but not always. Each element in the hosts array describes a single subdomain named according to the name field. The IP sub-array can contain zero or more entries which are IP addresses that have been or are associated with the particular subdomain. pg. 29

32 ISSUES OPERATIONS These operations are related to returning information about issues and all begin with the /issues endpoint. Example: RETURN KNOWN ISSUES FOR DOMAIN OPERATION: /GET FORMAT: /ISSUES/DOMAIN/DOMAIN_NAME ID/?KEY=API_KEY This tool requires a subscription to the Issues toolset. Example: <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="issues/domain/yncsp2qncd.asia"; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; The result object looks like the following: pg. 30

33 Array [issues] => Array [0] => Array [type] => report [severity] => 56 [info] => Array [abuse_type] => spam [when_detected] => [when_removed] => 0 [last_checked] => [1] => Array [type] => name [severity] => 0 There is a primary array named issues that contains a number of subarrays. There are two types of subarrays. The first type of subarray one of type report as shown in the example. It has a severity rating and a subarray of additional information about the type of report such as the abuse type and various timestamps Unixstyle timestamps: Seconds since 1/1/1970. The second type of issue subarray details the severity of the issue and the type of issue. In the above example the type is name which means the name of the domain was determined to be less than likely to be an actual domain name. Note that it could still be a non-malicious domain something like na8ur.com would score low, but it certainly bears looking at. The severity is a value that represents the seriousness of the issue on a sliding scale: 0 being the most serious, 1000 being the least. The age issue s severity is calculated on a sliding scale based on the age of the domain time since it was registered. The tld issue occurs when a domain is part of a notorious TLD such as.blue. The bad_domain issue arises when the domain has been identified as being associated with bad actors such as in.net. The hosts issue is raised when the domain has a large number of subdomains. Note that it is up to you to determine which issues are important to your application. pg. 31

34 PERSON SEARCH These operations are related to returning information about issues and all begin with the /issues endpoint. Example: SEARCH FOR PERSON DATA BY , TELEPHONE, TWITTER HANDLE OR FACEBOOK NAME OPERATION: /GET FORMAT: /PERSON/SEARCH/WHAT-DATA/?KEY=API_KEY This tool requires a subscription to the Person Search toolset. The single parameter is WHAT-DATA which is really two parameters separated by a dash -. WHAT can be one of the following: . Search for person data by an address. DATA must be a valid address. phone. Search for person data by a telephone number. DATA must be a valid telephone number. twitter. Search for person data by a Twitter handle. Data must be a valid telephone number. fb. Search for person data by a Facebook user name. Data must be a valid Facebook user name. Example: <?php?> $key="?key=<<< YOUR CTB ISSUED API KEY GOES HERE >>>"; $route="person/search/ -mlewis@xelent.net"; $base = " $url="$base$route$key"; $curl_result = $curl_err = ''; $ch = curl_init; curl_setopt$ch, CURLOPT_URL, $url; curl_setopt$ch, CURLOPT_RETURNTRANSFER, 1; curl_setopt$ch, CURLOPT_HEADER, 0; curl_setopt$ch, CURLOPT_VERBOSE, 1; curl_setopt$ch, CURLOPT_SSL_VERIFYPEER, FALSE; curl_setopt$ch, CURLOPT_SSL_VERIFYHOST, FALSE; curl_setopt$ch, CURLOPT_SSLVERSION, 1; curl_setopt$ch, CURLOPT_TIMEOUT, 30; $res $curl_err = curl_error$ch; curl_close$ch; if substr$res,0,1=="[" substr$res,0,1=="{" { print_rjson_decode$res,1; else { print "$res"; pg. 32

35 The result object looks like the following: Array [contactinfo] => Array [websites] => Array [0] => Array [url] => [familyname] => Lewis [fullname] => Michael Lewis [givenname] => Michael [demographics] => Array [locationdeduced] => Array [normalizedlocation] => Stroudsburg, Pennsylvania [deducedlocation] => Stroudsburg, Pennsylvania, United States [city] => Array [deduced] => [name] => Stroudsburg [state] => Array [deduced] => [name] => Pennsylvania [code] => PA [country] => Array [deduced] => 1 [name] => United States [code] => US [continent] => Array [deduced] => 1 [name] => North America [county] => Array [deduced] => 1 [name] => Monroe [code] => Monroe [likelihood] => 1 [gender] => Male [locationgeneral] => Stroudsburg, PA continued on next page pg. 33

36 [socialprofiles] => Array [0] => Array [followers] => 0 [following] => 0 [type] => klout [typeid] => klout [typename] => Klout [url] => [username] => myithreat [id] => [1] => Array [followers] => 0 [following] => 0 [type] => foursquare [typeid] => foursquare [typename] => Foursquare [url] => [id] => [2] => Array [bio] => We provide information from thousands of threat sources as well as the alerts generated by our users. [followers] => 6 [following] => 2 [type] => twitter [typeid] => twitter [typename] => Twitter [url] => [username] => myithreat [id] => [digitalfootprint] => Array [scores] => Array [0] => Array [provider] => klout [type] => general [value] => 11 [topics] => Array [0] => Array [provider] => klout [value] => Twitter [domains] => 13 pg. 34

37 APPENDIX A. WHIS RECORD FIELDS whois_id = CTB whois record ID created_date = Date the public whois record was created. updated_date = Date the public whois record was last updated. expires_date = Date the public whois record expires. status = Status of the whois record. = Primary associated with whois record. domain.name = Whois record domain name. registrar.name = Authoritative registrar for whois record. organization = Organization responsible for domain. contacts. = Contact for domain. contacts.name = Personal contact name for domain. contacts.organization = Organization responsible for domain. contacts.street1 = Street address for domain contact. contacts.street2 = Street address for domain contact. contacts.street3 = Street address for domain contact. contacts.street4 = Street address for domain contact. contacts.city = Street address for domain contact. contacts.state = Street address for domain contact. contacts.country = Country of domain contact. contacts.fax = Fax number for domain contact. contacts.fax_ext = Fax number extension for domain contact. contacts.telephone = Telephone number for domain contact. contacts.telephone_ext = Telephone extension for domain contact. contacts.postal_code = Postal code for domain contact. diff.* = Changes to the above records. pg. 35

38 APPENDIX B. DOMAIN INFO RECORD The following example shows the format of the domain info array after JSON decoding by a PHP json_decode function. The information is returned as an array with a number of sub-arrays of related information. The first of the sub-arrays we will look at is the $ra[ domain_info ] sub-array which contains information about the domain itself and any addresses that are related to the domain. [domain_info] => Array [name] => xelent.net [id] => The $ra[ domain_info ] contains the following elements: The Domain name in $ra[ domain_info ][ name ] The CTB domain ID number used in a number of API calls in the field $ra[ domain_info ][ id ] You then will have one of two different arrays representing the whois data. The first is in the $ra[ whoisnp ] sub-array. This sub-array is returned if we were unable to parse the whois data for any reason. For example, consider this returned data for google.com: pg. 36

39 [whoisnp] => Array [0] => Array [source] => ct [when_related] => [domain] => google.com [whois_server] => whois.markmonitor.com [whois_server_ip] => [ts] => :55:03 [raw] => Domain Name: google.com<br>registry Domain ID: <br>registrar WHOIS Server: whois.markmonitor.com<br>registrar URL: Date: T04:00: <br>Creation Date: T00:00: <br>Registrar Registration Expiration Date: T21:00: <br>Registrar: MarkMonitor, Inc.<br>Registrar IANA ID: 292<br>Registrar Abuse Contact Abuse Contact Phone: <br>Domain Status: clientupdateprohibited<br>domain Status: clienttransferprohibited<br>domain Status: clientdeleteprohibited<br>registry Registrant ID: <br>registrant Name: Dns Admin<br>Registrant Organization: Google Inc.<br>Registrant Street: Please contact 1600 Amphitheatre Parkway<br>Registrant City: Mountain View<br>Registrant State/Province: CA<br>Registrant Postal Code: 94043<br>Registrant Country: US<br>Registrant Phone: <br>Registrant Phone Ext: <br>registrant Fax: <br>Registrant Fax Ext: <br>registrant Admin ID: <br>admin Name: DNS Admin<br>Admin Organization: Google Inc.<br>Admin Street: 1600 Amphitheatre Parkway<br>Admin City: Mountain View<br>Admin State/Province: CA<br>Admin Postal Code: 94043<br>Admin Country: US<br>Admin Phone: <br>Admin Phone Ext: <br>admin Fax: <br>Admin Fax Ext: <br>admin Tech ID: <br>tech Name: DNS Admin<br>Tech Organization: Google Inc.<br>Tech Street: 2400 E. Bayshore Pkwy<br>Tech City: Mountain View<br>Tech State/Province: CA<br>Tech Postal Code: 94043<br>Tech Country: US<br>Tech Phone: <br>Tech Phone Ext: <br>tech Fax: <br>Tech Fax Ext: <br>tech Server: ns3.google.com<br>name Server: ns4.google.com<br>name Server: ns2.google.com<br>name Server: ns1.google.com<br>url of the ICANN WHOIS Data Problem Reporting System: Last update of WHOIS database: T16:47: <<<<br><br>the Data in MarkMonitor.com's WHOIS database is provided by MarkMonitor.com for<br>information purposes, and to assist persons in obtaining information about or<br>related to a domain name registration record. MarkMonitor.com does not guarantee<br>its accuracy. By submitting a WHOIS query, you agree that you will use this Data<br>only for lawful purposes and that, under no circumstances will you use this Data to:<br> 1 allow, enable, or otherwise support the transmission of mass unsolicited,<br> commercial advertising or solicitations via spam; or<br> 2 enable high volume, automated, electronic processes that apply to<br> MarkMonitor.com or its systems.<br>markmonitor.com reserves the right to modify these terms at any time.<br>by submitting this query, you agree to abide by this policy.<br><br>markmonitor is the Global Leader in Online Brand Protection.<br><br>MarkMonitor Domain ManagementTM<br>MarkMonitor Brand ProtectionTM<br>MarkMonitor AntiPiracyTM<br>MarkMonitor AntiFraudTM<br>Professional and Managed Services<br><br>Visit MarkMonitor at us at <br>In Europe, at <br>--<br> [id] => 2 Note that the newline characters in the $whoisnp[ raw ] field have been replaced with <br>. pg. 37