Considerations for Constructing Twitter Queries in SMA

Transcription

1 Considerations for Constructing Twitter Queries in SMA This document is intended for users who write BoardReader queries in SMA to trigger collection of Twitter content via a GNIP Power Track instance. This happens when writing queries to fetch against the Microblogs (Twitter) source. When Twitter is enabled as a data source for a BoardReader key, BoardReader collects Twitter records from the SMA user's GNIP PowerTrack instance by translating Twitter queries into GNIP rules. The collected results are then fetched via the issued queries and used in SMA analysis. For best results, the following rules and best practices should be considered. The rules and best practices take into account the following: The extended query syntax offered in BoardReader APIs The compatible Power Track operators and syntax for creating valid rules The challenging task of processing an infinite variety of pass-through queries and mapping these queries to functionally equivalent Power Track rules. Important: BoardReader queries that do not adhere to GNIP limitations as described in this document will be dropped from collection without real-time feedback to the SMA user. By contrast, SMA's automatically generated queries accommodate the limitations so that users don't have to think about BoardReader and GNIP syntax restrictions. Considerations for pass-through queries Term - A query is composed of one or more terms. A term can also be referred to as a keyword. Terms can be positive or negative No individual term should exceed 128 characters A term can start with either a digit (0-9) or any non-punctuation character. Current punctuation characters are defined as the ASCII characters:! % & \ ' ( ) * + -. / ; < = >? \\, : \t \r \n " [] _ and the Unicode ranges: U+007B -- U+00BF U+02B0 -- U+037F U U+2BFF U+FF00 -- U+FF03 U+FF05 -- U+FF0F Any term that needs to start with these characters must be quoted. Note that this consideration applies only if you intend the characters to be part of a term, not when they are being used as an actual operator (i.e. term for term exclusion). 1 of 5

2 Terms containing Unicode symbol, separator, or punctuation characters should be quoted to take advantage of the exact match operator. Do not use wildcards with pass-through queries. Query the value being passed to BoardReader s API (e.g. /Twitter/Search endpoint) The maximum rule length allowed by Power Track is 1,024 characters. However, the actual limit for pass-through queries is less than this. BoardReader must translate submitted queries into compatible GNIP rules and this may impact the number of characters required (e.g. the character used to designate an OR must be replaced with a literal OR which means each OR term requires one additional character). Also, during rule construction BoardReader may append the following operator to the query value passed in the API request: Sample This operator is used to specify the sampling rate (i.e. % of matches to be delivered). The default is 100% so this operator is only required if the client key is associated with some other sampling rate (e.g. 25% or 50%). When the operator is included, this will require 10 characters. To be safe, customers should limit query length to account for such manipulations to the original query. In addition, if languages are selected, 11 characters and 1 OR term must be reserved for each language. No additional considerations need to be given if Use all languages is selected (no language filters are specified). In summary, customers must allow 10 extra characters if a sampling rate applied to their key, one extra character for each OR term used, and 11 extra characters and 1 OR term for every language selected. A single Power Track rule can support up to 29 positive terms, and up to 50 negative terms. Exceeding these limits will cause the rule to be rejected. Negative terms cannot be part of an OR. Language Boolean Operators When Twitter queries are executed through SMA, these queries are registered (with GNIP Power Track) for collection. As of SMA 1.3 FP1 IF5, collection respects language selection and collects records matching only those selected languages. Queries invoked through SMA filter the content to your languages of choice. Selecting Use all languages in SMA results in collecting and fetching results matching the query in all languages. In some cases, there are multiple ways to write a BoardReader query involving Boolean operators. For pass-through queries we recommend the following operators be used: 2 of 5

3 AND Use a single space instead of the & character. NOT Use the - character (ASCII hyphen-minus sign) instead of the! character (also valid) or some Unicode dash character like Em or En (invalid). Use a space to separate the NOT term(s) from the rest of the query. These are the same operators that GNIP supports so following these recommendations will reduce variation and make query evaluation and mapping to an equivalent GNIP rule more reliable. The OR operator is not listed because the only option is to use the character. The + character should not be used to separate terms. A single space should be used for pass-through queries. Phrase/Exact Match A phrase counts as a single positive or negative term for the purpose of checking those limits A phrase must have a matching pairs of quotes or the pass-through query will be rejected. Please make sure to use the standard ASCII double quote character for phrases. Phrases cannot be embedded. For example outer inner phrase is not a valid phrase. Proximity Operator Power Track rules support the same proximity operator syntax as BoardReader, but the behavior differs slightly and is defined as follows: Syntax: "term_a term_b"~n Where terms are no more than N tokens from each other If keywords in opposite order, they cannot be more than N-2 tokens from each other. GNIP provides the following example: Assume the rule is love boulder ~4. This rule would match the following 2 sentences: Love everything about my town Boulder. Boulder, I love living here. But would not match these 2 sentences: I don t love hiking, but I really like to visit Boulder. Boulder is a place I love to visit. 3 of 5

4 Can have any number of terms in quotes N cannot be greater than 6 If a value of N > 6 is submitted in the pass-through query BoardReader will substitute a value of 6 in the derived Power Track rule. Grouping Stop words Parenthesis used for grouping terms must be balanced or the passthrough query will be rejected. The current list of Power Track stop words includes: "a", "an", "and", "at", "but", "by", "com", "from", "http", "https", "if", "in", "is", "it", "its", "me", "my", "or", "rt", "the", "this", "to", "too", "via", "we", "www", "you" These terms are not allowed unless in quotes. There must be at least one non-stop word term in the phrase. To prevent run-away data collection, we are ignoring any pass-through query with character length 1 (e.g. the letter M or number 1 ) Field Search and Position Operators The following Sphinx operators have no Power Track equivalent and will be silently ignored: Field search operator world) Field position limit modifier hello) Multiple-field search operator hello world) Ignore field search operator hello world) Ignore multiple-field search operator hello world) All-field search operator hello) In the above examples title and body are searchable fields in our index Other Advanced Sphinx Operators The following Sphinx operators have no Power Track equivalent and will cause the pass-through query to be rejected: Quorum matching operator Field-start and field-end modifier NEAR, generalized proximity operator 4 of 5

5 SENTENCE operator (not supported in BoardReader index) PARAGRAPH operator (not supported in BoardReader index) ZONE limit operator (not supported in BoardReader index) ZONESPAN limit operator (not supported in BoardReader index) These operators will be silently removed: Strict order operator (silently removed if not quoted) Exact form modifier (silently removed if not quoted) 5 of 5