Webtrekk Raw Data Export

Transcription

1 Webtrekk Raw Data Export

2 Contents 1 General information Prerequisites for use Configuration of the raw data export Raw data export FTP access data Export data Custom sending Reviewing the exports in System Monitor Content of the raw data tables getfullvisitors data in the context of visits getcontents page calls getfullreferrer access via referrers getfullsearchphrase / getfullsearchphraseintern search phrases getfullclicks clicks for page navigation getfullcampaigns campaign calls getfullbasket products in the shopping basket getfullorders orders getfullforms form data getformfields form field data getfullmedia media data getfullmediaaction media action data getfullcustomer customer identification getfullcustomerjourney customer journey data getfullcustpara tables custom parameter data getfullcustparasession_x (1..25) session parameters of 35

3 getfullcustparatimes_x (1..25) time-dependent parameters getfullcustparapage_x (1..25) page parameters getfullcustparaproduct_x (1..25) product parameters getfullcustparaclick_x (1..25) click parameters getfullcustparacampaign_x (1..25) campaign parameters Linking the raw data tables Assignment of the export tables to context levels Linking between the levels Contact us of 35

4 1 General information Webtrekk offers various export options. One possibility is the raw data export. Its functionality is described in detail in this document. The raw data export allows you to export raw data tables to an FTP server, for instance to populate your own data warehouse with the tracking data. CSV is used as the file format. 2 Prerequisites for use The raw data export may have to be activated before you can use it. Please contact the Webtrekk Support Team. You need administrator rights in Q3 to configure the raw data export. 3 Configuration of the raw data export To configure the raw data export, go to Configuration and click the Raw data export button. 3.1 Raw data export Figure 1 Raw data export button The basic settings for the raw data export are configured on the Raw data export tab. Figure 2 Raw data export The raw data export is activated/deactivated with the Activate raw data export (Yes/No) setting. 4 of 35

5 CSV field separator establishes the character used to separate the individual table entries from each other. The following can be selected: semicolon, comma, space, tab and custom characters. The dash (-) is not permitted; a tilde (~) used as the separator is replaced by a semicolon. The CSV field delimiter establishes the character used to delimit the table entries (this character comes before and after each table entry). The following can be selected: no character, apostrophe, quotation mark and custom characters. A - is not permitted as a delimiter. Please note that only custom characters not occurring in the table entries may be used. Furthermore, the field separator and field delimiter must be different. The Sending interval establishes when the raw data export takes place. Daily: The data tables contain the data from the previous day. An export is performed as soon as the data for the previous day have been fully written (end of day). Daily at 0 hours: The data tables contain the data from 10:00 pm on the day before last to 10:00 pm on the previous day. Hourly: The tables contain the data for the sessions concluded in the last hour. This option is not available by default. Please contact the Webtrekk Support Team for activation. Custom: The time period covered by the data tables can be customised. Initiate the process with the Send raw data now button. This process may be carried out up to 10 times per day. It can be performed again immediately. There is no need to wait until the tables for a previous export, if applicable, are received. The number of rows in a table is limited to 1,000,000. An error message is generated if this limit is exceeded. In this case, the time period has to be reduced and the export split into more than one. The 1,000,000 row limit does not apply to the other sending interval options. 3.2 FTP access data The data for the connection to the target server where the raw data tables will be sent are entered on the FTP access data screen. Figure 3 FTP access data The protocol used to send the data is chosen under Data source. FTP, SFTP, FTPS (SSL) and FTPS (TLS) are available. 5 of 35

6 The server access data must be entered in the remaining fields: Host: IP address or URL of the server where the data will be stored Remote path: Select the folder on the target server where the raw data files will be stored. The selected user must have write access to this folder. Specifying an absolute path is recommended. Login: The user for logging on to the server Password/private key: The password or private key (in case of SFTP with a key pair) Port: The standard ports are provided automatically. Alternative ports can be entered. Use the Connection test button to check whether the server can be accessed with the data provided and whether switching to the specified folder is possible. 3.3 Export data Under Export data, specify what raw data tables will be delivered by the export. Select them with Yes. Tables containing custom parameters each encompass up to 10 parameters and are numbered consecutively. An Export log file can be created in the first row. This is a CSV file that provides information about the tables sent, how many rows are in each table and what the unpacked file size is. The file is transferred last and only if all tables were sent successfully. It can be used as a signal for systems receiving the raw data to start further processing of the data. After all settings have been entered, click the Save button to store the changes. 3.4 Custom sending Custom sending ( Raw data export tab) makes it possible to send raw data with different settings, but without overwriting the stored settings. The FTP access data as well as the export data can be changed temporarily before sending with Send raw data now) according to the changed configuration, with no need to save the configuration. 6 of 35

7 7 of 35 Figure 4 Export data

8 Figure 5 Example of a log file The CSV files are packed (ZIP file) for transferring. Except for the custom sending of data, the file name contains the date of the day covered by the raw data. For custom sending, the file name contains the start and end dates. The file name also contains the corresponding hour in case of hourly exports. For example, the file getcontents_ _20.csv.zip contains the data for all page calls on 1 March 2016 from 20:00:00 to 20:59:59. 4 Reviewing the exports in System Monitor The status of the raw data exports can be viewed for a chosen period in the Q3 System Monitor. In order to do so, go to Configuration > System Configuration > System Monitor. Select Raw data export in the Service menu and enter the desired time period. You get a list with all raw data exports for the selected period. In the list, Object refers to the respective raw data table. There are two process types for objects. The rawdata type applies to the process of creating the respective CSV file, the rawdataftptransfer type applies to transferring the file to the FTP server specified under FTP access data. Start time states when the respective process began, End time when the status shown for the object was reached. Both times refer to the time zone configured in the account under Configuration > System Configuration > Data Collection > Time Zone. The following status values are possible: 8 of 35

9 Status Description Started The process was started. Waiting The process was added to the processing queue and is waiting for processes ahead of it to finish. Repeat The process could not be completed successfully and is being repeated. Finished The process was completed successfully. (An error message may be displayed. This happens when the process failed in an earlier attempt, but the last repetition was successful.) Error The process was not completed successfully even after it was repeated. An error message is displayed. 9 of 35

10 Figure 6 List of raw data exports 5 Content of the raw data tables Descriptions and technical details for the individual tables provided to you by Webtrekk with the raw data export are found below. In the table field overviews, primary key fields are shown in bold (if more than one field is shown in bold, this represents a composite primary key). Fields that serve as a foreign key are underlined (what tables are linked via the foreign keys is covered later in this document). In addition to specifying the basic data type, the Data type column in the table overview also contains the field size limit. for the Numeric data type for example means that numbers with no more than 20 characters and 0 decimal places can occur in the corresponding field. Unused decimal places are always filled with 0 (for instance ).). Fields with the Timestamp data type always have the following format: YYYY-MM-DD HH:MM:SS If required, the permitted values or value ranges are specified in curly brackets after the description. 10 of 35

11 5.1 getfullvisitors data in the context of visits The getfullvisitors table contains all data recorded in the context of visits. These include technical parameters, such as the name of the device that was used, as well geography data such as the country where the visit came from or the time zone in which the visit took place. EID Session ID for the unique identification of a session (visit) Ever-ID used to identify a visitor using a Webtrekk cookie VISIT_TIME Timestamp Time when the visit started NEW_VISITOR (1,0) Is this a new visitor? {1: yes, 0: no, 2: unknown} OS Text (100) The operating system of the visitor s device (for instance Android 4.4, Windows 10, ios 9) BROWSER Text (100) The user s browser (for instance Firefox 42, Internet Explorer 11) JAVASCRIPT (1,0) Does the user have JavaScript activated? {1: yes, 0: no} RESOLUTION Text (255) Screen resolution on the device (for instance 1920x1080) COLORDEPTH Text (255) Screen colour depth on the device (for instance 24-bit) COOKIE (1,0) Indicates whether cookies are activated on the user s device/browser {1: yes, 0: no} INNER_RESOLUTION Text (10) The resolution used internally (by the browser). It may deviate from the screen resolution due to operating system settings or scaling of the browser window. JAVA (1,0) Indicates whether the Java runtime environment and the corresponding browser plugin are installed and active. {1: yes, 0: no} BROWSER_LANG Text (50) The language used in the browser. 11 of 35

12 CITY Text (255) The city where the visitor is located during the visit. REGION Text (255) The region where the visitor is located during the visit. COUNTRY Text (255) The country where the visitor is located during the visit. ISP Text (255) The visitor s Internet service provider. ORGANISATION Text (255) The organisation from which the visit took place. NETSPEED TIMEZONE (1,0) (18,0) The speed category of the visitor s Internet connection. {1: unknown, 2: Modem/ISDN, 3: Cable/DSL, 4: Corporate, 5: Mobile, 6: Satellite, 7: Wireless} The visitor s time zone. MOBILE_CLIENT Text (255) Name of the visitor s device. The device name can only be determined for smart phones and tablets. When a PC or laptop is used, this field contains the string Miscellaneous PC/laptop. (for instance Google Nexus 7) VISIT_TIME_END Timestamp The end time of the visit. GLOBAL_END_DEVICE_VISITOR_ID Text (255) Device-specific, global cookie ID of the device. This column is optional and only exported if the corresponding feature is activated for your Webtrekk account. URM_VISITOR_ID Unique visitor ID. This ID is account-specific and applicable across devices. It is used for usercentric tracking. This column is optional and only exported if the corresponding feature is activated for your Webtrekk account. 12 of 35

13 5.2 getcontents page calls This table contains the data for all page calls. For instance, this table lets you follow the sequence and duration of visits by visitors to your website. REQUEST_ID ID for unique identification of a page call Session ID used to assign the page call to a visit TIMES Timestamp Time of the page call CONTENT Text (255) Name of the page that was called PAGE_DURATION PAGE_ORDER (18,0) (18,0) The dwell time on the page in seconds Used to rank the page call in the sequence of all page calls in reference to the visit ( 11 for instance means the 11th page call within the visit) 13 of 35

14 5.3 getfullreferrer access via referrers This table contains referrer information for access to your website. REQUEST_ID ID for unique identification of referrer access Session ID used for assignment to a visit TIMES Timestamp Time of access REFERRER Text (255) Name of the referrer URL Text (255) Referrer URL SEARCH_ENGINE (1,0) Is the referrer a search engine? {0: no, 1: yes, 2: social media, 3: } 5.4 getfullsearchphrase / getfullsearchphraseintern search phrases These two tables respectively contain external and internal search phrases. REQUEST_ID ID for unique identification of a search phrase entry Session ID used for assignment to a visit TIMES Timestamp Time of the search SEARCH_PHRASE Text (255) The search phrase that was entered (if the search phrase is not transferred for an external search, for instance because the search page uses https, not provided is shown) 14 of 35

15 5.5 getfullclicks clicks for page navigation This table contains all clicks on navigation elements on your website. CLICK_REQUEST_ID REQUEST_ID ID for unique identification of a click Session ID used for assignment to a visit ID of the corresponding page call (0 for the initial click of a session) TIMES Timestamp Time of the click CLICK Text (255) Name of the click element 5.6 getfullcampaigns campaign calls This table contains all clicks on ad media. REQUEST_ID ID for unique identification of a campaign click Session ID used for assignment to a visit CAMPAIGN Text (255) Media code of the ad media TIMES Timestamp Time of the click KEYWORD Text (255) SEM keyword if available 15 of 35

16 5.7 getfullbasket products in the shopping basket This table contains shopping baskets along with their products. The composite primary key is used to identify a product in a specific position in a shopping basket. REQUEST_ID BAS_POS (18,0) ID for unique identification of a shopping basket state Position of the product in the shopping basket Session ID used for assignment to a visit PRODUCT Text (255) Name of the product TIMES Timestamp Time of the click that changed the product status QUANTITY PRICE STATE (18,0) (12,2) (1,0) of product units in the shopping basket Price of the product Product status {0: Added to shopping basket, 1: Bought, 2: Product view} 16 of 35

17 5.8 getfullorders orders This table contains all orders. REQUEST_ID ID for unique identification of an order Session ID used for assignment to a visit ORDER_ID Text (255) ID for the order TIMES Timestamp Time of the order TOTAL_PRICE (12,2) Total order value 17 of 35

18 5.9 getfullforms form data This table contains all form tracking data. FORM_REQUEST_ID REQUEST_ID ID for unique identification of a form request Session ID used for assignment to a visit Used to assign a page call TIMES Timestamp Time of the request FORM Text (255) Name of the form SUBMITTED (1,0) Did the contents of the form get sent? {0: no, 1: yes} 18 of 35

19 5.10 getformfields form field data This table contains detailed information about individual form fields for form tracking. Since a field of the original, composite primary key from the original table is not included in the export, the FORM_REQUEST_ID merely serves as the foreign key for the getfullforms table here. Therefore all fields in the table together form a usable primary key. FORM_REQUEST_ID REQUEST_ID ID for unique identification of a form field Session ID used for assignment to a visit Used to assign a page call TIMES Timestamp Time of the request FORM Text (255) Name of the form FORM_FIELD Form ID FIELD_VALUE Text (255) Value of the form field CANCELLED (1,0) Was the form cancelled? {0: no, 1: yes} 19 of 35

20 5.11 getfullmedia media data This table contains information about media requests. MEDIA_REQUEST REQUEST_ID ID for unique identification of a media request Session ID used for assignment to a visit Used to assign a page call TIMES Timestamp Time of the request MEDIA Text (255) Media designation DURATION MAX_DURATION BANDWITH VOLUME MUTE MEDIA_TIME (18,0) (18,0) (18,0) (3,0) (1,0) (18,0) The actual playback time in seconds, including rewinding and so on. It may therefore be longer than the transferred overall media length. The maximum media playback time in seconds, without backward movements. The media bandwidth (in kbit/s) The volume during media consumption (in %, from 0 to 100) Was the medium muted? {0: no, 1: yes} Media time 20 of 35

21 5.12 getfullmediaaction media action data This table contains information on specific actions carried out on media. Here the MEDIA_ACTION_REQUEST_ID is the ID for the media call as a whole. However, this ID is not unique since the same ID is used for all sub-actions taken on a medium assigned to a media call. MEDIA_ACTION_REQUEST_ID REQUEST_ID ID for identification of a media action Session ID used for assignment to a visit Used to assign a page call TIMES Timestamp Time of the request MEDIA_ACTION Text (255) Designation of the media action A_START A_STOP (5,0) (5,0) The media position where the action was started. The media position where the action was stopped. MEDIA Text (255) Media designation MEDIA_TIME (18,0) Media time 21 of 35

22 5.13 getfullcustomer customer identification This table contains information on the customer ID, internal Ever-ID and the customer s sessions. EID CUSTOMER_ID Session ID used for assignment to a visit The Ever-ID is the internal Webtrekk ID for each visitor. This ID is unique. Customer ID transferred by you to Webtrekk TIMES Timestamp Timestamp for the start of the session. 22 of 35

23 5.14 getfullcustomerjourney customer journey data This table contains click events relevant for the customer journey. They are supplemented by other information relevant for the customer journey, such as the value and time of the respective corresponding conversion. REQUEST_ID CJ_ORDER (18,0) ID for the identification of a customer journey event (such as a click) that led to conversion and therefore ended the customer micro-journey Position of the event in the customer journey Session ID used for assignment to the visit during which the target was reached or the conversion took place. CAMPAIGN Text (255) The ad media belonging to the event (mediacode) KEYWORD Text (255) The advertisement name of the ad media SEARCH_PHRASE Text (100) The campaign search phrase CUST_PARA_STRING_[1..3] Text (255) Values of the custom parameters belonging to the ad media. The number of these columns (from 1 through 3) and their meaning depend on your account configuration. CJ_TIME Timestamp Position of the event in the customer journey TIME Timestamp Time of the conversion in the corresponding customer journey. CONVERSION_VALUE (12,2) Value of the conversion belonging to the customer journey. 23 of 35

24 5.15 getfullcustpara tables custom parameter data The tables listed below contain data for the custom parameters configured in your account. While the tables are very similar to each other, the request types for which they contain parameters are different. Note that the different tables therefore contain columns with the same names, but that these have a different meaning depending on the context of the table. The number of existing tables per parameter type and the number of existing columns per table depend on the number of the custom parameters respectively existing in your Webtrekk account. Each export table for custom parameters contains a maximum of 10 different parameters getfullcustparasession_x (1..25) session parameters This table contains custom parameters at the visit level. The session ID, for which one or more additional customer values are transferred, is used for assignment of the corresponding visit TIMES Timestamp Time within the session when the custom parameter is transferred. CUST_PARA_STRING_Y Text (255) Value of the corresponding custom parameter with string data type (optional, depending on custom parameter configuration) CUST_PARA_NUMBER_Y (12,2) Value of the corresponding custom parameter with numeric data type (optional, depending on custom parameter configuration) CUST_PARA_GENERIC_Y Text (255) Value of the corresponding generic custom parameter, represented as a string (optional, depending on custom parameter configuration) 24 of 35

25 getfullcustparatimes_x (1..25) time-dependent parameters This table contains time-dependent custom parameters. REQUEST_ID Unique ID for entering the timestamp, not used for further assignment. TIMES Timestamp Time of the request CUST_PARA_STRING_Y Text (255) Value of the corresponding custom parameter with string data type (optional, depending on custom parameter configuration) CUST_PARA_NUMBER_Y (12,2) Value of the corresponding custom parameter with numeric data type (optional, depending on custom parameter configuration) CUST_PARA_GENERIC_Y Text (255) Value of the corresponding generic custom parameter, represented as a string (optional, depending on custom parameter configuration) 25 of 35

26 getfullcustparapage_x (1..25) page parameters This table contains custom parameters at the page level. REQUEST_ID Used for the unique assignment of a dataset of page parameters to the page request that measured the parameter values. Session ID used for assignment to a visit TIMES Timestamp Time of the request CUST_PARA_STRING_Y Text (255) Value of the corresponding custom parameter with string data type (optional, depending on custom parameter configuration) CUST_PARA_NUMBER_Y (12,2) Value of the corresponding custom parameter with numeric data type (optional, depending on custom parameter configuration) CUST_PARA_GENERIC_Y Text (255) Value of the corresponding generic custom parameter, represented as a string (optional, depending on custom parameter configuration) 26 of 35

27 getfullcustparaproduct_x (1..25) product parameters This table contains custom parameters at the product level. REQUEST_ID BAS_POS (18,0) Used for the unique assignment of a dataset of product parameters to the product call contained in getfullbasket that measured this dataset. The combination of REQUEST_ID and BAS_POS can be linked to the same columns in getfullbasket. Position in the shopping basket Session ID of the product call, used for assignment to the corresponding visit TIMES Timestamp Time of the request CUST_PARA_STRING_Y Text (255) Value of the corresponding custom parameter with string data type (optional, depending on custom parameter configuration) CUST_PARA_NUMBER_Y (12,2) Value of the corresponding custom parameter with numeric data type (optional, depending on custom parameter configuration) CUST_PARA_GENERIC_Y Text (255) Value of the corresponding generic custom parameter, represented as a string (optional, depending on custom parameter configuration) 27 of 35

28 getfullcustparaclick_x (1..25) click parameters This table contains custom parameters at the click level. REQUEST_ID Used for the unique assignment of a dataset of click parameters to the click request that measured the parameter values. Session ID of the click request, used for assignment to the corresponding visit TIMES Timestamp Time of the request CUST_PARA_STRING_Y Text (255) Value of the corresponding custom parameter with string data type (optional, depending on custom parameter configuration) CUST_PARA_NUMBER_Y (12,2) Value of the corresponding custom parameter with numeric data type (optional, depending on custom parameter configuration) CUST_PARA_GENERIC_Y Text (255) Value of the corresponding generic custom parameter, represented as a string (optional, depending on custom parameter configuration) 28 of 35

29 getfullcustparacampaign_x (1..25) campaign parameters This table contains custom parameters for ad media. REQUEST_ID Used for the unique assignment of a dataset of campaign parameters to the campaign click that measured the parameter values. Session ID of the campaign call, used for assignment to the corresponding visit TIMES Timestamp Time of the request CUST_PARA_STRING_Y Text (255) Value of the corresponding custom parameter with string data type (optional, depending on custom parameter configuration) CUST_PARA_NUMBER_Y (12,2) Value of the corresponding custom parameter with numeric data type (optional, depending on custom parameter configuration) CUST_PARA_GENERIC_Y Text (255) Value of the corresponding generic custom parameter, represented as a string (optional, depending on custom parameter configuration) 29 of 35

30 6 Linking the raw data tables This section explains the relationship between the tables of the Webtrekk raw data export. For a reference of the fields in each table, see the previous section Assignment of the export tables to context levels In principle, Webtrekk data have three levels, each of which represents a different context: visits, page calls and clicks (or events). In general there are 1 : n relationships between the levels. Thus there is at least 1 page call within a visit, and at least 1 click is assigned to a page call. However, the initial click that subsequently leads to calling a page is not always assigned to a page call (when the visit was interrupted for a certain time and the visitor clicks a link, a new visit is started and the first click is not assigned to a page call since the page call occurred in a previous visit). Figure 7 Context levels The remaining export tables can also be assigned to the levels shown in the diagram. For example, the getfullcustomer represents a vertical partition of the getfullvisitors table (at the table column level). The columns contained in the getfullcustomer table but not in the getfullvisitors table could in theory also be integrated in the getfullvisitors table (same primary key). But since a customer ID is not measured/transferred for each visit, the field of the corresponding dataset would remain empty. Therefore this information is supplied in a separate table. In principle there is a 1:1 relationship between the getfullvisitors table and the getfullcustomers table, but the occurrence of a corresponding dataset in the getfullcustomer table is optional. A maximum of one customer ID can but does not have to be assigned to each visit. But if there is a dataset in the getfullcustomer table, there is always exactly one corresponding dataset in the getfullvisitors table. The same relationships exist respectively between the getfullvisitors table and the getfullcustparasession_x tables. The relationships are shown in Figure 8 in (min, max) notation: every visit has at least 0 and no more than one reference in the getfullcustomer table. This applies correspondingly for all getfullcustparasession tables. 30 of 35

31 Figure 8 Raw data level 1 These tables are linked via the Session ID (). In principle the relationships also exist on level 2 between the getcontents table and the other tables on this level, as illustrated by the following graphic: 31 of 35

32 Figure 9 Raw data level 2 All of the tables shown here can be linked to the getcontents table via their REQUEST_ID field. For example, it is possible to determine which page call triggered an order (normally the order is linked to a special page at the end of the ordering process), whether a referrer was recorded, or what page parameters were included with the page call. Corresponding relationships also apply on level 3, as shown in Figure 10. Some tables on level 3 can be linked to the getfullclicks table with the REQUEST_ID. However, this does not apply to the getfullforms, getfullformfields, getfullmedia and getfullmediaaction tables. These tables can be linked to the getfullclicks table (per CLICK_REQUEST_ID) via the FORM_REQUEST_ID, MEDIA_REQUEST or MEDIA_ACTION_REQUEST_ID fields (depending on the table, these are their primary keys). 32 of 35

33 33 of 35 Figure 10 Raw data level 3

34 6.2 Linking between the levels The basic relations between the tables on one level were presented in the preceding section. Now you will learn how the tables can be linked to each other in a meaningful way. Every raw data export table (aside from the getfullcustparatimes tables for the time categories) contains the (session ID) column. You can link the datasets to the corresponding visit via this column. For example, you can determine what pages a visitor called up during a certain visit by linking the getcontents table to the getfullvisitors table via the using a JOIN operation. To obtain information about the identity of the visitor who made the visit, you can add information from the getfullcustomer table. The getfullclicks, getfullforms, getfullmedia, getfullformfields and getfullmediaaction tables of the event context (raw data level 3) each contain the foreign key field REQUEST_ID. This field allows you to link the corresponding tables or datasets directly to the respective page calls. The REQUEST_ID primary key of the getcontents table is used here. This for example makes it possible to determine which forms or media are positioned on what page (implicitly via the information of what page call a corresponding form or media event is linked to). Naturally you can also determine the clicks belonging to a page call. If you require further information or support for certain raw data use cases, your consultant will be happy to help you! 34 of 35

35 7 Contact us Please do not hesitate to contact us if you have any questions: Webtrekk GmbH Robert-Koch-Platz Berlin/Germany Telephone Fax support-de@webtrekk.de 35 of 35