Schema for Gmail logs in BigQuery

Transcription

1 Schema for Gmail logs in BigQuery

2 Contents Schema for Gmail logs in BigQuery Field descriptions Sender Service and Selector Destination Service and Selector Action types Message set types Error codes SMTP Relay errors Upload error category Consequence action File category S/MIME content type String match type String Match Source Authentication type (message_info.connection_info.authenticated_domain.type) SMTP response reason (message_info.connection.smtp_response_reason) Gmail Spam classification disposition (message_info.spam_info.classification_disposition) Spam classification reason (message_info.spam_info.classification_reason) Custom rule spam classification disposition (message_info.triggered_rule_info.spam_label_modifier) Custom rule type (message_info.triggered_rule_info.rule_type) Gmail logs in BigQuery 2

3 Schema for Gmail logs in BigQuery Field descriptions This table defines the field name, type, and description for each field available in the logs. Field name Type event_info RECORD General information about the event. event_info.timestamp_usec Time when this event started, in the format of a UNIX timestamp, in microseconds. event_info.elapsed_time_usec Time period this event took, in microseconds. event_info.success Success is true, unless the event failed; for example, message was rejected by a policy. message_info RECORD General information about the message. message_info.action_type message_info.rfc2822_message_id Which action took place for this event. See Action type for a breakdown of what each number means. RFC 2822 message ID for the message, the same as found in "Show Original" of the message. message_info.subject Subject line of the message. message_info.payload_size Size of the payload; for example, the message, in bytes. message_info.link_domain List of advertised domains (domain names extracted from all urls present in message content). message_info.attachment RECORD List of attachments contained in the message message_info.attachment.sha256 SHA256 hash of the message message_info.attachment.file_extension_ty pe File extension type (not mime part type) message_info.attachment.malware_type Malware category, if detected. message_info.spam_info.classification_tim estamp_usec Message spam classification timestamp. message_info.source.from_header_addres s From header address. Eg: foo@domain.com message_info.source.from_header_display name From header display name: Eg: User Foo message_info.destination RECORD Information about one recipient. There could be more than one destination for this field. message_info.destination.address address of the recipient. message_info.destination.service message_info.destination.smime_signature _verification_success The service to where this message went. See Destination Service and Selector for description. When set, indicates that smime signature verification was attempted (not set if skipped), and the value indicates the completion status. Only meaningful for inbound messages. Gmail logs in BigQuery 3

4 message_info.destination.smime_decryptio n_success When set, indicates that smime decryption was attempted (not set if skipped), and the value indicates the completion status. Only meaningful for inbound messages. message_info.destination.smime_parsing_ success message_info.destination.smime_extractio n_success message_info.destination.selector message_info.flattened_destinations message_info.description message_info.connection_info RECORD When set, indicates that smime parsing was attempted (not set if skipped), and the value indicates the completion status. Only meaningful for inbound messages. When set, indicates that smime extraction was attempted (not set if skipped), and the value indicates the completion status. Only meaningful for inbound messages. Subcategory for each service. See Destination Service and Selector for description. String that has information of all recipients flattened, in the format of service_for_recipient1:selector_for_recipient1:address_f or_recipient1,service_for_recipient2:selector_for_recipi ent2:address_for_recipient2 Human-friendly description about what happened to the message. Information about the connection on which the message was transferred. message_info.connection_info.client_ip IP address of the mail client that started the message. message_info.connection_info.smtp_in_co nnect_ip Remote IP address for MTA client connections (inbound mail to smtp-in). message_info.connection_info.smtp_out_c onnect_ip Remote IP address for smtp-out connections. message_info.connection_info.failed_smtp _out_connect_ip message_info.connection_info.smtp_tls_st ate message_info.connection_info.smtp_reply_ code message_info.connection_info.tls_required _but_unavailable message_info.connection_info.smtp_out_r emote_host List of all IPs in the remote MX record that an smtp-out server attempted to connect to and failed. Type of connection made to the SMTP server, only set for logs of events handled by SMTP servers. 0 for no TLS, 1 for TLS. SMTP reply code returned to smtp-in or smtp-out server, for smtp-in and smtp-out connections. Generally, 2xx, 4xx, or 5xx. TLS required for an smtp-out connection, but no valid certificate was present. For smtp-out connections, the domain we started from the destination domain or the smarthost. message_info.connection_info.smtp_user_ agent_ip IP address of the mail user agent, in smtp-in time. message_info.connection_info.is_intra_do main Whether the message is a dasher intra domain message (sent within the customer domain? message_info.connection_info.dmarc_pass Whether the message pass DMARC policy evaluation? message_info.connection_info.dmarc_publi shed_domain Domain name used to evaluate the dmarc policy message_info.connection_info.client_host_ zone Client host name of the mail sender. message_info.connection_info.authenticate d_domain RECORD List of authenticated domain names and authentication mechanims. Gmail logs in BigQuery 4

5 message_info.connection_info.authenticate d_domain.name The authenticated domain name message_info.connection_info.authenticate d_domain.type Message authentication type. Eg: SPF, DKIM etc message_info.connection_info.is_internal message_info.connection_info.dkim_pass True, if the message is sent within domains owned by the customer. Whether the message authenticated using at least one DKIM signature. message_info.connection_info.spf_pass Whether the message authenticated using SPF mechanism message_info.connection_info.smtp_respo nse_reason Detailed reason for the smtp response type. message_info.connection_info.ip_geo_city Nearest city computed based on the relay ip. message_info.connection_info.ip_geo_cou ntry Country code based on the relay ip. message_info.structured_policy_log_info.e xchange_journal_info.rfc822_message_id message_info.structured_policy_log_info.d etected_file_types message_info.smime_sign_message message_info.smime_encrypt_message message_info.smime_packaging_success message_info.smime_extraction_success message_info.smime_content_type RECORD RFC 822 message ID of the journaled message. Information about the files detected in the message. When set and true, indicates message should be signed. Only meaningful for outbound messages. When set and true, indicates message should be encrypted. Only meaningful for outbound messages. When set, indicates that smime packaging was attempted (not set if skipped), and the value indicates the completion status. Only meaningful for outbound messages. When set, indicates that inbound S/MIME processing occurred (not set if skipped), and the value indicates the completion status. Only meaningful for outbound messages. The top-level S/MIME type of a message, as indicated by the Content-Type header. Only meaningful for outbound messages. See S/MIME content type for a breakdown of what each number means. message_info.triggered_rule_info RECORD Information about rules triggered for the message in different events. message_info.triggered_rule_info.policy_h older_address message_info.triggered_rule_info.consequ ence message_info.triggered_rule_info.consequ ence.action message_info.triggered_rule_info.consequ ence.reason message_info.triggered_rule_info.consequ ence.subconsequence message_info.triggered_rule_info.consequ ence.subconsequence.action RECORD RECORD address of the policyholder whose policy triggered the rules. Information about a consequence applied to the message due to this triggered rule. Action taken for the consequence. See Consequence action for a breakdown of what each number means. Reason why this consequence was applied. Usually contains the unique description of a rule that triggered the consequence. Information about one subconsequence of the primary consequence. Action taken for the subconsequence. See Consequence action for a breakdown of what each number means. Gmail logs in BigQuery 5

6 message_info.triggered_rule_info.consequ ence.subconsequence.reason message_info.triggered_rule_info.string_m atch message_info.triggered_rule_info.string_m atch.type message_info.triggered_rule_info.string_m atch.predefined_detector_name message_info.triggered_rule_info.string_m atch.source message_info.triggered_rule_info.string_m atch.attachment_name message_info.triggered_rule_info.string_m atch.matched_string message_info.triggered_rule_info.string_m atch.match_expression RECORD Reason why this subconsequence was applied. Usually contains the unique description of a rule that triggered the consequence. If the rule was triggered because of string match; for example,. content compliance rule, which contains the information about the string matches. Type of the match. See String Match type for a breakdown of what each number means. If this was a match of predefined detectors, shows the name of the predefined detector. Location of the string matched in the message. See String Match Source for a breakdown of what each number means. Name of the attachment where a matching string was found if in the text extracted from a binary file. String that caused the rule to fire. Sensitive information is hidden by "*" or "." Match expression that customer set in the Admin console. message_info.flattened_triggered_rule_info String that has information of all triggered rules info flattened, in JSON format. message_info.triggered_rule_info.spam_la bel_modifier Describes the custom rule spam classification verdict. message_info.triggered_rule_info.rule_type Custom rule type. Eg. Inbound gateway, Compliance rule etc message_info.triggered_rule_info.rule_nam e Custom rule description given by admin in control plane. message_owner.customer_domain Customer domain of the message recipient. Sender Service and Selector There are many Service and Selector pairs for a sender. You can use these 2 fields to determine which service the message was from and why the message was generated. Service Selector calendar send Notifications from Google Calendar gmail-ui read-receipt Gmail s read-receipt feature. gmail-ui autoforward Gmail's autoforward feature. gmail-ui unsubscribe Gmail's unsubscribe feature. gmail-ui canned-response Message sent by Gmail's Canned Response feature. gmail-ui vacation-response Gmail's vacation response feature. gmail-ui send Message sent from Gmail web UI. docs share Sharing notification from Google Drive. groups groups-ui Message sent from Google Groups. keep invites Invitation sent by Google Keep. mailing-list-server custom-replies Auto-replies from Google Groups. mailing-list-server null Sent from Google Groups. mailing-list-server moderation Sent from Google Group's moderation. Gmail logs in BigQuery 6

7 mailing-list-server to-archive Sent from Google Group's archive. google-apps-script user Sent from Google Apps Script. mail-fetcher null Message pulled by Gmail's Mail Fetcher gmail-for-work spam-redelivery User requests a (possibly a false positive) spam message to be redelivered to their non-gmail mailbox; or, this is a quarantine summary (spam folder summary) sent to the non-gmail mailbox. gmail-for-work qsum-delivery Periodic report is sent to the user detailing the contents of the Spam label and (optionally) the Inbox label. gmail-for-work quarantine-delivery Message released from the Quarantine Manager. gmail-for-work quarantine-notification Non-delivery response sent to the original sender of a denied quarantined message. gmail-for-work policy Message triggered some setting configured by the domain administrator. gmail-for-work comprehensiv -storag e Sent to Gmail's servers due to a Comprehensive Mail Storage setting. smtp-inbound null Message inserted from Google's SMTP servers to Gmail's delivery pipeline. smtp-msa null Message inserted from Google's SMTP servers, in authenticated mode, to the Gmail's delivery pipeline. smtp-relay gmail-for-work Messages routed through the SMTP Relay setting. google-spreadsheets google-forms-receipt Notifications from Google Sheets. google-spreadsheets google-forms-invite Sharing invites from Google Sheets. unified-notifications google-apps Notification from G Suite unified-notifications null Notification from a Google system. Destination Service and Selector There are many Service and Selector pairs for destination. You can use these 2 fields to determine to which service the message was sent. Service Selector gmail-ui sent-on-behalf-of-user Message was sent to Gmail UI and is kept as a copy in the user's Gmail Sent label. gmail-ui null Message was sent to Gmail UI. mailing-list-server spam-check Message was sent to Google Groups and was spam-checked. mailing-list-server null Message was sent to Google Groups. mailing-list-server moderation Message was sent to Google Groups and is pending administrator moderation. mailing-list-server archive Message was sent to Google Groups and is archived. gmail-for-work-catchall smtp-outbound gmail-delivery-server Message had unrecognized recipients and was delivered according to a catchall rule. Message was sent to outbound SMTP server and handled by Gmail delivery servers. Gmail logs in BigQuery 7

8 smtp-outbound smtp-outbound smtp-outbound smtp-outbound google-apps-for-work google-apps-for-work-starter gmail-notification relay Message was sent to outbound SMTP server and handled by G Suite basic. Message was sent to outbound SMTP server and handled by G Suite basic. Message was sent to outbound SMTP server and handled by Gmail notification. Message was sent to outbound SMTP server and handled by Gmail relay servers. smtp-outbound gmail Message was sent to outbound SMTP server smtp-outbound gmail-for-work Message was sent to outbound SMTP server and added by Gmail for business policies. smtp-outbound null Message was sent to outbound SMTP server. Action types You can use this field to determine which stage in the delivery pipeline the message went through. of relevant action types is detailed below: 1 Message received and handled by inbound SMTP server. 2 Message inserted to Gmail delivery server and prepared for delivery. This is usually the next step following 1, or the first step if you send from the Gmail UI. For incoming messages, policies with reject dispositions are usually evaluated here; for example, an attachment compliance policy that rejects incoming messages with archives. 3 After 2 (message inserted into Gmail delivery server), this means the message was handled by the Gmail delivery server and delivered to another server; for example, the outbound SMTP server. Policies with dispositions other than reject are evaluated here; for example, the attachment compliance policy that strips an attachment if it s an archive file. 10 Message handled and sent out by outbound SMTP server. 14 This means a temporary error occured when Gmail delivery server tried to deliver the message, and the message has been scheduled for retry. This is usually caused by external or internal servers that Gmail delivery server talks to being temporarily unavailable; retry later. For example, we tried to deliver the message to an external SMTP server, but got a temporary error (4xx) back. 18 Message bounced by Gmail delivery server. When we can t deliver the message, we bounce the message. Sometimes you can find out what happened by reading message_info.description. Some common reasons are: Gmail logs in BigQuery 8

9 The recipient server didn t accept our requests. We ve retried many times due to temporary errors, but still failed to deliver the message. We can t evaluate policies in insert time (message_info.action_type == 2), deferred the evaluation until delivery time, and found out the message triggered a reject disposition. The recipient is unrecognized and there s no policy triggered to change the primary delivery route. 19 Message dropped by the Gmail delivery server. This can occur: If a message being sent triggers admin quarantine consequences, the original message is dropped and a copy of the message is added in the admin Quarantine Manager. For exchange journaling message, we deliver the wrapped inner message, and dropped the original message. For inbound messages, Gmail can block and drop messages; for example, if: Not RFC 5322-compliant Sender violates bulk senders guidelines For this type of drop, you should be able to see the detail reason through message_info.description for that message where message_info.action_type = 1. There s policy triggered to remove the primary delivery route and add other routes. We dropped the original message and delivered copies to other routes. The recipient is an unrecognized address and there s policy triggered to add delivery routes for the message. We dropped the original message and deliver copies to other routes. 55 Message was re-inserted into Gmail delivery server, caused by policies that modify the primary delivery route or envelope recipient. 48 Message received and handled by inbound SMTP server in relay mode. 49 Message sent via outbound SMTP server in relay mode. 69 This action indicates manual spam correction (eg: report as phishing, report as spam/not-spam etc.) Gmail logs in BigQuery 9

10 Message set types You can use this field to see attributes of the message; for example, being sent or received. 9 Message being received; for example, an incoming message. 8 Message being sent; for example, an outgoing message. 11 Message had sender or recipients outside your domain. Or, for message being received, if message set 27 is missing, this means we couldn't authenticate the sender; will treat the message as having a sender outside your domain. 10 Message internal to your domain. 12 Message had some recipients internal to your domain and some recipients outside your domain. This message set may appear only when: There are multiple recipients. A message is being sent (for message being received, we enforce that the recipients all belong to the same domain). Action type for the message is 2 (we split multirecipient messages to single-recipient messages after that). Messages with this message set would have policies affecting both internal and external triggered. 1 Incoming message from outside your domain. This message set doesn t appear at the same time as message set Outgoing message sent to other domains. This message set doesn t appear at the same time as message set Sender has successfully passed SPF/DKIM/DMARC authentication. If the sender isn t authenticated, then we don't trust the sender domain; the message won t be considered internal. 7 Message was classified as spam by Gmail spam filter. 47 Message was detected to be spam by tag-and-deliver information in your inbound gateway settings. Gmail logs in BigQuery 10

11 Error codes The following are message codes and their respective descriptions. SMTP Relay errors Message 0 There was no error. 1 Authentication error. 2 Daily rate limit was exceeded. 3 Peak rate limit was exceeded. 4 SMTP relay was abused. 5 Per-user rate limit was exceeded. Upload error category Message 0 Uncategorized transient error. 1 Recipient account is too busy. 2 DNS error resolving recipient domain. 3 Recipient's server refused connection. 4 Recipient is out of storage quota. Consequence action Message 0 This consequence is a no-op. 6 Added a message header. 5 Add an additional delivery target. 9 Add message to specified message set. 3 Put message in the admin quarantine. 12 Add a footer to the message. 18 Archive to Google Vault for recipients. 10 Modify the message's labels 17 Can't be delivered. Bounce. 11 Prefix text to existing subject header. 7 Overwrite the envelope recipient. 4 Alter the primary delivery target. 16 Require secure delivery. 19 Skip spam checks when insert into Gmail delivery server. 15 Replace attachments with canned text. Gmail logs in BigQuery 11

12 13 Strip message body 14 Store a copy of the message in the user s mailbox, per Comprehensive mail storage setting File category 0 There s no mapping for this ID, which can happen if the categorization process is updated with new file types. 1 Unrecognized file type. 2 Microsoft Office documents like word processing, spreadsheet, presentation, database, PDF, and so on The file may or may not be password-protected. 3 Video and multimedia, like MPEG, Quicktime, WMV, and so on. 4 Music and sound like MP3, AAC, WAV, and so on. 5 Images JPEG, BMP, GIF, and so on. 6 Archives like ZIP, TAR, TGZ, and so on. 7 Executables like EXE, COM, JS, and so on. 8 Password-protected Office documents. 9 Not password-protected Office documents. S/MIME content type 0 The message does not have a recognized S/MIME Content-Type An S/MIME message with a detached signature, indicated by content type "multipart/signed" with parameter "protocol=application/pkcs7-signature". An S/MIME message with an opaque signature, indicated by content type /"application/pkcs7-mime" or "application/x-pkcs7-mime" with parameter "smime-type=signed-data". An S/MIME message that is encrypted, indicated by content type "application/pkcs7-mime" or "application/x-pkcs7-mime" with parameter "smime-type=enveloped-data". An S/MIME message that is compressed, indicated by content type "application/pkcs7-mime" or "application/x-pkcs7-mime" with parameter "smime-type=compressed-data". String match type 0 Undefined 1 Regular expression match 2 Predefined detector match 3 Simple content match 4 Non-ASCII match Gmail logs in BigQuery 12

13 String Match Source 0 Unknown 1 Message body, including text format attachments 2 Binary format attachments 3 Headers 4 Subject 5 Sender header 6 Recipient header Authentication type (message_info.connection_info.authenticated_domain.type) 1 SPF 2 DKIM 3 DKIM_PROXY 4 XOAR_SPF 5 XOAR_DKIM SMTP response reason (message_info.connection.smtp_response_reason) 0 UNKNOWN Response 1 Default response 2 Blatant Spam or Phishing 3 Attachment contains malware 4 Message rejected due to domain s DMARC policy 5 Unsupported (by GMail policy) attachment type Gmail logs in BigQuery 13

14 6 Receive limits exceeded 7 Account over quota 8 Bad or missing PTR record 9 Recipient doesn t exist 10 Message rejected to customer policy Gmail Spam classification disposition (message_info.spam_info.classification_disposition) 0 Classification disposition is unknown 1 Message classified as CLEAN 2 Message classified as SPAM. Message is delivered to Spam folder. 3 Message classified as PHISHING. Message is delivered to Spam folder 4 Message classified as SUSPICIOUS. Message is delivered to Inbox with a warning banner. 5 Message classified as MALWARE Spam classification reason (message_info.spam_info.classification_reason) 1 Default 2 Sender in Address Book 3?? 4?? 5?? Gmail logs in BigQuery 14

15 Custom rule spam classification disposition (message_info.triggered_rule_info.spam_label_modifier) 0 NO Action. Custom rules honor GMail spam classification verdict 1 SPAM. Custom rules classified the message as Spam 2 NOT_SPAM. Custom rules classified the message as not spam. Custom rule type (message_info.triggered_rule_info.rule_type) 0 Walled Garden 5 General Rule 7 Objectionable Content 8 Content Compliance 9 Unknown rule type 14 Blocked Senders 16 Attachment Compliance 17 TLS Compliance 18 Domain Default 21 Quarantine Summary 25 Routing Rule 26 Inbound Gateway Gmail logs in BigQuery 15