Table of Contents. Electronic mail. History of (2) History of (1) history. Basic concepts. Aka (or according to Knuth)

Transcription

1 Table of Contents Electronic mail Aka (or according to Knuth) Karst Koymans Informatics Institute University of Amsterdam (version 17.7, 2017/10/03 11:29:40 UTC) Friday, September 29, history Basic concepts Message agents in detail Message transfer Message format security and validity Message storage formats Validating at OS3 History of (1) History of (2) 1971 Tomlinson s first ( ?) Introduces the use of First based on the CPYNET and SNDMSG program Later piggybacked on FTP over ARPANET 1979 UUCP-based introduces the bang (!) not based on TCP/IP (or NCP) 1982 SMTP (Simple Mail Transfer Protocol) specified 1983 sendmail released (4.1c BSD) (late) 1983 DNS specified 1984 DNS toplevel domains specified sendmail knows (ARPANET)! (UUCP) : (BerkNet)

2 Survey October 2001 (Dan Bernstein) Survey May 2003 (Thomas Pircher) 401 UNIX (Sendmail) 176 Windows (Exchange/IIS) 167 UNIX (qmail) 57 Windows (Ipswitch IMail) 23 UNIX (smap) 15 UNIX (IBM Postfix, formerly VMailer) 14 UNIX (Exim) (35.59%) Sendmail 4537 (8.42%) qmail 4104 (7.62%) Postfix 2812 (5.22%) Microsoft (Exchange/IIS) 2464 (4.57%) Exim Source: Thomas Pircher Source: Survey October 2004 (SNE) Survey 2006/2007 (MailChannels) (35.3%) Sendmail (21.0%) Microsoft (Exchange/IIS) 6836 (11.8%) Exim 4008 (6.9%) imail 3669 (6.3%) qmail 3172 (5.5%) Postfix Source: lost Source: O Reilly SysAdmin

3 Survey (securityspace) Survey graph (securityspace) Description Number of Servers Percent Total Number of MX Servers Queried 2,440, Number of Servers that didn t respond 414, Server didn t open socket 402, Server didn t provide banner 11, Server provided banner 2,025, Server banner identifies software in use 1,108, Source: Source: Surveys... warnings concepts Important questions Who did the survey? Were there personal interests? What assumptions were made? What model was chosen? What methods were employed? Is this reproducible? Without answers to these questions the result is worthless, or at least questionable Message transfer (RFC 5321) Message format (RFC 5322) Message agents (RFC 5598) Message stores (RFC 5598)

4 Message Agents Message User Agent Agent acronym MUA MTA MDA MSA MRA 2 MAA 2 Agent use Message 1 User Agent Message Transfer Agent Message Delivery Agent Message Submission Agent Message Retrieval Agent Message Access Agent Interface for the user Reads and composes messages Thunderbird, Outlook, mutt, pine, mh Webmail (browser frontend; backend is more like an MRA 3 ) Often uses SMTP to send mail ( MSA) Often uses IMAP/POP3 to get mail ( MAA) May have direct access to message store Direct access is less and less common 1 According to RFC 5598, called Mail User Agent in RFC Not standardised in RFC5598: Internet Mail Architecture 3 Open for discussion Message Transfer Agent Message Delivery Agent Transfers across the Internet Uses SMTP as transfer protocol sendmail, Postfix, qmail, Exim, Often also operates as a Message Submission Agent Makes use of MX records to route Delivers into the message store (MS) mail, local (Postfix), rmail, procmail, sieve May do filtering, SPAM and virus checking, Has knowledge about mailbox formats Can use Global file space (for example /var/mail/mbox) User specific file space (for example $HOME/mbox) Database (often not directly accessible)

5 Message Submission Agent (1) Message Submission Agent (2) See RFC 6409 Injects message into the mail system Sanitizes message content, for instance Message-ID: header must be generated if absent Envelope domains must be FQDNs Date: header must be added or corrected Often combined with MTA Should bind to its own port (587), if possible Often acts as an MTA-frontend Is sometimes integrated into the MTA Can also operate locally sendmail (no daemon mode) postdrop without SMTP or with piped SMTP Message Retrieval Agent Message Access Agent Program that uses a MAA 4 to collect mail Webmail could be seen as a combination of an MUA-frontend and an MRA-backend Possibly reinjects mail into the mail system Typical example is fetchmail which can use an MTA SMTP TURN (Insecure!) SMTP ETRN (More secure variant 5, RFC 1985) SMTP ATRN 6 (Authenticated variant, RFC 2645) Often not considered an agent in the mail handling system probably because it doesn t use SMTP but it is needed to complete the whole picture Knows about and interacts with the Message Store (MS) Offers services to access mail to MUA (or MRA) POP3 (Post Office Protocol) IMAP (Internet Message Access Protocol, version 4) 4 Or sometimes even an MTA 5 Starts a new SMTP connection instead of reusing the old one 6 Providing ODMR (On-Demand Mail Relay) for clients with dynamic IP addresses

6 Message Agent Relationships Message transfer MUA MAA MRA MS MSA Q MTA MDA protocol local process variable SMTP (RFC 5321) Uses the Network Virtual Terminal (NVT) presentation layer from the TELNET RFC (854) Net-ASCII might be replaced in the future by Net-Unicode, see RFC 5198 Mail objects content (in message format ) envelope (SMTP parameters) Normal (E)SMTP session EHLO options EHLO (newer form of HELO ) (greeting, option negotiation) MAIL FROM: (envelope sender) RCPT TO: (envelope recipient) DATA (content, ended by <CRLF>.<CRLF>) QUIT (goodbye) Response 7 from the OS3 mail server smtp.os3.nl to EHLO 250-PIPELINING 250-SIZE ETRN 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN 7 Checked

7 Message format Augmented Backus Naur Form (ABNF) Formally specified by ABNF syntax in RFC 5322 Headers, empty line, body Only 7-bit US-ASCII 8 (1-127) allowed MIME extends this to possibly 8-bit Lines are delimited by <CR><LF> Lines should be no longer than 78 characters Derived from BNF concatenation, choice, literal extended with constructs from EBNF optionals, repetition, grouping, comments Any context free language can be specified by ABNF which is a larger class than the regular languages which are defined by basic 9 regular expressions 8 Or simply ASCII; IANA prefers US-ASCII as character set name in MIME 9 Extended regular expressions are much stronger! Line delimiters (1) Line delimiters (2) Newline, line break, end-of-line (EOL) CR (U+000D) Mac OS before Mac OS X LF (U+000A) Unix CRLF (U+000D, followed by U+000A) Windows Network Virtual Terminal NEL (U+0085) IBM s Next Line More options for newline, line break, end-of-line (EOL) LS (U+2028) Unicode Line Separator PS (U+2029) Unicode Paragraph Separator RS (U+001E) Record Separator (obsolete) LFCR (U+000A, followed by U+000D) Sic!

8 Message (specified in the ABNF formalism) Header format message = (fields / obs-fields) [CRLF body] body = *(*998text CRLF) *998text CRLF = %d13.10 In mathematical terminology there is an isomorphism text + = (text {CRLF}) This is an example of deflation Note the + instead of a CRLF is delimiter or separator, not terminator or initiator <Field name>:<field body> <Field name> printable US-ASCII (33-126) except : (58) <Field body> US-ASCII (1-127) except CR(13) and LF(10) but also (un)folding is allowed Some important headers Internationalized From:<originator mailbox> Sender:<sender mailbox> To:<recipient mailbox> Message-ID:<unique message identification> Received:<recording of message transfer> MIME is only used for the body Headers are still 7-bit US-ASCII RFC 6532 extends the header field format Introduces support for UTF-8 encoded Unicode Permits the direct use of UTF-8 in header field data/values, including mail addresses Also needs SMTP extension from RFC 6531 Introduces SMTPUTF8 EHLO extension

9 Securing (1) Securing (2) Use secure protocols imap (port 143) imaps (port 993) pop3 (port 110) pop3s (port 995) smtp (port 25) smtps, ssmtp (port 465)? 10 SMTP improvements Authenticated SMTP AUTH extension (RFC 4954) Based on SASL 11 (RFC 4422) STARTTLS extension (RFC 3207) Replaces (s)smtp(s) These mechanisms are often used on mail submission via port Not registered as such with IANA 11 Simple Authentication and Security Layer Securing (3) Message store Use MUA-based encryption and authentication PGP (Pretty Good Privacy) Inline or PGP/MIME GPG (GNU Privacy Guard) Uses a web of trust S/MIME Uses a hierarchy of trust (PKI) MIME Multipurpose Internet Mail Extensions Different character sets, binary attachments, multiple parts, internationalised headers In database Only accessible via IMAP, POP3 In flat files Also accessible via direct access Enables grepping the message store

10 Mbox format MMDF format Ordinary file with multiple messages separated by From at start of line has extra blank line at end of message quotes From to >From and >From to >>From a first characteristic line From <envelopesender> <date> <optionalinfo> Variant of mbox format UsesˆA ˆAˆA as separator Optionally has the mbox From information MH format Maildir format Mailbox is a directory Every message is a file with a numeric name Used by mh, nmh, xnmh MUA s Mailbox is again a directory Subdirectories tmp, new, cur Arriving mail: tmp/<time>.<pid>.<host> No mailbox locking needed Works reliably over NFS

11 Mailbox locking Validating (1) Uses flock, lockf, fcntl system calls Does not always work reliably over NFS Needed if delivery agents and/or access agents operate on the same file (mailbox) SPF: Sender Policy Framework RFC 7208 Uses DNS TXT records at the zone apex Specifies whitelist for clients and envelope sender addresses Can check both MAIL FROM: and HELO/EHLO Rather crude and simple system Can break mail forwarding SPF examples Validating (2) apple.com IN TXT "v=spf1 ip4: /8 -all" google.com IN TXT "v=spf1 include:_spf.google.com ~all" uva.nl IN TXT "v=spf1 ip4: /27 ip4: /27 include:_spfout.mf.surf.net include:_spf.aanmelder.nl include:spf.mailtopay.nl?all" DKIM: DomainKeys Identified Mail RFC 6376 Uses DNS TXT records at the domain name <selector>._domainkey.<apex> where <selector> is the s attrribute in the DKIM-Signature header Signs (part of) the header and body of an , which is being checked by the public key of the sending domain Does not depend on the SMTP envelope

12 DKIM examples Validating (3) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s= ; h=from:mime-version:subject:message-id:date:to; _domainkey.gmail.com. 300 IN TXT "k=rsa; p=miibijanbgkqhkig9w0baqefaaocaq8 AMIIBCgKCAQEAviPGBk4ZB64UfSqWyA <snip> svpbq4lxuw9ixaewyzgcsgz4n5q8x8t ndowsduzoccpfghdwidaqab" DMARC: Domain-based Message Authentication, Reporting and Conformance RFC 7489 Focuses on the RFC5322.From (From:) message header Uses SPF and/or DKIM Uses DNS TXT records at the domain name _dmarc.<apex> Enables feedback reports Interesting cases are _dmarc.{apple,facebook,google,microsoft,twitter}.com DMARC examples Cyrus MDA/MAA at OS3 _dmarc.apple.com IN TXT "v=dmarc1;p=none; ruf 12 rua 13 _dmarc.google.com. 300 IN TXT "v=dmarc1; p=reject; IMAP server with support for local delivery through LMTP LMTP: Local Mail Transfer Protocol Similar to ESMTP Uses LHLO instead of EHLO Reports separate status results for every RCPT TO: Therefore no queueing is needed on the receiver side Uses (improved) Maildir format as message store Does not support direct access to mail files 12 Reporting URIs for failure data 13 Reporting URIs for aggregate data

13 OS3 Mail Infrastructure (1) OS3 Mail Infrastructure (2) Separate incoming and outgoing mail services Incoming: smtp.os3.nl Listens on port 25 to the world for mail destined for os3.nl Includes SPAM checking Delivers local mail via SMTP/LMTP to Cyrus on imap.os3.nl Forwards outbound aliases to mail.serv.os3.nl Outgoing: mail.serv.os3.nl Listens on port 25 to the internal network for mail destined for the world (including os3.nl) Forwards local mail to smtp.os3.nl Enable relaying for authenticated users Via smtp.os3.nl Listen on port 587 as a mail submission agent First enforce STARTTLS Use username/password authentication inside the protected connection Enable webmail Via imap.os3.nl Listen on port 443 for https connections and user login