.. Electronic mail Aka e-mail (or email according to Knuth) Karst Koymans Informatics Institute University of Amsterdam (version 16.5, 2016/10/05 12:42:51 UTC) Tuesday, September 27, 2016 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 1 / 54
.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 2 / 54
Email history Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 3 / 54
Email history History of email (1) 1971 Tomlinson s first email (e-mail?) Introduces the use of the @-symbol First based on the CPYNET and SNDMSG program Later piggybacked on FTP over ARPANET 1979 UUCP-based email introduces the bang (!) not based on TCP/IP (or NCP) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 4 / 54
Email history History of email (2) 1982 SMTP (Simple Mail Transfer Protocol) specified 1983 sendmail released (4.1c BSD) (late) 1983 DNS specified 1984 DNS toplevel domains specified sendmail knows about @ (ARPANET)! (UUCP) : (BerkNet) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 5 / 54
Email history Survey October 2001 (Dan Bernstein) 401 UNIX (Sendmail) 176 Windows (Exchange/IIS) 167 UNIX (qmail) 57 Windows (Ipswitch IMail) 23 UNIX (smap) 15 UNIX (IBM Postfix, formerly VMailer) 14 UNIX (Exim) Source: http://cr.yp.to/surveys/smtpsoftware6.txt Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 6 / 54
Email history Survey May 2003 (Thomas Pircher) 19169 (35.59%) Sendmail 4537 (8.42%) qmail 4104 (7.62%) Postfix 2812 (5.22%) Microsoft (Exchange/IIS) 2464 (4.57%) Exim Source: Thomas Pircher Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 7 / 54
Email history Survey October 2004 (SNE) 20492 (35.3%) Sendmail 12172 (21.0%) Microsoft (Exchange/IIS) 6836 (11.8%) Exim 4008 (6.9%) imail 3669 (6.3%) qmail 3172 (5.5%) Postfix Source: lost Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 8 / 54
Survey 2006/2007 (MailChannels) Source: O Reilly SysAdmin
Survey 2006-2012 (securityspace) Description Number of Servers Percent Total Number of MX Servers Queried 2,440,105 100.0 Number of Servers that didn t respond 414,360 16.98 Server didn t open socket 402,406 16.49 Server didn t provide banner 11,954 0.49 Server provided banner 2,025,745 83.02 Server banner identifies software in use 1,108,213 45.42 Source: http://www.securityspace.com/s_survey/data/man.201208/mxsurvey.html
Survey 2006-2012 graph (securityspace) Source: http://www.securityspace.com/s_survey/data/man.201208/mxsurvey.html
Email history Surveys... warnings Important questions Who did the survey? Were there personal interests? What assumptions were made? What model was chosen? What methods were employed? Is this reproducible? Without answers to these questions...... the result is worthless, or... at least questionable Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 12 / 54
Basic concepts Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 13 / 54
Basic concepts Email concepts Message transfer (RFC 5321) Message format (RFC 5322) Message agents (RFC 5598) Message stores (RFC 5598) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 14 / 54
Basic concepts Message Agents Agent acronym MUA MTA MDA MSA MAA 2 MRA 2 Agent use Message 1 User Agent Message Transfer Agent Message Delivery Agent Message Submission Agent Message Access Agent Message Retrieval Agent 1 According to RFC 5598, called Mail User Agent in RFC 5321 2 Not standardised in RFC5598: Internet Mail Architecture Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 15 / 54
Message agents in detail Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 16 / 54
Message agents in detail Message User Agent Interface for the email user Reads and composes messages Thunderbird, Outlook, mutt, pine, mh... Webmail (browser frontend; backend is more like an MRA 3 ) Often uses SMTP to send mail ( MSA) Often uses IMAP/POP3 to get mail ( MAA) May have direct access to message store Direct access is less and less common 3 Open for discussion... Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 17 / 54
Message agents in detail Message Transfer Agent Transfers email across the Internet Uses SMTP as transfer protocol sendmail, Postfix, qmail, Exim,... Often also operates as a Message Submission Agent Makes use of MX records to route email Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 18 / 54
Message agents in detail Message Delivery Agent Delivers email into the message store (MS) mail, mail.local, rmail, procmail May do filtering, SPAM and virus checking,... Has knowledge about mailbox formats Can use Global file space (for example /var/mail/mbox) User specific file space (for example $HOME/mbox) Database (often not directly accessible) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 19 / 54
Message agents in detail Message Submission Agent (1) See RFC 6409 Injects message into the mail system Sanitizes message content, for instance Message-ID: header must be generated if absent Envelope domains must be FQDNs Date: header must be added or corrected Often combined with MTA Should bind to its own port (587), if possible Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 20 / 54
Message agents in detail Message Submission Agent (2) Often acts as an MTA-frontend Is sometimes integrated into the MTA Can also operate locally sendmail (no daemon mode) postdrop without SMTP or with piped SMTP Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 21 / 54
Message agents in detail Message Access Agent Can get message out of Message Store (MS) Offers services to access mail to MUA (or MRA) POP3 (Post Office Protocol) IMAP (Internet Message Access Protocol, version 4) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 22 / 54
Message agents in detail Message Retrieval Agent Program that uses a MAA 4 to collect mail Webmail could be seen as a combination of MRA and MUA Possibly reinjects mail into the mail system Typical example is fetchmail which can use an MTA SMTP TURN (Insecure!) SMTP ETRN (More secure variant 5, RFC 1985) SMTP ATRN 6 (Authenticated variant, RFC 2645) 4 Or sometimes even an MTA 5 Starts a new SMTP connection instead of reusing the old one 6 Providing ODMR (On-Demand Mail Relay) for clients with dynamic IP addresses Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 23 / 54
Message Agent Relationships MUA MRA MSA Q MTA protocol local process MAA. MS MDA variable
Email security and validity Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 25 / 54
Email security and validity Securing Email (1) Use secure protocols imap (port 143) imaps (port 993) pop3 (port 110) pop3s (port 995) smtp (port 25) smtps, ssmtp (port 465)? 7 7 Not registered as such with IANA Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 26 / 54
Email security and validity Securing Email (2) SMTP improvements Authenticated SMTP AUTH extension (RFC 4954) Based on SASL 8 (RFC 4422) STARTTLS extension (RFC 3207) Replaces (s)smtp(s) These mechanisms are often used on mail submission via port 587 8 Simple Authentication and Security Layer Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 27 / 54
Email security and validity Securing Email (3) Use MUA-based encryption and authentication PGP (Pretty Good Privacy) Inline or PGP/MIME GPG (GNU Privacy Guard) Uses a web of trust S/MIME Uses a hierarchy of trust (PKI) MIME Multipurpose Internet Mail Extensions Different character sets, binary attachments, multiple parts, internationalised headers Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 28 / 54
Message format Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 29 / 54
Message format Message format Formally specified by ABNF syntax in RFC 5322 Headers, empty line, body Only 7-bit US-ASCII 9 (1-127) allowed MIME extends this to possibly 8-bit Lines are delimited by <CR><LF> Lines should be no longer than 78 characters 9 Or simply ASCII; IANA prefers US-ASCII as character set name in MIME Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 30 / 54
Message format Augmented Backus Naur Form (ABNF) Derived from BNF concatenation, choice, literal extended with constructs from EBNF optionals, repetition, grouping, comments Any context free language can be specified by ABNF which is a larger class than the regular languages which are defined by basic 10 regular expressions 10 Extended regular expressions are much stronger! Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 31 / 54
Message format Line delimiters (1) Newline, line break, end-of-line (EOL) CR (U+000D) Mac OS before Mac OS X LF (U+000A) Unix CRLF (U+000D, followed by U+000A) Windows Network Virtual Terminal NEL (U+0085) IBM s Next Line Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 32 / 54
Message format Line delimiters (2) More options for newline, line break, end-of-line (EOL) LS (U+2028) Unicode Line Separator PS (U+2029) Unicode Paragraph Separator RS (U+001E) Record Separator (obsolete) LFCR (U+000A, followed by U+000D) Sic! Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 33 / 54
Message format Message (specified in the ABNF formalism) message = (fields / obs-fields) [CRLF body] body = *(*998text CRLF) *998text CRLF = %d13.10 In mathematical terminology there is an isomorphism text + = (text {CRLF}) This is an example of deflation Note the + instead of a CRLF is delimiter or separator, not terminator or initiator Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 34 / 54
Message format Header format <Field name>:<field body> <Field name> printable US-ASCII (33-126) except : (58) <Field body> US-ASCII (1-127) except CR(13) and LF(10)...... but also (un)folding is allowed Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 35 / 54
Message format Some important headers From:<originator mailbox> Sender:<sender mailbox> To:<recipient mailbox> Message-ID:<unique message identification> Received:<recording of message transfer> Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 36 / 54
Message transfer Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 37 / 54
Message transfer Message transfer SMTP (RFC 5321) Uses the Network Virtual Terminal (NVT) presentation layer from the TELNET RFC (854) Net-ASCII might be replaced in the future by Net-Unicode, see RFC 5198 Mail objects content (in message format ) envelope (SMTP parameters) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 38 / 54
Message transfer Normal (E)SMTP session EHLO (newer form of HELO ) (greeting, option negotiation) MAIL FROM: (envelope sender) RCPT TO: (envelope recipient) DATA (content, ended by <CRLF>.<CRLF>) QUIT (goodbye) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 39 / 54
Message storage formats Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 40 / 54
Message storage formats Message store In database Only accessible via IMAP, POP3 In flat files Also accessible via direct access Enables grepping the message store Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 41 / 54
Message storage formats Mbox format Ordinary file with multiple messages separated by From at start of line has extra blank line at end of message quotes From to >From and >From to >>From... a first characteristic line From <envelopesender> <date> <optionalinfo> Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 42 / 54
Message storage formats MMDF format Variant of mbox format Uses     as separator Optionally has the mbox From information Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 43 / 54
Message storage formats MH format Mailbox is a directory Every message is a file with a numeric name Used by mh, nmh, xnmh MUA s Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 44 / 54
Message storage formats Maildir format Mailbox is again a directory Subdirectories tmp, new, cur Arriving mail: tmp/<time>.<pid>.<host> No mailbox locking needed Works reliably over NFS Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 45 / 54
Message storage formats Mailbox locking Uses flock, lockf, fcntl system calls Does not always work reliably over NFS Needed if delivery agents and/or access agents operate on the same file (mailbox) Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 46 / 54
Validating email Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 47 / 54
Validating email Validating Email (1) SPF: Sender Policy Framework RFC 7208 Uses DNS TXT records Specifies whitelist for clients and envelope sender addresses Can check both MAIL FROM: and HELO/EHLO Rather crude and simple system Can break mail forwarding Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 48 / 54
Validating email Validating Email (2) DKIM: DomainKeys Identified Mail RFC 6376 Uses DNS TXT records Signs (part of) the header and body of an email, which is being checked by the public key of the sending domain Does not depend on the SMTP envelope Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 49 / 54
Validating email Validating Email (3) DMARC: Domain-based Message Authentication, Reporting and Conformance RFC 7489 Focuses on the RFC5322.From (From:) message header Uses SPF and/or DKIM Uses DNS TXT records Enables feedback reports Interesting cases are dmarc.{apple,facebook,google,microsoft,twitter}.com Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 50 / 54
Email at OS3 Outline.1 Email history.2 Basic concepts.3 Message agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message storage formats.8 Validating email.9 Email at OS3 Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 51 / 54
Email at OS3 Cyrus MDA/MAA at OS3 IMAP server with support for local delivery through LMTP LMTP: Local Mail Transfer Protocol Similar to ESMTP Uses LHLO instead of EHLO Reports separate status results for every RCPT TO: Therefore no queueing is needed on the receiver side Uses (improved) Maildir format as message store Does not support direct access to mail files Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 52 / 54
Email at OS3 OS3 Mail Infrastructure (1) Separate incoming and outgoing mail services Incoming: smtp.os3.nl Listens on port 25 to the world for mail destined for os3.nl Includes SPAM checking Delivers local mail via SMTP/LMTP to Cyrus on imap.os3.nl Forwards outbound aliases to mail.serv.os3.nl Outgoing: mail.serv.os3.nl Listens on port 25 to the internal network for mail destined for the world (including os3.nl) Forwards local mail to smtp.os3.nl Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 53 / 54
Email at OS3 OS3 Mail Infrastructure (2) Enable relaying for authenticated users Via smtp.os3.nl Listen on port 587 as a mail submission agent First enforce STARTTLS Use username/password authentication inside the protected connection Enable webmail Via imap.os3.nl Listen on port 443 for https connections and user login Karst Koymans (UvA) Electronic mail Tuesday, September 27, 2016 54 / 54