.. Electronic mail Aka e-mail (or email according to Knuth) Karst Koymans Informatics Institute University of Amsterdam (version 15.4, 2015/09/28 11:43:08 UTC) Friday, September 25, 2015 Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 1 / 52
.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 2 / 52
Email history Outline.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 3 / 52
Email history History of email (1) 1971 Tomlinson s first email (e-mail?) Introduces the use of the @-symbol First based on the CPYNET and SNDMSG program Later piggybacked on FTP over ARPANET 1979 UUCP-based email introduces the bang (!) not based on TCP/IP (or NCP) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 4 / 52
Email history History of email (2) 1982 SMTP (Simple Mail Transfer Protocol) specified 1983 sendmail released (4.1c BSD) (late) 1983 DNS specified 1984 DNS toplevel domains specified sendmail knows about @ (ARPANET)! (UUCP) : (BerkNet) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 5 / 52
Email history Survey October 2001 (Dan Bernstein) 401 UNIX (Sendmail) 176 Windows (Exchange/IIS) 167 UNIX (qmail) 57 Windows (Ipswitch IMail) 23 UNIX (smap) 15 UNIX (IBM Postfix, formerly VMailer) 14 UNIX (Exim) Source: http://cr.yp.to/surveys/smtpsoftware6.txt Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 6 / 52
Email history Survey May 2003 (Thomas Pircher) 19169 (35.59%) Sendmail 4537 (8.42%) qmail 4104 (7.62%) Postfix 2812 (5.22%) Microsoft (Exchange/IIS) 2464 (4.57%) Exim Source: Thomas Pircher Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 7 / 52
Email history Survey October 2004 (SNE) 20492 (35.3%) Sendmail 12172 (21.0%) Microsoft (Exchange/IIS) 6836 (11.8%) Exim 4008 (6.9%) imail 3669 (6.3%) qmail 3172 (5.5%) Postfix Source: lost Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 8 / 52
Survey 2006/2007 (MailChannels) Source: O Reilly SysAdmin
Survey 2006-2012 (securityspace) Description Number of Servers Percent Total Number of MX Servers Queried 2,440,105 100.0 Number of Servers that didn t respond 414,360 16.98 Server didn t open socket 402,406 16.49 Server didn t provide banner 11,954 0.49 Server provided banner 2,025,745 83.02 Server banner identifies software in use 1,108,213 45.42 Source: http://www.securityspace.com/s_survey/data/man.201208/mxsurvey.html
Survey 2006-2012 graph (securityspace) Source: http://www.securityspace.com/s_survey/data/man.201208/mxsurvey.html
Email history Surveys... warnings Important questions Who did the survey? Were there personal interests? What assumptions were made? What model was chosen? What methods were employed? Is this reproducible? Without answers to these questions...... the result is worthless, or... at least questionable Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 12 / 52
Basic concepts Outline.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 13 / 52
Basic concepts Email concepts Message transfer (RFC 5321) Message format (RFC 5322) Message agents (RFC 5598) Message stores (RFC 5598) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 14 / 52
Basic concepts Message Agents Agent acronym MUA MTA MDA MSA MAA 2 MRA 2 Agent use Message 1 User Agent Message Transfer Agent Message Delivery Agent Message Submission Agent Message Access Agent Message Retrieval Agent 1 According to RFC 5598, called Mail User Agent in RFC 5321 2 Not standardised in RFC5598: Internet Mail Architecture Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 15 / 52
Message Agents in detail Outline.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 16 / 52
Message Agents in detail Message User Agent Interface for the email user Reads and composes messages Thunderbird, Outlook, mutt, pine, mh... Webmail (browser frontend; backend is more like an MRA 3 ) Often uses SMTP to send mail ( MSA) Often uses IMAP/POP3 to get mail ( MAA) May have direct access to message store Direct access is less and less common 3 Open for discussion... Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 17 / 52
Message Agents in detail Message Transfer Agent Transfers email across the Internet Uses SMTP as transfer protocol sendmail, Postfix, qmail, Exim,... Often also operates as a Message Submission Agent Makes use of MX records to route email Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 18 / 52
Message Agents in detail Message Delivery Agent Delivers email into the message store (MS) mail, mail.local, rmail, procmail May do filtering, SPAM and virus checking,... Has knowledge about mailbox formats Can use Global file space (for example /var/mail/mbox) User specific file space (for example $HOME/mbox) Database (often not directly accessible) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 19 / 52
Message Agents in detail Message Submission Agent (1) See RFC 6409 Injects message into the mail system Sanitizes message content, for instance Message-ID: header must be generated if absent Envelope domains must be FQDNs Date: header must be added or corrected Often combined with MTA Should bind to its own port (587), if possible Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 20 / 52
Message Agents in detail Message Submission Agent (2) Often acts as an MTA-frontend Is sometimes integrated into the MTA Can also operate locally sendmail (no daemon mode) postdrop without SMTP or with piped SMTP Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 21 / 52
Message Agents in detail Message Access Agent Can get message out of Message Store (MS) Offers services to access mail to MUA (or MRA) POP3 (Post Office Protocol) IMAP (Internet Message Access Protocol, version 4) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 22 / 52
Message Agents in detail Message Retrieval Agent Program that uses a MAA 4 to collect mail Webmail could be seen as a combination of MRA and MUA Possibly reinjects mail into the mail system Typical example is fetchmail which can use an MTA SMTP TURN (Insecure!) SMTP ETRN (More secure variant 5, RFC 1985) SMTP ATRN 6 (Authenticated variant, RFC 2645) 4 Or sometimes even an MTA 5 Starts a new SMTP connection instead of reusing the old one 6 Providing ODMR (On-Demand Mail Relay) for clients with dynamic IP addresses Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 23 / 52
Message Agent Relationships MUA MRA MSA Q MTA protocol local process MAA. MS MDA variable
Email security and validity Outline.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 25 / 52
Email security and validity Securing Email (1) Use secure protocols imap (port 143) imaps (port 993) pop3 (port 110) pop3s (port 995) smtp (port 25) smtps, ssmtp (port 465)? 7 7 Not registered as such with IANA Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 26 / 52
Email security and validity Securing Email (2) SMTP improvements Authenticated SMTP AUTH extension (RFC 4954) Based on SASL 8 (RFC 4422) STARTTLS extension (RFC 3207) Replaces (s)smtp(s) These mechanisms are often used on mail submission via port 587 8 Simple Authentication and Security Layer Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 27 / 52
Email security and validity Securing Email (3) Use MUA-based encryption and authentication PGP (Pretty Good Privacy) Inline or PGP/MIME GPG (GNU Privacy Guard) Uses a web of trust S/MIME Uses a hierarchy of trust (PKI) MIME Multipurpose Internet Mail Extensions Different character sets, binary attachments, multiple parts, internationalised headers Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 28 / 52
Email security and validity Validating Email (1) SPF: Sender Policy Framework RFC 7208 Details are currently a lab exercise Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 29 / 52
Email security and validity Validating Email (2) DKIM: DomainKeys Identified Mail RFC 6376 Details are currently a lab exercise Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 30 / 52
Email security and validity Validating Email (3) DMARC: Domain-based Message Authentication, Reporting and Conformance RFC 7489 Focuses on the RFC5322.From (From:) message header Uses SPF and/or DKIM Uses DNS TXT records Enables feedback reports Interesting cases are dmarc.{apple,facebook,google,microsoft,twitter}.com Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 31 / 52
Message format Outline.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 32 / 52
Message format Message format Formally specified by ABNF syntax in RFC 5322 Headers, empty line, body Only 7-bit US-ASCII 9 (1-127) allowed MIME extends this to possibly 8-bit Lines are delimited by <CR><LF> Lines should be no longer than 78 characters 9 Or simply ASCII; IANA prefers US-ASCII as character set name in MIME Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 33 / 52
Message format Augmented Backus Naur Form (ABNF) Derived from BNF concatenation, choice, literal extended with constructs from EBNF optionals, repetition, grouping, comments Any context free language can be specified by ABNF which is a larger class than the regular languages which are defined by basic 10 regular expressions 10 Extended regular expressions are much stronger! Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 34 / 52
Message format Line delimiters (1) Newline, line break, end-of-line (EOL) CR (U+000D) Mac OS before Mac OS X LF (U+000A) Unix CRLF (U+000D, followed by U+000A) Windows Network Virtual Terminal NEL (U+0085) IBM s Next Line Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 35 / 52
Message format Line delimiters (2) More options for newline, line break, end-of-line (EOL) LS (U+2028) Unicode Line Separator PS (U+2029) Unicode Paragraph Separator RS (U+001E) Record Separator (obsolete) LFCR (U+000A, followed by U+000D) Sic! Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 36 / 52
Message format Message (specified in the ABNF formalism) message = (fields / obs-fields) [CRLF body] body = *(*998text CRLF) *998text CRLF = %d13.10 In mathematical terminology there is an isomorphism text + = (text {CRLF}) This is an example of deflation Note the + instead of a CRLF is delimiter or separator, not terminator or initiator Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 37 / 52
Message format Header format <Field name>:<field body> <Field name> printable US-ASCII (33-126) except : (58) <Field body> US-ASCII (1-127) except CR(13) and LF(10)...... but also (un)folding is allowed Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 38 / 52
Message format Some important headers From:<originator mailbox> Sender:<sender mailbox> To:<recipient mailbox> Message-ID:<unique message identification> Received:<recording of message transfer> Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 39 / 52
Message transfer Outline.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 40 / 52
Message transfer Message transfer SMTP (RFC 5321) Uses the Network Virtual Terminal (NVT) presentation layer from the TELNET RFC (854) Net-ASCII might be replaced in the future by Net-Unicode, see RFC 5198 Mail objects content (in message format ) envelope (SMTP parameters) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 41 / 52
Message transfer Normal (E)SMTP session EHLO (was HELO ) (greeting, option negotiation) MAIL FROM: (envelope sender) RCPT TO: (envelope recipient) DATA (content, ended by <CRLF>.<CRLF>) QUIT (goodbye) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 42 / 52
Message Store Outline.1 Email history.2 Basic concepts.3 Message Agents in detail.4 Email security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 43 / 52
Message Store Message store In database Only accessible via IMAP, POP3 In flat files Also accessible via direct access Enables grepping the message store Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 44 / 52
Message Store Mbox format Ordinary file with multiple messages separated by From at start of line has extra blank line at end of message quotes From to >From and >From to >>From... a first characteristic line From <envelopesender> <date> <optionalinfo> Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 45 / 52
Message Store MMDF format Variant of mbox format Uses     as separator Optionally has the mbox From information Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 46 / 52
Message Store MH format Mailbox is a directory Every message is a file with a numeric name Used by mh, nmh, xnmh MUA s Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 47 / 52
Message Store Maildir format Mailbox is again a directory Subdirectories tmp, new, cur Arriving mail: tmp/<time>.<pid>.<host> No mailbox locking needed Works reliably over NFS Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 48 / 52
Message Store Mailbox locking Uses flock, lockf, fcntl system calls Does not always work reliably over NFS Needed if delivery agents and/or access agents operate on the same file (mailbox) Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 49 / 52
Message Store Cyrus MDA/MAA at OS3 IMAP server with support for local delivery through LMTP LMTP: Local Mail Transfer Protocol Similar to ESMTP Uses LHLO instead of EHLO Reports separate status results for every RCPT TO: Therefore no queueing is needed on the receiver side Uses (improved) Maildir format as message store Does not support direct access to mail files Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 50 / 52
Message Store OS3 Mail Infrastructure (1) Separate incoming and outgoing mail services Incoming: smtp.os3.nl Listens on port 25 to the world for mail destined for os3.nl Includes SPAM checking Delivers local mail via SMTP/LMTP to imap.os3.nl/cyrus Forwards outbound aliases to mail.serv.os3.nl Outgoing: mail.serv.os3.nl Listens on port 25 to the internal network for mail destined for the world (including os3.nl) Forwards local mail to smtp.os3.nl Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 51 / 52
Message Store OS3 Mail Infrastructure (2) Enable relaying for authenticated users Via smtp.os3.nl Listen on port 587 as a mail submission agent First enforce STARTTLS Use username/password authentication inside the protected connection Enable webmail Via imap.os3.nl Listen on port 443 for https connections and user login Karst Koymans (UvA) Electronic mail Friday, September 25, 2015 52 / 52