Aka (or according to Knuth) Karst Koymans. Friday, September 25, 2015

Size: px

Start display at page:

Download "Aka (or according to Knuth) Karst Koymans. Friday, September 25, 2015"

Jayson Phelps
5 years ago
Views:

1 .. Electronic mail Aka (or according to Knuth) Karst Koymans Informatics Institute University of Amsterdam (version 15.4, 2015/09/28 11:43:08 UTC) Friday, September 25, 2015 Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

2 .1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

3 history Outline.1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

4 history History of (1) 1971 Tomlinson s first ( ?) Introduces the use of First based on the CPYNET and SNDMSG program Later piggybacked on FTP over ARPANET 1979 UUCP-based introduces the bang (!) not based on TCP/IP (or NCP) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

5 history History of (2) 1982 SMTP (Simple Mail Transfer Protocol) specified 1983 sendmail released (4.1c BSD) (late) 1983 DNS specified 1984 DNS toplevel domains specified sendmail knows (ARPANET)! (UUCP) : (BerkNet) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

6 history Survey October 2001 (Dan Bernstein) 401 UNIX (Sendmail) 176 Windows (Exchange/IIS) 167 UNIX (qmail) 57 Windows (Ipswitch IMail) 23 UNIX (smap) 15 UNIX (IBM Postfix, formerly VMailer) 14 UNIX (Exim) Source: Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

7 history Survey May 2003 (Thomas Pircher) (35.59%) Sendmail 4537 (8.42%) qmail 4104 (7.62%) Postfix 2812 (5.22%) Microsoft (Exchange/IIS) 2464 (4.57%) Exim Source: Thomas Pircher Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

8 history Survey October 2004 (SNE) (35.3%) Sendmail (21.0%) Microsoft (Exchange/IIS) 6836 (11.8%) Exim 4008 (6.9%) imail 3669 (6.3%) qmail 3172 (5.5%) Postfix Source: lost Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

9 Survey 2006/2007 (MailChannels) Source: O Reilly SysAdmin

10 Survey (securityspace) Description Number of Servers Percent Total Number of MX Servers Queried 2,440, Number of Servers that didn t respond 414, Server didn t open socket 402, Server didn t provide banner 11, Server provided banner 2,025, Server banner identifies software in use 1,108, Source:

Survey 2006-2012 graph (securityspace) Source:

11 Survey graph (securityspace) Source:

12 history Surveys... warnings Important questions Who did the survey? Were there personal interests? What assumptions were made? What model was chosen? What methods were employed? Is this reproducible? Without answers to these questions the result is worthless, or... at least questionable Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

13 Basic concepts Outline.1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

14 Basic concepts concepts Message transfer (RFC 5321) Message format (RFC 5322) Message agents (RFC 5598) Message stores (RFC 5598) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

15 Basic concepts Message Agents Agent acronym MUA MTA MDA MSA MAA 2 MRA 2 Agent use Message 1 User Agent Message Transfer Agent Message Delivery Agent Message Submission Agent Message Access Agent Message Retrieval Agent 1 According to RFC 5598, called Mail User Agent in RFC Not standardised in RFC5598: Internet Mail Architecture Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

16 Message Agents in detail Outline.1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

17 Message Agents in detail Message User Agent Interface for the user Reads and composes messages Thunderbird, Outlook, mutt, pine, mh... Webmail (browser frontend; backend is more like an MRA 3 ) Often uses SMTP to send mail ( MSA) Often uses IMAP/POP3 to get mail ( MAA) May have direct access to message store Direct access is less and less common 3 Open for discussion... Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

18 Message Agents in detail Message Transfer Agent Transfers across the Internet Uses SMTP as transfer protocol sendmail, Postfix, qmail, Exim,... Often also operates as a Message Submission Agent Makes use of MX records to route Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

19 Message Agents in detail Message Delivery Agent Delivers into the message store (MS) mail, mail.local, rmail, procmail May do filtering, SPAM and virus checking,... Has knowledge about mailbox formats Can use Global file space (for example /var/mail/mbox) User specific file space (for example $HOME/mbox) Database (often not directly accessible) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

20 Message Agents in detail Message Submission Agent (1) See RFC 6409 Injects message into the mail system Sanitizes message content, for instance Message-ID: header must be generated if absent Envelope domains must be FQDNs Date: header must be added or corrected Often combined with MTA Should bind to its own port (587), if possible Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

21 Message Agents in detail Message Submission Agent (2) Often acts as an MTA-frontend Is sometimes integrated into the MTA Can also operate locally sendmail (no daemon mode) postdrop without SMTP or with piped SMTP Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

22 Message Agents in detail Message Access Agent Can get message out of Message Store (MS) Offers services to access mail to MUA (or MRA) POP3 (Post Office Protocol) IMAP (Internet Message Access Protocol, version 4) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

23 Message Agents in detail Message Retrieval Agent Program that uses a MAA 4 to collect mail Webmail could be seen as a combination of MRA and MUA Possibly reinjects mail into the mail system Typical example is fetchmail which can use an MTA SMTP TURN (Insecure!) SMTP ETRN (More secure variant 5, RFC 1985) SMTP ATRN 6 (Authenticated variant, RFC 2645) 4 Or sometimes even an MTA 5 Starts a new SMTP connection instead of reusing the old one 6 Providing ODMR (On-Demand Mail Relay) for clients with dynamic IP addresses Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

24 Message Agent Relationships MUA MRA MSA Q MTA protocol local process MAA. MS MDA variable

25 security and validity Outline.1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

26 security and validity Securing (1) Use secure protocols imap (port 143) imaps (port 993) pop3 (port 110) pop3s (port 995) smtp (port 25) smtps, ssmtp (port 465)? 7 7 Not registered as such with IANA Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

27 security and validity Securing (2) SMTP improvements Authenticated SMTP AUTH extension (RFC 4954) Based on SASL 8 (RFC 4422) STARTTLS extension (RFC 3207) Replaces (s)smtp(s) These mechanisms are often used on mail submission via port Simple Authentication and Security Layer Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

28 security and validity Securing (3) Use MUA-based encryption and authentication PGP (Pretty Good Privacy) Inline or PGP/MIME GPG (GNU Privacy Guard) Uses a web of trust S/MIME Uses a hierarchy of trust (PKI) MIME Multipurpose Internet Mail Extensions Different character sets, binary attachments, multiple parts, internationalised headers Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

29 security and validity Validating (1) SPF: Sender Policy Framework RFC 7208 Details are currently a lab exercise Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

30 security and validity Validating (2) DKIM: DomainKeys Identified Mail RFC 6376 Details are currently a lab exercise Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

31 security and validity Validating (3) DMARC: Domain-based Message Authentication, Reporting and Conformance RFC 7489 Focuses on the RFC5322.From (From:) message header Uses SPF and/or DKIM Uses DNS TXT records Enables feedback reports Interesting cases are dmarc.{apple,facebook,google,microsoft,twitter}.com Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

32 Message format Outline.1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

33 Message format Message format Formally specified by ABNF syntax in RFC 5322 Headers, empty line, body Only 7-bit US-ASCII 9 (1-127) allowed MIME extends this to possibly 8-bit Lines are delimited by <CR><LF> Lines should be no longer than 78 characters 9 Or simply ASCII; IANA prefers US-ASCII as character set name in MIME Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

34 Message format Augmented Backus Naur Form (ABNF) Derived from BNF concatenation, choice, literal extended with constructs from EBNF optionals, repetition, grouping, comments Any context free language can be specified by ABNF which is a larger class than the regular languages which are defined by basic 10 regular expressions 10 Extended regular expressions are much stronger! Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

35 Message format Line delimiters (1) Newline, line break, end-of-line (EOL) CR (U+000D) Mac OS before Mac OS X LF (U+000A) Unix CRLF (U+000D, followed by U+000A) Windows Network Virtual Terminal NEL (U+0085) IBM s Next Line Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

36 Message format Line delimiters (2) More options for newline, line break, end-of-line (EOL) LS (U+2028) Unicode Line Separator PS (U+2029) Unicode Paragraph Separator RS (U+001E) Record Separator (obsolete) LFCR (U+000A, followed by U+000D) Sic! Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

37 Message format Message (specified in the ABNF formalism) message = (fields / obs-fields) [CRLF body] body = *(*998text CRLF) *998text CRLF = %d13.10 In mathematical terminology there is an isomorphism text + = (text {CRLF}) This is an example of deflation Note the + instead of a CRLF is delimiter or separator, not terminator or initiator Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

38 Message format Header format <Field name>:<field body> <Field name> printable US-ASCII (33-126) except : (58) <Field body> US-ASCII (1-127) except CR(13) and LF(10) but also (un)folding is allowed Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

39 Message format Some important headers From:<originator mailbox> Sender:<sender mailbox> To:<recipient mailbox> Message-ID:<unique message identification> Received:<recording of message transfer> Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

40 Message transfer Outline.1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

41 Message transfer Message transfer SMTP (RFC 5321) Uses the Network Virtual Terminal (NVT) presentation layer from the TELNET RFC (854) Net-ASCII might be replaced in the future by Net-Unicode, see RFC 5198 Mail objects content (in message format ) envelope (SMTP parameters) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

42 Message transfer Normal (E)SMTP session EHLO (was HELO ) (greeting, option negotiation) MAIL FROM: (envelope sender) RCPT TO: (envelope recipient) DATA (content, ended by <CRLF>.<CRLF>) QUIT (goodbye) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

43 Message Store Outline.1 history.2 Basic concepts.3 Message Agents in detail.4 security and validity.5 Message format.6 Message transfer.7 Message Store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

44 Message Store Message store In database Only accessible via IMAP, POP3 In flat files Also accessible via direct access Enables grepping the message store Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

45 Message Store Mbox format Ordinary file with multiple messages separated by From at start of line has extra blank line at end of message quotes From to >From and >From to >>From... a first characteristic line From <envelopesender> <date> <optionalinfo> Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

46 Message Store MMDF format Variant of mbox format Uses Â Â Â Â as separator Optionally has the mbox From information Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

47 Message Store MH format Mailbox is a directory Every message is a file with a numeric name Used by mh, nmh, xnmh MUA s Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

48 Message Store Maildir format Mailbox is again a directory Subdirectories tmp, new, cur Arriving mail: tmp/<time>.<pid>.<host> No mailbox locking needed Works reliably over NFS Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

49 Message Store Mailbox locking Uses flock, lockf, fcntl system calls Does not always work reliably over NFS Needed if delivery agents and/or access agents operate on the same file (mailbox) Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

50 Message Store Cyrus MDA/MAA at OS3 IMAP server with support for local delivery through LMTP LMTP: Local Mail Transfer Protocol Similar to ESMTP Uses LHLO instead of EHLO Reports separate status results for every RCPT TO: Therefore no queueing is needed on the receiver side Uses (improved) Maildir format as message store Does not support direct access to mail files Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

51 Message Store OS3 Mail Infrastructure (1) Separate incoming and outgoing mail services Incoming: smtp.os3.nl Listens on port 25 to the world for mail destined for os3.nl Includes SPAM checking Delivers local mail via SMTP/LMTP to imap.os3.nl/cyrus Forwards outbound aliases to mail.serv.os3.nl Outgoing: mail.serv.os3.nl Listens on port 25 to the internal network for mail destined for the world (including os3.nl) Forwards local mail to smtp.os3.nl Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

52 Message Store OS3 Mail Infrastructure (2) Enable relaying for authenticated users Via smtp.os3.nl Listen on port 587 as a mail submission agent First enforce STARTTLS Use username/password authentication inside the protected connection Enable webmail Via imap.os3.nl Listen on port 443 for https connections and user login Karst Koymans (UvA) Electronic mail Friday, September 25, / 52

Aka (or according to Knuth) Karst Koymans. Tuesday, September 27, 2016

Aka (or according to Knuth) Karst Koymans. Tuesday, September 27, 2016 .. Electronic mail Aka e-mail (or email according to Knuth) Karst Koymans Informatics Institute University of Amsterdam (version 16.5, 2016/10/05 12:42:51 UTC) Tuesday, September 27, 2016 Karst Koymans