SMTP, IMAP, POP

Email server

In order for a computer system to function as a mail server, it must include mail server software. This software allows the system administrator to create and manage email accounts for any domains hosted on the server. For example, if the server hosts the domain name "techterms.com," it can provide email accounts ending in "@techterms.com."

How does an MTA know where/how to transfer mail?

It finds the MX record from the recipient domain's DNS Zone, then uses SMTP to transfer the mail to another MTA (or to the MDA if the recipient's server has been reached)

Often, these two agents are instances of the same software launched with different options on the same machine.

MSA, MTA

MUA

Mail User Agent. A client application (desktop or webmail app) that allows the receiving and sending of emails.

Do MTAs know exactly where to send an email (EG, the final destination)?

No, MTAs forward email. MTA first asks a DNS server to map out a path Then sends email to a different MTA closer to the destination's inbox EG, the process of an MTA searching for destination for recipient [email protected]: Takes away "janedoe" Searches for MX DNS record for gmail.com

Port 465

Originally registered for SMTPS, which now uses STARTTLS Never published as an official SMTP transmission by the IETF

Port 25

Plaintext transmission Used in MSA to MTA communication, MTA to MTA communication In most cases, SMTP email clients should NOT use this port for submission of mail to MSAs.

Postfix

Postfix is a free and open-source mail transfer agent software that routes and delivers electronic mail.

Server administrators need to impose some control on which clients can use the server. This enables them to deal with abuse, for example spam. Two solutions have been in common use:

Restricting client access by location Requiring client authentication

SMTP Server Return Codes

Status codes are issued by a server in response to a client's request made to the server. Types: Basic Status Codes Enhanced Status Codes Full list: https://en.wikipedia.org/wiki/List_of_SMTP_server_return_codes

SMTP transaction steps

TCP connection on port 25/587 (?) See following flash cards for terms in pink

SMTP relaying

The process of transferring mail between MTAs or from an MTA to an MDA

Restricting client access by location

Under this system, an ISP's SMTP server will not allow access by users who are outside the ISP's network. More precisely, the server may only allow access to users with an IP address provided by the ISP, which is equivalent to requiring that they are connected to the Internet using that same ISP This method is largely obsolete

Web-based user agents

Web-based email clients (gmail, hotmail) This means the emails are stored on the web, and the user may access them from any device they log into

HELP command

With the HELP command, the client requests a list of commands the server supports. HELP may be used with an argument (a specific command). If the server supports this, it will provide the information accordingly to this request. Example: HELP

What happens if an MX record cannot be found?

a conformant relaying server (not all are) instead looks up the A record. Relay servers can also be configured to use a smart host

SMTP Banner

a greeting sent by the SERVER (Receiver) that starts with the code 220 to indicate that the client is ready to speak SMTP (or usually ESMTP, a superset of SMTP). Syntax: 220 your.f.q.d.n ESTMP... (e.g. S: 220 smtp.example.com ESMTP Postfix)

The initiating host, the SMTP client, can be either

as a mail user agent (MUA), or a relay server's mail transfer agent (MTA), that is an SMTP server acting as an SMTP client,

The main difference between connecting to an MTA and an MSA

connecting to an MSA requires SMTP Authentication.

SMTP Authentication (SMTP AUTH)

extension of SMTP used for client SMTP email submission (to an MSA), typically on TCP port 587

3 steps of an SMTP transaction

handshake - establishing a TCP connection email transfer - manipulations with the email termination - closing a TCP connection

For retrieving messages, IMAP and POP3 are the standard protocols in use, but proprietary servers also often implement

proprietary protocols, such as exchange ActiveSync

Message transfer can occur in a ___ between two MTAs, or via ________________________

single connection of series of hops through intermediary systems

A receiving SMTP server may be one of 3 things

the ultimate destination an intermediate "relay" (that is, it stores and forwards the message) a "gateway" (that is, it may forward the message using some protocol other than SMTP).

STARTTLS extended command

used to start a TLS handshake for a secure SMTP session. STARTTLS resets the SMTP protocol to the initial state. Once the response 220 is received from the server, the SMTP client should send HELO or EHLO to launch the session. In the case of a negative response (454), the client must decide whether to continue the SMTP session or not. Example: STARTTLS

Types of Mail Servers

2 categories: Outgoing email servers are called SMTP servers (Simple Mail Transfer Protocol). Incoming email servers are known as POP3 servers and IMAP servers

Incoming Mail Server

A POP3 Server or IMAP Server e.g. imap.gmail.com

SMTP Server

A mail server that uses an MTA application to transfer mail between MTAs or from MTA to MDA An SMTP server can be set up and hosted independently using dedicated hardware and software (such as Sendmail or Windows Server which had an SMTP Server feature) or paid for monthly for a provider. SMTP servers are automatically provided to webmail users, eg gmail has an SMTP server named smtp.gmail.com

SMTP

A protocol for relaying email between MTAs (Mail Servers) (port 25) and sending email out from client to server (port 587)

POP3

A protocol used by a desktop mail client (not webmail) to retrieve email from a server via TCP port 110 POP3 deletes emails from the sever after they've been downloaded to the client, and emails are only retrievable by that device

Mailbox provider (Mail service provider, email service provider)

A provider of email hosting. It implements email servers to send, receive, accept and store email for other entities on their behalf There are various kinds of email providers. There are paid and free ones (e.g. gmail, hotmail) Mailbox providers typically accomplish their task by implementing Simple Mail Transfer Protocol (SMTP) and possibly providing access to messages through Internet Message Access Protocol (IMAP), the Post Office Protocol, Webmail, or a proprietary protocol.

Smart host

A smart host or smarthost is an email server via which third parties can send emails and have them forwarded on to the email recipients' email servers

220 OK

A status code indicate that the SMTP/ESMTP server is ready to communicate (i.e. ready to receive HELO/EHLO)

IMAP Server

A type of incoming Mail Server As per the diagram, the final (second) email server (?) Contains an MDA which stores the email until client retrieves it (?)

Give an example of why restricting access by location is problematic

A mobile user may often be on a network other than that of their normal ISP, and will then find that sending email fails because the configured SMTP server choice is no longer accessible.

250 OK

An ESMTP server returns the code 250 OK in a multi-line reply with its domain and a list of keywords to indicate supported extensions.

How does an MDA deliver messages?

An MDA may deliver messages directly to storage, or forward them over a network using SMTP or other protocol such as Local Mail Transfer Protocol (LMTP), a derivative of SMTP designed for this purpose.

Outgoing Mail Server

An SMTP Server e.g. smtp.gmail.com

How does a mail sender communicate with a mail receiver?

An SMTP session which consists of commands originated by an SMTP client (the initiating agent, sender, or transmitter) and corresponding responses from the SMTP server (the listening agent, or receiver)

Port 2525

An alternative port which mirrors 587 Not endorsed by IETF or IANA

SMTP commands

Define a particular function within the SMTP session

Application-based user agents

Desktop email clients (e.g., outlook, thunderbird) These are also known as private email systems, and require downloading an email application This means the user may only access their emails from within their private network (or only from their PC)

After receiving 220 your.f.q.d.n ESTMP... greeting from server, what does the client send?

EHLO, followed by its own Fully Qualified Domain Name: EHLO peers.f.q.d.n

Service Extensions (Extended Commands)

Each Extension has its own RFC. These are keywords that provide additional mechansims. E.g., the AUTH extension (SMTP AUTH) provides an access control mechanism. See diagram for common extensions. The syntax for these keywords comes after an EHLO, when the server sends a list of supported extensions. Example where SIZE, HELP and PIPELINING are extensions used: S: 220 smtp2.example.com ESMTP Postfix C: EHLO bob.example.com S: 250-smtp2.example.com Hello bob.example.org [192.0.2.201] S: 250-SIZE 14680064 S: 250-PIPELINING S: 250 HELP

Types of SMTP Servers

Email addresses based on hosted email services such as Hotmail, Google, Yahoo, etc. will use their own outgoing mail servers (e.g. smtp.gmail.com) If your email is hosted on your own website, you may want to use your hosting provider's mail server. For example, websites hosted on example.com may use mail.example.com as both their incoming and outgoing mail servers. Some ISP's (Internet Service Providers) will not allow you to make use of your own outgoing mail server. In this situation, you'll need to use your ISP's outgoing mail server (e.g. smtp.comcast.net) Note that Some ISPs do not provide SMTP services 'at all' - expecting you to use webmail ISP email addresses (e.g. [email protected]) use the ISP's SMTP server (?)

Maildir vs mbox

Email formats which act as a directory for storing messages in email applications Mbox stores all email messages in a single file on the server, whereas, Maildir stores messages in individual files with unique names (in a directory tree).

Port 587

Encrypted transmission Used for submission by clients to MSAs

Examples of common MTAs

Exim, Postifx, Sendmail, Qmail, Microsoft Exchange

ESMTP

Extended Simple Mail Transfer Protocol. Specifies extensions to the original protocol for sending e-mail that supports graphics, audio and video files, and text in various national languages. Most commercial e-mail servers and clients support ESMTP. It is used as both an inter-server transport protocol and (with restricted behavior enforced) a mail submission protocol.

IMAP

Internet Message Access Protocol A protocol used by webmail to connect to an IMAP mail server on TCP port 143 in order to retrieve email. Stays connected, email is NOT deleted from server and is retrievable by any device the end user is authenticated on

MDA

Message Delivery Agent (aka Local Delivery Agent LDA). A server program that receives mail from the server's MTA application, and stores it in a mailbox. Responsible for the delivery of e-mail messages to a local recipient's mailbox

MSA

Message Submission Agent. Software on an SMTP server that receives mail from an MUA, checks for errors, and transfers it with SMTP to the MTA program on the same server

MTA

Message Transfer Agent An MTA. Software that transfers electronic mail messages from one computer to another using SMTP (specifically, a server application that receives email from the MSA or another MTA and forwards it to another MTA or an MDA) An MTA is a specific type of mail server--one that queues email and moves it along a delivery chain until it hits a Mail Delivery Agent (MDA) of some sort Usually, MTAs use a store-and-forward model of mail handling. This means that outgoing mail is put into a queue and waits for the recipient's server response.

Email is read from an email box with POP, IMAP or WEBMAIL OVER HTTP. How is email WRITTEN to an email box?

Messages sent to a mailbox are written by a mail delivery agent into the server's local mailbox, which, for remote users, is a remote mailbox that they own on that server. IMAP clients can copy, move, and delete messages in remote mailboxes.

Requiring client authentication

Modern SMTP servers typically require authentication of clients by credentials before allowing access, rather than restricting access by location as described earlier. This more flexible system is friendly to mobile users and allows them to have a fixed choice of configured outbound SMTP server. It uses SMTP AUTHENTICATION extension to accomplish this.

Does SMTP define message content?

NO. SMTP defines message transport, not the message content. Thus, it defines the mail envelope and its parameters, such as the envelope sender, but not the header (except trace information) nor the body of the message itself. STD 10 and RFC 5321 define SMTP (the envelope), while STD 11 and RFC 5322 define the message (header and body), formally referred to as the Internet Message Format.

SMTP ports

Port 25 is used for relaying between MTAs (mail servers) or from MSAs to MTAs 587 for client to MSA

Email process diagram

Ports = destination ports

Popular client-server protocols to retrieve email messages are

Post Office Protocol (POP): a method that is most suitable for reading messages from a single client computer. Usually messages are removed from the server mailbox after retrieval. Anyway, the master copy of a message is the one in the local mailbox. Internet Message Access Protocol (IMAP): designed to retrieve messages from multiple clients by allowing remote management of the server mailbox. Master copies stay on the server, but a copy can be saved locally. Webmail over HTTP: messages are served to a user's browser in a server-defined format. Master copies stay on the server, possibly in the original format, which may be downloadable.

Enhanced status code

RFC 3463 defines a separate series of enhanced mail system status codes which is intended to be better structured, consisting of three numerical fields separated by ".", as follows (see the diagram) The classes are defined as follows: 2.XXX.XXX Success: Report of a positive delivery action. 4.XXX.XXX Persistent Transient Failure: Message as sent is valid, but persistence of some temporary conditions has caused abandonment or delay. 5.XXX.XXX Permanent Failure: Not likely to be resolved by resending the message in current form. The subjects are defined as follows: X.0.XXX Other or Undefined Status X.1.XXX Addressing Status X.2.XXX Mailbox Status X.3.XXX Mail System Status X.4.XXX Network and Routing Status X.5.XXX Mail Delivery Protocol Status X.6.XXX Message Content or Media Status X.7.XXX Security or Policy Status

Dot-stuffing

Since a message body can contain a line with just a period as part of the text, the client sends two periods every time a line starts with a period; correspondingly, the server replaces every sequence of two periods at the beginning of a line with a single one. Such escaping method is called dot-stuffing. Recall that a "." indicates the end of an email message

ATRN command

The ATRN command replaced the obsolete TURN command. It was used to reverse the connection between the local and external SMTP servers (sender and receiver). TURN lacked authentication and hence was deprecated. ATRN is devoid of this drawback. Besides, it is available for dynamically assigned IP addresses. Example: ATRN client.net,client.com 250 OK now reversing the connection (server response)

AUTH extended command

The AUTH command is used to authenticate the client to the server. For this, it uses an argument that specifies different levels of security and login methods: PLAIN, LOGIN, and CRAM-MD5. The session is considered authenticated once the server provided a positive response. For more on this, read the SMTP authentication blog post. Example: AUTH CRAM-MD5

BDAT command

The BDAT command is used to submit mail contents. It can be an alternative to the DATA command. BDAT has two arguments. The first one defines the length of the data chunk in octets. The second one indicates that the data chunk is terminating. No need for a period to terminate mail transfer as it is in the DATA command. BDAT is widely used in Microsoft Exchange Server. At the same time, DATA is a must to support command for all servers. Example: BDAT 67 LAST To: [email protected] From: [email protected] Subject: How SMTP works 250 Message OK, 67 octets received (server response)

ETRN

The ETRN command is the request to start SMTP queue processing of a specified server host. Example: ETRN client.com 250 OK, queuing for client.com started (server response)

LHLO command

The LHLO command has identical semantics to the EHLO command of ESMTP

LMTP

The Local Mail Transfer Protocol (LMTP) is an alternative to (Extended) Simple Mail Transfer Protocol for situations where the receiving side does not have a mail queue, such as a message transfer agent acting as a message delivery agent. LMTP servers do not queue messages, so they must return an individual status reply for every recipient of a particular email message. For those recipients that could not be delivered, the MTA, and not the LMTP server, takes the responsibility of queuing the message and attempting redelivery. LMTP conversations can occur between mail subsystems on the same machine or on different machines on a local area network. It is not recommended for wide area networks, since the protocol depends on a quick response to indicate whether the message was delivered.

NOOP command

The NOOP command is used only to check whether the server can respond. "250 OK" reply in response Example: NOOP

QUIT command

The QUIT command send the request to terminate the SMTP session. Once the server responses with 221, the client closes the SMTP connection. This command specifies that the receiver MUST send a "221 OK" reply and then closes the transmission channel. Example: QUIT

RCPT TO command

The RCPT TO command specifies the recipient. As an argument, RCPT TO includes a destination mailbox (forward-path). In case of multiple recipients, RCPT TO will be used to specify each recipient separately. Example: RCPT TO "[email protected]"

RSET command

The RSET command resets the SMTP connection to the initial state. It erases all the buffers and state tables (both sender and recipient). RSET gets only the positive server response - 250. At the same time, the SMTP connection remains open and is ready for a new mail transaction. Example: RSET

Basic Status Code

The first digit denotes whether the response is good, bad, or incomplete.: 2yz (Posiive Completion) E.G. 250 3yz (Positive Intermediate) 4yz (Transient Negative Completion) 5yz Permanent negative completion The second digit encodes responses in specific categories: x0z (Syntax): These replies refer to syntax errors, syntactically correct commands that do not fit any functional category, and unimplemented or superfluous commands. x1z (Information): These are replies to requests for information. x2z (Connections): These are replies referring to the transmission channel. x3z : Unspecified. x4z : Unspecified. x5z (Mail system): These replies indicate the status of the receiver mail system. See full list: https://en.wikipedia.org/wiki/List_of_SMTP_server_return_codes

EHLO vs HELO

The main identification feature for ESMTP clients is to open a transmission with the command EHLO (Extended HELLO), rather than HELO Clients learn a server's supported options by using the EHLO greeting, as exemplified below, instead of the original HELO. Clients fall back to HELO only if the server does not support EHLO greeting. If the client used EHLO, the server knows that it is capable of handling multi-line responses, and so will normally send back several lines indicating the capabilities offered by your server: See diagram

In many cases, web servers and mail servers are combined in a single machine. True or false?

True. However, large ISPs and public email services (such as Gmail and Hotmail) may use dedicated hardware for sending and receiving email.

Why do VRFY and EXPN pose a security risk?

VRFY and EXPN implement SMTP authentication. Also, they are useful to perform an internal audit of the server. On the other hand, these commands are considered a security risk. Spammers can use them to harvest valid email addresses from the server. Therefore, messaging systems either install corresponding protections or disable the commands.

VRFY and EXPN commands

VRFY is used to verify whether a mailbox in the argument exists on the local host. The server response includes the user's mailbox and may include the user's full name. Example: VRFY user2 250 Samantha Smith [email protected] (server response) EXPN is used to verify whether a mailing list in the argument exists on the local host. The positive response will specify the membership of the recipients. Example: EXPN mail-list [email protected] (server response) [email protected] (server response) [email protected] (server response)

Does webmail use POP3 or IMAP?

Webmail clients may use either method, but the retrieval protocol is often not a formal standard (typically WEBMAIL OVER HTTP is used) EG, IMAP and POP3 are both disabled in GMAIL by default. Gmail likely stores gmails in their data centers.

How does SMTP mail queuing work?

When an MTA makes a delivery to an SMTP server, where the message is destined for multiple recipients, and one or more recipients cannot accept the message for some reason, the SMTP server takes the responsibility of queuing the message to deliver it later, and reports an overall successful delivery to the MTA.

DATA command

With the DATA command, the client asks the server for permission to transfer the mail data. The response code 354 grants permission, and the client launches the delivery of the email contents line by line. This includes the date, from header, subject line, to header, attachments, and body text. A final line containing a period (".") terminates the mail data transfer. The server responses to the final line. Example: DATA 354 (server response code) Date: Wed, 30 July 2019 06:04:34 From: [email protected] Subject: How SMTP works To: [email protected] Body text .

MAIL FROM command

With this SMTP command the mail transfer begin: the sender states the source email address in the "From" field Example: MAIL FROM "[email protected]"

Why is port 25 blocked by ISPs?

an ISP (Internet Service Provider) may block port 25 in order to prevent spamming by its customers. ISPs block clients from using destination port 25 to submit mail Note that most mailbox providers still allow submission on traditional port 25.

What may the response to an EHLO command be, from an ESMTP server?

server will respond with success (code 250), failure (code 550) or error (code 500, 501, 502, 504, or 421), depending on its configuration. A RFC 821 compliant server returns error code 500, allowing ESMTP clients to try either HELO or QUIT.