Application layer
Mail protocols (SMTP15, MIME16, POP317, IMAP18 and MOTIS
Electronic Mail was also one of the earliest Internet application protocols to be designed. It is a store and forward text messaging protocol supporting mail clients (user agents that send and receive messages) and mail servers that relay messages to each other and to and from mail clients. Email clients have five basic functions: • composition, which allows users to create messages • transfer, which allows users to transfer messages to and from the mail server • reporting, which allows the mail server to indicate such things as a message not having been delivered • displaying, which allows the mail client to display the headers of messages and their contents • disposition, which allows the user to delete messages or store them in folders. Email clients use two different protocols. One for sending messages to the mail server and one for retrieving messages from the mail server.
Ecoplexing requirement
It required a full-duplex link, but had the advantage of showing users when noise on the line was corrupting data. It is not well suited to modern packetswitched networks because there is a delay between typing a character and seeing it appear on the screen, and each character will be transmitted to and from the host in a single packet, incurring large protocol overheads. Also, with modern WAN links, the probability of characters being corrupted is quite low. Because of this, it is advisable to turn off the echo function at the remote server and enable a local echo at the client.
eal time applications are more tolerant to packet loss than they are to delay. Losses of occasiona
Losses of occasional packets will not make much difference to audio or video transmissions, as losses are relatively infrequent and the applications can interpolate missing data, so that losses can be hidden from users. Delays, however, and particularly variable delays, do cause problems that can be observed by users. They can result in a jerky effect which is quite disconcerting. Using TCP, as we will discover later, does give rise to extremely variable delays, while delays with UDP are less severe and are more consistent.
Ecoplexing
A further potential problem with Telnet that makes it very inefficient over WANs is that it was designed for use over asynchronous modem links. The standard method of remotely accessing a host at the time was for a character to be sent to the host and the host to echo it back to the terminal before it was displayed. This method of operation is known as echoplexing.
Domain Name System (DNS)
Access to directories is required by a number of network functions, as well as by other applications. A directory service called the Domain Name System (DNS) was developed for the Internet to allow applications to use host names, and then for these host names to be translated (or resolved) into network layer addresses. In the early days of the Internet, the mapping between flat host names and network layer addresses was done via a text file that was centrally managed and then distributed to all the hosts on the Internet. This method soon became too unwieldy as the Internet grew in size, and an alternative method using a hierarchical, fully distributed system was devised called the Domain Name System.
Addressing
Addressing is often thought of as a function of lower layer protocols, but many application protocols do require their own addressing function as well as having to pass down addresses to be used by lower layer protocols.
Internet Mail Access Protocol (IMAP)
An alternative protocol for retrieving messages from mail servers is Internet Mail Access Protocol (IMAP) which offers very similar functions to POP3, but also allows users to view message headers and select which messages to download. This is very useful for a dial-up connection where bandwidth is expensive and should not be wasted by downloading spam or other unwanted messages. IMAP also allows messages to be stored in folders on the server, which is particularly important if the user often accesses his email from different machines.
Data encoding
An important function of the application layer is to determine how data is to be encoded for transmission. This may involve the choice of character codes,6 the use of tags to define how data is displayed7 or to define data syntax or semantics.8 Abstract Syntax Notation 1 (ASN.1) has become an important standard for specifying PDU formats at all layers. It is an ISO standard, but it has also been used extensively by the IETF in specifying new Internet based protocols. It can be thought of as a type definition language where data is defined as belonging to either primitive types such as Boolean, Integer or Bitstring, or more complex user-defined types. It is similar to data type declarations in programming languages. ASN.1 is not only used for specifying protocols
application client simply retransmits its request. • If applications are very security conscious, they will not trust the transport service or anything else that was developed or is managed by other partie
The designers of such applications will want to detect and recover from errors within the application itself. In this case, it would be pointless to replicate this functionality in the transport layer and it would be much more efficient to use an unreliable transport protocol.
Functions of app layer
The main function of the application layer is to organise the necessary resources to allow an application process on one system to communicate with an application process on another system via a network. The application layer may also synchronise the application processes at both ends so that they can communicate successfully.
VT
Virtual Terminal (VT) is the ISO equivalent protocol that offers similar functionality to Telnet, but like many other ISO protocols, it has found it difficult to compete with the protocols designed for use on the Internet.
VTP
Virtual terminal protocols (Telnet,11 SSH and VT)
Error control
Where an application is using an unreliable transport service (or where the application does not trust a reliable transport service), the application must perform its own error detection and recovery. This will require a redundant error checking field in the application protocol header as well as sequence number and acknowledgement fields, so that the application layer can check that all the PDUs have been received and so that retransmission can be requested and PDUs re-ordered if necessary.
Post Office Protocol 3 (POP3)
One protocol that does this is Post Office Protocol 3 (POP3). The protocol has three phases. It has an authentication phase where the user is authenticated by a user name and password; a transaction phase where messages are downloaded from the mail server; and an update phase where the messages on the server will be deleted (if required) after they have been successfully downloaded to the client.
Security
Security is often an important function of the application layer, as many applications will assume that all networks are insecure, and application layers sometimes prefer to implement security at this level, rather than make use of security functions within the transport and network layers. Schemes are required to authenticate the parties involved in the communication and to prevent any other parties from being able to read or alter the data being transmitted.
Services of application layer
Services The application layer is the layer that provides communications functions for a network application to serve an end user or another application program
Telnet not used
Telnet is not commonly used today because of security concerns, but it is still often used by network managers to remotely manage and configure routers. Network managers can make Telnet much more secure by configuring the routers to only accept Telnet sessions from known IP addresses, corresponding to the network management workstations.
Telnet relays
Telnet just relays any characters typed by a user to the remote host, and allows commands to be entered at the remote host as if they came from a local terminal. It then relays any characters sent in response back to the user. These characters will include any user name and password requests, but it should be noted that passwords will be transmitted as plaintext and the protocol therefore is not secure.
Telnet messags consist of
Telnet messages do not have application layer headers. They consist of the characters being typed by the user or sent by the server together with occasional command codes generated by the Telnet client or server.
Telnet
Telnet was one of the original Internet application protocols. It is an example of a virtual terminal protocol that allows users on a character-mode terminal (or more commonly these days on a PC running a terminal emulator) to log into and execute commands on a remote host using a command line interface. It uses a reliable TCP connection.
Application layer and reliable communication
The application layer will often provide reliable communications to the application processes, especially when the application layer makes use of an unreliable transport service. The application layer is the layer of last resort which must correct all the problems that have not been dealt with by the lower layers. The application layer must therefore, unless it is using a reliable transport service, be able to detect the loss, corruption and duplication of messages and be able to recover from these problems. It must also be able to control the flow of data if the receiver or the network cannot handle the rate of data being transmitted.
Connection control
The connection control function is required in all application protocols that make use of a connection-oriented transport service. The application layer must be able to establish connections prior to transmitting data and to close them when there is no more data to be transmitted. The application layer must identify and determine the availability of the application processes which wish to communicate and establish their authority to do so. It must also determine the mode of communication (simplex, half duplex or full duplex). A facility is also required so that connections can be reset or reinitialised to a known state should serious problems be encountered.
Connection used for commands
The connection used for commands actually uses the Telnet protocol to transfer commands and responses to the remote host. For security purposes, hosts normally require a user name and password to be entered for all Telnet sessions. But this would hinder general public FTP access to a site, as users would have to pre-register. A convention that developed for public FTP access was for public FTP servers to accept the user name 'anonymous', but not to perform a password check. Instead, anonymous users are expected to enter their email address as the password, so that the host can, if it wants, collect some details on the users of its public FTP service.
SSH
Another way to improve security for remote login type applications is to use Secure Shell (SSH). The functionality of SSH is very similar to Telnet, but user names and passwords are encrypted for transmission from SSH clients to SSH servers.
Hyper-Text Transfer Protocol (HTTP)
Another, and increasingly popular method for sending and receiving email messages is to use the world wide web to access mail services such as Hotmail, making use of the web's Hyper-Text Transfer Protocol (HTTP) rather than using mail protocols. As with IMAP, messages can be organised in folders on the server. Web-based email has the advantage of allowing users to access their email from any machine that supports a web browser, such as a PC in an Internet Café.
Flow control
Applications which do not make use of a reliable transport service will also require an end-to-end flow control mechanism so that the receiver can regulate the flow of data from the transmitter. To do this the application protocol header will require a sequence number and an acknowledgement f ield so that the receiver can acknowledge each PDU transmitted. It can then slow the rate of transmission by not acknowledging PDUs until it is ready to receive some more.
Multipurpose Internet Mail Extensions (MIME)
Because SMTP was designed only to carry ASCII 7-bit characters, it cannot on its own be used to transfer 8-bit binary data which would be required if an executable file or a formatted text file (such as a Word document) were to be transmitted. When the designers of SMTP realised that users wanted to the ability to send data other than ASCII text, they were faced with two possible solutions. Either they could change the SMTP protocol and update all the clients and servers at the same time, or they could just create a new protocol to allow 8-bit data to be encoded as 7-bit ASCII characters. They chose the latter option, as it only involved upgrading the clients, and they designed a new protocol called Multipurpose Internet Mail Extensions (MIME) which allowed 8-bit data files to be attached to SMTP messages and be transmitted as 7-bit ASCII characters.
Data compression
Data compression is often required because bandwidth in wide area networks is a scarce (and hence expensive) resource, and some types of data (such as voice and video) require a large amount of bandwidth, although they use that bandwidth quite inefficiently. Typically voice and video signals (and to some extent also text) contain a large amount of redundant information and can be coded much more efficiently using data compression algorithms. These algorithms can either be lossy,9 where information cannot faithfully be reproduced at the receiver, or they can be lossless,10 where the quality of the information after decompression at the receiver is just as good as it was before it was compressed at the transmitter.
Encapsulation
Data from end users or other applications is encapsulated in an application layer PDU by prefixing the data with an application layer header specific to the application protocol.4
Data encryption
Data is often encrypted between application layer entities to ensure that it cannot be viewed or altered by third parties as it is transmitted across networks.
Ftp use
FTP can convert between different character codes. It converts data to the same Network Virtual Terminal (NVT) codes as Telnet for transmission. Unlike HTTP, FTP cannot be a stateless protocol. The FTP server has to remember which connections belong to which FTP users, and also which current working directories each FTP user is using.
1 File Transfer Protocol (FTP)
File Transfer Protocol (FTP) was also one of the original Internet application protocols. It allows three different types of file (unstructured, structured and random) to be transferred over a network between one host and another using a set of simple commands. File transfers are now quite often carried out using HTTP rather than FTP, but FTP is still used not least when web pages are published to a web server. Most web publishing software uses FTP, although the details of it are hidden from the users and few web designers will be aware that they regularly use FTP.
The fundamental decision that designers of applications must take is which transport service to use. It is the transport service that supports the differing levels of service that might be required.
For internet applications, there is a choice between a reliable service using the Transmission Control Protocol (TCP) or an unreliable or best efforts service using the User Datagram Protocol (UDP). The reliable service uses a complex connection oriented transport protocol, and the unreliable service uses a simple connectionless transport service. The choice of what sort of transport service to use will have a huge effect on what functions are required in the application layer. It may seem strange at first to think that some application developers would prefer to use an unreliable transport service, but there are several reasons why this might be the best choice.
Interfaces in app layer
For user agents, the interface to the application layer is today likely to be a Graphical User Interface (GUI), such as that provided by the Windows operating system. It could also be a command line interface, as provided by DOS or Unix. Application layers may also provide services to other applications by means of an Application Programming Interface (API), which will provide a set of library functions that can be called from application programs.
Characters of http
HTTP and HTML were studied in a Level 1 unit. You are strongly advised to revise this material, as the remainder of this section on web protocols will assume this knowledge and build upon it. HTTP assumes a client server model for communications. The web browser acts as the client and requests web pages from the web server. The page is referenced by a Uniform Resource Locator (URL) which can be thought of as an application layer address. It defines the protocol to be used, the location of the server and the file to be transferred as well as the transport layer address to be used. A URL has the following structure: protocol://hostname/filename:port number13
Http and security
HTTP has a simple security mechanism that developers can implement to help prevent unauthorised access to web pages. A web page can be set up so that authorisation is required. In this case the web server prompts the client for a user name and password. The server requests a user name and password with a 405 Authorization Required Response and the browser prompts the user for this information. Once the browser has obtained a user name and password it resends the request but this time includes the user name and password in the request header. The server will then check this and if satisfied, will download the page. The browser will cache the user name and password and will automatically include them in any further requests to the server during that browser session.
User agent
If an application layer entity is providing a service direct to an end user, then the software that provides the interface between the end user and the networked application is described as a user agent.
Mime
MIME defines a number of standard data types and sub-types.19 These MIME data types have become a standard in many Internet and other applications. SMTP assumes that mail servers operate continuously and are always available. If they are not available, the messages will be stored and forwarded when the mail server becomes available. Clients, however, are frequently not available, as users do not keep their mail clients running all day and often do not have a permanent connection to the Internet. For this reason SMTP is not well suited for delivering messages to clients. Instead, other protocols were designed to allow clients to connect to servers and request that messages are downloaded. Because these protocols were designed to work over dial-up networks, they also require security mechanisms to ensure that mail is being downloaded by valid users.
Message-Oriented Text Interchange Standard (MOTIS)
Message-Oriented Text Interchange Standard (MOTIS) is an ISO messaging standard. It is based on the ITU-T X.400 Message Handling Service (MHS) standard. This standard, unlike SMTP, is very complex and sophisticated and does many things that SMTP does not do. It has not been very successful and virtually all email today still uses SMTP. MOTIS/X.400 is perhaps too complex and users prefer the simplicity of SMTP, particularly with regard to email addresses. X.400 addresses consist of a set of keywords and values which are much harder to remember and more cumbersome to use than SMTP addresses.20
Network application and client server model
Network applications are often implemented as client server systems. Under the client server model, the client only runs when it is required and initiates a request to the server and the server replies with a response.2 The server will typically handle requests from many clients and will run continuously. Both the user agent (client) and the server run application processes that work together via a network to deliver the application service to the end user.
Network Management Protocols (SNMP28 and CMIP)
Network management protocols is another area where there was a battle between Internet and ISO standards which was conclusively won by the Internet standards. Yet again the Internet developers chose to launch a simple but effective network management protocol which could be implemented quickly, while the ISO developers created a technically complex protocol that had many more functions and would take a long time to implement and debug.
Ordered delivery
Ordered delivery is a function of a reliable transport or network service, but where an application is using an unreliable transport service, PDUs can be received in the wrong order as they can take different routes through the network. Where this happens, the application layer protocol must contain a sequence number field so that the application layer can determine if PDUs arrive out of order. The sequence numbers can also be used to request that lost PDUs are retransmitted, and to reorder them if necessary so that they can be handed in order to the appropriate application process or user agent. To do this the application layer needs to buffer the PDUs received, and therefore a certain amount of memory must be allocated for storing incoming PDUs while earlier PDUs are awaited.
Simple Mail Transfer Protocol (SMTP)
Simple Mail Transfer Protocol (SMTP) is used to transfer messages from a mail client (user agent) to a mail server (mail transfer agent) and is also used to transfer messages between mail servers. It is a very simple text based protocol. Messages comprise a set of headers and a body. There are two envelope headers which start with MAIL FROM:, used to identify the message originator and RCPT TO:, used to indicate the recipient(s) of the message. Unlike many more modern protocols, each envelope header is transmitted and acknowledged separately, rather than encapsulating the body. The message itself is prefixed by the keyword DATA followed by the text to be transmitted, but this also has its own headers such as FROM:, TO:, SUBJECT: and DATE: from which the envelope addresses are obtained. These headers are followed by a blank line and then the actual text of the message. The body is terminated by a new line with just a full stop on it andthen another new line. All bodies and headers are coded in 7-bit ASCII text. SMTP is not a real-time protocol. Messages are stored at clients and servers and forwarded at regular intervals using reliable TCP connections. SMTP PDUs do not really have application layer headers as normally understood. They consist of keywords followed by some data. SMTP does not offer any guarantees about delivery of messages, although it is quite robust and considered to be reliable.
Network File System (NFS)
Sometimes it will be more efficient to access files remotely rather than to transfer them in their entirety. A popular means of doing this is to use the Network File System (NFS), originally developed by Sun Microsystems for the Unix environment, but which has since been ported to most other commonly used environments. NFS allows physically remote directories to be mounted on local systems, so that the directories and their files appear to be local to the users. All the standard operations that are carried out by the Operating Systems on local directories and files are supported transparently on the remote directories and files. NFS is implemented using an application mechanism called Remote Procedure Calls (RPCs), also developed by Sun, where software that normally calls procedures on the local system can call equivalent procedures on the remote system. RPCs are implemented using a very simple protocol that packs the name of the procedure and any parameters required into a message using a coding system called External Data Representation (XDR). This is sent as a request to the remote system which unpacks it and calls the procedure. It then packs the return value and any other output parameters into a response message to be sent back to the calling system, which then returns these to the original calling process. NFS can use either TCP or UDP for its transport service but, because it is a simple client server application, it is best implemented on top of UDP. NFS is not very secure, but authentication services have been developed that offer improvements in this area. File Transfer Access and Management (FTAM) is the equivalent ISO protocol to FTP. It is not in common use today.
Telnet snd terminals
Telnet can support many different terminal types and translate between different character codes, if necessary. It does this by translating to a standard format, known as Network Virtual Terminal (NVT) for transmission acros the nereotks
Inband signaling
Telnet command codes can be embedded in the data stream. To achieve this, a special escape character (FF in Hexadecimal) is required so that the receiver knows to interpret the next character(s) as a command code. Embedding control information like this within data is known as in-band signalling
Original FTP
The original FTP implementations were command line interfaces for use on Unix hosts, and allowed users to view directories on remote hosts and the f iles they contained, change directories when necessary using Unix commands and then download or upload files between two hosts.24 FTP converts these user commands to standard FTP commands (three or four letter codes) which are sent via the control connection and elicit responses containing a three digit status code followed by some text from the server. Modern FTP implementation uses a graphical user interface where the local and remote directories can be viewed together and files to be transferred can be highlighted and then transferred by clicking an arrow button that indicates the direction of the transfer.
Service app layer nature
The precise service offered by the application layer will vary from application to application, but will often involve identification of the communicating partners and the agreement of the responsibility for error recovery, security aspects and data encoding. The application layer is also responsible for negotiating and meeting certain quality of service requirements for reliable data transfer, throughput or for delays. Some applications are loss tolerant while others are loss sensitive. Some applications are bandwidth sensitive
Protocol that used by FTP
The protocol actually makes use of two TCP connections: one for control commands and responses, and the other for actual file transfer. The control connection is left open for a whole FTP session, while the data connection is established to transfer a file and closed as soon as the file has been successfully received. This method of control using one connection for control and another for data is called out-of-band signalling as opposed to other protocols such as Telnet and HTTP which use in-band signalling. Because all control information in FTP is passed via the control connection, the data connection does not require any application layer headers and is a simple TCP connection.
Web protocols (HTTP12 and HTML)
The world wide web, Hypertext Mark-up Language (HTML) and the Hyper-Text Transfer Protocol (HTTP) were all invented by Tim Berners-Lee at CERN in 1989. The world wide web is now the most important and widely used Internet application.
Compression
There are many different complex data compression algorithms used for coding data prior to transmission to conserve network capacity. All of them apart from Huffman Coding are beyond the scope of this syllabus. Huffman Codes use variable length codes for different symbols depending on how frequently they are used. The ASCII character set uses 8-bit codes (including a parity bit) to define all the characters of the alphabet and other characters. But some characters occur much more frequently than others. Huffman Codes allow more frequently used characters to be represented by fewer bits, and less frequently used characters to be represented by more bits. By doing this a significant reduction can be achieved in transmitting a large amount of text. But, if characters are represented by variable length codes there must be a clear method for determining the start and end of the code for each character. Huffman Codes do this in a clever way by coding each character as the path from the root to a leaf of a binary tree called a Huffman Tree. Thus messages can be encoded in an unambiguous way, so that the receiver can always decode the message and knows that when it reaches a leaf node, a character has been received.
Huffman tree
There are many different complex data compression algorithms used for coding data prior to transmission to conserve network capacity. All of them apart from Huffman Coding are beyond the scope of this syllabus. Huffman Codes use variable length codes for different symbols depending on how frequently they are used. The ASCII character set uses 8-bit codes (including a parity bit) to define all the characters of the alphabet and other characters. But some characters occur much more frequently than others. Huffman Codes allow more frequently used characters to be represented by fewer bits, and less frequently used characters to be represented by more bits. By doing this a significant reduction can be achieved in transmitting a large amount of text. But, if characters are represented by variable length codes there must be a clear method for determining the start and end of the code for each character. Huffman Codes do this in a clever way by coding each character as the path from the root to a leaf of a binary tree called a Huffman Tree. Thus messages can be encoded in an unambiguous way, so that the receiver can always decode the message and knows that when it reaches a leaf node, a character has been received.
Trivial File Transfer Protocol
There are some situations where a complex file transfer protocol is inappropriate. Some network devices, such as low-cost routers, do not have sufficient memory and processing capability to justify the implementation ofsuch a protocol, if it has to sit on top of a complex connection-oriented transport protocol. Also, between two hosts on the same LAN the probability of errors is quite remote and it may be preferable to use a lighter weight file transfer protocol
Trivial file
disc-less work station, which has to download all its software from a server over a LAN, is a good example of a situation where only a light-weight file transfer protocol is desirable. Trivial File Transfer Protocol (TFTP) is such a protocol. As its name implies, it is extremely simple. It uses UDP as its transport service and provides quite a thin application layer. Each application layer message has to be explicitly acknowledged before another message can be sent. Each TFTP message carrying data contains an application header that includes a sequential block number. The receiver will then acknowledge the receipt of the message with an acknowledgement message that contains the block number. The transmitter will re-transmit a message if an acknowledgement is not received before a timeout expires. Unlike FTP, TFTP only supports file transfer. It does not support any interaction to locate files in directories. The files and the direction of transfer are specified in the command line that is used to call TFTP. TFTP also has no facility for authenticating users. For security purposes, network managers should therefore only allow TFTP traffic to and from known IP addresses. TFTP is often used by network managers to download or upload router configurations and software.
The client server model is well suited to using a connectionless transport service. If clients make occasional request to servers, then using a reliable connection-oriented service, such as that provided by TCP, can be very inefficient.
will be necessary to set up a connection and close it down afterwards. This will require a minimum of five packets and the server will have to hold state information in its memory about all the transport connections that are currently active. Communications operate much more efficiently, as do applications, if a connectionless transport service is used. There will only be a need for two packets to be exchanged and the server can minimise memory usage as it does not need to hold any state information about connections. If packets are lost or corrupted the application client simply retransmits its request.
