Unix network programming

What is thundering herd problem ?

All N children are awakened, even though only one will obtain the connection. It has negative impact on the number of spawned children within preforking.

What is IO multiplexing ?

Capability to tell the kernel that we want to be notified if one or moreI/O conditions (descriptors) are ready. It is provided by select(), poll() and newer posix variant called pselect().

Describe model "TCP Prethreaded Server, per-Thread accept"

Create a pool of threads and then let each thread call accept. Instead of having each thread block in the call to accept, use a mutex lock that allows only one thread at a time to call accept.

What are our choices for signal disposition ?

1. We can provide handler. 2. We can ignore the signal. 3. We can set the default disposition.

What are categories of ports ?

1. Well-known ports: 0-1023. Port numbers controlled by IANA. 2. Registered ports. 1024-49151. IANA lists uses of these ports as a convenience to the community. 3. Emphemeral ports: Dynamic private ports, short lived ports. These port numbers are assigned by transport protocol to the client.

How can we limit duration of connect ?

We may use non-blocking connect and specify the limit to select().

How can we design concurrent server ?

1. It can fork to spawn a child for every client. 2. Single process can use IO multiplexing. 3. It can use model one thread per client instead of process per client. Preforking has the server call fork when it starts, creating a pool of child processes. One process from the currently available pool handles each client request. Prethreading has the server create a pool of available threads when it starts, and one thread from the pool handles each client.

What is the difference between shutdown() and close() function ?

1. close decrements the descriptor's reference count and closes the socket only if the count reaches 0. With shutdown, we can initiate TCP's normal connection termination sequence , regardless of the reference count. 2.close terminates both directions of data transfer, reading and writing. Since a TCP connection is full-duplex, there are times when we want to tell the other end that we have finished sending.

Categorize non-blocking calls

1. input ops: they block if there is no data and return EWOULDBLOCK. 2. output ops: they block if socket's buffer is full and return EWOULDBLOCK. UDP sockets do not need buffer, but may block on some internal kernel struct. 3. accept(): blocks if there is no new connection available and returns EWOULDBLOCK. 4. connect() for TCP: connect function does not return until the client receives the ACK of its SYN. Returns EWOULDBLOCK or EAGAIN.

What buffers are involved when an application writes to a TCP socket ?

1. socket send buffer 2. datalink output queue Each datalink has an output queue. If the output queue is full, packet is discarded and an error is returned up the protocol stack. TCP will try to send the segment later. Application is not told the transient condition.

Give example of preforking architecture.

Apache preforks its children and then uses either the technique in the previous section (all children blocked in the call to accept), if the implementation allows this, or file locking around the accept.

What is the purpose of bind() ?

Assigns a local protocol address to a port. With TCP, bind lets us specify a port number, IP address, both, or neither.

What happens if server crashes while client is blocking in read () after it wrote some data ?

Assuming the server host crashed and there were no responses at all to the client's data segments, the error is ETIMEDOUT. But if some intermediate router determined that the server host was unreachable and responded with an ICMP "destination unreachable' message, the error is either EHOSTUNREACH or ENETUNREACH.

What are Unix models of IO operations ?

Blocking IO, nonblocking IO, IO multplexing, signal driven IO, asynchronous IO.

What is the recommended way to handle SIGPIPE ?

Depends on the application. If there is nothing special to do, then setting the signal disposition to SIG_IGN is easy, assuming that subsequent output operations will catch the error of EPIPE and terminate. If special actions are needed when the signal occurs (writing to a log file perhaps), then the signal should be caught and any desired actions can be performed in the signal handler. Be aware, however, that if multiple sockets are in use, the delivery of the signal will not tell us which socket encountered the error. If we need to know which write caused the error, then we must either ignore the signal or return from the signal handler and handle EPIPE from the write.

What return slow functions in non-blocking mode if their condition is not ready ?

EWOULDBLOCK

What does close() do ?

Marks the socket as closed and returns to process. The socket descriptor is no longer usable BY THE PROCESS. Close decreases reference count of the descriptor. TCP tries to send any queued data to the other end.

Describe common 32-bit and 64-bit programming models of UNIX systems (sizes of integers...).

ILP32: Integer, Longs and Pointers are 32bit. LP64: Longs and pointers require 64 bits. With LP64, we can not assume that a pointer can be stored in an integer.

What happens if a process dies without taking care of its zombies ?

If a process terminates, and that process has children in the zombie state, the parent process ID of all the zombie children is set to 1 (the init process), which will inherit the children and clean them up (i.e., init will wait for them, which removes the zombie). They take up space in the kernel and eventually we can run out of processes.

What is zombie state ?

If child terminates, the purpose of the zombie state is to maintain info about the child for the parent to fetch at some later time.

How can client detect that server has crashed ?

If client wrote some data, client TCP continually retransmits the data segment, trying to receive an ACK from the server. Client hits the rexmit limit. If we want to detect the crashing of the server host even if we are not actively sending it data, keepalive messages are required.

What happens if multiple processes blocks on the same decriptor using select() ? Which are woken up ?

If multiple processes are waiting for the same descriptor, the kernel must wake up all processes that are blocked in a call to select since it doesn't know which processes are affected by the descriptor that just became ready.

What are interrupted system calls and how we handle them ?

If the process is blocked in some system call and the signal is caught, system call is interrupted with EINTR. Process must handle this error (we restart the function). Rule: When a process is blocked in a slow system call and the process catches a signal and the signal handler returns, the system call can return an error of EINTR. Some kernels automatically restart some interrupted system calls. For portability, when we write a program that catches signals (most concurrent servers catch SIGCHLD), we must be prepared for slow system calls to return EINTR.

What are non-fatal errors that accept() can return and how do we handle them ?

Interrupted system call (EINTR) and ECONNABORTED (for POSIX). ECONNABORTED means that 3-way handshake was completed, but then the connection got RST.

What is an iterative server ?

It completely services the current client before processing to pending clients.

What does listen() do ?

It indicates, that kernel should accept incoming connection requests directed to this socket. The kernel will complete the three-way handshake for any additional clients, up to the listen backlog for this socket, and then pass the completed connections to the server when it calls accept.

What is a function connect() in case of TCP socket ?

It initiates three-way handshake.

What happens if the process writes to the socket socket that received FIN ?

It is okay to write to a socket that has received a FIN, but it is an error to write to a socket that has received an RST.

What is the SIGCHLD signal ?

It is sent to the parent when the server child terminates.

How can process wait for its children ?

It uses wait() and waitpid(). If there are no terminated children for the process calling wait, but the process has one or more children that are still executing, then wait blocks until the first of the existing children terminates. waitpid gives us more control over which process to wait for and whether or not to block. First, the pid argument lets us specify the process ID that we want to wait for. A value of -1 says to wait for the first of our children to terminate. (There are other options, dealing with process group IDs, but we do not need them in this text.) The options argument lets us specify additional options. The most common option is WNOHANG. This option tells the kernel not to block if there are no terminated children.

What is the goal of backlog argument of listen function ?

Kernel maintains two queues for a listening socket: 1. sockets in SYN_RCVD state 2. completed connection queue: ESTABLISHED sockets Historically, backlog limited size of both queues. Now it depends on implementation and it is difficult to say what the parameter means. If the queues are full, TCP ignores incoming SYN and does not send RST, as it is temporary condition.

What is signal IO ?

Kernel notifies us with the SIGIO signal when the descriptor is ready.

Do all architectures support preforking model without locking ?

No, it works only with Berkeley-derived kernels that implement accept within the kernel. System V kernels, which implement accept as a library function, may not allow this. The solution is for the application to place a lock of some form around the call to accept, so that only one process at a time is blocked in the call to accept. The remaining children will be blocked trying to obtain the lock.

Can server refuse the client before finishing 3way handshake ?

No.

Does client call bind() ?

Normally no and kernel assigns ephemeral port, but it can.

What is UNIX signal ?

Notification to process that some event has occurred. They occur asynchronously. They can be sent by process to another process (including itself) or by kernel.

What is reserved port ?

On Unix system, port less than 1024. The port usually requires superuser privileges.

Describe model "TCP Preforked Server, Descriptor Passing".

Only the parent call accept and then "pass" the connected socket to one child. This gets around the possible need for locking around the call to accept in all the children, but requires some form of descriptor passing from the parent to the children. This technique also complicates the code somewhat because the parent must keep

What happens if non-block connect is successful and we are waiting for it in select() ?

POSIX has the following rules: 1. When the connection completes successfully, the descriptor becomes writable. 2. When the connection establishment encounters an error, the descriptor becomes both readable and writable. Unfortunately, nonblocking connects are one of the most nonportable areas of network programming. Be prepared for portability problems, especially with older implementations. A simpler technique is to create a thread to handle a connection.

What is the Single Unix Specification ?

Standard that unifies POSIX and The Open Group UNIX systems.

How does TCP demultiplex packets arriving to the same server ?

TCP has to look at all four elements in the socket pair to determine which endpoint receives an arriving segment: IP addresses and ports.

What happens if the process writes to the socket that received RST ?

The first write elicits the RST and the second write elicits the signal. When a process writes to a socket that has received an RST, the SIGPIPE signal is sent to the process. The default action of this signal is to terminate the process, so the process must catch the signal to avoid being involuntarily terminated. If the process either catches the signal and returns from the signal handler, or ignores the signal, the write operation returns EPIPE.

Describe model "TCP Concurrent server, one thread per client".

The main thread blocks in a call to accept and each time a client connection is returned, a new thread is created by pthread_create.

What are disadvantages of architecture that spawns child ?

There is number of child processes. The amount of CPU time it takes to fork, this can be reduced by preforking, but then there is thundering herd problem.

What are signals SIGKILL and SIGSTOP ? Can we catch them ?

They can not be caught.

What buffers are involved when an application writes a UDP socket ?

UDP socket has a buffer, it's size is the upper limit on the max-sized UDP datagram that can be written to the socket. The UDP datagram is not split into MSS sized segments as in TCP.

How do we set socket to non-blocking ?

Using fcntl. At first we obtain socket's flags and then we add flag O_NONBLOCK.

What is the purpose of SIGTERM and SIGKILL ?

When a Unix system is shut down, the init process normally sends the SIGTERM signal to all processes (we can catch this signal), waits some fixed amount of time (often between 5 and 20 seconds), and then sends the SIGKILL signal (which we cannot catch) to any processes still running. This gives all running processes a short amount of time to clean up and terminate. If we do not catch SIGTERM and terminate, our server will be terminated by the SIGKILL signal.

When does blocking connect() return, how long does it take ?

When the callers gets ACK of its SYN packet. It takes one RTT.

Describe how preforking works.

When the program starts, N children are created, and all N call accept and all are put to sleep by the kernel. When the first client connection arrives, all N children are awakened. This is because all N have gone to sleep on the same "wait channel," the so_timeo member of the socket structure, because all N share the same listening descriptor, which points to the same socket structure. Even though all N are awakened, the first of the N to run will obtain the connection and the remaining N - 1 will all go back to sleep, because when each of the re

How do we avoid zombies ?

Whenever we fork children, we must wait for them to prevent them from becoming zombies. To do this, we establish a signal handler to catch SIGCHLD, and within the handler, we call wait.

What is difference between asynchronous IO and signal driven IO ?

With signal-driven I/O, the kernel tells us when an I/O operation can be initiated, but with asynchronous I/O, the kernel tells us when an I/O operation is complete.

What does accept() do ?

accept is called by a TCP server to return the next completed connection from the front of the completed connection queue. If the queue is empty, the process is put to sleep (blocking socket). accept takes listening socket as an argument and returns brand new descriptor for connected socket.

Describe model "TCP Prethreaded Server, Main Thread accept"

main thread create a pool of threads when it starts, and then only the main thread calls accept and passes each client connection to one of the available threads in the pool.

Name elementary socket functions for TCP server

socket(), bind(), listen(), accept(), read(), write(), close()

Name elementary socket functions for TCP client.

socket(), connect(), write(), read(), close()

How do we wait for multiple child processes ?

wait() blocks if there are running children that have not yet terminated. It can not be used in loop. Non-blocking waitpid() must be used within loop.