Distributed Systems

Consistency Security Issues

1. Integrity 2. Non-repudiation

Types of Measures of Failure Rate

1. Mean Time Before Failure (MTBF) 2. t-Fault Tolerance

Parts of a Message

1. Message text 2. Sequence number of the message 3. Identifier of the transmitter

Directory

A list of bindings

Binding

Association between different names

Handling Failure Modes

Failstop: easiest to handle, because we know that there is something wrong, so just throw an exception and fix it Crash: give a timeout period and if it hasn't responded by the end of the period, assume it has crashed and deal with it like a Failstop. This only works in synchronous systems, because in asynchronous systems the timeout method may not work because it may just be slow and not have crashed

Causal Odering

If a message is broadcast after another message is delivered at the same process, then every other process must also follow this ordering of the first message being delivered after then second

Migration Transparency

Moving a resource between hosts does not affect the logic needed to access the resource

t-fault Tolerance

Must be more than t failed components before a service fails Can be a more intuitive measurement Tells us that we can suffer t failures before losing a service

Namespace

Range of names supported

Reliability

System should always be in working order

Adaptability

System should be modifiable to account for changes is use and technology over time

Can you be causal but not total?

Yes, because total is stronger

Specifying a transaction

1. A transaction forms a program executed by a transaction manager 2. Often this program is specified using SQL 3. Program specifies a transaction type 4. Each execution produces a transaction 5. Unless the distinction is important transaction types often referred to as just transactions

Types of Transparency

1. Access transparency 2. Location transparency 3. Concurrency transparency 4. Replication transparency 5. Failure transparency 6. Migration transparency 7. Performance transparency 8. Scaling transparency

Reliable Broadcast Properties

1. Agreement 2. Validity 3. Integrity

Reliable Broadcast Properties Implemented

1. Agreement: on delivery of message at a process, the message is resent to all other processes 2. Validity: even if a link between 2 processes breaks, the message will still eventually reach the process through an intermediary process 3. integrity: messages are only delivered if they haven't already been recorded

Assumptions made for transactions

1. Any object is read by a transaction before being written to by the same transaction 2. Each transaction only reads and writes to an object once 3. Every operation can be uniquely identified as a read or write.

Mean Time Before Failure (MTBF)

1. Average period of time between failures, can assume a random distribution of failures, can also measure using the probability of a failure occurring in any period of time

Types of Algorithms that clients use to talk with name servers

1. Client Iterative Navigation: client iteratively goes from one name server to the next to the next until it resolves the name (puts lot of strain on client) 2. Server Recursive Navigation: user asks one name server to resolve the name, if that name server cannot, the name server asks its neighboring name servers if they can resolve the name, and so on recursively until the name is resolved and returned to the client. More work for the server. 3. Server Multicasting: client asks a name server to resolve name, if it can't then that name server iteratively asks each other name server to resolve the name until the name is resolves and returns that name to the client. More work for the server, but as soon as the name is resolved, the process of resolving the name stops, unlike the recursive case where the name could have been resolved and the subtweets are still being explored.

General Replication Model

1. Clients communicate with the service via front ends 2. The service is implemented by a set of replication manager (RMs) 3. Requests are reads if they do not alter the state of a replica, updates/writes if they do

Characteristics of a DS

1. Each host operates independently, so there is no central authority, although hosts may be allocated as authorities on particular tasks 2.Each host keeps time separately, so every host has a clock, synchronization requires an algorithm 3.Hosts fail independently of one another 4. Hosts exchange messages to: share state, use others' resources, notify remote changes. This is both the function of a DS and the means to control the DS

General model for replica manager connectivity

1. Each manager fails independently of others 2. Each pair of managers has its own link which fails independently of other links

Maintaining Consistency Between Replicas

1. Each manager has the same initial state. 2. Order in which requests are processed is the same between replica managers (FIFO, causal, total) To maintain these orderings we need: 1. Agreement between all non-faulty managers about the request they have been sent. 2. Ordering the requests according to the rules We achieve this agreement and ordering through: 1. Primary backup 2. State machine

Types of Message Orderings

1. FIFO 2. Causal (potential) Ordering 3. Total Ordering

Pros of Nested Transactions

1. In centralized systems the throughput of the transaction processing system may increase through the use of nested transactions 2. Because a parent transaction has alternatives in how to react to aborts more situations can be handled by transaction execution 3. For distributed databases with fragmented data nesting is useful in allowing transactions to be broken up into components executed on distinct servers

Advantages of Stateless Approach

1. Nothing is lost when either the client or server crashes, so greater fault tolerance a. If the server fails, it is rebooted, and then clients continue as before b. If a client fails, no action is needed by the server 2. The server does not need to keep tables of information about active users 3. No "too many files open" errors 4. No explicit open or close are required, reducing latency in gaining access to a file

Framework for Atomic commit protocols

1. One coordinator process 2. Several sever processes, which are executing the transaction. 3. At the end of of the transaction, the coordinator asks all servers to vote if they are able to commit the transaction. The servers vote to ether commit or abort. The coordinator commits the transaction only if all servers voted to commit

Security (design consideration)

1. Preventing users from doing things they shouldn't do, according to policy decisions 2. Preventing actions which by accident prevent the correct functioning of the system Prevent the action, not the intention There are: Identity security issues, and consistency of data security issues

Two phase locking

1. Prevents a read or write command if it would cause rules to be violated 2. Works by placing locks on the objects, preventing operations that are not permitted 3. Locks can restrict reading and/or writing

Other Design Considerations

1. Reliability 2. Adaptability 3. Time-criticality

Types of Update Models

1. Synchronous updates: data in replicas is exactly the same in order and content 2. asynchronous updates: data in replicas is identical in content, but not necessarily in order, some messages may be missing from either replica 3. Causal updates: same as asynchronous but with a partial order, so that if M1 causes M2 then M1 should appear before M2 at every replica

Objectives in replication

1. Transparency: issuing a request 2. Consistency: the result is the same regardless of which replicas are used

Total Ordering

1.The order of message delivery is the same at all processes 2. The transmission of messages forms atomic broadcasts

Atomic Broadcasts

A broadcast that requires a guarantee that: 1. A message delivered is delivered to all processes or to none at all 2. All messages are delivered in the same order to all processes

Distributed System

A collection of autonomous computers/devices which perform a task together

Execution of 2 phase commit (2PC)

A common protocol for ensuring atomic commitment. Phase 1: 1. Coordinator transmits C-PREPARE to all servers, informing them that the transaction should now commit 2. A server may reply C-READY if ready to commit. After that point the server may not abort the transaction 3. A server may reply C-REFUSE if it is unable to commit, then abort the transaction Phase 2: 1. If coordinator receives C-READY from all servers it transmits C-COMMIT to all servers. Each server commits. 2. If coordinator receives C-REFUSE from any server it transmits C-ROLLBACK to all servers. Each server aborts.

Conflicts

A conflict occurs in a transaction when it needs to obtain a lock on an object which the locking rules refuse In general we need to make the transaction wait until the other transaction ends its execution and releases the locks

General Omission

A failure mode that is either a send or receive omission

Receive Omission

A failure mode where either a crash or a processor fails to read some of the messages sent to it

Send Omission

A failure mode where either a crash or a processor fails to write some messages required by the protocol

Byzantine Failure

A failure mode where the processor behaves in a manner not permitted by the protocol (bug, Logic failure in the code)

File Systems

A file system is a system which provides a mechanism for the permanent storage of data in a manner that survives system crashes 1. Files are identified by file names 2. Provides access to a named file via primitives (open, read, write, close) 3. A file is modeled as an array of bytes

Waits-for graph

A graph whose nodes are transactions being executed, and whose arcs are those transactions which are waiting for others A cycle in the graph implies deadlock Local transaction managers (LTM) may detect deadlock using these wait-for graphs

Stable message

A message M is stable at a process P if P knows that no other message that should be delivered before M is yet to be received by P

FIFO ordering

A message n+1 may only be delivered to a particular process if message n has already been delivered to that process

Domain

A piece of the divided up system. Each domain had a system manager. What goes into a domain varies: everything associated with a workgroup under one manager, one manager looks after a particular type of resource over the whole domain

Correct Process

A process that has not failed.

Crash

A processor halts and remains halted, but cannot be detected by other processors.

Failstop

A processor halts and remains halted. This halt can be detected by other processors

Relaxing consistency rules

A read request does not alter the state of the resource A read coupd be sent to just one replica manger if: 1. The manager has only failstop failures 2. The request does not cause other requests. Some requests are commutative if: 1. Both requests are reads 2. Updates made on distinct parts of the data

Pending and Stable Requests

A replication manager may be in one of the following states with regards to a request Rci: Initial: not yet received Rci Pending: received Rci, but do not know if it is stable Stable: Rci is known to be stable Processed: Rci has been processed and any changes committed

Failure Transparency

A service remains available even if a resource which supplies that service fails, logic doesn't have to take into account failures, handled by DS

Nested transactions

A transaction which appears in the scope of another transaction i.e. a sub-transaction of a parent transaction Properties: 1. Message transactions obey ACID properties 2. Multiple nested transactions may be executed concurrently 3. The abort of a nested transaction is reported to its parent which may then either: 1. abort itself 2. take other actions 4. A nested transaction always aborts if its parent aborts

Write-through

A write to a local cache (on the side of the client file server) copies the change made in the cache to the remote file server. This helps maintain consistency between the local caches and remote. This does not get rid of all inconsistency, though We fix this and obtain consistency by refreshing all local caches on each read. We implement this by having timestamps for each block On a read, the block's local timestamp is sent to the server, if the block's timestamp is the same or newer than the server's, don't download the data

Concurrency Transparency

Access to a resource is not affected by number of clients, Logic will be the same

Location Transparency

Access to resource does not require specifying its location (where it is stored)

Transaction states

Active: the initial state; the transaction stays in this state while executing Partially committed: right after the final statement has been executed (end of transaction) Failed: after the Discovery that normal execution can no longer proceed Aborted: after the transaction has been rolled back and the database restored to its state prior to the start of the transaction Committed: after successful completion

Resolving Deadlock

After being detected, a deadlock can be resolved by aborting one of the transactions and trying it again later

Consensus

An application of atomic broadcasts. A consensus between processes involves them all excepting a single value based on values they propose

Maintaining Atomicity and durability for transactions

Atomicity and durability can only hold for a certain t-fault tolerance Can be achieved by having copies of actions in a log file allowing: 1. Partially complete transactions to be undone 2. Completed transactions can be redone In a DS, recovery must be coordinated between different hosts

ACID Properties

Atomicity: all requests happen or none of them do Consistency: consistent input leads to consistent output Isolation: all transactions are independent of any other Durability: once complete system failure will not lose results

Name Context

Based on a path within a naming hierarchy, a name can be: 1. Context-dependent: may resolve to a different identifier depending on the context. Only the context name together with the object name forms a unique identifier 2. Absolute always resolves to the same identifier regardless of context

Distributed Locking Algorithm

Because conflicts only involve write locks, if t replica managers fail (are in some kind of conflict) you only need one more replica manager to be able to detect the conflict, so you need t+1 hosts in a t-fault tolerant system If there are n hosts and no failures: If we have placed read locks on an object at k hosts, place write locks on the object at n-k+1 hosts If there may be t failures as well: Write locks must be on n-k+1+t hosts

Disadvantages of 2PC

Blocking: a process will block while it is waiting for a message. Locks get held by the blocked process, causing competing processes to have to wait for the locks to be released Conservative behavior: all processes must commit or none. Therefore biased to abort rather than commit. If any one process fails, it leads to a time-out, which always leads to abort, so favors aborting.

Remote Access

Changing the configuration and monitoring the behavior of the system can be performed remotely

When to lock

Choice of when to lick affects transaction throughput In practice: 1. Locks obtained only as required 2. Growing phase covers most of the transaction 3. Common for all locks to be held until commit or abort, so locking is strict

Non-repudiation

Clients cannot deny communication took place

Confidentiality

Clients may limit the access to the information they own: data confidentiality, traffic confidentiality

Primary Backup Approach

Clients only communicate with one manager. This manager is called the primary. Other managers are back ups, which the primary updates.

Host

Computer or device that contains one or more CPUs, memory, and possibly disk storage

Problem with Causal ordering implementation

Concatenation of all messages can lead to sending around very long messages Solution: concatenate the message identifiers instead of entire messages, or keep track of deliveries instead of broadcasts (see lecture 2 discussion slide)

Concurrency (design consideration)

Concurrency is important because: 1. Even if the DS is comprised only of hosts running a single process, the distributed system as a whole will be multi-process 2. Multiple clients may try to access a service 3. Duplicated resources may provide equivalent services So issues similar to those in multi-process operating systems must be addressed: locking, synchronization

Duplication of Data

Copies of the same data are maintained at several hosts. Two types of duplication: 1. Replication: maintain copies in a proactive manner 2. Caching: maintain copies in a reactive manner Duplication of data is more complicated to manage than fragmentation

Names

Data items used to refer to resources

State Information

Data that alters the resource's future responses to requests (e.g. The State of a file server is the data held on disk, alterations to the state alter the response given to a read request)

Maintaining Consistency for transactions and databases

Databases will initially be in a consistent state. When we cannot trust transactions, we use consistency constraints: 1. Checked at the end of each transaction 2. If violated than the transaction should abort. This is a database problem and changes little in a DS

Types of Conflicts

Deadlock: two transactions are waiting for a lock on an object which the other has a lock on Livelocks : some objects are in high demand, and one transaction finds that some other transaction always obtains the lock first.

Failure Modes

Different ways a system can fail. We can evaluate systems based on how well they perform in the presence of different kinds of failures 1. Failstop 2. crash 3. crash+link 4. Receive omission 5. Send omission 6. General omission 7. Byzantine failure

Data Distribution Models

Distributing a service means replicating hardware/or distributing data. This is done in 2 ways: 1. Fragmentation: the data is divided into parts and split between hosts 2. Migration: the data moves from one host to another over time

Implementing Resolution

Each request for a resolution of a name requires a request upon the NS being made over the network. This leads to heavy load on the network and NS. It is not scalable or fault tolerant. To reduce the load on the network and NS: 1. Use caches on local hosts to store the results of lookup requests. Cache shared by all processes 2. Spread the directory amongst several NS, either through replicating or partitioning

Lower Bounds on Primary Backup Replication

Failstop: in the worst case scenario, only one manager is left working, this manager realizes from the lack of activity from the other mangers that it must be the primary, so you need a number of replicas greater than T Crash: same as failstop Crash+link: in a crash+link failure, managers may remain active, but because links have broken they cannot know whether each other has crashed. A new primary may be elected while the original is still active leading to multiple primaries. So for t-fault tolerance, we need t backups, so more than t+1 replica managers Byzantine failure: can lead to total failure of a primary backup protocol, so no t-fault tolerance is possible for this kind of failure. Send omission: same as Failstop and crash Receive omission: 3t/2 rounded down General Omission: more than 2t replica managers

Handling Failure in the Primary Backup model of Replication

Failure: occurs when the primary manager itself fails, causing all requests to be ignored for a period of time called failover time. If the failure is a Failstop or the system is synchronous, then one of the backup servers can tell that the primary has failed and can take over the role of the primary. This scenario is called a failover. Changing one of the backup servers to the primary is called primary change. To handle this primary change, the front end must detect outages and the re-transmit the requests to the new primary. Requests are lost if the primary replies to them before it has received replies from backups

Fault Isolation

Fault occurring in one resource does not affect operation of other resources (need hardware redundancy)

Services

File Service: 1. Handles the basic operation of reading and writing data from a file 2. Works with binary names 3. Deals with details of physical layout of data Directory Service: 1. Handles structuring of files into some user oriented catalogue system 2. Works with symbolic names 3. Provides a map between symbolic and binary names

Stateless File System

For a DS, the state of a file system is split between client and server, so maintaining consistency can be difficult A stateless file system requires each request to the server with the file on it to be self-contained, providing: 1. The identification of the user for authorization 2. The identification of the file to operate on (file name) 3. The position in the file on which to operate (cursor) 4. The operation to perform (primitive) 5. Any data associated with the operation

Virtual File System

For a client program to access both local and remote files, on a distributed file system, access transparency is violated, so The answer is to create a virtual file system layer. Operations on files in local file system are simply passed through the local system. Operations on files in a remote directory are processed by the virtual file system layer, which maintains state info for remote file access. Client user process sees, via the virtual file system, a uniform file model, creating access transparency

Serially equivalent

For a combined sequence to meet the ACID properties, The result of execution must be the same as executing one transaction then the other. Such a sequence is called serially equivalent

How we know if the message is stable

For a request RAi to be stable at RMn, the following must be true: 1. RMn must have been informed of the unique identifier of RAi 2. There must be no other request for which RMn has generated a lower candidate identifier but not yet been informed of its unique identifier

Reliable Broadcast Algorithm for Send

For each process: 1. Send(msg, sequence number, transmitter identifier) 2. Increase sequence number

Use of Hard Links

For every binary name that is valid in the file service, there must be at least one hard link pointing to it Only exception is the binary name of the root directory. For the terror directory, either the binary name or the file itself must be located at a predefined place on disk

Achieving Location Transparency while mounting file systems

For location transparency we want the directory structure to be the same on all hosts We can achieve this using symbolic links: 1. We mount the external file systems in a separate directory (/external) 2. Symbolically link /external into the local host's directory structure.

Locking Distributed Data

Fragmented data: lock can be held by the server holding the fragment Replicated data: should we just replicate the locks? Still use priorities for Livelock detection. Use distributed wait-for graphs for detecting deadlock

Name scope

Global: name alone gives the type of object it is Name-space specific: Same name may be used in another name space for a different object. Object type can be different based on the name space

Two phases of locking

Growing phase: 1. Operations which require additional locks to be obtained 2. Transactions may obtain locks may not release locks Shrinking Phase: 1. Operations which do not require additional locks to be obtained 2. Transactions may release locks, may not obtain locks

Linking Files to Directories

Hard link: 1. Binary name of the file is contained in the directory 2. Advantage is that the file may be found rapidly Symbolic Link: 1. Symbolic name of file is contained in the directory 2. Must be passed to directory service to get binary name 3. Advantage is that not limited by binary name space of a particular file system

Resolving Livelock

Have a priority associated with transactions, a priority that gradually increases as it is delayed, until eventually its priority is high enough that it obtains the lock and executes

Stability problem

How does process know when it can deliver a message? If the process delivers the messages in clock order, consistency can be maintained, but if a process receives a message with a clock value of 10, how does that process know whether or not a message with a lower clock value might be received later? So how long should a process wait before delivering the message with clock value 10? If using vector clocks, then a process knows not to deliver the message if the clock value of the message is not the lowest in the vector. (Need some other form of metadata to be completely sure of a message is stable)

Reliable Broadcast Algorithm for Receive

If message has not been recorded in record array: 1. Deliver the message to application 2. Add message to record array 3. For every process, send the message (again) If the message has already been received and recorded, do nothing Each message that is received is re-broadcast to every other message (helping with validity)

Robustness of 2PC

If no servers crash and there are no network errors, then 2PC is reliable and robust Some crash failures are easily handled by having time-outs, e.g. The coordinator may not receive a reply from one server, and then decides to abort the transaction using C-ROLLBACK or ask a server to try again using C-RESTART If the coordinator fails after asking for a vote: 1. All the servers will time-out 2. They will contact each other to perform an election 3. The newly elected coordinator will continue the transaction

Atomic commitment

In a DS, atomic commitment is: 1. Ensuring that all updates are written to log files 2. Ensuring write operation to disk which marks the transaction as complete is atomic 3. All servers involved in the transaction agree to either all commit, or all abort

Transaction manager

In a centralized DBMS the transaction manager: 1. Executes the transaction 2. Ensures that ACID properties are maintained In a distributed environment: 1. Local maintenance of ACID properties is performed by a local transaction manager (LTM) 2. A global transaction manager (GTM) distributes requests and coordinates execution of LTMs 3. To avoid having a single point of failure, we use several GTMs in the system

State in a File System

Inconsistency of state between hosts of the file system is a problem for distributed systems. State could contain things like: 1. Cursor position that the user process has reached in the file 2. Who is reading and writing to the file, making it possible to perform some locking So why not have a stateless file system for DS?

Fault Tolerance (design consideration)

Increased number of components leads to an increased likelihood that at least one component fails at any given time, so design the DS to continue to work even if any component fails. Use fault isolation, and fault masking Need hardware redundancy and software recovery There are different measures of failure rate Fault tolerance is costly There are different failure modes

Why use replication?

Increases fault tolerance, because if one replica fails, others can be used by clients Spreading the load of requests between replicas can mean that clients have better response times

Performance Transparency

Increases or decreases in resource performance do not affect the logic of the accessor methods

Openness (design consideration)

Interfaces to the OS, network, and services must be based on published specifications (such as standards) Allows for new hosts to be added to the network 1. Makes it possible for one vendor to build systems which interact with those of another vendor 2. Without such openness, user has to buy everything from the same vendor

Maintaining Isolation for transactions

Isolation is maintained by executing transactions one after another (serially) However to allow for concurrent execution of transactions we need concurrency control. Concurrency control must be performed to ensure transactions are serializable

Middleware

Layer of software whose purpose is to mask the heterogenerity of the distributed system, and to provide a convenient programming model for programmers

Network

Links hosts together

Replication Transparency

Logic within methods that access a resource does not change according to how many copies of that resource are maintained

Management (design consideration)

Management is the supervising and controlling of the system so that it satisfies the requirements of its owners and users System is divided into a set of domains, with each domain having a system manager

Programmatic Inrerface

Management must be achievable through a program interface

System Manager

Manages a domain. Role of the system manager: 1. Configuration of the domain 2. Decide what resources are available at any one time 3. Decide where the resources are physically located 4. Decide which users are allowed to access the resources

Partitioning a directory

Means using a hierarchy of names, resulting in zones

Access Transparency

Methods to access a resource do not vary according to means of access e.g. Java method call, or SOAP message use same method signature

Transactions in distributed systems

Must ensure that the ACID properties of the execution are maintained over the whole system. This means modifying the concurrency control algorithms to communicate between hosts

T-fault tolerance of a State machine replication - non-Byzantine failures

Must have a number of replicas greater than t so that the front end always has at least one non-failed replica to look at

Naming domain

Name space with a single authority (like @kcl.ac.uk all have that in them)

Crash+link

Network fails to deliver message because the link is broken

Access Control

Only certain authenticated clients Are allowed access to a resource

Relevance

Only information necessary for management should be presented at any one time, because there is so much informstion

Design Considerations

Openness, scalability, transparency, management, heterogeneity, concurrency, fault tolerance, security, performance, other issues

Can you be causal but not FIFO?

People assume that you cannot because causal is a stronger ordering than FIFO, but without some assumptions, it can be true

Performance (design consideration)

Performance characteristics: 1. Responsiveness 2. Throughout 3. Load balancing

Throughout

Performance metric where as much processing as a possible should occur throughout the system per time

Load Balancing

Performance metric where balance the load between hosts so that no host does all the work while others stand idle

Responsiveness

Performance metric where operations should return responses as Quickly as possible

Locking Replicated Data

Primary backup replication: 1. Apply the locks only on the primary copy 2. The primary controls ordering on the backup copies State-machine replication: 1. We can make one replication manager responsible for each given activity: reduces to primary backup situation 2. Alternatively we must distribute the locking algorithm

Pros and cons of 3PC vs. 2PC

Pros: non-blocking Cons: poorer performance than 2PC, more situations to handle

Name Servers (NS)

Provides access to a directory (just something that lets look up information about names, such as what names/objects the name resolves to (refers to))

Pure vs. Impure Names

Pure name: no structure, can only compare for equality with another name Impure name: has some structure. (Like IP address, full path name)

Mount Points

Rather than naming files using the name of the host the file is stored on, we can mount directories of file servers locally as directories of a client. In essence, locally the client makes space in its directory structure for the remote directory. This point of entrance to the remote host's directory is called the mount point. For this to work, the remote host must have its file system export its

Buffering Data Blocks

Rather than reading a file one byte at a time from disk (which requires a lot of overhead) a practical file system buffers the data from disk in fixed units called blocks On request, the file system first checks to see if the block containing the byte of interest is already held in a buffer in main memory. If present, return the byte. If not, fetch the block of interest from memory, temporarily suspending the process

Two types of lock

Read locks: placed in an object whenever a process wants to read its contents. 1. May be obtained by any number of processes 2. Cannot be obtained if write lock exists on object Write locks: placed on an object whenever a process wishes to change its contents. A write lock on an object may be obtained by only one process at any particular time and only if either: 1. There are no read locks present or 2. The only read lock is held by the process itself

Order of Strength of Ordering

Reliable < FIFO < Causal < Total < Atomic

Integrity

Reliable Broadcast property Deliver(m) occurs at most once at each correct process, and only if broadcast(m) occurred in some process

Validity

Reliable Broadcast property If a correct process executes broadcast(m) then all correct processes will eventually deliver m

Agreement

Reliable Broadcast property If any correct process delivers m, then all correct processes eventually deliver m

Replication Issues

Replication issues depend on whether the replicated resource has a state. If it does, then not only do we need duplicate hardware, we need to maintain duplicate data as well. Delays in updates can mean that one resource gets the messages in a different order

Uniform Reliable Broadcast

Same as reliable broadcast, except the agreement and integrity properties apply to all processes, including failed processes This is important if we want to allow failed processes to recover

Scalability (design consideration)

Scalability pros of DS: 1. relatively cheap components may be used to build a small system at first 2. Additional components can be easily added as demand on the system increases For this to work we need duplicated resources, allowing us to avoid: the notion of single master services, short fixed names, performance bottlenecks Scalability and state: Some resources have state information, and the duplication of state information leads to the problem of propagating changes of state to the various copies

How do the Reliable Broadcast Algorithms for send and receive change for a Uniform Reliable Broadcast?

Send does not change. Receive changes the order so that delivering the message only happens after the message is successfully resent to all other processes 1. Add message to recorded 2. Resend message to every process 3. Deliver message to application This protects against crashes that could happen after the message is delivered but before resending the message

Uniform FIFO Algorithm

Send is the same. A new array called next is created storing the information about the next sequence number to deliver for each process Receive: 1. Add the message to the record While the record contains the message with the next sequence number for this process: 1. Resend the message to all processes 2. Deliver the next message 3. Increment the next value for this process

Uniform Causal Algorithm

Send: Keeping track of all messages received since last broadcast, called causes. 1. Concatenate new message to causes. 2. Clear causes. 3. Send to each process 4. Increment sequence number Receive: 1. Process each message in the concatenation individually as you would do in FIFO. 2. Add all messages received to causes.

Sending and Receiving a Broadcast Message

Separate the middleware from the application to allow for Transparency (application does not need to know how many processes there are), so application uses the methods: 1. broadcast(m) 2. deliver(m) Where broadcast tells the middleware to send the message to all the processes, and those messages are received by the middleware and then delivered to the application

Why does 2PC Block?

Server Blocks because if there is a failure with the coordinator the server does not know whether it should commit or abort, so blocks and waits. This allows one of the servers that blocked to send a restart message to the failed servers, although this make take a long time

Failure Mode Illustration

Service 1 fails: Failstop: client knows that service 1 Has failed, So uses service 2 Crash: client does not know service 1 has failed, so client blocks, waiting for a response In failure, service 1 may or may not be still active: Crash: service 1 does not process message from client Receive omission: service 1 does not receive clients messages Send omission: service 1 processes message from client but unable to respond to client to let client know that message was processed, so client may continue to send requests/messages

T-fault tolerance of a State machine replication -Byzantine failures

Since some of the responses may be invalid,take a majority vote on those responses, therefore you will need a number of replicas greater than 2t

Local Waits-for graph

Stored outside the local server being marked an EXT node When a request is received, the local manager sends its waits-for graph to the current coordinator node, time stamped so the coordinator can distinguish the most recent information. The coordinator constructs a global waits-for graph and detects cycles

Naming of files

Symbolic names: 1. Given by user processes to identify a file 2. Usually impure name (like /etc/password) Binary Names: 1. Used inside the file server to identify the file's physical location 2. Either actual disk address, or address of record containing physical location of various parts 3. Usually pure name

Fault Masking

Take action to restore the service when a particular resource of the service fails without the user noticing (need software recovery)

Consensus protocol properties

Termination: every correct process will decide on a value. Validity: if all proposals are for V then all correct processes eventually decide V. agreement: if one correct process decides V then all correct processes eventually decide V Integrity: every correct process decides at most one v, and either V = FAIL or some process must have proposed V

Integrity

The actions of a client cannot affect the overall integrity of a resource

Authentication

The client knows that the service really is what it says it is, and vice versa

Use of Symbolic Links

The file linked to need not exist, allowing the file that is linked to the symbolic link to be deleted and replaced, without needing to change the directory containing the symbolic link. Files that are pointed to by a hard link may only be deleted once every hard link that was pointing to the file has been deleted

Scaling Transparency

The logic of accessor methods does not need to account for or worry about changes in the overall size of the system or the number of instances of a resource

Failover time

The period of time for which there is no primary replication manager

Uniformity

The same style of control for each type of resource (resource information must be in a uniform format for all types of resources or it cannot be managed)

Naming scheme

The structure/syntax from which names are constructed

Identity Security Issues

These are the ways that we try to implement security 1. Authentication 2. Confidentiality 3. Access Control

Requirements for Management

These requirements must be met to allow a system to be managed 1. Remote access 2. Programmatic Interface 3. Relevance 4. Uniformity

Blocking Time

Time between a client making a request and receiving a reply

Transparency (design consideration)

Transparency is about making the service provided independent of how it is provided. Want to make the user's life easier by hiding information that is unnecessary for using the services of the DS, making it simpler to use There are different types of transparency

State machine approach

Treate all managers on an equal footing, communicatijg with all managers when making requests. More robust than primary backup

Heterogeneity (design consideration)

Variation in resources Want to allow for variations in use of the DS, variations in the types of hosts (operating environments, hardware and software)

View equivalence

View equivalence between sequences of action ensures the relevant state of database objects is as expected for any one action. A sequence of actions interleaved from multiple transactions is then serializable if it is view equivalent to a serial execution of those transactions

Enforcing serializability rules

Ways of enforcing the rules across a distributed system: 1. Timestamping: a. Add a timestamp to each object and transaction b. On a write, the object's time stamp is set to the transaction's time stamp c. A transaction may only access an object with an earlier timestamp 2. Optimistic concurrency control: a. Transactions perform whatever actions they like b. Check for violation of the rules at commit time

3 phase commit

We can make 2PC non-blocking by adding another phase Phase 1: same as for 2PC Phase 2: 1. If coordinator receives C-READY from all servers it transmits C-PRECOMMIT (telling all the servers the result of the vote) 2. Each server replies with a C-PRECOMMIT-ACK 3. If coordinator receives a C-REFUSE from any server it transmits C-ROLLBACK to all servers 4. Each server aborts on receiving this message Phase 3: 1. If the server receives a C-PRECOMMIT-ACK from all servers, it then transmits C-COMMIT to servers 2. If the coordinator is missing a C-PRECOMMIT-ACK from any server it transmits C-ROLLBACK to all servers This gets rid of blocking because if a server misses a C-PRECOMMIT the server can ask another server for the result of the vote, and can either commit or abort based on that result of the vote. If all other servers have failed then the server can abort because none f the other servers could have committed

Time-criticality

Where certain operations have strict time constraints, the system should be able to meet these constraints

View equivalence rules

the two sequences S1 and S2 are view equivalent if: 1. For each data item Q, if transaction Ti reads the initial value of Q in S1, then Ti must, in S2, also read initial value of Q 2. For each data item Q, if Ti executes read(Q) in S1 and that value was produced by transaction Tj, then transaction Ti must in S2 also read the value of Q produced by transaction Tj 3. For each data item Q, the transaction (if any) that performs the final write(Q) operation in S1 must perform the final write(Q) operation in S2.