Final review
What is a distributed system? ( 1-2 sentences max)
A distributed system is a group of computers working together as to appear as a single computer to the end-user
Name three characteristics of the Data Parallel Model
Address space is treated globally Most of the parallel work focuses on performing operations on a data set. The data set is typically organized into a common structure, such as an array or cube. A set of tasks work collectively on the same data structure, however, each task works on a different partition of the same data structure. Tasks perform the same operation on their partition of work, for example, "add 4 to every array element".
There are a number of important factors to consider when designing your program's inter-task communications: Communication overhead Latency vs. Bandwidth Visibility of communications Synchronous vs. asynchronous communications Scope of communications Efficiency of communications Overhead and Complexity For each of the factors above write two sentences the basic details about each factor that you need to be concerned about when designing your program's inter-task communications.
Communication overhead : The amount of time required to coordinate parallel tasks, as opposed to doing useful work. This can be effected by ,Task start-up time, SynchronizationsData, communications ,Software overhead imposed by parallel compilers, libraries, tools, operating system, etc. latency is time it takes to send a minimal (0 byte) message from point A to point B. Commonly expressed as microseconds. Bandwidth is the amount of data that can be communicated per unit of time. Commonly expressed as megabytes/sec. Visibility of communications : With the Message Passing Model, communications are explicit and generally quite visible and under the control of the programmer.Synchronous vs. asynchronous communications : Synchronous communications require some type of "handshaking" between tasks that are sharing data. This can be explicitly structured in code by the programmer, or it may happen at a lower level unknown to the programmer.Synchronous communications are often referred to as blocking communications since other work must wait until the communications have completed.Asynchronous communications allow tasks to transfer data independently from one another. For example, task 1 can prepare and send a message to task 2, and then immediately begin doing other work. When task 2 actually receives the data doesn't matter.Asynchronous communications are often referred to as non-blocking communications since other work can be done while the communications are taking place.Interleaving computation with communication is the single greatest benefit for using asynchronous communications. Scope of communications : Knowing which tasks must communicate with each other is critical during the design stage of a parallel code. Both of the two scopings described below can be implemented synchronously or asynchronously.Point-to-point - involves two tasks with one task acting as the sender/producer of data, and the other acting as the receiver/consumer.Collective - involves data sharing between more than two tasks, which are often specified as being members in a common group, or collective. Efficiency of communications : Very often, the programmer will have a choice with regard to factors that can affect communications performance. Only a few are mentioned here.Which implementation for a given model should be used? Using the Message Passing Model as an example, one MPI implementation may be faster on a given hardware platform than another.What type of communication operations should be used? As mentioned previously, asynchronous communication operations can improve overall program performance. Overhead and Complexity : If granularity is too fine it is possible that the overhead required for communications and synchronization between tasks takes longer than the computation.
What is the difference between deadlock and starvation?
Deadlock occurs when a process can't readily access a shared resource, where's starvation occurs when all processes cannot access shared resources.
What does Amdahl's Law tell us about adding more processors to solve a problem?
Introducing the number of processors preforming the parallel fraction of work, the relationship can be modeled by speedup = 1PN+S1PN+S where P = parallel fraction N = number of processors S = serial fraction Overall the maximum amount of speedup you can get from adding an ungodly amount of processors is 20x (assuming your code is also 95% parallel as well)
What are the similarities and differences between SPMD and MPMD?
Like SPMD, MPMD is actually a "high level" programming model that can be built upon any combination of the previously mentioned parallel programming models. MPMD: Tasks may execute different programs simultaneously. The programs can be threads, message passing, data parallel or hybrid. All tasks may use different data MPMD applications are not as common as SPMD applications, but may be better suited for certain types of problems, particularly those that lend themselves better to functional decomposition than domain decomposition SPMD: All tasks execute their copy of the same program simultaneously. This program can be threads, message passing, data parallel or hybrid. All tasks may use different data
List the 4 main components of von Neumann Architecture and what each component does.
Memory : read/write, random access memory is used to store both the program instructions and data Control Unit : this fetches the instructions/data from memory, decodes the instructions and then sequentially coordinates operations to accomplish the programmed task Arithmetic Logic Unit : this preforms basic arithmetic operations ( +, - , / , % ) Input/Output :is the interface to the human operator
Name three characteristics of the Distributed Memory / Message Passing Model
Multiple tasks can reside on the same physical machine and/or across an arbitrary number of machines. Tasks exchange data through communications by sending and receiving messages. Data transfer usually requires cooperative operations to be performed by each process. For example, a send operation must have a matching receive operation.
What is the difference between concurrency and parallelism?
Parallelism is when multiple sets of instructions are executed at the same time in a completely different process (utilizing the machines multiple cores) . This all can be run synchronously or asynchronously. Concurrency, while similar in use for decreasing runtime, is when multiple tasks can start, run, and complete in overlapping time periods. This doesn't mean they'll necessarily be running at the same instance, however.
In your own words define the following: ( 1-2 sentences max) Race Condition Critical Section Mutual exclusion Deadlock Starvation Semaphore Monitor Condition Variable
Race Condition : This occurs when two or more threads can access shared data and they try to change it at the same time. Since the thread scheduling can swap between threads at any time, you don't know the order in which the threads will attempt to access said shared data. Critical Section : When two or more processes share data in common, the code that updates this data is called a critical section. To ensure integrity of the shared data, we require that at most one process is executing in its critical section for a given data structure at one time. Mutual exclusion : Under no circumstances can two processes be in their critical sections (for the same data structure) at the same time. Deadlock : where a thread/process cannot access a resource it needs to continue. When an active application hits a deadlock, it may "hang" or become unresponsive. Starvation : where threads/processes cannot access shared resources because another thread/process is holding onto it for too long. Monitor : A monitor provides synchronization between threads through mutual exclusion and condition variables. Condition Variable : Condition variables provide operations wait and signal similar to those in a semaphore, but there are some important differences:
In one sentence.... what is scalability
Refers to a parallel system's ability to demonstrate a proportionate increase in parallel speedup with the addition of more resources.
List and describe the four classifications of parallel computers in Flynn's Classical Taxonomy.
SISD (single instruction stream single data stream): a serial (non-parallel) computer Only one instruction stream is being acted on by the cpu during any one clock cycle, and only one data stream is being used as input during one clock cycle. The execution of the instructions is in a deterministic manner. This is the oldest type of computer SIMD (single instruction stream multiple data stream): parallel computer, all processing units execute the same instruction at any given clock cycle, and each processing unit can operate on a different data element. This is best for specialized problems characterized by a high degree of regularity. It is synchronous and deterministic execution MISD (multiple instruction stream single data stream): parallel computer, each processing unit operates on the data independently via separate instruction streams, and a single data stream is fed into multiple processing units. There are few, if any, examples of this class of parallel computer have ever existed. MIMD (multiple instruction stream single data stream): parallel computer, every processor may be executing a different instruction stream, and every processor may be working with a different data stream. The execution can be synchronous of asynchronous, deterministic or non-deterministic. This is the most common type of parallel computer, most modern supercomputers fall into this category.
What are safety and liveness?
Safety - concurrent processes may corrupt shared data Liveness - processes may starve if not properly coordinated these are two forms of complexity introduced by concurrent applications.
What is the CAP Theorem?
The CAP theorem states that is impossible for a distributed system to simultaneously provide more than two of the three following guarantees. Consistency - What you read and write sequentially is what is expected Availability - The whole system does not die - every non-failing node always returns a response Partition Tolerance - The system continues to function and uphold its consistency/availability guarantees in spite of network partitions
In one sentence.... what is parallel overhead ?
The amount of time required of coordinate parallel tasks, as opposed to doing useful work.
Name the steps that you take to design and implement a programmer directed to solve a problem.
The first step in developing parallel software is to first understand the problem that you wish to solve in parallel. If you are starting with a serial program, this necessitates understanding the existing code also. Before spending time in an attempt to develop a parallel solution for a problem, determine whether or not the problem is one that can actually be parallelized. Identify the Hotspots of where the real work is being done. Identify bottlenecks, areas that are slow or cause the parallelizable work to halt or be deferred.
Why use a distributed system? ( 1-2 sentences max)
There are quite a few reasons to use a distributed system, a few would be for Horizontal scaling, reducing system latency, and increasing fault tolerance.
What is Vertical and Horizontol Scaling?
Vertical scaling: is when you increase performance on the same system by upgrading its capabilities. Such as adding more ram, upgrading its storage like capacity or switching to a ssd rather than using a hdd, or even upgrading the cpu. The advantages to this is there isn't a lot of change to the software design, and in the short term it is a low cost to benefit ratio. However it is limited to hardware capabilities, and the gains taper off harshly in the long run. Horizontal Scaling: This is when you add more computers instead of increasing the capabilities of the system. The key advantages to this, is that it is cheaper in the long run, there is no cap over the scaling, the fault tolerance is increased, and lowers latency by distributing the systems over a wide area. The disadvantage of this is that it has a high initial investment, and it does require a software redesign.
List 4 reasons to use parallel computing
We can solve larger and more complex problems, since we can break up and calculate said problem with multiple processes Takes advantage of non-local resources, using a network or internet to allocate resources from other machines to compute problems. It provides concurrency, multiple compute resources can do many things simultaneously. It takes advantage of the underlying parallel hardware in modern computers, most all computer have multiple core processors with typically multiple threads per core (although intel had shied away from multithreading in their 9700 series mid to lower end processors)
Why do we need synchronization mechanisms?
We need synchronization mechanisms to keep data integrity throughout a program. For example, in a producer-consumer relationship, the consumer process is dependent on the producer process until the necessary data has been produced. If said system didn't wait for the producer to finish calculating the data, there would be discrepancies with passed around values and information.
In one sentence.... what is pipelining?
breaking a task into steps preformed by different processor units, with inputs streaming through, much like and assembly line; a type of parallel computing.
What is the shared memory model for concurrent programming?
concurrent modules interact by reading and writing shared objects in memory. For example two programs running on the same computer, sharing a common filesystem with files they can read and write.
What is the message passing model for concurrent programming?
concurrent modules interact by sending messages to each other through a communication channel. Modules send off messages, and incoming messages to each module are queued up for handling.
In two sentences.... what is parallel computing?
parallel computing is the simultaneous use of multiple compute resources to solve a computational problem. This means we are using multiple processors simultaneously to execute multiple, smaller problems/calculations broken down from an overall larger complex problem.