COP 4521
Examples of Parallelism?
- Multiple Independent Jobs (STUPID EASY) - Scientific - Databases - Artificial Intelligence
How do you write parallel programs?
- Use old serial code - Compiler turns it to parallel - Serial Language plus Communication Library - No Compiler changes, used by PVM (Parallel Virtual Machines) - New language - Brand new code, hard to optimize for all systems -
What is a function closure?
A closure is a function object that remembers values in enclosing scopes regardless of whether they are in memory or not. That means if you define a function inside of a function and assign it to a variable x, if you delete the outer function the inner function can still do its job.
What is a Database?
A collection of data stored electronically, usually organized and structured. Data in a database is usually stored in tables (Relational) or through XML or JSON-Based collections (NOSQL)
What is a decorator?
A decorater essentially decorates an existing functions value. It does this without modifying it functionality. EX. def say_hello(name): return "Hello, " + str(name) + "!" def decorate(func): def func_wrapper(name): return "<p>" + func(name) + "<p>" return func_wrapper decorated_say_hello = decorate(say_hello) print(decorated_say_hello("John")) # Output: <p> Hello, John <p>
How do dictionaries work in python?
A dictionary is essentially the same as a hash table in C++. They take in a key, value pair and store them. In order to access the value you must put the key into the variable[ key ] -> value. You can call a dict by typing variable = dict( ) or by typing variable = { key1 : value1 , key2 : value2, ...}
What is a transaction?
A group of tasks. We need to determine the order of the process and make sure that the atomicity of the database is still intact. This is very similar to OS Scheduling.
What are common methods found on sockets?
#s is a socket s.connect(addr): connects to the addr. addr is a tuple containing hostname and port (hostname, port) s.send(string): sends a string to wherever the socket is connected to. The return value is the number of bytes sent. It may not be all sent! Check the return value!
What is a processing element (PE)?
A unit of usually commodity hardware that is capable of performing computation, typically supported by some type of memory.
What is a socket?
An endpoint for communication. This abstracts the details for connecting to networks.
What is an interconnect?
An interconnect is a communication network used to connect processing elements. It also allows the PE to interact with the memory/handle I/O etc.
How can you tell the users of a module the intended types that should be passed into functions?
Annotating them! We can define a function in this matter: def func(arg: arg_type, optarg: arg_type = default)->return_type
What is an iterable?
Any Python object with the following properties: It can be looped over, Can be used as an argument for iter( ), which returns and iterator, Must define __iter__() or __getitem__() EX. List1 = [1,2,3,4,5] it = iter(List1) print(next(it)) #Output 1 print(next(it)) #Output 2
What the 4 statements for manipulating control flow tools?
Break: terminate current loop continue: Immediately begin the next iteration of the loop pass: Do nothing else: Represents a set of statements that should be executed when a loop concludes.
What are some common TCL Commands?
COMMIT: Save Changes ROLLBACK: Do not save changes SAVEPOINT: Go back to this point when doing a ROLLBACK SET TRANSACTION: Places name on transactions.
How do you GRANT/CREATE roles?
CREATE ROLE manager; GRANT create table, create view TO manager; GRANT manager TO John Smith
What are entities?
Entities are theoretical constructs that help us conceptualize relations. Think of an entity as anything that has meaning and we want to add to our database. We then use these entities to relate them to other entities such as tables relations to other tables.
What is the filter function?
Filter is a function that takes in a function and a sequence and returns the sequence passed in but only with the values that come back from the function passed in. The first function passed in can be looked as the actual filter and the sequence is the item being filtered. EX. filter(getEvens, range(0,30))
How do you create a list?
MyList1 = [] MyList2 = list() MyList3 = []; MyList4 = []
What is Flask?
Web application platform made by Python. Uses Werkzeug and Jinja2
How to define inheritance?
class DeriveClassName(BaseClassName) if BaseClassName is in another module define it using: class DeriveClassName(module_name.BaseClassName)
What are the scoping rules in Python?
Python follows the LEGB Rules: Local namespace Enclosing namespace: check non local names in the local scope of the enclosing functions from inner to outer Global namespace __builtins__: Names python assigned in built-in module
What is a database management system (DBMS)?
The intermediary between the user and the database. It is also used for the communication of the application and the database.
How do strings work in python?
They are easily assigned to variables by enclosing a statement in either double or single quotes. They are immutable meaning that they cannot be changed, and altering them would mean creating an entirely new string. In addition, strings support commands such as \n and \t. You can find the raw value of a string by putting r in front of it.
When do we use tuples?
When storing elements that will not need to be changed. When performance is a concern. When you want to store your data in a logical immutable pairs, triples, etc.
When do you use a Set?
When the elements must be unique When you need to be able to modify the collection When you need support for mathematical set operations (Think Union, Intersection) When you don't need to store nested sequence types as elements. (You can still store tuples in sets though)
What are keyword arguments in functions?
When the formal parameter of a function is specified when calling the function this is known as a keyword argument. EX. def func ( do, work, bad, good) func( work = True, do = False, good= False, bad =True) Here you can see that we are not inputting the values in the correct order but since we are calling the argument we can do that just fine. They must also go at the end of the function if they are not all defined in this manner.
Can you pass *args into a function if it doesn't have it in the definition?
Yes! It is called unpacking and it works by unpacking a tuple of arguments and filling in the parameters with the contents of the tuple Ex. args = (1,2,3) func(*args)
Can you pass **kwargs into a function if it doesn't have it in the definition?
Yes! You can pass a dictionary into a function that accepts dictionary style attributes.
How do you add an element/list to the end of a list?
You add an element to a list by calling the .append( ) method on the list. EX. MyList.append(4) You add a list to the end of a list by calling the .extend( ) method. EX. MyList.extend(MyList2)
How do you insert and remove elements from a list?
You can insert an element into a list with the .insert( ) method. EX. MyList.insert( 1, 56) where 1 is the index we are inserting into and 56 is the element being inserted You can remove the first occurence of an element in a list with the .remove( ) method. EX. MyList.remove(56)
How do you obtain the name of a module?
You can obtain a modules name as a string with __name__, and if the module is executed directly its name will be main allowing to have a main like in C++.
What are built in attributes in all classes?
__dict__: dictionary containing the object's namespace. __doc__: class documenting string __name__: name of class __module__: module name in which the class is defined, this is "__main__" in interactive mode __bases__: A tuple that may be empty containing the base classes.
How do you make a regular function?
def func( args ): code Similar to C++ but with no need for a return type in front of the function.
How do we add default arguments to a function?
def func( regargs, args=value) You must put the default argument at the end of the function. You can put more than one just like in C++.
How do you create a socket?
from socket import * s = socket(AF_INET, SOCK_STREAM) This now becomes an endpoint for other machines.
How do you build a constructor for a class?
use def __init__(self, list of params) self is the name of the instance of the object. EX. a = myClass( ) a is the instance that fills the requirement for self.
How do lambda work?
It essentially works as an inline defined function. Instead of writing def for a function we use lambda. You can only use one expression. It can also be used to make a function object. EX. g = lambda x : x**2 print(g(8))
How does a set no if there are no duplicates?
It gets each value and hashes it. This means that the only values that can be passed into a set are immutable values such as strings or integers. This is why mutable objects such as lists can not be passed into a set. However, since tuples are immutable you can pass them into a set.
Functions are first-class objects in Python. What does this mean?
It means that whgatever you can do with a variable you can do with a function: Assign a name to it Passing it as an argument Return it as a result Storing it in a data structure
What are the two modes supported by Python?
Normal Mode: entire .py files are read and compiled Interactive Mode: read-eval-print loop (REPL) executes statements piecewise.
What are primary and foreign keys?
Primary Keys are given to columns that uniquley identify a table. If there are more than one Primary Key in a table it is called a composite key Foreign Keys are primary keys in other tables stored inside of your current table.
How is Python implemented?
Python is interpreted, similar to Java it is compiled into byte code and then executed by the Python VM.
How is Python typed?
Python is strongly, dynamically typed language. It is strongly typed since we cannot perform integer operations on a string and vice-versa and it is dynamically typed because all type checking is done at run-time. You do not need to tell a variable its type, Python will assign it from the R-Value provided.
What are the different forms of Normalization?
1NF: Each Table has a single value and each record is unique (atomic) 2NF: MUST BE 1NF and every non-key is fully dependent on the primary key. There can be no Partial dependencies 3NF: Has no transitive functional dependencies ^These three are needed the rest probs arent -------------------------------------------------------------- BCNF: Remove all redundancy-based functional dependencies. 4NF: Eh 5NF: GL 6NF: not defined
What is sqllite3?
A DBMS available when installing python. You can run it from inside of the terminal. We can use sqlite3 inside of a setup.py file in order to make a database. Follow the steps for HW2 for more information on how.
How are HTTP Requests and Responses handle in Flask?
A client(Chrome) sends an HTTP request to the Server(Flask). The Server sends back an HTML page. HTTP commands can be set to the Server via POST/GET requests. POST: More use in production. Safer. No limit on data being sent GET: Use in smaller scale. Less Safe. Limited data being sent.
What is name mangling?
Any attribute prefixed with two __ and no more than one trailing _ will automatically be replaced with _classname__nameofattribute.
What is ACID?
Atomicity: All or nothing transactions Consistency: Only valid data served Isolation: Transactions should not affect other transactions Durability: Written data will not be lost
How can lists be used as a queue?
For this we need to get the module collections by adding from collections import deque to the top of the file. With this we can turn any sequential data structure into a deque. We can add to the left of the deque using .appendleft( ) and pop from the left using .popleft( ) methods. Adding to the right of the deque works the same as a stack. EX. queue = deque( [2,3,4,5]) queue.appendleft(1) queue.popleft()
What are generators?
Generators are iterators defined on a function. They use yield in order to give way to the next element in the function. They can be used wherever iterators are used. They are desirable because they are LAZY! They do nothing until the first value is requested and then only do enough work only to produce the next value, causing less resources to be used. EX. def gen(): n = 1 while true: yield n n+=1 next_gen = gen print(next(next_gen)) # Output 1 print(next(next_gen)) # Output 2 ...
How do joins work in DBMS?
If two tables have a common data type the new table combines only the rows from the given table that matches the fields.
What are the 5 main operations that can be done to tables?
Select: Takes rows from tables and makes a temporary table out of it Project: Build a new table out of the current attributes of another table. You specify the columns and rows you want in the new table. Not implemented in SQL but select works in its place Union: Combines two tables Difference: Remove rows from one table also listed in the second table. Product: Gives the Cartesian product of two tables. Joins: Combine two tables like the Product operation but do not necessarily copy all rows and columns.
How do you index/slice a list?
Index the same as C++: myList[i], you can find i by saying myList.index(element_searching_for) Slicing is used to grab sublists. This can be done by seperating the beginning index and the last index of the sublist with a : EX. myList2 = myList[1:4] where 1 index one is included and index 4 is not We can get the length of a list using the len() function.
How do unicode strings work in python?
Just add a u to the beginning of a regular string. Use ur at the beginning of a string to get the raw value of a Unicode string. In order to get a regular string from a Unicode string use .encode() method. Use unicode() to go from regular string to Unicode string.
What are the 3 levels of Data?
Level 1 - The actual database File that stores all information where format is based on the DB being used. Level 2 - The DataBase Managament System (DBMS) This allows interaction between the data and the application. Level 3 - The Application Any software that will need this data to perform tasks.
What are the most common data structures?
Lists, Tuples, Sets and FrozenSets, Dictionaries, From Collections: Deques, Stacks, and OrderedDicts
What are the two protocols for ensuring transaction consistency?
Locks: Resource gets locks. Nothing else can go until it is finished. TimeStamps: Every transactions have a timestamp and the transactions are executed in the order of the timestamps.
What are modules in Python?
Modules are just text files containing Python definitions and statements that can be executed directly or imported from another module.
Why do we need multiprocessing in python?
Multiple CPU's have become standard so we want a way to use them effectively.
What are the 4 major styles of Parallel Computing?
SIMD - Single Instruction Multiple Data - One master program counter (PC) MIMD - Multiple Instruction Multiple Data - Separate Code for each processor SPMD - Single Program Multiple Data - Same code on each processor, separate PC's on each Dataflow - instruction waits for operands - "Automatically" finds parallelism
What is Scheduling/SerialScheduling?
Scheduling: a chronological execution sequence of transactions. Serial Scheduling: Transactions are aligned and ordered. When the first transaction finishes the next one starts.
What is a **kwargs in a function?
Similar to *args, **kwargs will create dictionary of the values passed into it. EX. func( **kwargs ): for key in kwargs.keys(): print(key, ":", kwargs[key] func(doWork=doWork, Why=Why, server='localhost') This will print each key (LValue) and each value (RValue)
What is the map function?
Similar to the filter function, map takes in a function and a sequence and applies the function action to each element in the sequence and returns the value as strictly a list. You can provide multiple sequences if the function passed in supports it. EX. map(square, range(0,11)) EX. map(multiply, range(0,4), range(0,4)
How do sets work in python?
Similar to the real-world definition of a set, a set is an unordered collection of unique objects. You can call a set by either passing a list into the set( ) function or by setting a variable = { things in set } ( Note, very similar to how to set up dictionaries ). There is also something known as a frozen set which is an immutable version of a set.
How can lists be used as stacks?
Simply call .append( ) and .pop( ) exclusively on your list and it will act like a stack.
What are some sources of parallelism?
Statements - Called "Control Parallel" - Can perform a series of steps in parallel Loops - called "Data Parallel" - Most common source of parallelism where each processor gets 1+ iterations to perform
How can we coordinate parallel processes?
Synchronization: - Protection of a single object (locks) - coordination of processors (barriers) Size of a unit of work by a processor - Here we manage to manage two issues - Load Balance - processors have equal work - Coordination Overhead - Communication and Sync
How do you manage ROLES?
The ON Clause can be used to specify objects for access. GRANT update ON transactions TO managers The WITH Clause allows users to give privileges to more users GRANT insert ON Employees TO managers WITH GRANT select Revoke Privileges: REVOKE create table FROM manager; Drop Roles: DROP ROLE manager;
What can be used in order to have a larger scale server in python?
The SocketServer library allows us to use a higher level networking library for making servers.
What is the topology in terms of Processing Elements?
The arrangment of PE's using a communication network is called a "Topology", where the key performance issues are: - Latency: time for first byte - Throughput: average bytes/second And the key design issues are: - Where is the memory? - Can all processors access all the memory?
What is Parallel Computing and Why do we need it?
The use of more than one processing element/nodes working together to solve a single problem. We use it for the speed advantages that arise from more processing elements/memory/disks. It is also more cost effective to have many weaker machines than one very powerful one.
What are the inherit sequence data types found in Python?
There are 7 which include: strings, unicode strings, lists, tuples, bytearrays, buffers, xrange objects.
How do private variables work in Python?
There are no strict private variables in python. However, ig a variable has a single underscore preceding its name it should be treated as private.
How do you sort lists?
There are two methods: Call the .sort( ) method on the list, this will sort items from smallest to largest, while .reverse( ) will sort them largest to smallest. Call the sorted( ) function which can take any iterable be it a deque, stack, set, etc. and sort it, with additional arguments such as reverse allowing it to be in reverse order.
What are the properties of entities?
They are unique, no two rows can be the exact same. They have atomic data, meaning the data can not be split up into smaller pieces. They have keys, keys are used to determine a unique characterstic of an entity. You make this unique field in a entity the Primary Key. If no unique field exits make a unique ID field and make that the Primary Key.
How do lists work in python?
They work very similarly to C++ arrays however, you can store any type of data in to it even if its not of the same type. You can mutate any of the data inside of the list unlike sets and you can even nest a list inside of a list.
What is list comprehension?
This is a way for us to construct lists by running loops with certain conditions. In its simplest form, it can be written as [expr for x in sequence]. It can also accept if statements for better filtering. EX. squares = [x**2 for x in range(0,11)] EX. squares = [(x,x**2,x**3) for x in range(0,9) if x % 2 == 0]
What is the "GIL" in python?
This is the Global Interpreter Lock, which is a thread-safe mechanism that prevents conflicts between other threads and executes one statement at a time.
What is the clien-server model?
This model follows the trend of the client (Web Browser) sending a request to a server (BackEnd Framework (EX.FLASK)) and having the server respond back. This model is defined by a protocol. The most common one is the TCP/IP protocol. This protocol instructs how data should be sent and received throughout the internet.
What is Role-Based Access Control (RBAC)?
This restricts access to the database by assigning roles to users. Different roles have different levels of access thus have different roles that give them that access.
What is a *args in a function?
This will take all the values found from a certain point in a function and concatonate them into a list and allow you to read them one by one using a for loop. This is called packing. EX. func( args*) for arg in args: print(arg) func( arg1, arg2, arg3, arg4) This will print each argument passed into it.
How do you modify a set?
Though its various methods/operations: .add( x ) will add x to the set if it is not already in there .remove( x )/.discard( x ) will remove x from the set. .pop( ) will remove an arbitrary element from the set. .clear( ) will clear the contents of the set. set |= other | ... Works like Union of all other sets set &= other & ... Works like intersection of all other sets set -= other | ... We remove all elements from a set found in other sets. set ^= other We obtain only the values found in either set but not in both. set >= other will check if set is a superset of other set <= other will check if set is a subset of other
What are decorators used for?
Timing execution of functions, Memoization, Logging, Debugging, Any pre/post function processsing
How do you flash a message to a client in Flask?
Use the flash( ) function which takes in message as the first parameter and the second parameter is the category, 'info', 'error'...
What are the contents of a DataBase?
User Data: Actual data being stored, Metadata: Data about the data, The structure(schema) of the database, Think name of columns and any constraints you would want on the data Application Metadata: Metadata on user settings Index Data: Used for performance improvement, includes logs, tracks, security, etc.
How do initialize tuples?
Using ( )/ calling tuple( ) constructor Ex. myTuple = (1,2,3) myTuple = (1, ) myTuple = () myTuple = 1,2,3,4
How do we return new sets from old sets?
Using these operations: UNION set | other | ... returns a set with all elements of other sets INTERSECTION set & other & ... returns a set with only the same elements found between all sets. DIFFERENCE set - other - ... returns a set with only elements found in the original set SYMMETRIC DIFFERENCE: set ^ other returns a set with elements in either other or set but not both.
What are database Schemas?
Ways to construct and define a table. They contain a list of table definitions that any table in the DB must abide by.
How do we create a server socket?
We can create them similarly to connecting sockets. This socket will act as the other endpoint of the connection. First have the socket get the hostname of the current machine it is running on socket.gethostname() #returns a string Second, allow the socket to listen to connections to it. Add the backlog to see how many users can to be connected before any more users get denied. s.listen(backlog) Last connect a socket to an addr block (tuple containing hostname and port (can be obtained from socket.gethostname())) but use bind instead of connect. s.bind(addr)
How do you declare a class?
class ClassName: statement 1 ... statment2
How do you import things from a module?
from {module_name} import {Thing to import} To import all run: from {module_name} import *
How do we use SocketServer?
import SocketServer SocketServer.TCPServer((HOST, PORT), RequestHandler ) RequestHandler is an class that overrides the
What are the numeric types in Python?
int, long, float, and complex. Each one can be called as a function to change a variables inherit typing. int( ), float( ), long( ), complex( ). An int is unlimited in 3.X and the same as a long in C in 2.X. float is the same as C doubles. long is not available in Python3.X and unlimited in 2.X.
Example of a socket connecting to a server.
s = socket() s.connect(('www.fsu.edu',80)) s.send("GET /index.html HTTP/1.0\n\n") data = s.recv(2048) s.close( ) print(data) #Prints the size of the buffer gotten in s.recv(), could be less or more.
How do we continuously wait for a connection to be made?
s.accept( ). This returns a tuple with conn, which is a new socket that can receive and send data from the connection, and addr which is the address bound to the socket on the other of the connection. EX. s.bind(("",9000)) s.listen(5) while True: c,a = s.accept() print(f"Received connection from {a}") c.send(f"Hello {str(a[0])}") c.close()
How do you bind to localhost?
s.bind(('localhost',9000)) OR s.bind(('127.0.0.1',900))
What are common methods found on sockets? Cont.
s.recv(buffer): recieves a stream of bytes from the address the socket is currently connected too. It returns the bufsize. s.close(): closes the connection.
Useful socket methods.
socket.gethostbyname("www.python.org") -> returns IP Addr socket.gethostbyaddr("82.94.237.218") -> returns names static methods^ ---------------------------------------------------------------- socket.sendall(string) -> blocks until all data has been sent. socket.setblocking(flag)-> determines whether a socket should be blocking or not. flag is true by default. socket.settimeout(t)-> socket will raise an exception if the time it takes to finish a connection exceeds t. If t is not assigned then it will never timeout. socket.makefile()-> returns a file object which allows us to write and read from a socket as a file object. VERY USEFUL (According to sharanya) ^instance methods
What is the socket module?
socket.socket([family, [type]]) creates a socket object. This creates an endpoint of a two-way communication. The family argument is the address family. socket.AF_INET is the most used since it is IPV4. The type is the type of socket we are using. The most common one socket.SOCK_STREAM which is TCP and socket.SOCK_DGRAM which is for UDP.
