Data Science Quizlet
Xpath which outputs cities located between 2E and 2W in format "city: latitude, longitude"
/mondial/country//city[@longitude<2 and @longitude >-2]/concat(name,':',@latitude, ',', @longitutde);
Map/Reduce
A computational paradigm for distributed processing of large-scale datasets. A simple 2-step parallel programming paradigm (Map, Reduce). Takes a big file as input. Not a database system.
Foreign key
An attribute K (or collection of attributes) in relation B that uniquely identifies a type in a relation A. Typically K is a primary key in A.
Horizontal Partitioning
Distributing the rows of a logical relation into several separate tables
Round-robin (Block) partitioning
Each next group of k rows are assigned to the next node. The simplest strategy. uniform distribution
Inter-query parallelism
Each query runs on one node. Several different queries execute concurrently on multiple processors.
Supervised Learning
Explicit data labels. Goal is to learn that, given a sample of data and desired outputs, best approximates the relationship between input and output observable in the data (image tagging, sentiment analysis)
Xquery differs from Xpath because
FLW expresssion
Page rank (definition and equation)
PageRank works by counting the number and quality of links to a page to determine a rough estimate of how important the website is. The underlying assumption is that more important websites are likely to receive more links from other websites Page Rank Formula: On Desktop
Benefits to Relational Model
Physical Data Independence (Oblivious to data organization on disk), Logical Data Independence (Scheme can evolve without affecting the users/applications).
Pros and Cons of Shared Disk Architecture
Pros: Easy to use, maintain and program Cons: Expensive, difficult to scale past a certain configuration, changes to codebase might be needed during migration
Pros and cons of shared-everything architecture
Pros: Easy to use, maintain, and program. Can migrate old codebase without major changes. Cons: Expensive, difficult to upgrade, difficult to scale past a certain configuration
Pros and Cons of Shared Nothing Architecture
Pros: Less contention for resources, the most scalable architecture. Cons: Difficult to administer, configure, and maintain. Changes to codebase required during migration
Inter-operator parallelism
Query runs on multiple nodes in parallel. Each operator runs of one node. enables different operators of the query to be executed in parallel. The more complex the query, the more opportunities there are for inter-operator parallelism.
Ranking Relevance Evaluation: F-Measure and equation
Recall + Precision. Equation on desktop
What can a learning algorithm do?
Recognizing Patterns, Generating Patterns, Recognizing Anomalies, Prediction
Main components of a relation schema:
Relation name, name of each attribute, domain of each field
Hash partitioning
Row r assigned to the node r.K mod P. applies a hash function to some attribute that yields the partition number. This strategy allows exact-match queries on the selection attribute to be processed by exactly one node and all other queries to be processed by all the nodes in parallel. May be skewed if k has the same value for all rows
Write a SQL query that lists the average length of movies played at Landmark Neptune Theatre
SELECT AVG(Length) FROM movies,movies2theaters, theaters WHERE theaters.Name = 'Landmark Neptune Theatre' AND theaters.TheaterCode = movies2theaters.TheaterCode AND movies2theaters.MovieCode = movies.MovieCode
Write a SQL query that shows how many times the Metro Cinemas theater has played a movie that is shorter than 2 hours
SELECT COUNT (*) FROM movies, theaters, movies2theaters WHERE theaters.name='Landmark Metro Cinemas' AND theaters.TheaterCode=movies2theaters.TheaterCode AND movies2theaters.MovieCode=movies.MovieCode AND movies.Length<120;
Company(name, city) Product(pname, maker) Purchase(id, product, buyer) Return cities having companies that manufacturer products bought by John (use nested query)
SELECT Company.city FROM Company WHERE Company.name IN ( SELECT Product.maker FROM Purchase, Product WHERE Product.pname = Purchase.product AND Purchase.buyer = 'John')
Indexing and two files that are created
Take the web pages, create an inverted index, output the inverted index. Split index into two files. Lexicon - hash table on disk (one read), main memory (has the words) Occurrence List: On Disk, distributed file system
Machine Learning
The study of algorithms that: improve their performance P, at some task T, with experience E. A well-defined learning task is given by <P,T,E>
what's a tuple
Tuple: is an element of S1 x S2 x ...x Sn,
is Kmeans supervised or unsupervised
Unsupervised
Normalized Discounted Cumulative Gain (NDCG) and equation
Using a graded relevance scale of documents in a search-engine result set, DCG measures the usefulness, or gain, of a document based on its position in the result list. equation on desktop
Where can subqueries be found
WHERE clause, FROM clause, HAVING clause (less common)
What a relational Database
a collection of relations
An inverted index in a web search is
a data structure designed to help find web pages relevant to the user query
Dummy Variables
a numerical variable used in regression analysis to represent subgroups of the sample in the study
Sentiment Analysis
a technique that allows marketers to analyze data from social media sites to collect consumer comments about companies and their products
kmeans
an iterative algorithm that tries to partition the dataset into Kpre-defined distinct non-overlapping subgroups (clusters) where each data point belongs to only one group. It tries to make the inter-cluster data points as similar as possible while also keeping the clusters as different (far) as possible. · Assign each point to its closest mean · Recompute the means to be the mean of the points assigned to each cluster
MongoDB
an open source document-oriented DBMS
Shared nothing architecture
architecture where nodes communicate over the network
K nearest neighbors does all the work during:
classification time
MongoDB Query Language Find
db.collection.find( <condition>, <projection>).limit (N)
MongoDB Query Language insert
db.collection.insert( <document> )
XPath one-liner which outputs the names and population of the countries with population less than 5 thousand
doc("world.xml")//country[@population<5000]/data((name, @population))
Xpath, which outputs all seas shallower than 200
doc("world.xml")//sea[@depth<200]/data(@name)
Xpath which outputs the population of Guadalajara
doc("world.xml")/mondial//city[contains(.,'Guadalajara')]/data(normalize-space(population))
Xpath that returns cities in Brazil location nearby seashore along with their population
doc("world.xml")/mondial//country[@name='Brazil']//city[located_at[@type='sea']]/data((name, population))
dimensionality reduction
find features that differentiate individuals. methods used to represent data using less columns or features,
FLWR expression which outputs the name and population of countries with population less than 5 thousand in order of descending population
for $x in doc("world.xml")/mondial/country where $x/@population < 5000) order by xs:int($x/@population) descending return <name> {$x/@name, $x/@population}</name>
FLWR expression which outputs a list of capitals of countries, where national inflation is greater than 10
for $x in document("world.xml")//country[@inflation>10] where $x/@capital=$x//city/@id return ($x//city/name/text())
FLWR expression which outputs a list of river names in ascending order
for $x in document("world.xml"//river/data(@name) order by $x ascending return $x
FLWR expression which outputs the population for each city in Spain
for $x in document("world.xml"/mondial//country[@name='Spain']//city/data(population) return $x
Ranking Relevance Evaluation: Precision
fraction of the documents retrieved that are relevant to the user's information need. (Relevant and Retrieved)/Retrieved
Ranking Relevance Evaluation: Recall
fraction of the documents that are relevant to the query that are successfully retrieved. (Relevant and Retrieved)/relevant
Goal of Data Science
improve decision making by extracting insights from large, heterogenous data sets.
Bayes Naive Algorithm assumes the features are
independent (but conditioned on the class)
Pagerank (static feature)
is an algorithm to calculate rank of a web page in a web graph
Reinforcement learning
label on the result of a sequence of actions, but not on each action (game, robotics). concerned with how software agents ought to take actions in an environment in order to maximize some notion of cumulative reward.
String similarity measure
number of characters you need to change in one string to modify into another string
A ranking feature to help rank search results:
page rank
A web page is a good authority if
pointed to by many good hubs
A web page is a good authority if it is
pointed to by many good hubs
A web page is a good hub
points to many good authorities
Data becomes dirty from
primary key violations, diff data representations, formatting differences
Parallel Database
seeks to improve performance through parallelization of various operations, such as loading data, building indexes and evaluating queries. The distribution is governed solely by performance considerations. Parallel databases improve processing and input/output speeds by using multiple CPUs and disks in parallel. In parallel processing, many operations are performed simultaneously.
Range Partitioning
selects a partition by determining if the partitioning key is within a certain range. An example could be a partition for all rows where the "zipcode" column has a value between 70000 and 79999. May be skewed - difficult to partition k into equal width chunks.
XML data is:
semi-structured
Parallel-Join
split the pairs to be tested over several processors. Each processor computes part of the join, and then the results are assembled (merged).
Stragglers in map-reduce
take a long time to finish
Stragglers
tasks that take very long time to complete and drag overall job progress (bad sectors on disk -> significant read/write performance degradation, scheduler assigned too many tasks on a single machine)
Omitted Variable Bias
the specific type of bias that results from the failure to include a variable that belongs in our regression model. We assume changes in the dependent variable that are correlated with the explanatory variable are because of the explanatory variable. We assume that changes in the dependence variable that are not explained by the explanatory variables is "noise."
clustering
the task of identifying subgroups in the data such that data points in the same subgroup (cluster) are very similar while data points in different clusters are very different. EX: find groups of customers with similar tastes, find topics within a set of news articles
Weakly/Distantly supervised
using noisy labels or partial labels (bootstrapping, automatically labeled data) This approach alleviates the burden of obtaining hand-labeled data sets, which can be costly or impractical. Instead, inexpensive weak labels are employed with the understanding that they are imperfect, but can nonetheless be used to create a strong predictive model
Relational Algebra expression for "Addresses and phone numbers of theaters that play 'Frozen' and query plan
Πtheaters.Address, theaters.Phone ((theaters ⋈ movies2theaters ⋈ (σTitle = "Frozen" (movies))) tree on desktop
Reduce in Map/Reduce
Sum, Filter, Transform, Aggregate. hich performs a summary operation (such as counting the number of students in each queue, yielding name frequencies)
Naiive Bayes practice session Positive Reviews: 1. Not bad, it is very fast 2. This is an awesome laptop 3. it has exceeded my expectations Negative Reviews 1. The fan noise is very bad. 2. Turned out to be factory defect. 3. You are going to hate it! P(+) = 1/2, P(-) = 1/2 Predict the class for 1. It is very bad. 2. Hate the noise P(+|words) P(-|words)
1. It is very bad. P(+|words) = 2/81. P(-|words) = 1/162 2. Hate the noise P(+|words) = 0, P(-|words) = 1/54
Google File System (GFS)
A distributed file system Google developed to meet rapidly increasing storage requirements; offers scalability to petabyte storage, the capability to handle extremely large files, data storage on distributed commodity servers, and simultaneous file access by multiple distributed applications. each file is N blocks (>= 64 MB), Blocks are randomly distributed among machines and replicated for fault tolerance. File system is optimized for append.
What is a relation
A subset of S1 x S2 x ...x Sn, where si is Domain of attribute i ' [1,n], and n is a number of attributes of R. A set of tuples
K-Nearest Neighbor
All work is done at classification time. Arguably the simplest ML algorithm. "Non-parametric" - no assumptions about the form of the classification model. Works with tiny amounts of training data. Assumes similar things exist in close proximity. The algorithm gets significantly slower as the number of examples and/or predictors/independent variables increase.
Broadcast Join
Broadcast join can be very efficient for joins between a large table (fact) with relatively small tables. Do not reshuffle R or S, Replicate S to all nodes.
Inverted index for the following: Doc with ID 1 = Data Science Studies methods of recording, storing, and deriving insights from data. Doc w ID 2 = Machine learning studies algorithms that learn from data and improve with experience. Lexi Columns = Word, Ndocs, PTR Occurrence list columns = DocId, occurs, Pos1, Pos2
Check phone
Semi-supervised learning
Combining large amount of unlabeled with smaller amount of labelled (pre-training)
Map in Map/Reduce
Computes a function based on the information read from each record in a file. Performs filtering and sorting (such as sorting students by first name into queues, one queue for each name).
Web Search - crawling
Crawl the web, store documents, check for duplicates, extract links -> web pages
Data Science Workflow
Data collection, Data processing, exploration/visualization, Analysis/machine learning, insight/policy decision
shared nothing architecture
Many nodes share nothing; each node has its own memory and disk. Nodes do not share memory or storage. Ex: HP Vertica running on a cluster. each update request is satisfied by a single node (processor/memory/storage unit).
Shared Disk Architecture
Many nodes which share the same disk(s) Ex: Oracle running on a cluster. All disks are accessible from all cluster nodes.
what takes into account order of returned search results
NDCG (Normalized discounted cumulative gain)
Unsupervised
No explicit labels. Its goal is to infer the natural structure present within a set of data points. "Finding structure in data" which is "exploratory analysis" · Used extensively for "pre-training" (dimensionality reduction, language modeling, auto-encoding) (clustering, dimensionality reduction)
What is shared-everything architecture
Nodes share disk and memory, Up to hundreds of CPUs. Ex: MySQL running on a single powerful machine.
Goal of classification
One goal: P(Y|X) where Y is label and X is features. P(email is spam | words in the message)
Intra-operator parallelism
Operator runs on multiple nodes in parallel Ex: Join running on several nodes. Intra-operator parallelism is based on the decomposition of a relational operator into a set of independent operator instances, each processing a different relation partition.
Gradient Descent
Optimization algorithm for finding the input to a function that produces the optimal value; iterative
Oversimplified model of ML and with more detail
Oversimplified Machine Learning: Data -> Model -> Goal/Task In more detail Machine Learning: Training Text Documents, Images, Sounds -> Features vectors, Labels -> Machine Learning Algorithm -> Predictive Model, Expected Label. Also, New Text Document Image, sound, features vector -> Predictive Model -> Expected Label.
Formula for likelihood estimation
P(C)Pi(n, I=1) P(xi, c)
how to declare key constraints when creating a table
PRIMARY KEY (sno, pno) FOREIGN KEY(pno) REFERENCES Part
SQL query that lists title, length, and movie time of movies that are played in the Cineplex Odeon Uptown theaters and relational algebra and tree
SELECT DISTINCT movies.Title, movies.Length, movies2theaters.movietime FROM movies, movies2theaters, theaters WHERE movies.MovieCode = movies2theaters.MovieCode AND movies2theaters.TheaterCode = theaters.TheaterCode AND theaters.Name = "Cineplex Odeon Uptown"; 3. Πmovies.Title, movies.Length, movies2theaters.movietime((movies ⋈ movies2theaters ⋈ (σname = "Cineplex Odeon Uptown" (theaters))) tree on desktop
Write a SQL query that lists the name and address of all theaters showing less than five different movies
SELECT DISTINCT theaters.Name, theaters.Address FROM theatersNATURAL JOIN movies2theaters GROUP BY theaters.NameHAVING COUNT(DISTINCT movies2theaters.MovieCode) < 5;
Write SQL and Relational Algebra that lists only the length of all movies with an R rating
SELECT Length FROM movies WHERE Rating ="r' Πlength σrating = 'R' (Movies)
SQL query that lists theater names, phone numbers and the number of different movies that are shown in these theaters sorted by this number (of different movies) in descending order
SELECT theaters.Name, theaters.Phone, COUNT(DISTINCT movies2theaters.MovieCode) FROM theaters NATURAL JOIN movies2theaters GROUP BY theaters.Name ORDER BY COUNT(DISTINCT movies2theaters.MovieCode) DESC;
Write a SQL query that lists all theaters in Capitol Hill District that play the longest R-Rated movie and out their names (in ascending order) and telephone numbers
SELECT theaters.Name,theaters.Phone FROM ((((districts INNER JOIN theaters2districts ON districts.DistrictCode=theaters2districts.DistrictCode)INNER JOIN movies2theaters ON theaters2districts.TheaterCode=movies2theaters.TheaterCode)INNER JOIN theaters ON movies2theaters.TheaterCode= theaters.TheaterCode) INNER JOIN movies ON movies2theaters.MovieCode= movies.MovieCode) WHERE districts.Name='Capitol Hill' AND Rating='R' GROUP BY theaters.Name HAVING MAX(movies.Length) ORDER BYtheaters.Name;
Write SQL and Relational Algebra that lists only the phone number and district name of Boeing Imax Theater
SELECT theaters.Phone, districts,Name FROM theaters INNER JOIN theaters2districts ON theaters.TheaterCode = theaters2districts.TheaterCode INNER JOIN districts ON districts.DistrictCode = theaters2districts.District Code WHERE theaters.Name = "Boeing Imax Theater" Πtheaters.Phone, district.Name(( σname = 'Boeing Imax Theater' (theaters)) ⋈theaters2districts ⋈ district)
Extended Relation Algebra. Duplication elimination, grouping, sorting. What are the operators?
Screenshot on laptop
Web Search Static and Dynamic Ranking Features
Static: Spam, Pagerank Dynamic: Term matching, Headings, Fonts, Anchortext, Title/Summary/Keywords/Body
What is a result in statistical linear regression vs a result in machine learning linear regression
Statistics: A "result" is typically in the form of a significant relationship and/or practically relevant effect size ML: A "result" is typically in the form of an improvement in prediction performance on a (held out) test set Regression in Statistics, makes claims about whether there is a meaningful relationship between X and Y Regression in ML, Focused on prediction accuracy; exploiting correlation is totally fine
While training linear regression this is minimized:
Sum of square errors
