Data Science Final
How can you use Excel's VLOOKUP() function to transform data from one value to another?
-=VLOOKUP(value, array, column index) -Result is lookup value OR #N/A, depending if it was able to find the value
What is the "Quantified Self" movement and how does it relate to data? To KPIs?
-Apps that help us learn more about ourselves -Quantified all your moves/seconds/ amount you sleep -ex. fitbit -Insurance companies can change your premiums based on these numbers
What is a Database?
-Bytes grouped into columns (aka fields) -Columns grouped into rows (aka records) -Group of similar records, table or file -DB- collection of tables plus relationships among rows in tables, plus special data called metadata, which describes DB structure
What is data science? What is data (vs. information)?
-Data - raw facts, raw numbers, don't mean anything unless you perform action -Information - putting data in meaningful context -Knowledge - information in action
What is data integration? Why is it necessary?
-Data integration involves combining data residing in different sources and providing users with a unified view of them. -It is necessary in order to connect data sources together in order to merge
What to do to narrow confidence interval?
-Decrease confidence interval -99% to 95%
How do you clean it?
-Fix how you collect data (fix the structure) instead of going back and fixing the data itself -Correct spellings, redo calculations, look to the raw data.
Explain the "tyranny of success" and how a reliance of KPIs and exacerbate the problem. What can you do in those situations?
-Grey area -not everything is a yes or no -You can provide more options when collecting data to ensure all bases are covered
Tableau Functions
-IF milk/soda > 1 then set the value of MostExpensive to "Milk" -Else if milk/soda <= 1 then set the value of MostExpensive to "Soda" -IIF (test,then,else [unkownn])
What to do to widen the confidence interval?
-Increase confidence interval -95%-99%
What is the difference between infographics and data visualization? What are the similarities?
-Infographics may contain data visualizations -Infographics tell stories -Data visualizations explain data
How can you use Excel's MATCH() function to find invalid values?
-MATCH (E2,Lookup!Names$A$3:$A$54, 0) = row # -Value searches array, if there is a match then it will output row number
What is open data? The three "V"s of big data?
-Open data- free and accessible to everyone -Velocity, Volume, and Variety
Explain why each of the "SMART" criteria are important
-Specific purpose for the business (clearly define goal) -Measurable (results should be tangible) -Achievable by the organization (Goal should stretch you slightly enough to challenge you) -Relevant to success (goals should measure outcomes, not activities in the process) -Time-phased (Time frame must be incorporated)
Differences between statistics and machine learning
-Statistics deals with all assumptions of the validity of the model Always need to justify the model in statistics -ML deals with predicting the outcome of a model (model validation is accuracy)
What is the difference between structured and unstructured data?
-Structured is easily searchable with basic algorithms. Data is formatted in a way that all the related values is under separate columns with the same type and format -Unstructured is more of human language and doesn't fit well in a relational database. Images, sound files, etc. They do not follow any structure; there is not one single type. It is difficult to manage unstructured data.
What are some examples of structured and unstructured data?
-Structured: numbers, dates, strings -Unstructured: emails, videos, pictures, presentations, audio files
What is sentiment analysis? How does it work?
-The process of computationally identifying and categorizing opinions expressed in a piece of text, especially in order to determine whether the writer's attitude towards a particular topic, product, etc. is positive, negative, or neutral . Singular words within the piece of text are evaluated as positive or negative
Best practices of dealing with dirty data
-focus on getting new data right -limit time fixing old data -data producers should communicate with data consumers -check work
Characteristics of open data
-it can be redistributed to others -it can come from any source -it can be available for free
How does data get dirty
-missing values -inconsistent data -nonintegrated data -wrong granularity (too fine or too coarse) -too much data
4 Types of Sentiment analysis
-sentence-level -document level -aspect-based -comparative-level
What does it mean for data to be dirty
-spelling errors -punctuation errors -incorrect data in specified field -duplication data -non integrated data
According to Redman, which is cause of bad data
-users of data create "work arounds" instead of addressing root causes -people creating the data don't understand how others will use it -Creators and users of data have poor communication
What are the steps for communicating an analysis (and what do they mean)?
1)Understand the problem 2)How will I measure? 3)What data is available? 4)Initial solution hypothesis 5)Solution 6)Impact of solution
According to Hayes, what percentage of business leaders do not trust the information they use to make decisions
33%
the dangers of big data analytics
33% of Managers don't trust data
People spend ____ of the time on data cleaning, the rest is on _____. They are:
50% data analysis Searching for data correcting errors verifying correctness
Stein was able to eventually predict the gender of a caller
80% of the time
What is a (good) hypothesis? What are the criteria?
A hypothesis does not always have to be true - (testable predictions) it needs to be testable, falsifiable, and grounded in rationale.
What is the difference in definition and purpose between a scorecard and a dashboard?
A scorecard only shows you binary responses - a check or an x. A dashboard can be interactive and has a scale - showing you exact instances of what is being measured.
How do tables of data become associated in a relational database?
By creating relationships between them. Think: we do Tableau to connect. There should also be a primary key (EmpNo, for example) to connect all tables
What are the benefits of sentiment analysis?
Can help determine how people feel towards a brand, product, etc.
Gandel cites the case of Barclays effort to purchase Lehman Brothers, where the following Excel error resulted in the accidental purchase of 179 toxic assets
cells were hidden instead of deleted
What is the best data visualization to find outliers? To create scorecards?
Data visualization for outliers is scatter plot.
According to Unwin, one issue with map-based graphical visualizations is that
Distance is not directly related to similarity
According to Hoven, which of the following is NOT one of Few's 8 core principles of data visualizations?
Explain (some that are include Attend, simplify, be skeptical)
According to Acohido, Microsoft uses all of the following data to combat cybercrime EXCEPT
FBI watchlist (some that are included are threat reports, malicious files, early warning reports)
How do you resolve conflicts in data (i.e., PA versus Penna. Versus Pennsylvania)?
Find which name was recorded in the Lookups tab and change any differing names to match it
What are the uses of forecasting?
Forecasting sales amount for the future (predictive) To widen the range means better accuracy (moving from 95% to 99% accuracy, increasing the number of data points for better accuracy)
According to Farmer, the purpose of creating an information scent is to
Give the user hints where they should explore further
What is the difference between Hadoop and MapReduce?
Hadoop allows you to store the big data while MapReduce is a software that allows you to perform tasks on the slices of data across servers.
According to Peck, the use of analytics to determine workers' potential is most widely used in
Hourly work, where the jobs are standardized
How do you create a KPI for a scorecard using a calculated field in Tableau?
IIF
When should you not bother resolving conflicts or even fixing your data?
If it's too damaged
What are the uses of association mining?
Items that are usually purchased together (eg. diapers and beer)
What is a KPI
Key Performance Indicator: a quantifiable measure used to evaluate the success of an organization, employee, etc., in meeting objectives for performance
According to Crawford, a key problem of Boston's StreetBump app is that
Low income residents have less access to smartphone
Hadoop is often paired with another piece of software called
MapReduce
How do they facilitate the analysis of Big Data?
MapReduce operates on slices of data (the stations are the workers). Data is processed by each slice with the main finding reported back to the main machine (the queen). It allows big data to be analyzed more efficiently
What does Crawford propose was the reason for Google's overestimation of flu outbreaks
Media coverage of the flu season
What is the difference between metadata (data dictionary) and data?
Metadata- self-describing data
The Tableau article suggests the following to when creating dashboards
Minimize distracting text and formatting
In the article by Weisberg, Eli Pariser argues that the Filter Bubble is caused by
Personalization of web content
MapReduce
Programming framework that allows us to perform distributed and parallel processing on large data sets in a distributed environment
According to Strickland, which of the following is a type of data integration approach
common user interface common data storage manual integration
Which of the following is the most appropriate technique to analyze the statement "Coke tastes better than Pepsi"
comparative sentiment analysis
What is the difference between relational databases and data formatted for Pivot Table analysis?
Relationship-separate themes in different files. In a flat file- everything is merged into one big file so you can make pivot tables and analyse the data as a whole. Same structure (column/row), same number of records, etc.
According to Feldman, the most common application of sentiment analysis is
Reviews of consumer products and services
Which of the following are part of the SMART criteria for KPIs
Specific Relevant Measurable Relevant Attainable(?)
According to Bialik, a key issue with counting steps as a measure of fitness is
Steps are not the only kind of exercise All steps do not burn the same calories Tools like Fitbit don't accurately count steps
How is data arranged in a relational database? For Pivot Table analysis?
Stored in a group of similar records (table (file))
What is the difference between Tableau and Excel?
Tableau: data visualization tool, organizes data Excel: Spreadsheet tool
The Ashley Madison hack is different from previous hacks in that
The Ashley Madison hack resulted in more personal damages to users
According to Krum, the fact that people remember messages with images more often than ones with just text is called
The Picture Superiority Effect
According to Peck, "people analytics" is
The application of predictive analytics to people's careers
According to Di Justo's article, "telephony metadata" includes
The callers duration
In what situations can sentiment analysis be inaccurate?
The meaning of the words can change in respect to how they are used
Data transformation for association mining
The two conditions we need to apply: -order IDs should be the same -product names should differ
According to Olson, insurance companies are seeking real-time customer data so that
They can build better customer risk profiles
FiveThirtyEight's search for America's best burrito began with data from
Yelp
How should you deal with outliers?
You hide them or you fix the incorrect formula
What is a scoreboard
a statistical record used to measure achievement or progress toward a particular goal.
A basic rule of the Pivot Table data structure is that
all values of the same type need to be in one column
According to Silver's article "What the Fox Knows," the "explanation" step involves
answering the questions "why" and "how"
The most detailed type of sentiment analysis is
aspect-based sentiment analysis
Explain the impact of bad data on future decision-making processes
could damage the creator and consumer relationship
Too many attributes (in dirty data)
curse of dimensionality
According to Olsen, real-time customer data could result in the ability to adjust premiums
daily
According to Taber, what is a common way Excel corrupts imported data
data are often converted into integers (a single number)
According to Bertolucci, the simplest type of analytics is
descriptive
What is a theory? What is the "direction of causality"?
direction of causality - A correlation between two variables does not indicate which variable is causing which.
Predictive analysis
ex. stock exchange prices (because it is unknown you need to try to predict it based on trends)
According to Wohlsen, Facebook's recent study of its users revealed
exposure to fewer positive messages led to fewer positive posts
Hadoop
framework that allows us to process and store huge data sets. The data is stored on multiple servers
Stein developed a model that could determine the gender of a caller using
his phone records from Google Voice
According to Unwin, a scale is 'really nice' if it
includes 0
According to Krum, the relationship between infographics and data visualizations is best described as
infographics can include visualizations within them
According to Hayes, a benefit of large samples is that
it minimizes sampling error
Which of the following an example of unstructured data
stock prices
Gandel cites the case of Reinhart and Rogoff, where an excel spreadsheet error:
led them to accidentally exclude five countries from their analysis
According to Gallagher, the RNC wants to customize the experience for visitors to GOP.com by
looking at past interactions with the GOP.com site
According to Microsoft, before cleaning your data you should
make a backup copy of the data in a separate workbook
According to Krum, good infographics should
make sure the relative size of chart elements are proportional to the data values
Hadoop is a platform that
makes big data easier to manage
Bertolucci claims an ongoing problem with Hadoop for companies is that
manages and executives don't really understand what it does
Prescriptive analysis
one step ahead of predictive, gives options on what to do
In Matlin's article, Whong states that his NYC Taxi Cab visualization is part of a larger movement for
open data and transparency
According to Paine, people have been tracking soccer data for
over 60 years
According to Davenport, what is an example of something that should not be included while storytelling with data
sequence of activities used in the analysis
Too many data points (in dirty data)
statistical sampling
According to Davenport, the "essence" of analytical communication includes
the data the model the relationships among the variables
According to Schambra, the problem with scoring non-profit outcomes as "success" or "failure" is
the distinction between success and failure is not always clear
Relational databases follow to a set of practices called
the rules of normalization
According to Paine, an analysis found that a team's probability of scoring increases as
they string together more successful passes
According to Gallagher, what is a key goal of the RNC's new data platform?
to create a single data platform for all Republican candidates
According to Unwin, the reason for using graphic displays is:
to present or explore data
According to Hurwitz, Hadoop is capable of handling _______ data.
unstructured
"The Agency Problem"
when the data creator is usually not the data consumer (creator and consumer must have a connection)
