What is Data Science?

Historically, it can be said that [x] have focused on generative modeling (inference), while [y] have prioritized predictive modeling .

x=generative model or statistics y=DS, CS, predictive model,2%,

What is Data Science?

"The scientific process of extracting value from data" . procedures for analyzing data, techniques for interpreting the results of such procedures, ways of planning the gathering of data to make its analysis easier, more precise or more accurate, and all the machinery and results of (mathematical) statistics which apply to analyzing data

While at Bell Labs, William S. Cleveland suggested effort be allocated into six areas What was % ofeach area?

(*) Multidisciplinary investigations (25%) (*) Models and Methods for Data (20%) (*) Computing with Data (15%) (*) Pedagogy (15%) (*) Tool Evaluation (5%) (*) Theory (20%)

Data Modeling

Practiced in using inferential and predictive approaches to answer interesting questions

CTF has these ingredients

(a) A publicly available training dataset involving, for each observation, a list of (possibly many) feature measurements, and a class label for that observation. (b) A set of enrolled competitors whose common task is to infer a class prediction rule from the training data. (c) A scoring referee, to which competitors can submit their prediction rule. The referee runs the prediction rule against a testing dataset which is sequestered behind a Chinese wall. The referee objectively and automatically reports the score (prediction accuracy) achieved by the submitted rule.

According to Donoho, which of the following is NOT true? 1-Doing data science requires that the data you're working with are 'big data.' 2-Being a data scientist requires coping skills for dealing with the pains of large-scale computing. 3- Doing data science requires that the data you're working with are 'big data.' 4-Many statisticians argue that they already are data scientists and question the need for a new field of data science. 5-Data science should be a true science that uses scientific rigor to answer interesting questions. 6-Often the technologies taught in university are different than those used by the data scientist in their job.

1,3

What are the six divisions described in Donoho's 50 Years of Data Science?

1. Data Exploration and Preparation 2. Data Representation and Transformation 3. Computing with Data 4. Data Modeling 5. Data Visualization and Presentation 6. Science about Data Science

What are the four major influences of DS today, as defined by Tukey?

1. The formal theories of statistics 2. Accelerating developments in computers and display devices 3. The challenge, in many fields, of more and ever larger bodies of data 4. The emphasis on quantification in an ever wider variety of disciplines

What are the arguments for and against Data Science existing as a discipline?

AGAINST: (The "Jobs", "Skills","Big Data Meme")Statisticians claim that they are analyzing data too, which is the same thing. You can't say that 'big data' is a criterion for meaningful distinction between statistics and data science because statisticians have been using big data for hundreds of years. Mathematical statistics researchers have pursued the scientific understanding of big datasets for decades. Due to larger constraints posed by the multiprocessors and networks and a lot of artifacts, data science skills have evolved not because of solving the real problem of inference from data. FOR: Data Science is not math because it is defined by a problem, not a concrete subject. DS is intellectual content, organization in an understandable form, reliance upon the test of experience as the ultimate standard of validation.Tukey identified the four MAJOR influences on DS today

Where do data scientist work?

AirbnB. use image processing to make sure every picture is correct. and use A/B Testing as a way to figure out whether you should do A or B.

What do data scientists do?

Ask interesting questions and then use data to answer them!

Data Visualization and Presentation

Generation of plots that help you explore your data as well as those that help you effectively communicate your findings

Data Exploration and Preparation

Getting the data into a usable format, using detective-like work to check the data, and identifying artifacts.

Who is Tukey?

He wanted DS to become it's own field

In 1962, John Tukey, a famed statistician, predicted that data science was on the horizon. Which of the following did Tukey NOT see as a driver in the field that would push us toward data science? General shift toward quantification Increasing Data Availability Advances in computing Correct Impressive salaries Statistical theory

Impressive salaries

Computing with Data

Knowledge of a number of programming languages/packages as well as approaches to computational efficiency.

Dr. Pepper Snapple

Outcome:Execution sell-inland improved hand improved by 50% since the previous month after using MyDPS on iPad. convincing the president to green light the project

two goals in analyzing the data:

Prediction. To be able to predict what the responses are going to be to future input variables;predictive MODELING • [Inference].23 To [infer] how nature is associating the response variables to the input variables.GENERATIVE MODELING

What's so good about CTF?

The CTF is known for improving predictive modeling, unemotional judging, and many other machine learning real-world applications like the Netflix challenge

Science about Data Science

Studying how data analysis is done by others.

Did Tukey state that statistics is only a fraction of what DS is?

Yes

How do I improve ride sharing?

The longer that a person has the wait for the ride, the more likely they are to cancel the ride. Measuring how impatience affects someone ride?People are getting more impatient

Describe the studying coordinated migration example:

To study between-city coordinated migration, we examined aggregate, anonymized data on all users who lists both their hometown and their current city on their Facebook profile?

Describe the OkCupid example:

Took all of the questions from their database and got rid of questions that were not indicative of a long relationship. If you want to know... Do my date and I have long-term potential? Ask your date and yourself:Do you like horror movies?Have you ever traveled around another country alone? Wouldn't it be fun to chuck it all and go live on a sailboat?

Data Representation and Transformation

Understanding the data you have and in what form the data will need to be in order to answer your question of interest

Did Tukey call for a much broader field of statistics?

Yes

Shell:

reduced inventory analysis from over 48 hours to les than 45 minutes, shaving 45 minutes, shaving millions of dollars a year off the cost moving and re-allocating inventory

What does FoDA stand for?

the future of data analysis

While at Bell Labs, William S. Cleveland suggested effort be allocated into six areas. Which of these six areas does Donoho propose many academic statistics departments focused the vast majority of their efforts?

theory

Describe the OkCupid graph

y-axis:percent of people that agree on a question.The blocks represent the top three questions that people ranked as being the most important to them in a partner. light blue: expected from pure chance. the right side shows that if you agree on those 3 questions you are more likely to be in a long term relationship with that person