MISY 160 Exam 3 University of Delaware
How to classify as big data
5 Vs: Volume, Velocity, Variety, Veracity, and Value
Veracity
Accuracy and trustworthiness of the data
Skill data analysts need to have
Analytical skills
What does a database administrator do?
Architecting how data is shared
Velocity
Big data is generated at a very high speed
Frameworks to store and process big data
Cassandra, Hadoop, Spark
Second phase in data science project
Data acquisition, which is gathering and scraping data from multiple sources like web servers, log files, databases, APIs, online
Roles/positions of a data scientist
Data analyst; Machine learning engineer; Deep learning engineer; Data engineer, Data scientist
Fifth phase in data science project
Data modeling
Third phase in data science project
Data preparation, which involves data cleaning and data transformation
What data cleaning is
Dealing with inconsistent data types, misspelled attributes, missing and duplicate values
Seventh phase in data science project
Deploying and maintaining the model
What does a software developer do?
Develop applications
Values of data science
Discovering the best time and route to ship for logistics companies like DHL, Fedex and predicting employee attrition
What Map Reduce is
Doing parallel processing: Lengthy task is broken into smaller tasks, one machine takes up one task, multiple machines complete
Example of structured data
Excel records
Fourth phase in data science project
Exploratory data analysis
How Hadoop distributed file systems
File is broken into smaller chunks, making copies of them, and storing them in various machines
What does a quality assurance analyst do?
Finding problems within the applications, testing the applications to find problems - what the system is supposed to accomplish
Benefits of big data
Improving users experience in the game industry, predicting a hurricane landfall earlier in weather forecasting and disaster
Example of semi-structured data
Log files
What does a data analyst do?
Look at data to find meaning to solve real business problems
What does a project manager do?
Managing people, keeping track of schedules, communicating risks
Big data
Massive amounts of data
First phase in data science project
Meeting with clients, asking relevant questions, understanding and defining objectives for the problem that needs to be tackled
Tools for data modeling
Python, R, SAS
Various data types
Structured, semi-structured, unstructured data
Tools for data visualization and communication
Tableau, Power BI, QlikView
Tools for complex data transformation
Talend, Informatica
Value
The benefit from analyzing the data
Skill software developer needs to have
Understanding progragramming languages and databases
What does a business analyst do?
Understanding what the business is trying to accomplish, the processes, the stakeholders, the goals, and the current system
Sixth phase in data science project
Visualization and communication
Example of unstructured data
X-ray images
