Big Data I Final
Machine Learning
*Field of study that gives computers the ability to learn without being explicitly programmed.* - Allows computers to find hidden insights without being explicitly programmed where to look. - AI that learns predictive models from data. Is a method of data analysis that automates analytical model building.
NOSQL
- NON RELATIONAL- Designed to meet scalability requirements of architectures. Most scalable--> Up to 10PB - Used for analysis and HANDLES big data structures. - New breed of non-realtional database products - Rejections of fixed table schema and join operations. - Schema-less data management requirements. - Handles structured, semi-structured, non-structured
Bit
0 or 1
Gigabyte
1 billion bytes
Megabyte
1 million bytes
Terabyte
1 trillion bytes
Reinforcement Machine Learning
1. Decision Process 2. Reward System 3. Learn series of actions -Area of machine learning concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. - Learns from environment and experience NOT DATA.
Drivers of Big Data
1. Digitization of society 2. Lowering of technology costs 3. Connectivity through cloud computing 4. Social media applications 5. Upcoming Internet of Things
Data Science encompasses what 3 skills?
1. Hacking Skills 2. Math and Stats skills 3. Domain Expertise - Hard to find because the complete skill set is hard to find. It's better to have a TEAM to equate to a data scientist.
Supervised Machine Learning
1. Labeled data 2. Direct feedback 3. Predict outcome/future- Based on input and output data. - FINALIZES into CLASSIFICATION and REGRESSION
Unsupervised Machine Learning
1. No labels 2. No feedback 3. Find the hidden structure - Helps find previously unknown patterns in data set without pre-existing labels. - Based on only input data. - FINALIZES into CLUSTERING
3 Types of Machine Learning
1. Supervised 2. Unsupervised 3. Reinforcement
Kilobyte
1000 bytes
Petabyte
10^15 bytes or quadrillion
Byte
8 bits
Data Mart
A data warehouse that stores only data relevant to a division or subject.
Business Intelligence
A framework for decision support .- Includes architecture for the data management cycle. -Includes tools for mining, analysis, and interpretation sometimes.
Variety
A number of different kinds; assortment
Algorithm
A process or set of rules to be followed during calculations and other problem solving operations. Ex. A GPS calculation of the fastest route
How is AI related to Big Data?
AI encompasses all the powerful techniques that big data uses and it used to train the most powerful techniques in machine learning.
Descriptive Analytics
AKA: Reporting analytics or exploratory analytics.- Refers to knowing what is happening in an organization and understanding some underlying trends and causes of such occurrences.- VISUALIZATION IS THE KEY.Ex. (Business reporting, Dashboards, Trend Analysis, Scoreboards, DW)What actually happened in the data? NO ASSUMPTIONS, JUST REALITY. What is happening?
Scalability
Capability of distributed computing. Platform to handle a growing amount of work-its potential to be scaled up to accommodate more data.
Structured Data
Data that can be fit in tables, Excel SS, Databases, RDMS.
Unstructured Data
Data that cannot be fit into tables. Ex. Images, Text data
Noisy Data
Data with a large amount of additional meaningless information.
NewSQL
Designed to scale out and have ACID properties .- Used for OLTP & HTAP- Designed to meet scalability requirements of architectures. - RELATIONAL - Handles structured Data - Atomic transactions
ELT vs ETL
ETL* can be very time consuming and costly and ELT* is faster and more scalable. Extraction, Loading, and Transformation - More common for internal organizational data- Examples of when ELT are better are: - When there are big volumes of data - When the source database and target database are the same. - When the database is well adapted for that kind of processing, such as NoSQL databases and paralleled data warehouses.
Data Warehouse
For storing and querying data. It's a pool of data produced to support decision making. It is a repository of current and historical data of potential interest to managers throughout the organization.
Hadoop vs. Spark
Hadoop requires developers to hand code each and every operation whereas Spark is easy to program with RDD - Resilient Distributed Dataset. ... Hadoop is designed to handle batch processing efficiently whereas Spark is designed to handle real-time data efficiently. - Spark can be 100x faster and expensive $$$ than Hadoop. - Spark allows you do data analysis on Jupiter notebooks. (Easier to use.)- Program runs on HDFS but puts it into the memory of each individual computer, instead of a mapping paradigm.
Semi-Structured Data
In-between Structured and Unstructured data. Some of it has structure and can fit into a Relational Database Management System and the half can't. Ex. Twitter Data
Facebook and Twitter use ____ _____ databases.
No SQL
Data Lake
Repository of data stored in its natural format, usually object blobs or files. A data lake is usually a single store of all enterprise data included raw copies of source system data and transformed data used for tasks such as reporting, visuals, analytics, or ML.- Hadoop can act as a data lake.- Data lakes are independent.
Prescriptive Analytics
Seeks to recognize what is going as well as the likely forecast and to make decisions to achieve the best performance possible.Ex: (Optimization, Simulation, Decision modeling, Expert Systems.)- Historically studied as OPERATIONS RESEARCH or MANAGEMENT SCIENCE and has generally been aimed at optimizing the performance of a system.- GOAL is to provide a decision or recommendation of an action.
SQL
Structured Query Language - Programming language for RELATIONAL database systems. - SCALES UP to 10TB. - Handles structured data. - Atomic transactions
Volume
The amount of space an object takes up
Business Analytics
The application of models to business data to support decision making
Noise
The meaningless information in noisy data.
Signal
The valuable information from noisy data.
Predictive Analytics
Use of tools to predict future unknown events.- Uses techniques from data mining, statistics, modeling, machine learning and AI to analyze current data to make such predictions.
4 V's of Big Data
Volume, Velocity, Variety, Veracity
Dirty Data
is erroneous or flawed data
Velocity
the speed and direction of a moving object
Veracity
truthfulness, honesty
