Big Data Exam 2

Machine learning

"is a current application of AI based around the idea that we should really just be able to give machines access to data and let them learn for themselves"

Consistency

- Values are the same/agree between data sets - best controlled with application and database level constraints

Data in Data Mining

- a collection of facts obtained as the result of experiences, observations or experiments - consist of numbers, letters, words, images, voice recordings - structured, unstructured, semi-structured

Documentation

- a data dictionary should be available to anyone in the organization who collects or works with data (data field names, types, ranges, constraints, defaults) - who is responsible for which data - where is the data collected - what is the data collection process

Operational data stores

- a database which integrate corporate data from different data sources in order to facilitate operational reporting in real-time or near real-time - used for short- term decisions rather than medium or long-term decisions - unlike the static content of a data warehouse, the contents of a ODS are updated throughout the course of business operation

Enterprise data warehouse

- a large-scale data warehouse for the enterprise - provides integration of data from many sources into a standard format - provides data for many types of Decision Support Systems (DSS) including supply chain management, product life-cycle management, revenue management and knowledge management systems

Data Warehouse

- a physical repository where relational data are specially organized to provide enterprise-wide, cleansed data in standardized format - a pool of data produced to support decision making. it is also a repository of current and historical data of potential interest to managers throughout the organization

Business Intelligence

- a technology-driven process for analyzing data and presenting actionable information to help executives, managers and other corporate end users make informed business decisions -organizations use them to make sense of data and to make better decisions - term that includes databases, applications, methodologies and other tools used for executive and managerial decision making

Data Warehousing

- accessing, organizing and integrating key operational data in a form that is consistent, reliable, timely and readily available wherever and whenever needed

6 quality dimensions

- accuracy - completeness - consistency - conformity - integrity - timeliness

Nonvolatile

- after data entered into a DW users cannot change or update them; any changes are recorded as a new data - possible to delete

Predictive analytics

- aims to determine what is like to happen in the future. it is based on statistical techniques as well as other more recently developed techniques that fall under the categories of data mining or machine learning

Benefits of DW

- allows end users to perform extensive analysis - allows a consolidated view of corporate data - better and more timely information - enhanced system performance - simplification of data access

2-tier architecture

- application server runs on the same hardware platform as the data warehouse - advantage: more economical

Timeliness

- as up to date as possible - time should be reflected in the data and/or report - the more timely the data, the more costly and difficult to produce - very context oriented

Data Mining

- certain names are more prevalent in certain US locations (O'Brien, O'Reilly) - group together similar documents returned by search engine according to their context

Correction

- change data values not recognized - misspellings (auto correct) - default values to correct data type errors

Analysis techniques

- classification - regression - clustering - association analysis

Abbreviation expansion

- clearly define - INC for incorporated - ST for street - USA for United States of America

Completeness

- contains all the required information - nothing is missing - all data is usable (no errors, all data is "understood") - if it doesn't exist at the time of the analysis, recreation is rarely successful - best controlled when planning process for data collection occurs

Include metadata

- data about data - contains information how data is organized and can be used effectively

Organization

- data can be examined for errors more quickly if it is reasonably organized (sorted by entity, date, location)

Typographical and transcription

- data entry errors - misspelling and abbreviations erros - miskeyed letters

Big data quality

- data is inherently unclean (typically, the more it is unstructured, the less clean it is) - data can speak volumes but have little to say (noise) - real world data is messy

Planning

- data management is a process that must be guided from start to end - understanding the organization's needs and the data that supports those needs is the place to start - data structures and data collection should be controlled to best facilitate the needs of the organization

Types of data warehousing

- data marts - operational data stores - enterprise data warehouses

Subject oriented

- data organized by subject such as sales, products, customers - enables users to determine how their business is performing and why - provides a more comprehensive view of the organization

Time-variant (time series)

- data saved over multiple time periods (daily, weekly) - enables decision makers to detect trends, deviations and long-term relationships - every DW should support time dimension of data quality

Separation of Duties

- data should be collected at a minimum number of places in the organization - student data should be entered once by the registrar and then used by other areas

Data cleansing framework

- define and determine error types - search and identify error instances - correct errors - document error instances and error types - modify data entry procedures to reduces future errors

Act (feedback)

- determine what action to take - figure out how to implement the action - monitor and measure the impact of the action - evaluate the action based on success criteria you defined at the beginning - any revision?

Accuracy

- does the data correctly reflect what is true? - should agree with an identified source - may be difficult to detect because of data errors - best controlled when data is entered as close to the source as possible

Data cleansing process

- eliminate duplicate records - sorting method to find duplicates - problem with finding non-exact duplicates

Education

- everyone in the process should be responsible for ensuring data quality - a good understanding of why data quality is important and ways to manage quality is critical - everyone in the process must be proactive not only with the data of which they are in charge, but anything unusual they may see

Format Conformance

- expected format - ex: date formats differ between countries - month/day/year - day/month/year

Updating missing fields

- fill fields that are missing data if reasonable - may be caused by errors in original data

Data cleansing

- has a definite process but that process is flexible - not all organizations view quality in the same way, so not all organizations clean in the same way - all processes include first finding errors, and then correcting them - not all organizations do it the same way, but its important nevertheless

Data quality software

- helps with optimizing data quality and data profiling software is available that helps understanding data structure, relationships and content

Analyze

- includes selection of analytical techniques to use, building a model of the data, evaluation of analytical results, and creating reports and data visualizations to showcase the results - input data > select analysis techniques > model > model output

Three goals of Integrate

- integrate all data that is essential for our problem - clean the data to address data quality issues - transform the raw data to make it suitable for analysis

Decision support systems

- interactive computer-based systems which help decision makers utilize data and models to solve unstructured problems - couple the intellectual resources of individuals with the capabilities of the computer to improve the quality of decisions. it is a computer-based support system for management decision makers

Organize

- involves looking at the data to understand its nature, what it means, its quality and format - it is a part of the two-step data preparation process - aims for some preliminary explore in order to gain a better understanding of the specific characteristics of data

Implicit and explicit Nullness

- is absence of a value allowed? - implicit nulls: missing allowed - explicit nulls: what value is to be used if data is missing? (telephone number)

Prevention

- it can be very difficult to change data so quality data collection methods are necessary - includes both application and data structure design (dropdown boxes to minimize text entries, ranges, data types)

Data mining

- knowledge discovery from data - extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patters of knowledge from huge amounts of data

Floating Data

- lack of clarity as to what types of data go into specific fields - data placed in wrong field - consider address 1 and address 2 fields: which is for the street address or are both for the street address?

Not data mining

- look up phone number in phone directory - query a web search engine for information about "Amazon"

Concerns about real-time DW

- not all data should be updated continuously - mismatch of reports generated minutes apart - may be cost prohibitive - may also be infeasible

3-tier architecture

- operational systems contain the data and the software for data acquisition in one tier, the application server is another tier, and the third tier includes data warehouse - advantage: less recourse constraint, enables data marts

Data cleansing guidelines

- planning - education - organization - separation of duties - prevention - documentation

Transformation errors

- reconstruct data field size may truncate existing data (jones becomes jon) - changing data field type may change existing data (date becomes a number)

Descriptive analytics

- refers to knowing what is happening in and organization understanding some underlying trends and causes of such occurrences - first involves consolidation of data sources - visualization is key to this exploratory analysis step - ex: dashboard

Active Data Warehouse

- strategic and tactical decisions - results measured with operations - only comprehensive detailed data available within minutes - high number of users accessing and querying the system simultaneously - flexible and hoc reporting, as well as machine-assisted modeling to discover new hypotheses and relationships

Traditional Data Warehouse Environment

- strategic decisions only - results sometimes hard to measure - daily, weekly, monthly data currency acceptable; summaries often appropriate - moderate user currency - highly restrictive reporting used to confirm or check existing processes and patterns; often uses pre developed summary tables or data marts

Association analysis

- the goal is to come up with a set of rules to capture associations within items or events. the rules are used to determine when items or events occur together - ex: famous diaper beer connection

Best Practices for implementing DW

- the project must fit with corporate strategy - it is important to manage user expectations - the data warehouse must be built incrementally - adaptability must be built incrementally - adaptability must be built in from the start - the project must be managed by both IT and business professionals - only load data that have been cleansed/high quality - do not overlook training requirements

Integrity

- this primarily applies to relationships within the data structure - all data in the structure should be retrievable regardless of where it is - this is why keys are so important in the data structure - best controlled at the time of data structure development

Parsing

- to divide a group of tokens.. used to find patterns in the data in order to standardize values - look for patterns (find two spaces in the name field, divide field into first, middle and last)

Overloaded Attributes

- too much information in one field - Joe Smith IRA - Joe Smith in Trust for Mary Smith - John and Mary Smith (implies they are married)

Regression

- when your model has to predict a numeric value instead of a category, then the task becomes a regression - example: predict the price of a stock

Steps to start a big data project

1. Define the problem 2. Assess the situation 3. Define the purpose

DW architectures

1. Information independence between organizational units 2. upper management's information needs 3. Urgency of need for a data warehouse 4. Nature of end-user tasks 5. constraints on resources 6. strategic view of the data warehouse prior to implementation 7. Compatibility with existing systems 8. Perceived ability of the in-house IT staff 9. Technical issues (particularly ETL) 10. Social/political factors

Important criteria in selecting an ETL tool

1. ability to read from and write to an unlimited number of data sources/architectures 2. automatic capturing and delivery of metadata 3. a history of conforming to open standards 4. an easy-to-use interface for the developer and the functional user

Issues affecting the purchase of an ETL tool

1. data transformation tools are expensive 2. data transformation tools may have a long learning curve

4 components to a BI system

1. data warehouse- storing and querying data 2. business analytics- for manipulating, mining and analyzing data 3. business performance management- monitoring and analyzing performance 4. user interface- controlling the system and visualizing data

Indirect benefits of data warehouse

1. enhance business knowledge 2. present competitive advantage 3. enhance customer service and satisfaction 4. facilitate decision making 5. help in reforming business processes

Conformity

For instances of similar/same data - same data type - same format - same size (date of graduation) - best controlled at the time of data structure creation

Pre-processing

Has two steps: organize and integrate

Summary statistics

Mode, mean, median and standard deviation provide numerical values to describe your data

Classification

Predicting the weather as being sunny, rainy, windy or cloudy based on various factors involved

Statistics

_____ and data mining both look for relationships within data - we first make hypothesis and collect sample data to test our hypothesis

Mathematical modeling

______ and data mining both look to make decisions to achieve the best possible performance of the system - can use data mining results as an input to provide optimal actions based on the restrictions and limitations of the system

Sorting

a common method for eliminating duplicates, but non-exact duplicates can pose a large problem

Outlier

a data point that's distant from other data points. Plotting outliers will help check for errors or rare events in the data

Data mart

a data warehouse that stores only relevant data (smaller and subject-oriented)

Independent data mart

a small data warehouse designed for a strategic business unit or a department (not dependent on the larger data warehouse); an alternative to high cost data warehouses

Dependent data mart

a subset that is created directly from a data warehouse (dependent on the larger data warehouse)

Correlation graphs

can be used to explore the dependencies between different variables in the data

Artificial intelligence

enabling machines to become "smart"

Real-time DW

enabling real-time data updates for real-time analysis and real-time decision making

Clustering

grouping a company's customer base into distinct segments for more effective targeted marketing like seniors, adults and teenagers

Capture

includes anything that makes us retrieve data including: finding, accessing, acquiring and moving data

Integrate

includes integration of multiple data sources, cleaning data, filtering data, creating datasets which programs can read and understand, such as packaging raw data using a specific data format

Quality

is a moving target-less important when doing exploratory data analysis; critical when using data for decision support

Transform

is the process of converting the extracted data from its previous form into the form it needs to be in so that it can be placed into another database. Transformation occurs by using rules or lookup tables or by combining the data with other data

Integrated

places data from different sources into a consistent format

Visualization

provide a quick and effective way to look at data - give you the idea of where the hotspots are - what the distribution of data is - show correlations between variables

Web-based

provides efficient computing environment for web-based applications

Real time

provides real time or active data access and analysis capabilities

Business analytics

refers to the application of models directly to business data. it involves the use of DSS tools, especially models, in assisting decision makers

Act

reporting insights from analysis and determining actions from insights based on the purpose you initially defined

Prescriptive analytics

seeks to recognize what is going on as well as the likely forecast and to make decisions to achieve the best performance possible

Data mining

serves as the foundation for AI and machine learning

Definition of Data Cleansing

the assessment of data to determine quality failures (inaccuracy, incompleteness) and then improving the quality by correcting as possible any errors found

Normalization

the process of organizing tables and columns in a relational database to reduce redundancy and improve data integrity

Extract

the process of reading data from a database

Load

the process of writing the data into the target database

Extract, Transform, Load

three data base functions that are combined into one tool to pull data out of one database and place it into another database

Standardization

use legal name instead of nickname (use robert for Bob, Rob, Bobby and Robbie)

Graphing the general trends

will show you if there is a consistent direction in which the values of these variables are moving towards, like sales prices going up or down