Data Mining Test 1

Tables

Simplest way of representing output: Use the same format as input! Main problem: selecting the right attributes Have to experiment to decide which are not relevant to the concept to be learned

Trees for numeric prediction

Regression: the process of computing an expression that predicts a numeric quantity Regression tree: "decision tree" where each leaf predicts a numeric quantity Predicted value is average value (of the class attribute) of training instances that reach the leaf Model tree: combine regression tree with linear regression equations at the leaf nodes Linear patches approximate continuous function

Trees

"Divide-and-conquer" approach produces tree Value of one attribute rules out certain classes Nodes involve testing a particular attribute Usually, attribute value is compared to constant Other possibilities: Comparing values of two attributes Using a function of one or more attributes Leaves assign classification, set of classifications, or probability distribution to instances New, unknown instance is routed down the tree for classification / prediction 9

Linear models

Another simple representation Also called a regression model Inputs (attribute values) and output are all numeric Output is the sum of weighted attribute values The trick is to find good values for the weights that give a good fit to the training data Easiest to visualize in two dimensions Straight line drawn through the data points represents the regression model / function Can be applied to binary classification Notonlynumericalestimation Line separates the two classes Decision boundary ‐ defines where the decision changes from one class value to the other Prediction is made by plugging in observed values of the attributes into the expression Predict one class if output 0, and the other class if output < 0 Boundary becomes a high‐dimensional plane (hyperplane) when there are multiple attributes

Association learning

Can be applied if no class is specified and any kind of structure is considered "interesting" Difference from classification learning: Unsupervised I.e., not told what to learn Can predict any attribute's value, not just the class, and more than one attribute's value at a time Hence: far more association rules than classification rules Thus: constraints are necessary Minimum coverage and minimum accuracy

Preparing the input

Denormalization is not the only issue Problem: different data sources (e.g. sales department, customer billing department, ...) Differences: styles of record keeping, conventions, time periods, primary keys, errors Data must be assembled, integrated, cleaned up "Data warehouse": consistent point of access External data may be required ("overlay data")

Missing values

Does absence of value have some significance? Yes "missing" is a separate value No "missing" must be treated in a special way Solution A: assign instance to most popular branch Solution B: split instance into pieces Pieces receive weight according to fraction of training instances that go down each branch Classifications from leaf nodes are combined using the weights that have percolated to them

Clustering

Finding groups of items that are similar Clustering is unsupervised The class of an example is not known Success often measured subjectively

Ordinal quantities

Impose order on values Butnodistancebetweenvaluesdefined Example: attribute "temperature" in weather data Values: "hot" > "mild" > "cool" Note:additionandsubtractiondon'tmakesense Example rule: temperature < hot play = yes Distinction between nominal and ordinal not always clear (e.g. attribute "outlook" - is there an ordering?) also called "numeric", or "continuous"

Interval quantities

Interval quantities are not only ordered but measured in fixed numerical units Example: attribute "year" Difference of two values makes sense Sum or product doesn't make sense

Ratio quantities

Ratio quantities are those for which the measurement scheme defines a zero point Example: attribute "distance" Distance between an object and itself is zero Ratio quantities are treated as real numbers All mathematical operations are allowed

Inaccurate values

Reason: data has not been collected for the purpose of mining Result: errors and omissions that don't affect original purpose of data but are critical to mining E.g. age of customer in banking data Typographical errors in nominal attributes values need to be checked for consistency Typographical, measurement, rounding errors in numeric attributes outliers need to be identified What facility of Weka did we learn in lab that might be useful here? Errors may be deliberate E.g. wrong zip codes Other problems: duplicates, stale data

Reasons: Missing values

Reasons: malfunctioning equipment changes in experimental design collation of different datasets measurement not possible user refusal to answer survey question Missing value may have significance in itself (e.g. missing test in a medical examination) Most schemes assume that is not the case: "missing" may need to be coded as additional value 25

Numeric estimation

Variant of classification learning where the output attribute is numeric (also called "regression") Learning is supervised Algorithm is provided with target values Measure success on test data

What are the 3 main structural descriptions?

-Rules: classification and association -Decision trees -Linear regression formulas

What are the 3 main learning strategies in Data Mining?

-Unsupervised clustering -supervised learning (classification, numerical estimation, prediction) -Association leanring (market basket analysis)

Data and Information

Data is recorded facts; whereas, information is patterns underlying the data. Information tells us what is useful about the data.

attribute

Each column in a collection of training data The attributes can be divided into two types: - the output attribute - the one we want to determine/predict - the input attributes - everything else Each instance is described by a fixed predefined set of features, its "attributes" But: number of attributes may vary in practice Example: table of transportation vehicles Possible solution: "irrelevant value" flag Related problem: existence of an attribute may depend on value of another one Example:"spousename"dependson"married?" Possiblesolution:methodsofdatareduction Possible attribute types ("levels of measurement"): Nominal, ordinal, interval and ratio Simplifies to nominal and numeric

example or instance.

Each row in a collection of training data

Data Mining

Employing machine learning techniques in order to find useful trends and patterns in data

Classification learning

Example problems: weather data, medical diagnosis, contact lenses, irises, labor negotiations, etc. Can you think of others? Classification learning is supervised Algorithm is provided with actual outcomes Outcome is called the class attribute of the example Measure success on fresh data for which class labels are known (test data, as opposed to training data) In practice success is often measured subjectively How acceptable the learned description is to a human user

Overfitting

In general,workingtoohardtomatchthetraining examples can lead to an overly complicated model that doesn't generalize well.

Supervised Learning and Unsupervised Learning

In supervised learning, the outputs are given to train the machine to the desired outputs. In unsupervised learning, outputs are not given, so data is clumped into different classes.

Input Attribute and Output Attribute

Input Attribute help us formulate our output attribute. An output attribute is the attribute we want to predict.

Shallow Knowledge and Hidden Knowledge

Shallow Knowledge is factual and easily manipulated in the database. Hidden knowledge is patterns in the data that are not easy to find. (That's why we have Data Mining.)

test examples

To test how well a model generalizes, we typically withhold some of the examples

Training Data and Test Data

Training Data is the data we used to formulate the patterns of the data. We use test data to see how correct the pattern is.

Output: representing structural patterns

Understanding the output is the key to understanding the underlying learning methods

Nominal attributes

have values that are "names" of categories. - there is a small set of possible values attribute Fever Diagnosis Outlook possible values {Yes, No} {Allergy, Cold, Strep Throat} {sunny, overcast, raining} • In classification learning, the output attribute is always nominal. • Nominal comes from the Latin word for name • No relation is implied among nominal values • No ordering or distance measure • Can only test for equality also called "categorical", "enumerated", or "discrete"

Numeric attributes

have values that come from a range of numbers. attribute Body Temp Salary possible values any value in 96.0‐106.0 any value in $15,000‐250,000 - you can order their values (definition of "ordinal" type) $210,000 > $125,000 98.6 < 101.3

Concepts

kinds of things that can be learned Goal: intelligible and operational concept description E.g.: "Under what conditions should we play?" This concept is located somewhere in the input data

Attributes

measuring aspects of an instance We will focus on nominal and numeric attributes

Classification rule

predicts value of a given attribute

Association rule

predicts value of arbitrary attribute (or combination)

machine learning

the area of computer science that aims to build systems & algorithms that learn from data

Instances

the individual, independent examples of a concept Note: more complicated forms of input are possible

error rate of a model

the percentage of test examples that it misclassifies

Data mining vs. data query

• Database queries in SQL are not the same thing as data mining. • Queries allow us to extract factual information. - "shallow knowledge" • In data mining, we attempt to extract patterns and relationships that go beyond mere factual information. - "hidden knowledge"

Why Data Pre‐processing?

• Dataintherealworldisdirty • incomplete: lacking attribute values, lacking certain attributes of interest - e.g., occupation=" " • noisy: containing errors or outliers - e.g., Salary="‐10" • inconsistent: containing discrepancies in codes or names - e.g., Age="42" Birthday="03/07/1997" - e.g., Was rating "1,2,3", now rating "A, B, C" - e.g., discrepancy between duplicate records • No quality data, no quality mining results! • Quality decisions must be based on quality data • Data extraction, integration, cleaning, transformation, and reduction comprises the majority of the work of building target data

Data Integration

• In designing a database, we try to avoid redundancies by normalizing the data • As a result, the data for a given entity (e.g., a customer) may be: - spread over multiple tables - spread over multiple records within a given table • In designing a database, we try to avoid redundancies by normalizing the data. • As a result, the data for a given entity (e.g., a customer) may be: - spread over multiple tables - spread over multiple records within a given table • To prepare for data warehousing and/or data mining, we often need to denormalize the data. - multiple records for a given entitya single record

Why is Data Dirty?

• Incomplete data (missing values) may come from - "Notapplicable"datavaluewhencollected - Differentconsiderationsbetweenthetimewhenthedatawas collected and when it is analyzed - Human/hardware/softwareproblems • Noisydata(incorrectvalues)maycomefrom - Faultydatacollectioninstruments - Humanorcomputererroratdataentry - Errorsindatatransmission • Inconsistentdatamaycomefrom - Differentdatasources(resultingfromintegration) - Functionaldependencyviolation(e.g.,modifysomelinkeddata)

Handing Noisy Data

• Noise: - randomerrororvarianceinameasuredattribute - outliervalues - moregenerally:non‐predictivevalues • Combined computer and human inspection - detectsuspiciousvaluesandcheckbyhuman - datavisualizationthekeytool • Clustering - detectandremoveoutliers - alsoemploysdataviz • Regression - smoothbyfittingthedataintoregressionfunctions • Clustering - employstechniquessimilartodiscretizing

Noise

• Noisy data is meaningless data • The term has often been used as a synonym for corrupt data • Its meaning has expanded to include any data that cannot be understood and interpreted correctly by machines - unstructured text for example

Handling Missing Values

• Options: - Ignore them • PRISMandID3won'tworkatall • Naïve Bayes handles them fine • J48 and nearest neighbor use tricks to get around - Remove all instances with missing attributes • Unsupervised RemoveWithValues attribute filter in Weka - Replace missing values with the most common value for that attribute • Unsupervised ReplaceMissingValues attribute filter in Weka • Only works with nominal values • Issues?

Data Reduction

• Problematic attributes include: - irrelevant attributes: ones that don't help to predict the class • despite their irrelevance, the algorithm may erroneously include them in the model - attributes that cause overfitting • example: a unique identifier such as Patient ID - redundant attributes: those that offer basically the same information as another attribute • example: in many problems, date‐of‐birth and age provide the same information • some algorithms may end up giving the information from these attributes too much weight • WecanremoveanattributemanuallyinWekaby clicking the checkbox next to the attribute in the Preprocess tab and then clicking the Remove button - How to determine? • Experimentation • Correlation analysis (filters in Weka) • Undoing preprocess actions: - InthePreprocesstab,theUndobuttonallowsyoutoundoactionsthatyou perform, including: • applying a filter to a dataset • manually removing one or more attributes - IfyouapplytwofilterswithoutusingUndoinbetweenthetwo,thesecond filter will be applied to the results of the first filter - Undocanbepressedmultipletimestoundoasequenceofactions

Discretizing Numeric Attributes

• We can turn a numeric attribute into a nominal/categorical one by using some sort of discretization • This involves dividing the range of possible values into subranges called buckets or bins. - example: an age attribute could be divided into these bins: child: 0‐12 teen: 12‐17 young: 18‐35 middle: 36‐59 senior: 60‐ • What if we don't know which subranges make sense? • Equal‐width binning divides the range of possible values into N subranges of the same size. - bin width = (max value - min value) / N - example: if the observed values are all between 0‐ 100, we could create 5 bins as follows: width = (100 - 0)/5 = 20 bins: [0‐20], (20‐40], (40‐60], (60‐80], (80‐100] - problems with this equal‐width approach? • Equal‐frequency or equal‐height binning divides the range of possible values into N bins, each of which holds the same number of training instances. - example: let's say we have 10 training examples with the following values for the attribute that we're discretizing: 5, 7, 12, 35, 65, 82, 84, 88, 90, 95 To select the boundary values for the bins, this method typically chooses a value halfway between the training examples on either side of the boundary final bins: (‐inf, 9.5], (9.5, 50], (50, 83], (83, 89], (89, inf)

Transforming the Data

• We may also need to reformat or transform the data. - we can use a Python program to do the reformatting - Weka also provides several useful filters • One reason for transforming the data: many machine‐learning algorithms can only handle certain types of data • We may also need to reformat or transform the data. - we can use a Python program to do the reformatting - Weka also provides several useful filters • One reason for transforming the data: many machine‐learning algorithms can only handle certain types of data - some algorithms only work with nominal attributes - attributes with a specified set of possible values • examples: {yes, no} {strep throat, cold, allergy} - other algorithms only work with numeric attributes

What are 3 problems that often occur in Data Mining?

◆ Problem 1: most patterns are not interesting (noise!) ◆ Problem 2: patterns may be inexact (or spurious) ◆ Problem 3: data may be garbled or missing