BI exam 2
All of the following are challenges associated with natural language processing EXCEPT
dividing up a text into individual words in English
Clickstream analysis does not need users to
enter their perceptions of the Web site or other feedback directly to be useful in determining their preferences.
Data mining tools' capabilities and ease of use are
essential
to perform text mining
first, impose structure to the data, then mine the structured data
85-90 percent of all corporate data is
in some kind of unstructured form (e.g., text)
Data mining can be very useful in detecting patterns such as credit card fraud and
is very helpful in increasing sales
What does the robustness of a data mining method refer to?
its ability to overcome noisy data to make somewhat accurate predictions
If using a mining analogy, _________________ would be a more appropriate term than "data mining."
knowledge mining
Data mining versus statistics
main difference is that statistics starts with a well-defined proposition and hypothesis, wheras data mining starts with a loosely defined discovery statement Data mining looks for data sets that are as "big" as possible; statistics looks for the right size of data
Tapping into these information sources is not an option, but a
need to stay competitive
The data field "ethnic group" can be best described as
nominal data.
unstructured data
nonnumeric information that is typically formatted in a way that is meant for human eyes and not easily understood by computers
What are the two main types of Web analytics?
off-site and on-site Web analytics
DM extract ________ from data
patterns
Myth or reality: If the data accurately reflect the business or its customers, any company can use data mining.
reality
Striking it rich
requires creative thinking
In text mining, if an association between two concepts has 7% support, it means
that 7% of the documents had both concepts represented in the same document.
Categorization and clustering of documents during text mining differ only in
the preselection of categories.
All of the following statements about data mining are true EXCEPT
the process aspect means that data mining should be a one-step process to results.
Sentiment analysis projects require a lexicon for use. If a project in English is undertaken, you must generally make sure to
use an English lexicon appropriate to the project at your discretion.
In text analysis, what is a lexicon?
a catalog of words, their synonyms, and their meanings
In the opening vignette, the architectural system that supported Watson used all the following elements EXCEPT
a core engine that could operate seamlessly in another domain without changes
Why Data Mining?
-More intense competition at the global scale -Recognition of the value in data sources -Availability of quality data on customers, vendors, transactions, Web, etc. -Consolidation and integration of data repositories into data warehouses -The exponential increase in data processing and storage capabilities; and decrease in cost -Movement toward conversion of information resources into nonphysical form
Data mining is
-the novel aspect means that previously unknown patterns are discovered. -the potentially useful aspect means that results should lead to some business benefit. -the valid aspect means that the discovered patterns should hold true on new data.
In the Influence Health case study, what was the goal of the system?
Increasing service use
Text Analytics =
Information Retrieval + Text Mining
What does advanced analytics for social media do?
It examines the content of online conversations.
I B M Watson going head-to-head with the best of the best in
Jeopardy
All of the following are challenges associated with natural language processing
-understanding the context in which something is said. -distinguishing between words that have more than one meaning. -recognizing typographical or grammatical errors in texts
Area under the ROC Curve (AUC)
-works with binary classification -Produces values from 0 to 1.0 -Random chance is 0.5 and perfect classification is 1.0 -Produces a good assessment for skewed class distributions too!
Companies understand that when their product goes "viral," the content of the online conversations about their product does not matter, only the volume of conversations.
False
Data mining can be very useful in detecting patterns such as credit card fraud, but is of little help in improving sales.
False
Search engines are only used in the context of the World Wide Web (WWW).
False
Since little can be done about visitor Web site abandonment rates, organizations have to focus their efforts on increasing the number of new visitors.
False
Text analytics is the subset of text mining that handles information retrieval and extraction, plus data mining.
False
In the cancer research case study, data mining algorithms that predict cancer survivability with high predictive power are good replacements for medical professionals.
False Are good at helping doctors
Association consists of
Market Basket Link Analysis Sequence Analysis
Myth or reality: Data mining is not yet viable for mainstream business applications.
Myth
Myth or reality: Data mining is only for large firms that have lots of customer data.
Myth
Myth or reality: Data mining provides instant, crystal-ball-like predictions.
Myth
Myth or reality: Data mining requires a separate, dedicated database.
Myth
Myth or reality: Only those with advanced degrees can do data mining.
Myth
In sentiment analysis, which of the following is an implicit opinion?
The customer service I got for my TV was laughable
In sentiment analysis, it is hard to classify some subjects such as news as good or bad, but easier to classify others, e.g., movie reviews, in the same way
True
•A representative application of association rule mining includes
-*In business:* cross-marketing, cross-selling, store design, catalog design, e-commerce site design, optimization of online advertising, product pricing, and sales/promotion configuration -*In medicine:* relationships between symptoms and illnesses; diagnosis and patient characteristics and treatments (to be used in medical D S S); and genes and their functions (to be used in genomics projects) •*Commercial* -I B M S P S S Modeler (formerly Clementine) -S A S Enterprise Miner -Statistica - Dell/Statsoft -... many more •*Free and/or Open Source* -K N I M E -RapidMiner -Weka -R, ...
Types of patterns
-Association -Prediction -Cluster (segmentation) -Sequential (or time series) relationships
Most common standard processes
-C R I S P-D M (Cross-Industry Standard Process for Data Mining) -S E M M A (Sample, Explore, Modify, Model, and Assess) -K D D (Knowledge Discovery in Databases)
k-Fold Cross Validation (rotation estimation)
-Estimation methodology for classification -Data is split into k mutual subsets and k number training/testing experiments are conducted
single/simple split method
-Estimation methodology for classification -Simple split (or holdout or test sample estimation) -Split the data into 2 mutually exclusive sets: - training (~70%) and testing (30%) -For Neural Networks, the data is split into three sub-sets (training [~60%], validation [~20%], testing [~20%])
•Clustering results may be used to
-Identify natural groupings of customers -Identify rules for assigning new cases to classes for targeting/diagnostic purposes -Provide characterization, definition, labeling of populations -Decrease the size and complexity of problems for other data mining methods -Identify outliers in a specific domain (e.g., rare-event detection)
Electronic communization records (e.g., e-mail)
-Spam filtering -E-mail prioritization and categorization -Automatic response generation
Cluster analysis methods
-Statistical methods (including both hierarchical and nonhierarchical), such as k-means, k-modes, and so on. -Neural networks (adaptive resonance theory [A R T], self-organizing map [S O M]) -Fuzzy logic (e.g., fuzzy c-means algorithm) -Genetic algorithms
What steps compose of 85% of the total CRISP-DM project time
-Step 1: Business Understanding -Step 2: Data Understanding -Step 3: Data Preparation
CRISP-DM composed of six consecutive phases
-Step 1: Business Understanding -Step 2: Data Understanding -Step 3: Data Preparation -Step 4: Model Building -Step 5: Testing and Evaluation -Step 6: Deployment
Structured versus unstructured data
-Structured data: in databases -Unstructured data: Word documents, P D F files, text excerpts, X M L files, and so on
K-means clustering algorithm
-k : pre-determined number of clusters -Algorithm (Step 0: determine value of k) Step 1: Randomly generate k random points as initial cluster centers. Step 2: Assign each point to the nearest cluster center. Step 3: Re-compute the new cluster centers. Repetition step: Repeat steps 3 and 4 until some convergence criterion is met (usually that the assignment of points to clusters becomes stable).
Other names for data mining
-knowledge extraction, -pattern analysis, -knowledge discovery, -information harvesting, -pattern searching, -data dredging
In the opening vignette, the architectural system that supported Watson used all the following
-massive parallelism to enable simultaneous consideration of multiple hypotheses. -an underlying confidence subsystem that ranks and integrates answers -integration of shallow and deep knowledge.
Unstructured corporate data is doubling in size every
18 months
pattern
A mathematical (numeric and/or symbolic) relationship among data items
Predictions consists of
Classification Regression Time Series
Classification versus clustering
Classification: learns the function between the characteristic of things and their membership through a supervised learning process where both input and output variables are presented to the algorithm Clustering: learns the function between the characteristic of things and their membership through an unsupervised learning process where only the input variables are presented to the algorithm
Classification versus regression
Classification: what is being predicted is a class label (e.g. "sunny," "rainy," "cloudy) Regression: what is being predicted is a numeric value (e.g. temperature)
Which broad area of data mining applications partitions a collection of objects into natural groupings with similar features?
Clustering
Segmentation consists of
Clustering Outlier analysis
CRISP-DM stands for
Cross Industry Standard Process for Data Mining
structured data
Data that (1) are typically numeric or categorical; (2) can be organized and formatted in a way that is easy for computers to read, organize, and understand; and (3) can be inserted into a database in a seamless fashion.
Decision Trees
Employs a divide-and-conquer method
Definition of Data Mining
The nontrivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns in data stored in structures databases.
Myth or Reality: Data mining is a multistep process that requires deliberate, proactive design and use.
Reality
Myth or reality: Because of the advances in database technology, a dedicated database is not required.
Reality
Myth or reality: Newer Web-based tools enable managers of all educational levels to do data mining
Reality
Myth or reality: The current state of the art is ready to go for almost any business type and/or size.
Reality
____________ present challenges for natural language processing.
Regional accents
Clustering partitions a collection of things into segments whose members share
Similar Characteristics
Text analytics versus text mining
Text analytics: a broader concept that includes information retrieval as well as information extraction, data mining, and web mining Text mining: primarily focused on discovering new and useful knowledge from the textual data sources
How many clusters?
There is not an optimal way to configure the number needed
What do voice of the market (VOM) applications of sentiment analysis do?
They examine customer sentiment at the aggregate level
Search engine optimization (SEO) is a means by which
Web site developers can increase Web site search rankings
In the Miami-Dade Police Department case study, predictive analytics helped to identify
a holistic view of the world of crime and criminals for better and faster reaction and management. NOT to identify the best schedule to pay the least overtime
Corpus (and corpora)
a large and structured set of texts prepared for the purpose of conducting knowledge discovery
The miner is often
an end user
In text mining, tokenizing is the process of
categorizing a block of text in a sentence
Current use of sentiment analysis in voice of the customer applications allows companies to
change their products or services in real time in response to customer sentiment.
D M environment is usually a
client-server or a Web-based information systems architecture.
In classification problems, the primary source for accuracy estimation is the
confusion matrix
Source of data for DM is often a
consolidated data warehouse (not always!)
Data is the most
critical ingredient for D M which may include soft/unstructured data
Data Mining Process
•A manifestation of the best practices •A systematic way to conduct D M projects •Moving from *Art to Science* for D M project Everybody has a different version
Association Rule Mining (market basket analysis)
•A very popular D M method in business •Finds interesting relationships (affinities) between variables (items or events) •Part of machine learning family •Employs unsupervised learning •There is no output variable •Also known as *market basket analysis* •Often used as an example to describe D M to ordinary people, such as the famous *"relationship between diapers and beers!"*
Data Mining versus Text Mining
•Both seek for novel and useful patterns Both are semi-automated processes
Classification techniques
•Decision tree analysis •Statistical analysis •Neural networks •Support vector machines •Case-based reasoning •Bayesian classifiers •Genetic algorithms Rough sets
Association ruling mining
•Input: the simple point-of-sale transaction data •Output: Most frequent affinities among items •Example: according to the transaction data... "Customer who bought a lap-top computer and a virus protection software, also bought extended service plan 70 percent of the time." •How do you use such a pattern/knowledge? -Put the items next to each other -Promote the items as a package Place items far apart from each other!
Data Mining Goes to Hollywood: Predicting Financial Success of Movies
•Goal: Predicting financial success of Hollywood movies before the start of their production process •How: Use of advanced predictive analytics methods •Results: promising
Text Mining Application Area
•Information extraction •Topic tracking •Summarization •Categorization •Clustering •Concept linking •Question answering
Additional Estimation Methodologies for Classification
•Leave-one-out -Similar to k-fold where k = number of samples •Bootstrapping -Random sampling with replacement •Jackknifing -Similar to leave-one-out •Area Under the R O C Curve (A U C) -R O C: receiver operating characteristics (a term borrowed from radar image processing)
Classification
•Most frequently used DM method •Part of the machine-learning family •Employ supervised learning •Learn from past data, classify new data •The output variable is categorical (nominal or ordinal) in nature
Assessment methosds for classification
•Predictive accuracy -Hit rate •Speed -Model building versus predicting/usage speed •Robustness •Scalability •Interpretability -Transparency, explainability
Ensemble Models for Predictive Analytics
•Produces more robust and reliable prediction models
Cluster Analysis for Data Mining
•Used for automatic identification of natural groupings of things •Part of the machine-learning family •Employ unsupervised learning •Learns the clusters of things from past data, then assigns new instances •There is not an output/target variable •In marketing, it is also known as segmentation
