Transcript Document 7117425
Premiere Products Team Project SAS Enterprise Miner (Part I) BCIS 4660 Spring 2006
BCIS 4660 Project Overview
• Assignment #7 • Assignment #8 • Assignment #9 • Each team will present their work (10-12 slides, 10 minutes) • All team members will submit PEER EVALUATIONS to the instructor at the conclusion of the project. These will be part of your grade
BCIS 4660 Project Team Assignments
• Team assignment will be centered around the team leaders • The traits of a good leader (
according to Nick Evangelopoulos
): – Knowledge on the class project topic – Track record on producing results/ meeting deadlines – Ability to push people when they are lazy, without being insensitive to special needs/ temporary problems – Leadership by example: Do what you preach – Being proactive in resolving communication/ management problems at a very early stage, before they escalate into conflicts
BCIS 4660 Project Team Assignments
What to do if you have already teamed up:
• If it’s less than 4 in your team, select a leader and let us know how many more people you need in your team • If it’s already 4 of you in the team, select a team leader and have him/her come up and let us know your team is full
Tasks for Team Leaders
• Come on up here and introduce yourself by FIRST NAME. Bring pen and paper.
• Use ONE SENTENCE to elaborate on your leadership style • Use ONE SENTENCE to instill confidence and convince the team members this project is something you can handle • • Invite team members to come talk to you and then take their full names and contact info (phone#, e-mail). Send everybody a confirmation/”thank you” e-mail ASAP
E-MAIL the instructor
your Team Number and the names of all team members (including your own)
Data Mining
Some slide material taken from: Groth, Han and Kamber, SAS Institute
The UNT/SAS® joint Data Mining Certificate: New in 2006
• •
Just approved!
Free of charge!
• Requires: – DSCI 2710 – DSCI 3710 – BCIS 4660 – DSCI 4520
Overview of this Presentation
• • • • • •
Introduction to Data Mining The SEMMA Methodology Regression/Logistic Regression Decision Trees Neural Networks SAS EM Demo: The Home Equity Loan Case
• • •
Important DM techniques Not Covered today:
Market Basket Analysis Memory-Based Reasoning Web Link Analysis
Introduction to DM
“It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of theories to suit facts.” (Sir Arthur Conan Doyle: Sherlock Holmes, "A Scandal in Bohemia")
What Is Data Mining?
• Data mining (knowledge discovery in databases): – A process of identifying hidden patterns and relationships within data (Groth) • Data mining: – Extraction of interesting ( non-trivial, implicit, previously unknown and potentially useful) information or patterns from data in large databases
Data
11
DM and Business Decision Support
– Database Marketing • Target marketing • Customer relationship management – Credit Risk Management • Credit scoring – Fraud Detection – Healthcare Informatics • Clinical decision support
Databases
Multidisciplinary
Statistics Pattern Recognition Neurocomputing Data Mining Machine Learning AI KDD
On the News: Data-mining software digs for business leads
SAN FRANSCISCO, March 8, 2004.
Last year, Congress shut down Pentagon’s Total Information Awareness programs. Now entrepreneurs are selling new data- mining tools that do similar things on a smaller scale.
Spoke
and
Visible Path
sell their software primarily to corporations. The idea is to provide tools for finding helpful business partners and making blind introductions - allowing, say, a lawyer for Silly Computers Inc. to electronically ask a former classmate from Harvard who once did legal work for Microsoft to help him pitch a business deal to Bill Gates.
How does it work?
Both Spoke and Visible Path send so-called
crawlers
around a corporation's internal computer network --
sniffing telltale clues
, say, from employee Outlook files about who they e-mail and how often, who replies to particular messages and who doesn't, which names show up in electronic calendars and phone logs. Then it
cross-references
those
snippets
with information from other company databases, including sales records from PeopleSoft and Salesforce.com.
Data Mining: A KDD Process
Pattern Evaluation
– Data mining: the core of knowledge
Data Mining
discovery process.
Task-relevant Data Data Warehouse Data Cleaning Data Integration Selection Databases
Data Mining and Business Intelligence
Increasing potential to support business decisions Making Decisions End User (Manager) Data Presentation
Visualization Techniques
Data Mining
Information Discovery
Data Exploration
Statistical Analysis, Querying and Reporting
Data Warehouses / Data Marts
OLAP, MDA
Data Sources
Paper, Files, Information Providers, Database Systems, OLTP
Business Analyst Data Analyst DBA
Architecture of a Typical Data Mining System
Graphical user interface Pattern evaluation Data mining engine Database or data warehouse server
Data cleaning & data integration Filtering
Knowledge-base
Databases Data Warehouse
Introducing SAS Enterprise Miner (EM)
The SEMMA Methodology
– Introduced By SAS Institute – Implemented in SAS Enterprise Miner (EM) – Organizes a DM effort into 5 activity groups: Sample Explore Modify Model Assess
Sample
Input Data Source Sampling Data Partition
Distribution Explorer
Explore
Association Multiplot Variable Selection Insight Link Analysis
Modify
Data Set Attributes Transform Variables Filter Outliers Replacement Clustering Self-Organized Maps Kohonen Networks Time Series
Regression Tree Neural Network Princomp/ Dmneural
Model
User Defined Model Ensemble Memory Based Reasoning Two-Stage Model
Assess
Assessment Reporter
Other Types of Nodes – Scoring Nodes, Utility Nodes
Group Processing Data Mining Database Score SAS Code C*Score Control Point Subdiagram
DATA MINING AT WORK:
Detecting Credit Card Fraud
–
Credit card companies want to find a way to monitor new transactions and detect those made on stolen credit cards. Their goal is to detect the fraud while it is taking place.
–
In a few weeks after each transaction they will know which of the transactions were fraudulent and which were not, and they can then use this data to validate their fraud detection and prediction scheme.
DATA MINING AT WORK: Telstra Mobile Combats Churn with SAS® As Australia's largest mobile service provider, Telstra Mobile is reliant on highly effective churn management.
In most industries the cost of retaining a customer, subscriber or client is substantially less than the initial cost of obtaining that customer. Protecting this investment is the essence of churn management. It really boils down to understanding customers -- what they want now and what they're likely to want in the future, according to SAS.
"With SAS Enterprise Miner we can examine customer behaviour on historical and predictive levels, which can then show us what 'group' of customers are likely to churn and the causes," says Trish Berendsen, Telstra Mobile's head of Customer Relationship Management (CRM).
DATA MINING AT WORK: Reducing armed robberies in South Africa SAS helped Absa, a Major South African Bank reduce armed robberies by 41 percent over two years (2002-2003), netting a 38 percent reduction in cash loss and an 11 percent increase in customer satisfaction ratings.
Absa, one of South Africa's largest banks, uses SAS' data mining capabilities to leverage their data for better customer relationships and more targeted marketing campaigns. With SAS analytics, the bank can also track which branches are more likely to fall victim to a robbery and take effective preventive measures.
"Absa used to be one of the banks that was targeted more than other banks; now we're at the bottom of the list," says Dave Donkin, Absa group executive of e-business and information management.
DATA MINING AT WORK: Strategic Pricing Solutions at MCI MCI now has a solution for making strategic pricing decisions, driving effective network analysis, enhancing segment reporting and creating data for sales leader compensation. Before implementing SAS, the process of inventorying MCI's thousands of network platforms and IT systems products they support – determining what each one does, who runs them, how they help business and which – was completely manual. The model created with SAS has helped MCI to catalog all that information and map the details to products, customer segments and business processes.
"That's something everyone is excited about," says Leslie Mote, director of MCI corporate business analysis. "Looking at the cost of a system and what it relates to helps you see the revenue you're generating from particular products or customers. I can see what I'm doing better."
Our own example: The Home Equity Loan Case
• HMEQ Overview • Determine who should be approved for a home equity loan.
• The target variable is a binary variable that indicates whether an applicant eventually defaulted on the loan.
• The input variables are variables such as the amount of the loan, amount due on the existing mortgage, the value of the property, and the number of recent credit inquiries.
HMEQ case overview
– –
The consumer credit department of a bank wants to automate the decision-making process for approval of home equity lines of credit. To do this, they will follow the recommendations of the Equal Credit Opportunity Act to create an empirically derived and statistically sound credit scoring model. The model will be based on data collected from recent applicants granted credit through the current process of loan underwriting. The model will be built from predictive modeling tools, but the created model must be sufficiently interpretable so as to provide a reason for any adverse actions (rejections).
The HMEQ data set contains baseline and loan performance information for 5,960 recent home equity loans. The target (BAD) is a binary variable that indicates if an applicant eventually defaulted or was seriously delinquent. This adverse outcome occurred in 1,189 cases (20%). For each applicant, 12 input variables were recorded.
The HMEQ Loan process
1. An applicant comes forward with a specific property and a reason for the loan (Home Improvement, Debt-Consolidation) 2. Background info related to job and credit history is collected 3. The loan gets approved or rejected 4. Upon approval, the Applicant becomes a Customer 5. Information related to how the loan is serviced is maintained, including the Status of the loan (Current, Delinquent, Defaulted, Paid-Off)
The HMEQ Loan Transactional Database
• Entity Relationship Diagram (ERD), Logical Design: APPLICANT becomes CUSTOMER Loan Reason Date Applies for HMEQ Loan on… using… Approval PROPERTY OFFICER Status Balance ACCOUNT MonthlyPayment has HISTORY
HMEQ Transactional database: the relations
• Entity Relationship Diagram (ERD), Physical Design:
Officer
OFFICERID OFFICERNAME PHONE FAX
HMEQLoanApplication
OFFICERID APPLICANTID PROPERTYID LOAN REASON DATE APPROVAL
Property
PROPERTYID ADDRESS VALUE MORTDUE
Applicant
APPLICANTID NAME JOB DEBTINC YOJ DEROG CLNO DELINQ CLAGE NINQ
Customer
CUSTOMERID APPLICANTID NAME ADDRESS
Account
ACCOUNTID CUSTOMERID PROPERTYID ADDRESS BALANCE MONTHLYPAYMENT STATUS
History
HISTORYID ACCOUNTID PAYMENT DATE
The HMEQ Loan Data Warehouse Design
• We have some slowly changing attributes:
HMEQLoanApplication:
Loan, Reason, Date
Applicant:
Job and Credit Score related attributes
Property:
Value, Mortgage, Balance • An applicant may reapply for a loan, then some of these attributes may have changed.
– Need to introduce “Key” attributes and make them primary keys
The HMEQ Loan Data Warehouse Design
STAR 1 – Loan Application facts • Fact Table:
HMEQApplicationFact
• Dimensions:
Applicant, Property, Officer, Time
STAR 2 – Loan Payment facts • Fact Table:
HMEQPaymentFact
• Dimensions:
Customer, Property, Account, Time
Two Star Schemas for HMEQ Loans
Applicant
APPLICANTKEY APPLICANTID NAME JOB DEBTINC YOJ DEROG CLNO DELINQ CLAGE NINQ
Officer
OFFICERKEY OFFICERID OFFICERNAME PHONE FAX
Property
PROPERTYKEY PROPERTYID ADDRESS VALUE MORTDUE
HMEQApplicationFact
APPLICANTKEY PROPERTYKEY OFFICERKEY TIMEKEY LOAN REASON APPROVAL
Time
TIMEKEY DATE MONTH YEAR
Customer
CUSTOMERKEY CUSTOMERID APPLICANTID NAME ADDRESS
HMEQPaymentFact
CUSTOMERKEY PROPERTYKEY ACCOUNTKEY TIMEKEY BALANCE PAYMENT STATUS
Account
ACCOUNTKEY LOAN MATURITYDATE MONTHLYPAYMENT
The HMEQ Loan DW: Questions asked by management
• How many applications were filed each month during the last year? What percentage of them were approved each month?
• How has the monthly average loan amount been fluctuating during the last year? Is there a trend?
• Which customers were delinquent in their loan payment during the month of September?
• How many loans have defaulted each month during the last year? Is there an increasing or decreasing trend?
• How many defaulting loans were approved last year by each loan officer? Who are the officers with the largest number of defaulting loans?
The HMEQ Loan DW: Some more involved questions
• Are there any patterns suggesting which applicants are more likely to default on their loan after it is approved? • Can we relate loan defaults to applicant job and credit history? Can we estimate probabilities to default based on applicant attributes at the time of application? Are there applicant segments with higher probability? • Can we look at relevant data and build a predictive model that will estimate such probability to default on the HMEQ loan? If we make such a model part of our business policy, can we decrease the percentage of loans that eventually default by applying more stringent loan approval criteria?
Selecting Task-relevant attributes
Applicant
APPLICANTKEY APPLICANTID NAME JOB DEBTINC YOJ DEROG CLNO DELINQ CLAGE NINQ
Officer
OFFICERKEY OFFICERID OFFICERNAME PHONE FAX
Property
PROPERTYKEY PROPERTYID ADDRESS VALUE MORTDUE
HMEQApplicationFact
APPLICANTKEY PROPERTYKEY OFFICERKEY TIMEKEY LOAN REASON APPROVAL
Time
TIMEKEY DATE MONTH YEAR
Customer
CUSTOMERKEY CUSTOMERID APPLICANTID NAME ADDRESS
HMEQPaymentFact
CUSTOMERKEY PROPERTYKEY ACCOUNTKEY TIMEKEY BALANCE PAYMENT STATUS
Account
ACCOUNTKEY LOAN MATURITYDATE MONTHLYPAYMENT
HMEQ final task-relevant data file
Name BAD REASON JOB LOAN MORTDUE VALUE DEBTINC YOJ DEROG CLNO DELINQ CLAGE NINQ Input Input Input Input Input Input Input Input Input Input Input Model Role Target Input Measurement Level Binary Binary Nominal Interval Interval Interval Interval Interval Interval Interval Interval Interval Interval Description
1=defaulted on loan, 0=paid back loan HomeImp=home improvement, DebtCon=debt consolidation Six occupational categories Amount of loan request Amount due on existing mortgage Value of current property Debt-to-income ratio Years at present job Number of major derogatory reports Number of trade lines Number of delinquent trade lines Age of oldest trade line in months Number of recent credit inquiries
HMEQ: Modeling Goal
– The
credit scoring model
should compute the
probability
of a given loan applicant
to default
on loan repayment. A
threshold
is to be selected such that all applicants whose probability of default is in excess of the threshold are recommended for rejection. – Using the HMEQ task-relevant data file,
three competing models
will be built: A logistic Regression model, a Decision Tree, and a Neural Network –
Model assessment
will allow us to select the best of the three alternative models
Cases
Predictive Modeling
Inputs ...
...
...
...
...
...
...
...
...
... ... ... ... ... ... ... ... ...
...
... ...
Target
Modeling Tools
Logistic Regression
•
Modeling Techniques: Separate Sampling
Benefits: • Helps detect rare target levels • Speeds processing Risks: • Biases predictions (correctable) • Increases prediction variability
Logistic Regression Models
Training Data log 1.0
p
0.5
0.0
=
w
0 + w 1
x
1 +…+ w
p x p
logit(p) 0
Changing the Odds
Training Data log =
w
0 + w 1
x
1 +…+ w
p x p
log
( )
´
=
w
0 log ) (x 1
( )
p p x x p p
odds ratio
Modeling Tools
Decision Trees
Divide and Conquer the HMEQ data
The tree is fitted to the data by
recursive partitioning
. Partitioning refers to segmenting the data into subgroups that are as homogeneous as possible with respect to the target. In this case, the binary split (Debt-to Income Ratio < 45) was chosen. The 5,000 cases were split into two groups, one with a 5% BAD rate and the other with a 21% BAD rate.
yes n = 3,350 5% BAD n = 5,000 10% BAD Debt-to-Income Ratio < 45 no n = 1,650 21% BAD The method is recursive because each subgroup results from splitting a subgroup from a previous split. Thus, the 3,350 cases in the left child node and the 1,650 cases in the right child node are split again in similar fashion.
The Cultivation of Trees
– Split Search • Which splits are to be considered?
– Splitting Criterion • Which split is best?
– Stopping Rule • When should the splitting stop?
– Pruning Rule • Should some branches be lopped off?
Possible Splits to Consider: an enormous number
500,000 400,000 300,000 Nominal Input Ordinal Input 200,000 100,000 1 2 4 6 8 10 12 14 16 18 20 Input Levels
Splitting Criteria
How is the best split determined? In some situations, the worth of a split is obvious.
If the expected target is the same
in the child nodes as in the parent node, no improvement was made, and the split is
worthless!
In contrast, if a split results in pure child nodes, the split is undisputedly best. For classification trees, the three most widely used splitting criteria are based on the Pearson chi squared test, the Gini index, and entropy. All three measure the difference in class distributions across the child nodes. The three methods usually give similar results.
Benefits of Trees
– Interpretability • tree-structured presentation – Mixed Measurement Scales • nominal, ordinal, interval – Robustness (tolerance to noise) – Handling of Missing Values – Regression trees, Consolidation trees
Modeling Tools
Neural Networks
Neural network models
(
multi-layer perceptrons
) Often regarded as a mysterious and powerful predictive modeling technique.
The most typical form of the model is, in fact, a natural extension of a regression model: • • A generalized linear model on a set of derived inputs These derived inputs are themselves a generalized linear model on the original inputs The usual link for the derived input’s model is
inverse hyperbolic tangent
, a shift and rescaling of the logit function Ability to approximate virtually any continuous association between the inputs and the target • You simply need to specify the correct number of derived inputs
Neural Network Model
x
2 Training Data
x
1 log tanh -1 ( H 1 =
w
00 + w 01
H
1 + w 02
H
2 +
w
03
H
3 ) = w 10 + w 11
x
1 + w 12
x
2 tanh -1 ( H 2 ) = w 20 + w 21
x
1 + w 22
x
2 tanh -1 ( H 3 ) = w 30 + w 31
x
1 + w 32
x
2 1 tanh(x) 0
x
-1
Input
layer,
hidden
layer layer,
output
Multi-layer perceptron models were originally inspired by neurophysiology and the interconnections between neurons. The basic model form arranges neurons in layers. The
input layer
connects to a layer of neurons called a
hidden layer
, which, in turn, connects to a final layer called the
target
, or
output
, layer.
The structure of a multi-layer perceptron lends itself to a graphical representation called a
network diagram
.
Neural Network Diagram
x
2 Training Data
x
1 tanh -1 ( H 1
w w
00 03 +
H
3
w
01
H
1 +
w
02
H
2 + ) = w 10 + w 11
x
1 + w 12
x
2 tanh -1 ( H 2 ) = w 20 + w 21
x
1 + w 22
x
2 tanh -1 ( H 3 ) = w 30 + w 31
x
1 + w 32
x
2 Inputs Target
H
1
x
1
H
2
p x
2
H
3
Hidden layer
Objective Function
Predictions are compared to the actual values of the target via an objective function.
An easy-to-understand example of an objective function is the mean squared error (MSE) given by :
MSE
1
N
training
(
y l cases
y
ˆ
i
( ) ) 2 Where: –
N
– y i –
i
– is the number of training cases. is the target value of the ith case. is the predicted target value. is the current estimate of the model parameters.
Overgeneralization
A small value for the objective function, when calculated on training data, need not imply a small value for the function on validation data.
Typically, improvement on the objective function is observed on both the training and the validation data over the first few iterations of the training process.
At convergence, however, the model is likely to be highly
overgeneralized
and the values of the objective function computed on training and validation data may be quite different.
Training Overgeneralization
x
2 log tanh -1 ( H 1 =
w
00 + w 01
H
1 + w 02
H
2 +
w
03
H
3 ) = w 10 + w 11
x
1 + w 12
x
2 tanh -1 ( H 2 ) = w 20 + w 21
x
1 + w 22
x
2 tanh -1 ( H 3 ) = w 30 + w 31
x
1 + w 32
x
2 Objective function (w) Training Data
x
1 0 10 20 30 40 50 60 70
Final Model
To compensate for overgeneralization, the overall average profit, computed on
validation
data, is examined.
The final parameter estimates for the model are taken from the training iteration with the
maximum validation profit
.
Neural Network Final Model
x
2 log tanh -1 ( H 1 =
w
00 + w 01
H
1 + w 02
H
2 +
w
03
H
3 ) = w 10 + w 11
x
1 + w 12
x
2 tanh -1 ( H 2 ) = w 20 + w 21
x
1 + w 22
x
2 tanh -1 ( H 3 ) = w 30 + w 31
x
1 + w 32
x
2 Profit Training Data
x
1 0 10 20 30 40 50 60 70