ArrowModel Blog

Agile Scoring



Agile Scoring Blog



Predictive Modeling from the Trenches

More Data Beats Better Algorithms
Apr 07, 2008 | /jeff | Link

Anand Rajaraman argues that more data > better algorithm. Hear, hear.

Good software is still required to crunch all this additional data, though.

New site design
Apr 02, 2008 | /jeff | Link

We're rolling out the redesigned ArrowModel web site today.

ArrowModel 0.2
Jan 11, 2008 | /jeff | Link

Second beta of ArrowModel is out. Registered users can download it now.

If you are not a registered user, but would like to give ArrowModel a try, please sign up.

Highlights of the new version include:

ARM file format remains unchanged. You will be getting warning messages when opening models created with previous versions, but they should work.

Thank you for the feedback and support.

Wir sprechen Deutsch
Sep 17, 2007 | /jeff | Link

ArrowModel site in German.

Second beta (build 888)
Jul 19, 2007 | /jeff | Link

Second beta of ArrowModel is out. The biggest new feature is ODBC connectivity.

New hosting
May 09, 2007 | /jeff | Link ArrowModel moved to new hosting. We apologize for the downtime and inconvenience.

ArrowModel beta FAQ
May 04, 2007 | /jeff | Link

ArrowModel goes through its first beta testing. Here are some of the frequently asked questions so far:

[UPDATE 5/4] Pictures added to illustrate the differences in ROC curves.

Not Quite Normal
Apr 19, 2007 | /jeff | Link

A lot of statistical magic relies on the premise that stuff is normally distributed.

Normal distribution
Normal distribution

The normal distribution has nice properties that make things easy analytically, but chances are that, most of the time, you'll see distributions that look like this:

Not quite normal distribution
Not quite normal distribution

Of course I'm generalizing and there are exceptions, but it's clear that the good old normal distribution belongs on the endangered species list.

There are several reasons why:

So what is the poor modeler to do?

There are more elaborate ways of dealing with not quite normally distributed data such as Johnson's SU functions and multivariate adaptive regression splines (MARS) which this margin is too narrow to contain.

Spanish translation
Apr 16, 2007 | /jeff | Link

ArrowModel speaks Spanish, too.

Site translations
Apr 01, 2007 | /jeff | Link

The main ArrowModel site now has French and Russian translations. Really. More i18n is underway.

Information Value
Feb 25, 2007 | /jeff | Link

Deciding which predictors to use is one of the key steps in model building. A good place to start is to examine predictors individually to see how good they are in a univariate sense.

Information value is a metric that is often used to tell how good a predictor is. Let's follow the calculations step by step.

  1. Start by ranking the data by the predictor in question. The number of ranks is not very critical and, in most cases, deciles will do just fine.
  2. Calculate the total number of goods (total_good_ct)
    and the total number of bads (total_bad_ct);
  3. For each rank
    • Calculate the number of goods (good_ct)
      and the number of bads (bad_ct);
    • good_pct = good_ct / total_good_ct,
      bad_pct = bad_ct / total_bad_ct;
    • diff_pct = good_pct - bad_pct;
    • info_odds = good_pct / bad_pct;
    • Weight of evidence: woe = log(info_odds);
    • Information value: inf_val = diff_pct * woe;
  4. Finally, sum up inf_val for all the ranks. This is the predictor's information value.

As you can see, the information value for each rank reflects log odds, but the order of ranks does not have any effect. This nicely takes care of nonlinearity and outliers.

Ordering predictors by information value and taking the top N is a tempting strategy, but not a very prudent approach. The predictors selected this way can turn out to be redundant, regression is rather sensitive to outliers, and we haven't done anything about nonlinearity yet. But it's a good way to screen out the least likely candidates.

Receiver Operating Characteristic
Feb 24, 2007 | /jeff | Link

ROC curves were first used during World War II to graphically show the separation of radar signals from background noise. They are commonly used to graphically show the added value of any predictive model. To plot the receiver operating characteristic, or ROC curve, one plots B(s) vs. G(s) for all values of s. This curve goes from (0, 0) to (1, 1). The curve of an ideal model (complete separation) goes through (0, 1), while the curve of a totally useless model (no separation) is a straight diagonal line. The curve looks like a banana, hence the nickname banana chart.

Very strong separation Weak separation
Excellent model Mediocre model

The KS query from this post can be easily modified to return coordinates of the points on the ROC curve: 

SELECT s
     , cdf.b "Sensitivity"
     , cdf.g "1-Specificity"
FROM ( SELECT a.s                                          "s"
            , SUM(distr.bad_cnt) /
              ( SELECT COUNT(*) FROM t WHERE outcome = 1 ) "b"
            , SUM(distr.good_cnt) /
              ( SELECT COUNT(*) FROM t WHERE outcome = 0 ) "g"
       FROM ( SELECT DISTINCT s FROM t ) a
       JOIN (
              SELECT s                "s"
                   , SUM(outcome)     "bad_cnt"
                   , SUM(1 - outcome) "good_cnt"
              FROM t
              GROUP BY s 
            ) distr
         ON distr.s <= a.s
         GROUP BY a.s 
     ) cdf
;

In the context of an ROC plot, B(s) is often called sensitivity or true positive fraction, and G(s) is called 1-specificity or false positive fraction.

Kolmogorov-Smirnov Test
Feb 23, 2007 | /jeff | Link

One of the most widely (mis)used measures of scorecard performance is the Kolmogorov-Smirnov test (KS), colloquially known as the vodka test. In this post, I'll explain what KS is, and show a way to calculate it in SQL.

Given two samples of a continuous random variable, the two sample K-S test is used answer the following question: did these two samples come from the same distribution or didn't they? The idea is simply to compute the largest absolute difference between the two empirical cumulative distributions and to conclude that there is a significant difference if the difference is large enough.

Consider a risk score that predicts the probability of a customer defaulting (we'll call that 'going bad'). KS is the greatest difference between the cumulative distribution functions of the scores of the good and the bad populations:

KS = maxs|B(s) - G(s)|,

where

KS is often multiplied by 100 for convenience. In many contexts 40 is considered to be a good KS.

Let's try an example. Start with the table t that contains initial data:

Column Description
id Unique identifier
s Score
outcome 1 is bad, 0 is good

The following query calculates the KS: 

SELECT MAX(cdf.b - cdf.g) * 100                            "KS"
FROM ( SELECT a.s                                          "s"
            , SUM(distr.bad_cnt) /
              ( SELECT COUNT(*) FROM t WHERE outcome = 1 ) "b"
            , SUM(distr.good_cnt) /
              ( SELECT COUNT(*) FROM t WHERE outcome = 0 ) "g"
       FROM ( SELECT DISTINCT s FROM t ) a
       JOIN (
              SELECT s                "s"
                   , SUM(outcome)     "bad_cnt"
                   , SUM(1 - outcome) "good_cnt"
              FROM t
              GROUP BY s 
            ) distr
         ON distr.s <= a.s
         GROUP BY a.s 
     ) cdf
;

The easiest way to understand how the query works is by decomposing it into smaller pieces. In this case there are five uncorrelated subqueries.

This subquery returns distribution of goods and bads by score: 

SELECT s                "s"
     , SUM(outcome)     "bad_cnt"
     , SUM(1 - outcome) "good_cnt"
FROM t
GROUP BY s

Note how it relies on the fact that outcome can be either 0 or 1.

This subquery returns the list of all possible score values: 

SELECT DISTINCT s FROM t

This subquery returns the total number of bads: 

SELECT COUNT(*) FROM t WHERE outcome = 1

This subquery returns the total number of goods: 

SELECT COUNT(*) FROM t WHERE outcome = 0

Finally, this subquery (abbreviated for clarity) makes the distributions cumulative: 

SELECT a.s                              "s"
     , SUM(distr.bad_cnt) / total_bad   "b"
     , SUM(distr.good_cnt) / total_good "g"
FROM a
JOIN distr
  ON distr.s <= a.s
GROUP BY a.s

Note that it is rather inefficient because the join results in a partial Cartesian product. There's a better way to do the cumulation if your flavor of SQL supports online analytical processing (OLAP) functions: 

SELECT s                                                  "s"
     , SUM(FLOAT(bad_cnt)) OVER (ORDER BY s) / total_bad  "b"
     , SUM(FLOAT(good_cnt)) OVER (ORDER BY s) / total_bad "g"
FROM distr

Now the only thing left to do is to pick the maximum difference. This is the KS.

I'm new at predictive modeling. Help!
Feb 22, 2007 | /jeff | Link

It's true that there does not seem to be a lot of information on scoring and predictive modeling available online, and that many articles are written in rather heavy language, peppered with statistical jargon. But don't panic. To help you navigate the unchartered waters, here are some good places to start.

There are also a few exceptionally good books. My favorites are:

Finally, these two classes by the SAS Institute are worth taking: