Discussion:
DIRECT MARKETING CAMPAIGN--CONTROL GROUP question
(too old to reply)
Nick .
2005-09-06 18:35:17 UTC
Permalink
Hello,

I have a question regarding the selection of a CONTROL GROUP in a direct marketing campaign (banking) setting.

I have built a (logistic) model and I have deciled my PROSPECT population from DECILE1 (low probability of rersponse) to DECILE10 (high probability of response). For example, if I have a prospect population of 1 million, then each decile gets 100K. I want to campaign to DECILE7 through DECILE10. At the same time, I wish to put aside a CONTROL GROUP to see how the model will perform against it. Here is my thought:

Since I have 1 million prospects, I will select 400K prospects--100K from DECILE7, 100K from DECILE8, 100K from DECILE9, and 100K from DECILE10. That's 400K, the population I will campaign to. I want to put aside 10% as a control group.

Do I randomly select the 10% from the 400K records (so I will end up campaigning to 360K) or do I randomly select the 10% from the 1 million records?

If I select the 10% from the 400K prospects (my high probability of response prospects according to the model), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control about the same, say about 5% response for the mailed (modeled) group and the control group, then am I saying that the model is not a good model because it has not provided a higher response than the control?

If I select the 10% from the 1 million prospects (my high probability of response prospects), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control much different, say 5% response for the mailed (modeled) group and 2% for the control group, then am I saying that the model is a good model because it has provided a (much) higher response than the control?


I guess, I don't quite know how to select the control group so as to compare model response rate to control group response rate.

Your thoughts are much needed and appreciated.

NICK


--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm
Talbot Michael Katz
2005-09-06 19:42:03 UTC
Permalink
Hi, Nick.

As Peter Luo has advised you, you should choose your control group from
across your entire target population, if you want to measure the
difference between using a model and targeting at random. You might wish
to check out a previous SAS-L thread concerning "Control Group Assignment"
from the second week of July. I gave the standard formula for computing
sample size needed to discriminate between the test and control response
rates (http://listserv.uga.edu/cgi-bin/wa?A2=ind0507B&L=sas-l&P=R7421).
But you will also want to pay close attention to David Cassell's critique
of that (http://listserv.uga.edu/cgi-bin/wa?A2=ind0507B&L=sas-
l&D=0&P=11278).

Good luck!

-- TMK --
"The Macro Klutz"
Post by Nick .
Hello,
I have a question regarding the selection of a CONTROL GROUP in a direct
marketing campaign (banking) setting.
Post by Nick .
I have built a (logistic) model and I have deciled my PROSPECT population
from DECILE1 (low probability of rersponse) to DECILE10 (high probability
of response). For example, if I have a prospect population of 1 million,
then each decile gets 100K. I want to campaign to DECILE7 through
DECILE10. At the same time, I wish to put aside a CONTROL GROUP to see how
Post by Nick .
Since I have 1 million prospects, I will select 400K prospects--100K from
DECILE7, 100K from DECILE8, 100K from DECILE9, and 100K from DECILE10.
That's 400K, the population I will campaign to. I want to put aside 10% as
a control group.
Post by Nick .
Do I randomly select the 10% from the 400K records (so I will end up
campaigning to 360K) or do I randomly select the 10% from the 1 million
records?
Post by Nick .
If I select the 10% from the 400K prospects (my high probability of
response prospects according to the model), execute the campaign, the
results come back and I see a response rate from the campaign (i.e. the
model) and the control about the same, say about 5% response for the
mailed (modeled) group and the control group, then am I saying that the
model is not a good model because it has not provided a higher response
than the control?
Post by Nick .
If I select the 10% from the 1 million prospects (my high probability of
response prospects), execute the campaign, the results come back and I see
a response rate from the campaign (i.e. the model) and the control much
different, say 5% response for the mailed (modeled) group and 2% for the
control group, then am I saying that the model is a good model because it
has provided a (much) higher response than the control?
Post by Nick .
I guess, I don't quite know how to select the control group so as to
compare model response rate to control group response rate.
Post by Nick .
Your thoughts are much needed and appreciated.
NICK
--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm
Susie Li
2005-09-07 14:24:00 UTC
Permalink
You would set up your control group based on what you are testing against.
If you want to test the response performance of using your model versus that
of using no model at all (namely, a random selection of prospects to mail
to), then you would randomly select 10% from your entire 1 million prospects
as control, and perhaps 10% from the top model-based 3 deciles as your test
group. Then compare the response rates between these 2 groups - the test
versus control group.

To establish the 10 deciles for your 1 million prospects, you would first
find the 9 cutpoints from your modeling sample population - i.e., the cutoff
probabilities that would divide your sampling population into 10 equal size
groups of ranked estimated individual probabilities. These are the
estimated probability of response coming from your model for each individual
in your sample.

Second, run your model against the entire prospect population of 1 million,
and then use the sample-based cutpoints to group the population into 10.
This becomes your decile-segmented universe. Don't be surprised the
resulting 10-decile groups are not of equal sizes.



Susie Li
TV Guide
1211 Avenue of the Americas
New York, NY 10036
Tel 212.852.7453
Email ***@tvguide.com

-----Original Message-----
From: SAS(r) Discussion [mailto:SAS-***@LISTSERV.UGA.EDU] On Behalf Of Jonas
Bilenas
Sent: Wednesday, September 07, 2005 7:33 AM
To: SAS-***@LISTSERV.UGA.EDU
Subject: Re: DIRECT MARKETING CAMPAIGN--CONTROL GROUP question

Somewhat related:

How do you setup your deciles? Do you use PROC RANK and TIES=LOW option to
prevent the same score being split over 2 deciles?

Also, why just Test vs Control? Why not set up an experimental design to
test multiple factors?

Jonas Bilenas
JP MORGAN CHASE BANK
Post by Talbot Michael Katz
Hi, Nick.
As Peter Luo has advised you, you should choose your control group from
across your entire target population, if you want to measure the
difference between using a model and targeting at random. You might wish
to check out a previous SAS-L thread concerning "Control Group Assignment"
from the second week of July. I gave the standard formula for computing
sample size needed to discriminate between the test and control response
rates (http://listserv.uga.edu/cgi-bin/wa?A2=ind0507B&L=sas-l&P=R7421).
But you will also want to pay close attention to David Cassell's critique
of that (http://listserv.uga.edu/cgi-bin/wa?A2=ind0507B&L=sas-
l&D=0&P=11278).
Good luck!
-- TMK --
"The Macro Klutz"
Post by Nick .
Hello,
I have a question regarding the selection of a CONTROL GROUP in a direct
marketing campaign (banking) setting.
Post by Nick .
I have built a (logistic) model and I have deciled my PROSPECT population
from DECILE1 (low probability of rersponse) to DECILE10 (high probability
of response). For example, if I have a prospect population of 1 million,
then each decile gets 100K. I want to campaign to DECILE7 through
DECILE10. At the same time, I wish to put aside a CONTROL GROUP to see how
Post by Nick .
Since I have 1 million prospects, I will select 400K prospects--100K from
DECILE7, 100K from DECILE8, 100K from DECILE9, and 100K from DECILE10.
That's 400K, the population I will campaign to. I want to put aside 10% as
a control group.
Post by Nick .
Do I randomly select the 10% from the 400K records (so I will end up
campaigning to 360K) or do I randomly select the 10% from the 1 million
records?
Post by Nick .
If I select the 10% from the 400K prospects (my high probability of
response prospects according to the model), execute the campaign, the
results come back and I see a response rate from the campaign (i.e. the
model) and the control about the same, say about 5% response for the
mailed (modeled) group and the control group, then am I saying that the
model is not a good model because it has not provided a higher response
than the control?
Post by Nick .
If I select the 10% from the 1 million prospects (my high probability of
response prospects), execute the campaign, the results come back and I see
a response rate from the campaign (i.e. the model) and the control much
different, say 5% response for the mailed (modeled) group and 2% for the
control group, then am I saying that the model is a good model because it
has provided a (much) higher response than the control?
Post by Nick .
I guess, I don't quite know how to select the control group so as to
compare model response rate to control group response rate.
Post by Nick .
Your thoughts are much needed and appreciated.
NICK
--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm
r***@gmail.com
2018-07-12 15:37:44 UTC
Permalink
Post by Nick .
Hello,
I have a question regarding the selection of a CONTROL GROUP in a direct marketing campaign (banking) setting.
Since I have 1 million prospects, I will select 400K prospects--100K from DECILE7, 100K from DECILE8, 100K from DECILE9, and 100K from DECILE10. That's 400K, the population I will campaign to. I want to put aside 10% as a control group.
Do I randomly select the 10% from the 400K records (so I will end up campaigning to 360K) or do I randomly select the 10% from the 1 million records?
If I select the 10% from the 400K prospects (my high probability of response prospects according to the model), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control about the same, say about 5% response for the mailed (modeled) group and the control group, then am I saying that the model is not a good model because it has not provided a higher response than the control?
If I select the 10% from the 1 million prospects (my high probability of response prospects), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control much different, say 5% response for the mailed (modeled) group and 2% for the control group, then am I saying that the model is a good model because it has provided a (much) higher response than the control?
I guess, I don't quite know how to select the control group so as to compare model response rate to control group response rate.
Your thoughts are much needed and appreciated.
NICK
--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm
Nick,

I know its been long but did you get the answer ?
How did you end up selecting your control group ?

I am currently going through same issue.

1. I created a model based on previous DM campaign responders and non-responders.
2. Applied it on the new prospects and calculated their probability of response.
3. Ranked them on Probability and created 10 DECILES with DECILE1(Highest probability of response) to DECILE10(Lowest probability of response).
4. Created Mail and Holdout groups from TOP4 DECILES taking 10% as Holdout.
5. Sent DM to Mail groups and Nothing to Holdout.

Now I am comparing performance of MAIL and HOLDOUT groups.
MAIL(TOP4 DECILE) = Response rate 1.16%
Holdout(TOP4 DECILE) = Response rate 1.02%
Incremental response rate = 1.16%-1.02%= 0.15%

Problem: This is the same Incr. Response we were getting without model.

My assumption is, the way I am selecting Holdout Group is not correct.

How should I tackle it ?
How can I judge my model performance ?
Post by Nick .
Hello,
I have a question regarding the selection of a CONTROL GROUP in a direct marketing campaign (banking) setting.
Since I have 1 million prospects, I will select 400K prospects--100K from DECILE7, 100K from DECILE8, 100K from DECILE9, and 100K from DECILE10. That's 400K, the population I will campaign to. I want to put aside 10% as a control group.
Do I randomly select the 10% from the 400K records (so I will end up campaigning to 360K) or do I randomly select the 10% from the 1 million records?
If I select the 10% from the 400K prospects (my high probability of response prospects according to the model), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control about the same, say about 5% response for the mailed (modeled) group and the control group, then am I saying that the model is not a good model because it has not provided a higher response than the control?
If I select the 10% from the 1 million prospects (my high probability of response prospects), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control much different, say 5% response for the mailed (modeled) group and 2% for the control group, then am I saying that the model is a good model because it has provided a (much) higher response than the control?
I guess, I don't quite know how to select the control group so as to compare model response rate to control group response rate.
Your thoughts are much needed and appreciated.
NICK
--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm
Hello,
I have a question regarding the selection of a CONTROL GROUP in a direct marketing campaign (banking) setting.
Since I have 1 million prospects, I will select 400K prospects--100K from DECILE7, 100K from DECILE8, 100K from DECILE9, and 100K from DECILE10. That's 400K, the population I will campaign to. I want to put aside 10% as a control group.
Do I randomly select the 10% from the 400K records (so I will end up campaigning to 360K) or do I randomly select the 10% from the 1 million records?
If I select the 10% from the 400K prospects (my high probability of response prospects according to the model), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control about the same, say about 5% response for the mailed (modeled) group and the control group, then am I saying that the model is not a good model because it has not provided a higher response than the control?
If I select the 10% from the 1 million prospects (my high probability of response prospects), execute the campaign, the results come back and I see a response rate from the campaign (i.e. the model) and the control much different, say 5% response for the mailed (modeled) group and 2% for the control group, then am I saying that the model is a good model because it has provided a (much) higher response than the control?
I guess, I don't quite know how to select the control group so as to compare model response rate to control group response rate.
Your thoughts are much needed and appreciated.
NICK
--
___________________________________________________________
Sign-up for Ads Free at Mail.com
http://promo.mail.com/adsfreejump.htm
Loading...