Problem 🎯

An interviewer at CVS asked:

Suppose CVS wants to increase sales of an over-the-counter test kit that screens for colon cancer. To promote the kit, CVS is considering three methods of outreach:

Mail – A letter sent to the customer’s mailing address.
Email – An email sent to the customer’s email address.
Walk-In – A customer at the CVS pharmacy is notified by the pharmacist.

How would you design an experiment?

Tips💡

Consider randomization at the aggregate levelCustomer-level analysis may not work if you cannot measure and track individual customer behavior. Rather than analyzing the experiment at the customer level, is there a way to analyze it at an aggregate level?
Choose the primary KPI carefullyDon’t naively presume you can measure the conversion rate across all the methods. Consider what the control variable is given that the variants are mail, email, and walk-in.
Consider using a multivariate model to measure the effectOnce you collect your data, you may want to consider using the multivariate approach to account for confounding variables in your experiment.

Solution ✍️

Interviewer: Alright, using the scenario you are given about the CVS outreach. How would you design an experiment?

Commentary: A naive response might look something like this: “Well, I would just randomly send out each of the promotional methods and see which method gets the most purchases of the test kit.”

This simplistic answer neglects many potential variables and factors that might influence the outcome, such as timing, frequency, or the differences in the populations receiving each method.

Let’s take a look at a more thought-out approach to the question.

Candidate: Thank you for the question. I’d like to start with clarifying questions if that’s okay with you.

Interviewer: Sounds good.

Candidate: I’d like to consider the business context of the problem. Could I have some more information about the main goal of this experiment? For instance, is the main goal to increase awareness or profit margin of the kit? The reason I ask is that, if the former is the aim, we should not be concerned about the cost of an outreach employed. If it is the latter, we need to consider the cost.

Interviewer: That’s a great question. Yes, it would be best if we account for the cost of the outreach as well. To provide more context, let’s just say that the retail price of the kit is $100. The costs of mail, email, and walk-in are $5, $1, and $10, respectively. You can ignore any other costs (e.g. tax) related to the kit.

Candidate: Understood. So, the profit margin per each of the three methods, mail, email, and walk-in are $95, $99, and $90. I have another question – is this kit available to purchase online as well? I suppose some customers may order the kit on the CVS online store for convenience. And, we need to account for that in our experiment.

Interviewer: Yes, the kit is available to purchase online. I do think it’s a good idea that you may want to account for online purchases. How would you consider what the control group should be for this experiment?

Candidate: I suppose it depends on whether the kit is already sold in stores. Customers without any outreach can be considered as the control group for the experiment. So, in essence, there are four groups for this experiment – no outreach, mail, email, and walk-in.

Interviewer: Okay. Now, how would you decide which customers are chosen for the experiment?

Candidate: I believe this is the challenging aspect of the experiment. In Online experiment on mobile apps or browsers, we can track user’s behavior from the moment a treatment is applied to the response. And, we can randomly assign a user to a group based on the user ID. But, this may not be feasible given that most CVS customers do not have accounts established with CVS. For instance, let’s consider the following scenarios:

If a user has a CVS card or CVS account that can be retrieved at the time of purchase in-store, we can identify whether the customer successfully purchased the kit based on the group he or she was assigned to.
If a user does not have a profile, we don’t have a way to track their behavior in the aftermath of the purchase. To elaborate, if a user receives a mail about the cancer screen kit and purchases the kit, we don’t have a way to link that purchase to the user who was assigned to the experiment.

So, randomization and analysis at the user-level may not work in this case.

Commentary: This is an excellent point presented by the candidate! Indeed, you cannot assume that a typical AB experiment can work in this case as randomization at the user level is not feasible. The only way this could be feasible is if you find a way to attribute a kit purchase to the customer who is part of the experiment. One such way could be mailing or emailing a voucher with a discount code – let’s say a $10 discount.

For instance, if a customer receives an email with a voucher and walk into the store a few days later and applies the discount from the voucher, the kit sale can be attributed to email. However, this may not be feasible in tracking customers in the control group – those who do not have any outreach. Hence, user-level analysis may not be feasible in this case.

Interviewer: Okay, fair point. Since user-level analysis may not be feasible, what would be the alternative?

Candidate: The alternative is to design the experiment at the aggregate level – let’s say store or zip code. Customers in the surrounding area of a store or within zip code would all receive the same treatment applied to the area. Hence, the randomization unit could be performed at either the store or zip code level. Then, you could look at the total profit margin, sales, or quantity of the kit sold per unit.

But, depending on the choice of the randomization unit: store and zip code, there are drawbacks to consider. For instance, when the aggregation is performed at the store level, there’s a spillover effect.

CVS has over 9,000 locations nationwide. In NYC alone, there are over a dozen locations – sometimes just a few blocks away. A customer may live near a store that’s assigned to no outreach, but may purchase the kit at another store assigned with a treatment during a commute to work.

To mitigate the spill over effect, we would want to apply the randomization at the zip code level. The spillover effect could still exist, but this risk is greatly reduced.

Interviewer: Could you consider any challenges with the zip code level aggregation?

Candidate: I suppose that the challenge with this zip code aggregation is that we are now prone to confounding factors in relation to the zip code. For instance, the control group may primarily consist of zip codes that have low population size and low income where as one of the variant groups may primarily consist of zip codes that have large population size and high income. We may observe a noticeable difference in the total sales between the control and variant groups. But, we cannot establish the difference due to the outreach. It could be that the larger population size and higher income is, in fact, the contributing factor to the difference. Not only that, some zip codes may have multiple CVS locations. So, accommodate for the differences, we need to include them as co-variates in the model.

Interviewer: Okay, could you elaborate on what you mean by including the factors as co-variates?

Candidate: Certainly. In a typical AB test, you apply T-Test or some other univariate test (e.g. chi-squared test or non-parametric test) to run your experiment. But, to account for confounding variables like population size and income, you need a multivariate model such as the multivariate linear regression model.

For instance, suppose that we have the following data gathered from the experiment:

Zipcode	Method	Population Size	Median Income	No. of Stores	Sales Volume
23033	No outreach	9,000	30,000	3	10
55011	Email	5,000	45,000	2	8
32011	Mail	8,000	24,000	1	5
44033	Walk-in	9,000	20,000	2	15
23090	No outreach	10,000	25,000	3	15
66066	Email	4,000	35,000	2	10
90102	Mail	7,000	12,000	2	3
44033	Walk-in	19,000	20,000	4	18
…	…	…	…	…	…

We can formulate our model with the following equation:

Y (Sales Volume) ~ B0 + B1Population Size + B2Median Income + B3No-of-Stores + B4Method

B0 ~ Intercept

B1 – B3 ~ Covariates

B4 ~ Main factor of interest

Commentary: It’s great to see that the candidate has fully fleshed out the data and the model equation. However, the candidate should elaborate more on the main KPI used as a proxy for measuring the effectiveness of each campaign. There are three candidate KPIs to consider:

Sales Volume ~ Quantity of Items sold
Sales Revenue ~ Total $ amount generated
Net Income ~ Sales minus expense (assume that the expense is just the cost of each delivery)

For instance, sales volume may not be the best measure if net income is the main goal of improving. Each method has a varying net income – mail, email, and walk-in are $95, $99, and $90, respectively. Thus, instead of sales volume, the candidate could consider net income as the independent variable measured in this model.

Y (Net Income) ~ B0 + B1Population Size + B2Median Income + B3No-of-Stores + B4Method

Interviewer: Okay, thank you for that. How do you decide how long you should run the experiment and how do you determine the sample size?

Candidate: Thank you for the question. Since we are using a regression model, in this case, the same sample size calculation ( sample size per group = 16*Variance / delta^2) applied in T-Tests does not hold here. There are some calculations that could be performed based on the effect size comparing the r-squared value of the reduced model vs the full model:

Reduced model: Y (Sales Volume) ~ B0 + B1Population Size + B2Median Income + B3No-of-Stores*

Full model: Y (Sales Volume) ~ B0 + B1Population Size + B2Median Income + B3No-of-Stores + B4Method

But, the exact calculation of this isn’t something I know readily, but I can find out upon research. Now, in terms of the experiment duration, tracking the sales per zip code for 30 days should be sufficient enough for us to detect a noticeable difference among the methods.

Interviewer: Okay, great. I’d like to discuss the confounding variable again. When you consider each of the methods, you should expect varying amount of customer who received the treatment. For instance, customers in zipcode assigned to the email group will receive more treatments compared to the walk-ins. So, how do you know if it’s the actually delivery method that is influencing the purchase decision, or simply because the volume of people who actually receive the treatment is different?

Candidate: That’s a great question. I do think that the volume of treatment received can pose an issue. For instance, if the aggregate number of customers in the email group is 10,000 while that of customers in the walk-in group is 1,000, the sales volume in the email group would most likely be higher than the walk-in group – simply because the treatment size is much larger. One way to mitigate this is to normalize the data using conversion rate – out of those who received the treatment within a zipcode, what % of the customers converted. But, the problem with this approach lies in the fact that we do not know any information about the control group customers. We cannot know what is the size of the control group to calculate the conversion rate when there’s no outreach among customers in this group the first place. So, we can include the outreach size (the number of individuals in the zipcode out reached in the experiment) in the model, but set the outreach size as 0 in the no outreach zipcode group. For instance, our new data would be the following:

Zipcode	Method	Outreach Size	Population Size	Median Income	No. of Stores	Sales Volume
23033	No outreach	0	9,000	30,000	3	10
55011	Email	400	5,000	45,000	2	8
32011	Mail	4000	8,000	24,000	1	5
44033	Walk-in	50	9,000	20,000	2	15
23090	No outreach	0	10,000	25,000	3	15
66066	Email	500	4,000	35,000	2	10
90102	Mail	4000	7,000	12,000	2	3
44033	Walk-in	80	19,000	20,000	4	18
…	…		…	…	…	…

And, our new model equation is the following: Full model: Y (Sales Volume) ~ B0 + B1Population Size + B2Median Income + B3No-of-Stores + B4Method + B5Outreach Size*

Interviewer: Okay great! Now, how would evaluate the result and provide a recommendation on which method CVS should go with?

Candidate: We would want to evaluate the statistical and practical significance of each method. And, the significance would be compared against the MDE and significance level set prior to the experiment. Let’s say that the values are +10 additional sale is considered to be practically significant, and the significance level is 0.05. And, let’s presume that the result from the regression model is as follows:

Method	Increase in Sales Volume	P-Value
No outreach	—	—
Email	+15	0.04
Mail	+3	0.07
Walk-In	+20	0.02

Based on this result, I would say that email and walk-in are found to be statistically significant. Therefore, we should consider these methods. The walk-in is the most effective in increasing sales volume, and we could choose this as the winning method.

However, unlike the AB testing approach where we may apply a one-size fits all approach. Given that we can presume that depending on the demography of each zip code, customers may react differently to each treatment method. So, given the characteristics of each zip code – median population size, income and no-of-stores, using the model, I would estimate what’s the expected sales from email and walk-in, and choose the campaign method that produces the highest sales.

Interviewer: Are there any other type of considerations?

Candidate: Yes, I’d recommend conducting a pilot study first to iron out any operational challenges, especially with the walk-in method where the pharmacist needs to notify customers. It’s essential to ensure consistent messaging across all methods, and feedback loops to quickly identify and rectify any issues during the promotional period.

Assessment 📝

The candidate’s overall performance is outstanding across the evaluated categories. They showed a robust understanding of technical aspects, married it well with business context, and communicated with clarity.

Business Sense & Insights

Rating: Proficient

Overall, the candidate displayed a good grasp of the business context by not only addressing the technical task. But, the candidate could have evaluated considered other KPIs to measure. The candidate chose the sales volume as the primary metric, but sales volume may not be the best method when the cost of promote a particular method outweighs the bottomline of the business. Considering other metrics such as the net income.

Communication & Presentation Skills

Rating: Outstanding

The candidate communicated complex methods, intentions, and insights clearly. They effectively broke down the information, ensured comprehension at each stage, and articulated their thought process. They successfully bridged technical aspects with business concerns.

Technical Methodology

Rating: Outstanding

The candidate offered an outstanding experimentation design. Most solutions presumes that the experiment can be conducted using a similar procedure such as the ones seen in AB testing: randomization at the user level and T-test for comparison of groups.

However, as the candidate recognized, the experimentation requires a randomization at the aggregate level (e.g. zip code). In addition, the candidate provided how she would consider confounding variables by including those variables in the regression model.

Problem Solving & Critical Thinking

Rating: Outstanding

The candidate was able to come up with quick and sound responses to follow-up questions. For instance, when asked about how to deal with covariates in regards to the impact that an outreach size has on confounding the effect of the outreach method, the candidate was able to quickly realize that she could include the size as one of the factors in the regression model. In addition, the candidate understands that there’s no method that is one-size fits all. She suggested the idea of the estimate from the model to find the method that generates the maximum sales.

[CVS] Marketing Outreach Experiment

Problem 🎯

Unlock now by joining.