Last Updated: August 2021

Problem

Duration: 15 Minutes
Difficulty: Easy

An interviewer at LinkedIn asked:

Two email versions were tested in San Francisco (SF) and New York City (NYC).

The total number of emails sent was 100,000 with 80,000 of those being email A while the remaining 20,000 being email B. The click-through rate (CTR) of email A was 15% while that of email B was 30%.

However, in SF, the CTR of email A was 15% while that of email B was 12.5% while, in NYC, the CTR of email A was 15% while the CTR of email B was 41.7%.

What is your recommendation on whether to launch or not?

Tips & Hints

This section provides tips and hints to help you practice this case problem, effectively. The practice tip section highlights how to practice this question. The solution hint section provides clues on how to solve this problem if you get stuck. Skip the solution hint if you want to try this problem on your own.

Practice Tip

To practice this problem effectively, set a timer for 15 minutes and explain your solution out loud to yourself or a practice buddy.
Make sure to use a whiteboard, sheet of paper, or word document to jot down your ideas. Jotting down your ideas will help structure your response.
Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.

Solution Hint

First, assess the problem. Recognize the statistical phenomenon that explains the shifts in the CTR trends when comparing versions and versions x cities.

Solution

[Interviewer] Two email versions were tested in San Francisco (SF) and New York City (NYC).

The total number of emails sent was 100,000 with 80,000 of those being email A while the remaining 20,000 being email B. The click-through rate (CTR) of email A was 15% while that of email B was 30%.

However, in SF, the CTR of email A was 15% while that of email B was 12.5% while, in NYC, the CTR of email A was 15% while the CTR of email B was 41.7%.

What is your recommendation on whether to launch or not?

[Candidate] Before I provide a recommendation on which email should be launched, I would like to make sense of the campaign results. I see that the sample sizes of email versions within each city are missing. Could I have more information on this?

[Interviewer] Good question. Let’s say that the sample sizes are the following. Among the 80,000 emails being version A, 60,000 of those are sent to SF while the remaining 20,000 are sent to NYC. Among 20,000 emails being version B, 12,000 of those are sent to SF while the remaining 8,000 are sent to NYC.

[Candidate] Great, thank you. I would like to organize the results in a summary table to help me diagnose the problem. Please, let me know if my summary table is valid based on the problem you stated.

	Total		NYC		SF
Variations	Size	Conversions	Size	Conversions	Size	Conversions
A	80,000(80%)	12,000 (15%)	20,000	3,000(15%)	60,000	9,000 (15%)
B	20,000(20%)	6,000 (30%)	12,000	5,000 (41.7%)	8,000	1,000 (12.5%)

Commentary: Note that the candidate summaries the number in a table format. Case rounds are often verbal. In this particular case, the interviewer posed a lengthy question filled with numbers such that crucial details could be forgotten. The candidate takes the time to organize the details in a summary table as this approach aids her comprehension and shows strength in her data intuition to the interviewer.

[Interviewer] You got it. Do you notice any phenomenon that explains why the emails perform differently when considering cities? *Address this question.

[Candidate] Well, I see that when the emails are combined across cities, email B with 30% CTR clearly performs better than email A with 15% CTR. However, the performance flips in SF as email B CTR is 12.5% while email A CTR is 15%. I am sure that the performance is due to random chance. If you were to re-run the campaign, the city-level result could become consistent with the combined-level result.

Commentary: The candidate missed an opportunity to identify Simpson’s Paradox. Simpson’s Paradox is a phenomenon when trends changed directions when groups are combined. The phenomenon often occurs in experiments when control and treatment groups are evaluated across sub-groups such as countries and devices. Given that the paradox is basic in statistics, the failure to identify this can cost a point in the interview.

[Interviewer] Do you see any potential issues with the sample sizes not being consistent across groups? *Address this question.

[Candidate] Not at all. Given that we evaluate the results using proportions, this allows us to compare the email performances even though the sample sizes among the groups vary.

Commentary: The candidate made a faulty assumption that the varying sample sizes across groups do not pose any issues. The candidate could have pointed out two potential issues with the experiment.

First of all, was the sample size per group calculated using the statistical power before testing the emails? If so, the sample sizes should have been the same in versions A and B. Given the mismatch, the experimenter could have skipped the crucial step in determining the sample sizes before running the experiment. As a consequence, the standard errors of the CTRs vary greatly, potentially leading to a false conclusion about which version performs better.

Secondly, if the sample sizes were determined using the statistical power, the ratio between the two A and B should have been, respectively, 1 to 1, not 4 to 1. Such a condition in AB testing is called the Sample Mismatch Ratio (SMR) when the actual ratio of the sample sizes is different from the expected ratio, causing bias in results. The underlying cause could be an instrumentation error when users are not correctly assigned to the groups randomly.

[Interviewer] Okay. Now, let’s suppose that the different samples sizes do not pose issues. What could be other reasons that cities reacted to the emails differently? *Address this question.

[Candidate] I can think of two factors. First, the content in email B was more appealing to users in NYC than those in SF. Let’s presume that the email recommended jobs to users. The job openings were more appealing to users in NYC than those in SF. The poor email result in SF might indicate that the jobs recommender system requires calibration.

Another reason could be the macroeconomic factor such as the recession. Perhaps, the recession led to lay-offs in an industry. Jobs in SF are generally tech while jobs in NYC are generally finance. The recession might have afflicted a particular industry such that users who are geographically close became unemployed and receptive to emails with job openings.

Commentary: The candidate receives points on data intuition and communication. She provided a thorough explanation of the potential factors that explain the variability in the email results across cities.

[Interviewer] What is your recommendation on whether to launch or not? *Address this question.

[Candidate] I recommend email B to launch since it had a higher conversion rate of 30%. Even though the results vary by city, the variability can be ignored given that the global average for email B is higher than A. The ratio should generalize if the campaign was rolled out to more markets.

Commentary: The candidate made several key mistakes in her launch decision.

First of all, she failed to mention potential issues with low statistical power and sample mismatch ratio. Unless those are addressed, the results are biased, requiring a re-run with corrections.

Secondly, she disregarded the statistical and practical significance of the results. Though the conversion rate in email B was higher than that in email A, were the results significant statistically and practically to justify launching?

Lastly, she ignored the possibility that emails could perform differently by geography. The candidate should have proposed to investigate the underlying factors that led to the discrepancy in email results by city and propose re-running a separate AB test per city.

Assessment

The candidate is assessed across three attributes: statistics, data intuition, and communication. Each attribute is rated based on the following scale:

Outstanding – Quick response with a sound solution.

Good – Minor mistakes, but converged toward a final solution that is sound.

Borderline – Required several hints before providing a sound solution.

Inadequate – Incorrect response.

Based on the rubric above, the candidate receives the following remarks:

Statistics – Inadequate

The candidate made several critical mistakes throughout the interview. (1) She failed to recognize Simpson’s Paradox which is basic in statistics, (2) assumed that sample sizes being noticeably different do not pose issues to the experimental results, and (3) disregarded validity checks and statistical/practical significance before providing a launch decision.

Data Intuition – Borderline

In the beginning of the interview, the candidate showed competency in understanding the problem by creating a summary table of the email results. However, she faltered in making sound decisions based on the results as she was overly optimistic about the performance of email B generalizing well once launched. She should have assessed the results with skepticism, requesting more clarification and proposing to investigate further before making a launch decision.

Communication – Good

The candidate receives a good remark given that she clearly understood the problem and expressed her thoughts clearly for the most part of the interview. However, she failed to receive an outstanding remark given that her justifications in some answers were shallow. For instance, she failed to provide a comprehensive explanation of her launch decision.

[LinkedIn] Email Conversions

Problem

Unlock now by joining.