Problem

LinkedIn is considering a change on the conversion funnel to the premium subscription. When a user clicks “activate premium” from the user homepage, they are directed to a page with product-tiers, displaying four products as shown below:

A new funnel considered recommends just one product after a user answers a series of questions. Based on their responses, they are recommended a product that can best serve them. How would you know which funnel the product team should use?

Tips

To address this product case question, structure the response in the following manner:

Clarifying Questions – Ask questions to understand how the conversion funnel works.
Success Metric – What are the primary and secondary metrics that should be considered in the experiment?
Experiment Design – How would you design the experiment? Consider randomization, statistical parameters of an experiment, statistical tests, and such.
Decisioning – How would you know whether to launch or not?

Solution

[Interviewer] Can you recommend which conversion funnel LinkedIn should use?

[Candidate] Thank you for the question. I have some questions I would like to clarify. I see that the product-tier page contains “Learn more” button on each product card. Does this direct users to another page?

[Interviewer] No, once the user clicks “Learn more,” they will be displayed with details below and a button that takes them to a checkout page.

[Candidate] Thank you for the clarification. So, just so I understand the funnel end-to-end, is this correct?

Control funnel: Product Page => Checkout => Convert

Treatment funnel: Questions Page => Product Page => Checkout => Convert

[Interviewer] Yes, that’s correct.

[Candidate] Great thank you. I’d like to start with a discussion on the success metric of the experiment.

Clearly, the end goal of the experiment is to increase conversion as it benefits LinkedIn’s revenue stream. Conversion reflected by the enrollment of one of the four products provided. Secondly, the unit of analysis should be per-user.

Given these aspects, I believe the primary metric of the experiment should be:

Conversion Rate = # of conversions# of users

[Interviewer] How would you account for users who may have entered the product page, and they didn’t subscribe the first time, but subscribed later? Would those users be counted as “converted?”

[Candidate] That’s a good question. We want to consider the immediacy of the experiment. Given that the user received either the control or treatment condition, did they respond by converting, right away? If they converted in the first visit session, then we know that the condition had an influence on the users. If the user converts, three or five sessions later, then there could be other factors that led them to the decision. This introduces extraneous variables that compromise the external validity of the experiment.

So, I believe we need to constrain the success metric based on the following:

Conversion Rate = # of conversions / # of first user sessions

In summary, we only consider the first user visits immediately after going through the funnel.

[Interviewer] A follow-up question on that. What if some users in the treatment do not complete the questionnaire? Would such users be tagged with failed conversions?

[Candidate] That’s another good question. We only want to count the users who have received the treatment – the questions – end-to-end before entering the product page. So, the # of first user sessions would only include users who have actually entered the product page.

[Interviewer] Great, are there any other metrics you want to consider?

[Candidate] Yes, there is one other metric we want to consider: (1) the % of users who complete the funnel, and (2) average revenue per user. These two metrics matter because even if the conversion rate is higher in the treatment than the control, if the % of users who finish the funnel and products selected are low-tiers, then the revenue may decrease. So, making a decision with these metrics in mind is vital.

[Interviewer] Great. Now, let’s talk about how you would set up the randomization for the experiment. How would you do this?

[Candidate] The randomization is performed at the user level, and the allocation is based on a 50-50 split of users into either the control or treatment bucket.

[Interviewer] Thanks. A quick follow-up on that, would you want to randomize any users on LinkedIn or a subset of users who enter the funnel?

[Candidate] That’s a good question, it makes the most sense to randomize users who enter the funnel. So, as soon as they click the “activate premium” button from the homepage, they are randomly assigned to either the control or treatment funnel.

[Interviewer] Great. Now, walk me through your experiment parameters like power, alpha, and e.t.c.

[Candidate] Sure thing. I’d like to consider the following parameters of the experiment:

Significance Level (Alpha) = 0.05

Power = 0.80

MDE = 1% Lift

Using these parameters and a statistical test, T-Test for proportions, I can compute the sample size required per group to achieve 80% power.

[Interviewer] Can you elaborate on why you chose 1% for MDE?

[Candidate] Well, the reason is that LinkedIn has 500M+ active users and millions of dollars in revenue generated from the subscription. 1% lift in the conversion rate can potentially equate to millions of dollars of additional revenue added. This should be considered practically significant for the business. I do want to emphasize that, ultimately, this requires a conversation with stakeholders to determine what is practically significant for them. Based on that, this will inform what would be a reasonable lift. But, I will just use 1% for now in this example.

[Interviewer] How many weeks should you run the experiment?

[Candidate] Typically, experiments on online platforms are run for at least a week to capture the days-of-the-week effect. And, at most, it should run for one or two weeks. Ultimately, this is based on various factors such as the urgency that a stakeholder has in gathering results and decisioning and the volume of traffic that can be allocated to the experiment.

[Interviewer] Great, now let’s talk about decisioning. How would you know whether to launch the new funnel or not?

[Candidate] After gathering the results, the first thing is to check the internal and external validity of the experiment. If there’s no violation, then I can proceed with the analysis of the result.

I would look at the relative lift, the p-value and confidence interval of the experiment. If the new funnel improved the conversion rate along with the total revenue generated, then it should be ramped up to wider use. For instance, we can ramp up the experiment to 5% of the user population, followed by 10%, 20%, 50%, and finally 90%. We hold out the remaining 10% at the final stage of the ramp-up to observe the long-term effect of the new funnel.

[Interviewer] What if the p-value on the lift is greater than 0.05, what does it mean for decisioning?

[Candidate] We want to consider the direction of the lift. If it’s negative, then, unless there’s a strong justification that the funnel should work, we stop the experiment there. If it’s slightly or extremely positive, and the p-value is close to 0.05, then this could be the consequence of the type II error rate – the chance of failing to reject when there’s an effect. To mitigate this, we re-run the experiment with increase power from 0.80 to 0.90 and assess whether the lift is significant.

Interviewer Assessment

The candidate is assessed based on three criteria: business/product sense, statistical methodology, and communication. Each dimension contains the following rating: (1) not competent, (2) marginal, (3) adequate, (4) good, (5) superior.

Business and Product Senses – 5

The candidate receives a superior remark based on the following:

The candidate demonstrated a strong product sense in understanding how the product funnel works. She understands the funnel process end-to-end, which helped her determine sound metrics for the experiment.
The candidate understands the practical significance of an experiment. She understands that MDE should be chosen based on what’s practically significant for the business, not 1% MDE just because it’s considered “industry-standard.”

Statistical Methodology – 5

The candidate receives a superior remark based on the following:

The candidate understands the framework of designing AB testing. She picked reasonable statistical parameters (i.e. 0.05 for the significance level, 0.80 for power).
The candidate provided metrics that were sound. Rather than just saying that the experiment should track the conversion rate, she understands the constraints of the metrics and ensured to call them out.

Communication – 5

The candidate receives a superior remark based on the following:

The candidate structured her responses well. She started with questions to clarify her assumptions about the problem. This is vital in experiments when business problems are often open-ended. This allowed her to provide a solution that was sound.
She expressed her ideas very clearly.

[LinkedIn] Premium Subscription Funnel

Problem

Unlock now by joining.