Last Updated: September 2021

Problem

Duration: 15 Minutes
Difficulty: Medium

An interviewer at Google asked:

Suppose that an experiment on the search result page replaced ads displayed on the side of the search result to the top. What metric(s) would you use for your experiment? Assume that ad content itself remains constant, and the only chance is the position.
Suppose that revenue from ad clicks increased, but the clicks on web pages (non-Ads) decreased. Should the test be launched?

Tips & Hints

This section provides tips and hints to help you practice this case problem, effectively. The practice tip section highlights how to practice this question. The solution hint section provides clues on how to solve this problem if you get stuck. Skip the solution hint if you want to try this problem on your own.

Practice Tip

To practice this problem effectively, set a timer for 15 minutes and explain your solution out loud to yourself or a practice buddy.
Make sure to use a whiteboard, sheet of paper, or word document to jot down your ideas. Jotting down your ideas will help structure your response.
Share your solution either on the course discussion or Slack channel to receive feedback from peers and instructors.
As you read the solution dialogue, sometimes you will see “*Address this question” after an interviewer asks a follow-up to the candidate. Pause the reading for a minute to respond to this follow-up as though you are the candidate.

Solution Hint

If you are struggling to frame the response, note the following elements:

Clarifying questions – Ask clarifying questions to address assumptions before diving into a solution.
Product knowledge – How does Google search monetize? Understanding this will help you define the key metric for question #1.

Solution

[Interviewer] Suppose that an experiment on the search result page replaced ads displayed on the side of the search result to the top. What metric(s) would you use for your experiment? Assume that ad content itself remains constant, and the only chance is the position.

[Candidate] I’d like to start with clarifying questions to identify key metrics for the experiment.

[Interviewer] Sounds good.

[Candidate] If I understand the problem correctly, the ads are initially displayed on the side of the search result. The change now provides a list of ad and non-ad links in the search result with the ad links displayed at the top. Is the goal of the change meant to increase the advertisement revenue?

[Interviewer] Yes, that’s correct.

[Candidate] Okay, I am visualizing the user experience as this will help me define metrics. Let’s say that a user queries the word, “cat.” The search result may display two ad links from Petsmart and Petco. Below the ads would just be web pages. The user may click the ad link or non-ad link.

Commentary: Notice that the candidate is concrete with his understanding of Google search. When addressing a product question that requires defining a set of metrics, analysis, and experiment, define the user experience first before proposing a solution. This step helps you brainstorm a solution and demonstrates to the interviewer that you have product knowledge.

[Interviewer] That sounds right. Do you want to define metric(s) for your experiment?

[Candidate] Yes, the primary goal of the experiment is to test whether changing the advertisement display would increase the ad revenue. So, the desired behavior is a user clicking ads displayed in a search session. So, the primary metric is the following:

Average ad revenue per user query session

I want to note that a query session accounts for multiple clicks within a query. Typically, once a user searches a term, one may click multiple web pages before finding the information they are seeking.

[Interviewer] How would you account for the fact that a user clicks multiple ad revenues in a query session? Do you think this could violate the assumption that the observations should be independent as seen in most statistical tests? *Address this question.

[Candidate] To account for this, I can sum the total ad revenue generated per user query session. So for instance, user A may click a single ad post with $0.10 CPC while user B may have clicked two ad posts with $0.20 and $0.30, respectively. The total ad revenue generated for user A is $0.10 and for user B is $0.50. I can average the total ad $ to get the metric that compares the control group (side banner) vs. the test group (top result).

Commentary: The candidate responds to the interviewer’s follow-up effectively as he addresses a potential violation of observation independence.

A common mistake among candidates is failing to address multiple observations collected per unit of analysis. When using statistical tests such as T-Tests and Chi-Squared Tests, observations are assumed to be independent. However, in many online experiments, a single user may generate multiple observations that bias the experimental outcomes.

For instance, on Google search, a user may click advertisement posts multiple times per query. Another example is, on Amazon, a user may check out multiple items per session.

There are a couple of ways to remove the violation. (1) normalize by taking the sum or average of the observations per user (2) only consider the first or last observation per user (3) use a different statistical test such as the generalized linear model accounts for multiple observations per user.

[Interviewer] Okay, sounds good. Now, Suppose that revenue from ad clicks increased, but the clicks on web pages (non-Ads) decreased. Should the test be launched?

[Candidate] I would say that, as long as the clicks in the non-Ad pages do not decrease, the change should be launched.

Commentary: The candidate provided an inadequate response.

First of all, the candidate failed to ask for details about the statistical results. The ad $ increased while the clicks in the non-Ad pages decreased, but were there statistical significance in both metrics? A decision cannot be made solely on the metric directions.

Secondly, the candidate failed to determine if the two metrics are equal in importance or one is primary while the other is secondary. Depending on the relative importance of each metric, decisions should vary. In this example, given that the aim of the experiment is to increase Ad revenue by testing a new layout, the primary metric is the Ad $ while the secondary, or guardrail metric, is the # of clicks in the non-Ad pages. Ideally, you would want to increase the Ad $ metric with statistical significance without affecting the clicks in the non-Ad pages.

To solve this problem effectively, it helps to map out how a decision should be made given the various outcomes of the metric directions and statistical significance.

Outcomes	Metric 1 – Ad $		Metric 2 – Non-Ad Clicks		Decision
Outcomes	Direction(Lift)	Alpha < 0.05	Direction(Lift)	Alpha < 0.05	Decision
1	+	N	–	N	Do not launch
2	+	N	–	Y	Do not launch
3	+	Y	–	N	Do not launch
4	+	Y	–	Y	Do not launch
5	–	N	–	N	Do not launch
6	–	N	–	Y	Do not launch
7	–	Y	–	N	Do not launch
8	–	Y	–	Y	Do not launch
9	–	N	–	N	Do not launch
10	–	N	+	Y	Do not launch
11	–	Y	+	N	Do not launch
12	–	Y	+	Y	Do not launch
13	+	N	+	N	Launch with caution
14	+	N	+	Y	Launch with caution
15	+	Y	+	N	Launch
16	+	Y	+	Y	Launch

*Launch with caution means that evaluate the practical significance of the primary metric direction. Even though the statistical significance isn’t achieved, if the lift observed is high, then proceed with the launch with caution.

Assessment

The candidate is assessed across two attributes: AB Testing and communication. Each attribute is rated based on the following scale:

Outstanding – Quick response with a sound solution.

Good – Minor mistakes, but converged toward a final solution that is sound.

Borderline – Required several hints before providing a sound solution.

Inadequate – Incorrect response.

Based on the rubric above, the candidate receives the following remarks:

AB Testing – Good

The candidate provided a solid response to the first problem. He started by understanding the business objective and the UI/UX of the change in a concrete manner. This enabled him to brainstorm a primary metric that could be useful when evaluating the experiment that changed the ad display position. In addition, the metric he proposed was precise with constraints. Instead of vaguely saying, “measure revenue from ads,” he was specific about the unit of analysis. For instance, he proposed taking the average of total revenue generated per user, which accounts for multiple ad clicks within a user’s query session.

However, the candidate could not receive an outstanding remark given the failure to provide justification in the follow-up question about the metrics shifting in the opposite directions.

Communication – Good

The candidate received a good remark given that he asking clarifying questions and provided a structured response overall. He comprehended the problems quickly and provided responses concretely in most of the questions. However, he could not receive an excellent remark given that in the last follow-up question, he seemed to struggle to provide a concrete justification to his assumption.

[Google] Search Metric Shifts

Problem

Unlock now by joining.