| Course | MKT 834 Data-Driven Marketing Management |
|---|---|
| Topic | Topic 3 |
| Paper type | Online experiment protocol |
| Length | About 980 words, 2 pages |
| Format | APA 7 student paper |
| School | Grand Canyon University (GCU) |
| Program | DBA |
| Updated | October 2026 |
Free sample paper for MKT 834 Topic 3
A Test Built to Deliver Bad News: An Experiment Protocol for Showing All Fees Upfront on a Ticket Resale Marketplace
[Author Name]
College of Doctoral Studies, Grand Canyon University
MKT-834: Data-Driven Marketing Management
Topic 3 Experiment Protocol
[Instructor Name]
[Date]
The organization, setting and figures below are a composite written as a model document. No real employer, client, colleague or patient is described.
A Test Built to Deliver Bad News: An Experiment Protocol for Showing All Fees Upfront on a Ticket Resale Marketplace
The Change and the Expectation
A composite ticket resale marketplace currently shows a ticket's base price in search results and adds service and delivery fees, typically about 25 percent of the base price, at checkout. The product lead proposes showing every fee in the first price a buyer sees and expects buyers' trust, and eventually sales, to rise. This protocol is written so that the test can deliver bad news and still be believed. That matters here, because there is good reason to expect bad news: Blake et al. (2021) studied a large ticket resale platform that switched between upfront and back-end fee display and found that buyers shown fees only at checkout spent more and were more likely to complete a purchase. The protocol must also handle three threats specific to this setting: peeking, novelty and buyers competing for the same seats. Figures are illustrative.
Hypotheses and Metrics, Fixed in Advance
The primary hypothesis is that showing all fees upfront changes completed purchases per visitor. The test is two-sided, because the change could lower or raise purchases. The primary metric is completed purchases per visitor; the secondary metrics are revenue per visitor, average ticket price chosen, the share of buyers who return within 90 days and a short post-purchase survey item on whether the price was what the buyer expected. Kohavi et al. (2020) recommend fixing the primary metric and decision rule before a test starts, so that the result cannot be reinterpreted by choosing the metric that looks best afterward.
The Unit of Randomization and Competing Buyers
Randomizing individual buyers is the usual approach, but on a resale marketplace buyers compete for the same seats. If buyers in the upfront-fee group see higher displayed prices and buy less, the seats they leave are bought by buyers in the other group, inflating that group's purchases. The difference between groups would then overstate the effect of the change. To limit this interference, the protocol randomizes by event: all buyers viewing a given event see the same fee display. With about 6,000 events listed in a typical month, randomizing by event still provides ample units, and it keeps buyers competing for the same seats in the same condition.
Sample Size and the Peeking Problem
The test will run for six weeks, a length chosen in advance from the number of events and visitors needed to detect a 3 percent change in purchases per visitor with conventional confidence. The team will not stop the test early because a result looks significant. Johari et al. (2017) showed that repeatedly checking a standard test and stopping when it crosses a significance threshold inflates the rate of false positives far above the nominal level. If the team needs to monitor the test for harm, it will use a sequential method designed for continuous monitoring, with its stopping rules fixed in advance.
Table 1
Protocol Summary
| Element | Choice | Reason |
|---|---|---|
| Randomization unit | Event | Buyers compete for the same seats |
| Duration | Six weeks, fixed in advance | Sample size for a 3% change; covers novelty |
| Primary metric | Completed purchases per visitor | The outcome the change most affects |
| Monitoring | Sequential method with preset rules | Peeking inflates false positives |
| Decision rule | Written before launch | Prevents bending to the result |
Novelty and Learning
Buyers used to back-end fees may react to upfront prices at first with surprise that fades, or regular buyers may take time to notice that the new prices include everything. Either would make the first weeks unrepresentative. The protocol therefore runs six weeks rather than two and reports results by week, so a shrinking or growing effect is visible. It also reports results separately for first-time and returning buyers, since only returning buyers have a habit to unlearn.
Guardrail Metrics
Beyond the primary metric, the protocol sets guardrails, measures that must not deteriorate beyond a stated limit regardless of the primary result. Seller listings must not fall by more than 3 percent in the upfront-fee events, since sellers may react if their tickets appear more expensive beside other events. Customer service contacts about fees must not rise. And page load times must stay unchanged, so that a slower page is not mistaken for an effect of fee display. If a guardrail is breached, the test is reviewed before it continues.
Checks Before Launch
Before any buyer sees the change, the protocol runs two checks. An A/A test, in which both groups see the current display for a week, confirms that randomization by event produces groups with similar purchase rates, and that the analysis pipeline does not report a difference where none exists. A sample ratio check confirms that events are split in the planned proportions; a mismatch would signal a bug in assignment, which Kohavi et al. (2020) describe as one of the most common causes of untrustworthy results.
The Decision Rule
The rule is written now. If purchases per visitor fall by more than 2 percent with upfront fees and the survey and return-rate measures do not improve, the marketplace keeps back-end fees for now and reports the result to the product lead and leadership as tested. If purchases fall but returning-buyer rates and expectation scores improve, leadership weighs the trade-off with the full results in front of it. If purchases hold or rise, upfront fees are adopted. Regulatory requirements for fee disclosure in particular markets override the test in those markets.
Why the Protocol Is Built This Way
A test that can only deliver good news is not a test. The product lead expects a gain, and the research evidence suggests a loss is plausible. By fixing the metric, the duration, the randomization unit and the decision rule in advance, and by guarding against peeking and novelty, the protocol ensures that whichever result arrives will be believed, including the one the product lead does not want.
References
Blake, T., Moshary, S., Sweeney, K., & Tadelis, S. (2021). Price salience and product choice. Marketing Science, 40(4), 619-636. https://doi.org/10.1287/mksc.2020.1261
Johari, R., Koomen, P., Pekelis, L., & Walsh, D. (2017). Peeking at A/B tests: Why it matters, and what to do about it. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 1517-1525). Association for Computing Machinery. https://doi.org/10.1145/3097983.3097992
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.
The MKT-834 Topic 3 assignment instructions
Topic 3 of MKT-834 asks doctoral students to write a protocol for an online controlled experiment. Strong protocols fix hypotheses, a primary metric, secondary metrics and guardrails in advance, choose a randomization unit that limits interference, and set sample size and duration with a plan for monitoring that does not inflate false positives. Students should address novelty effects, run pre-launch checks such as an A/A test and sample ratio checks, and write the decision rule before launch so an unwelcome result will be believed. Guardrail metrics should be stated alongside the primary metric.
How the MKT-834 Topic 3 example is put together
The finished protocol is a document the team signs before any traffic is split. It names one primary metric, completed purchases per visiting buyer, and two guardrails, refund requests and seller listing volume, with the decision each result would trigger. Randomization is by event rather than by buyer, because treated and control buyers bidding for the same limited seats would otherwise contaminate each other, a violation of the no-interference assumption in Rubin's framework. A power calculation sets the number of events and the fixed end date, with illustrative figures. Peeking is handled either by committing to that end date or by using sequential methods of the kind Johari and colleagues developed for continuous monitoring. A novelty check compares effects by weeks since first exposure. The sample-ratio check that Kohavi, Tang and Xu describe runs daily.
Where the marks sit in the MKT 834 Topic 3 rubric
Faculty reward a protocol designed to survive bad news. An Excellent protocol cites Blake and colleagues' finding that back-end fees raised spending on a ticket platform. Credit goes to randomizing by event because buyers compete for seats. The rubric values a fixed six-week duration and Johari and colleagues on peeking, with sequential monitoring. Strong protocols add guardrails, an A/A test and sample ratio checks from Kohavi, Tang and Xu. Top marks go to weekly reporting for novelty and a decision rule fixed before launch. Protocols that randomize individual buyers on a competitive marketplace overstate effects.
MKT 834 Topic 3 help: mistakes that cost marks
Protocols give up the most when they randomize individual buyers on a marketplace where treated and control buyers compete for identical seats, since the comparison then measures a reallocation between arms rather than an effect. Close behind is the plan with no fixed reading date and no sequential method, which invites stopping on the first favorable day. Many drafts list several success metrics without naming a primary one, leaving room to report whichever moved. Novelty is often ignored, although a changed display can draw attention simply by being new. Sample-ratio checks are commonly missing, and a broken split can produce a convincing but meaningless difference. Doctoral readers also expect the stopping result written down before any traffic is split, because a test that cannot disappoint anyone has not been designed to inform a decision.
Get an MKT-834 Topic 3 example written to your instructions
Send the MKT-834 Topic 3 instructions and your classroom's rubric, with the product change or scenario your section is testing. A custom example is written to them, with the stopping result stated first, the randomization unit chosen for interference, a power-based horizon, monitoring rules, novelty and ratio checks and exploratory cuts labeled, back in 24 to 48 hours. The first one is free.
More MKT 834 papers and DBA samples
- MKT 834 Topic 1: Evidence Standard Position Paper
- MKT 834 Topic 2: Causal Identification Essay
- MKT 834 Topic 4: Disconfirming Result Brief
- MKT 834 Topic 5: Test Abandonment Dq Post
- MKT 834 Topic 6: Gaming-resistant Metric Design
- MKT 834 Topic 7: Program Counterfactual Evaluation
- MKT 834 Topic 8: Contrary Evidence Decision Defense
- MKT 832 Topic 3: Platform Social Influence Paper
- MKT 830 Topic 3: Managerial Turn Critique
- MGT 830 Topic 3: Backfire Case Critique
MKT 834 Topic 3 questions, answered
What is peeking in an A/B test?
Checking a running test repeatedly and stopping when the result looks significant. Standard significance tests assume a single analysis at a planned sample size, so each extra look raises the chance of declaring an effect that is not there. Teams avoid the problem by fixing the reading date in advance or by using sequential methods designed for continuous monitoring, such as the always-valid approach Johari and colleagues described.
What is interference in a marketing experiment?
A situation in which one unit's treatment affects another unit's outcome, violating the assumption that each outcome depends only on its own assignment. Marketplaces are prone to it: if treated buyers purchase more of a limited supply, control buyers find less available, and the measured difference exaggerates the effect. Randomizing larger units, such as whole events or markets, is the usual remedy, which is why the example assigns whole events.
What is a novelty effect?
A temporary change in behavior caused by the fact that something is new rather than by what it does. Users may click a redesigned button more at first simply because they notice it, and the effect fades as it becomes familiar. The opposite, a primacy effect, occurs when users initially resist a change. The example checks for both by comparing results across groups exposed for different lengths of time.
Where can I find a free MKT 834 Topic 3 sample paper?
The experiment protocol is shown here in full: the change and expectation, hypotheses and metrics, the randomization unit, sample size and peeking, novelty, guardrails, pre-launch checks, the decision rule and why it is built this way. Bring your test idea and we will draft a free protocol.
Why randomize by event in MKT 834 Topic 3?
Buyers compete for the same seats, so seats left by one group are bought by the other, inflating its purchases. Showing one fee display per event keeps competing buyers in the same condition.