| Course | MKT 834 Data-Driven Marketing Management |
|---|---|
| Topic | Topic 4 |
| Paper type | Disconfirming result brief |
| Length | About 830 words, 1 pages |
| Format | Memo with APA 7 references |
| School | Grand Canyon University (GCU) |
| Program | DBA |
| Updated | October 2026 |
Free sample paper for MKT 834 Topic 4
Memorandum
To: Founder and Leadership Team
From: Director of Customer Analytics
Date: [Date]
Re: Results of the handwritten-note holdout test
The Result
For six months, a random fifth of orders went out without the handwritten thank-you note that has gone into every order for nine years. Among about 24,000 first orders in the test, customers who received a note made a second purchase within 90 days at about 31.8 percent; customers who did not received one at about 31.2 percent. The difference, about 0.6 points, sits comfortably inside the range random variation would generate. We found no detectable difference in repeat purchases. That finding rules out a large effect, not every effect: marketing effects are often small relative to the natural swings in customer purchases, which makes small ones expensive to detect (Lewis & Rao, 2015). All figures in this brief are from the test and are illustrative of its scale.
What the Result Does and Does Not Say
The result does not prove the notes have no effect. It says that if they have one, it is small: the test's 95 percent interval runs from about 0.9 points below to about 2.1 points above zero. An effect as large as the founder has described, notes as the reason customers come back, would have shown up clearly in a test of this size. An effect of one or two points could still exist and would be hard to see.
The test also measured only repeat purchases within 90 days. It did not measure longer-term loyalty, word of mouth or reviews, where the notes might matter. These limits are real and should be weighed, but they do not restore the belief that notes are the main reason customers return.
Why the Test Was Done This Way
The test randomized which orders received notes, so customers with and without notes were alike in every other respect, including what they bought, where they lived and how they found us. Kohavi et al. (2020) treat randomized tests as the strongest evidence that a change produces an effect, because they remove the differences between groups that make simple comparisons misleading. Before the test, we compared repeat rates of customers whose orders happened to go out without a note, for example during staff shortages, and those comparisons suggested a large effect. The randomized test shows those comparisons were misleading: orders without notes were concentrated in busy periods, when first-time buyers were less likely to return anyway.
Why the Belief Was Reasonable
The belief in the notes was not foolish. Customers mention them in reviews and emails, and the founder has read hundreds of messages thanking the company for them. But messages come from customers who noticed and valued the notes, not from those who did not, and evidence that fits a prior belief is the evidence people remember (Nickerson, 1998). The test was designed precisely because such evidence, however heartfelt, cannot show whether the notes change behavior.
What the Notes Cost
Each note costs about 0.40 dollars in cards and ink and about a minute of staff time, roughly 0.75 dollars per order in total. At about 120,000 orders a year, the notes cost about 90,000 dollars annually. If the 0.6-point difference were real, it would add about 240 repeat customers a year among roughly 40,000 first-time buyers, worth perhaps 5,000 dollars in margin, far short of the 90,000-dollar cost.
Table 1
Test Results and Cost
| Measure | With notes | Without notes |
|---|---|---|
| Orders in the test | About 19,200 | About 4,800 |
| Repeat purchase within 90 days | About 31.8% | About 31.2% |
| Difference and 95% interval | About 0.6 points (minus 0.9 to plus 2.1) | |
| Annual cost of notes | About 90,000 dollars | None |
Options
Three options deserve the leadership team's discussion. Stop the notes and redirect the 90,000 dollars to something that can be tested in the same way. Keep notes only for customers' first orders, where any welcome effect is most plausible, and test that version for six months, at roughly a third of the current cost. Or keep the notes as an expression of the brand's values, accepting that their purpose is not to drive repeat purchases. The third option is legitimate, but it should be chosen as a values decision, not justified by an effect the test did not find.
How We Propose to Discuss This
We will present the result first and take questions on the method before discussing options, so that doubts about the test are aired and answered rather than carried into the decision. We have also prepared the customer messages that mention notes, so the founder's evidence is on the table beside the test's. Both kinds of evidence deserve a hearing; they answer different questions.
Recommendation
We recommend the second option: keep notes for first orders only, test it for six months with the same randomized design, and decide on the full result. The meeting should not be about whether the founder was right, but about what the company does next with what it now knows.
References
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press.
Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4), 1941-1973. https://doi.org/10.1093/qje/qjv023
Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175-220. https://doi.org/10.1037/1089-2680.2.2.175
Reading the GCU MKT 834 Topic 4 instructions
In Topic 4, MKT-834 asks doctoral students to present a disconfirming result to a decision maker who believes the opposite. Strong briefs lead with the result, state precisely what it does and does not show with its uncertainty, and explain why the test design is more reliable than earlier evidence. Students should treat the belief as reasonable, explain how it formed, price what is at stake, set out options including a values-based choice, recommend a next step and plan how the meeting will run. The brief should keep the discussion on what to do next rather than on who was right.
Inside the MKT 834 Topic 4 example
The finished brief is addressed to a room that includes the program's creator. Its first page restates what leadership agreed before the test: notes would continue if ninety-day repeat purchases rose by at least an illustrative two points among eligible orders. The result follows with its interval rather than as a bare null, showing that effects as large as the threshold are unlikely while small ones cannot be ruled out. Staw's research on escalating commitment to a chosen course of action is cited to explain why the founder's reaction is predictable rather than personal, and Argyris's account of defensive routines frames the likely challenges to method. The brief anticipates three of them and answers each with the test's design. Its recommendation is deliberately smaller than the belief it contradicts: pause notes for first orders and test them on high-value customers.
MKT 834 Topic 4 rubric: what Excellent looks like
Faculty reward a brief built to survive the meeting. An Excellent brief leads with no detectable difference and states the interval rather than claiming no effect. Credit goes to explaining why earlier comparisons during staff shortages misled and why randomization fixes that, with Kohavi, Tang and Xu. The rubric values Nickerson on confirmation bias, used without blaming the founder. Strong briefs price notes at about 90,000 dollars against roughly 5,000 in margin. Top marks go to three options and a first-order test. Briefs that overstate the result as proof of no effect lose credit.
MKT 834 Topic 4 help: mistakes that cost marks
Graders punish first the brief that reports a null result as proof that the notes do nothing, an overstatement the founder's allies will spot at once and use to discredit the whole test. Nearly as costly is presenting the result without reference to any standard agreed in advance, which turns the meeting into a debate about what should count. Many drafts treat resistance as irrationality to be overcome, when the research on commitment makes it an expected response that a brief should plan for. Objections are frequently left for the meeting instead of answered on the page. Some papers recommend ending the program outright, a larger step than the evidence requires and one that invites rejection. Omitting the result that would reinstate the notes leaves the brief looking like advocacy rather than inquiry.
Get an MKT-834 Topic 4 example written to your instructions
Send the MKT-834 Topic 4 instructions and the rubric shown in your classroom, with the case or test result your section is working from. We write a custom example to those instructions, with the advance agreement restated, the finding reported as an interval, objections answered from the design and a recommendation sized to the evidence, in 24 to 48 hours. The first one is free.
More MKT 834 papers and DBA samples
- MKT 834 Topic 1: Evidence Standard Position Paper
- MKT 834 Topic 2: Causal Identification Essay
- MKT 834 Topic 3: Online Experiment Protocol
- MKT 834 Topic 5: Test Abandonment Dq Post
- MKT 834 Topic 6: Gaming-resistant Metric Design
- MKT 834 Topic 7: Program Counterfactual Evaluation
- MKT 834 Topic 8: Contrary Evidence Decision Defense
- MKT 832 Topic 4: Personalization Narrowing Appraisal
- MKT 830 Topic 4: Relationship Paradigm Appraisal
- MGT 830 Topic 4: Systems Archetype Analysis
MKT 834 Topic 4 questions, answered
How should a null test result be reported?
As an interval rather than a verdict. A test that finds no significant difference has usually ruled out large effects while leaving small ones possible, and the range of plausible effects says exactly how small. Reporting that range lets decision makers judge whether any effect still possible would matter commercially. The example shows the interval against the threshold leadership agreed, so the conclusion follows from a standard set in advance.
What is escalation of commitment?
A pattern Barry Staw documented in which decision makers who are responsible for a course of action tend to invest further in it after receiving negative feedback, partly to justify the original choice. It is strongest when the person is personally identified with the decision. The example uses it to explain the founder's likely response and to shape a recommendation that does not require a public reversal.
Why propose a smaller change than the evidence allows?
Because a recommendation has to be adopted to matter, and one that demands complete reversal from the program's creator is likely to be refused. Pausing the notes for first orders captures much of the saving, and testing them among high-value customers leaves open the possibility the founder is right about a segment. The follow-up test also gives the organization a way to learn more without reopening the first one.
Where can I find a free MKT 834 Topic 4 sample paper?
The brief appears here complete: the result, what it does and does not say, why the test was done this way, why the belief was reasonable, the cost, options, how the meeting will run and the recommendation. Bring your test result and we will draft the brief free.
Does the test prove the notes have no effect in MKT 834 Topic 4?
No. It shows any effect is small, within about minus 0.9 to plus 2.1 points, far from the founder's belief that notes are the reason customers return.