BY ROBERT REYNOLDS
Nudging—the application of behavioral insights to public policy interventions—is among today’s fastest growing public policy fields. In the last few years, organizations from the White House to Google to the World Bank have launched behavioral science teams tasked with running randomized controlled trials (RCTs) that identify powerful nudges.
During this time, Kennedy School students have flocked to new behavioral courses and to Behavioral Insights Study Group events. Yet, very few HKS students actually test nudges using RCTs. Why is this? My initial struggles to launch RCTs taught me that the primary challenge facing first-time experimentalists is identifying a problem well-suited for experimentation.
During my MPP program, I designed and led multiple experiments at Harvard and MIT. In these situations, to decide which experiment idea my team and I would focus on, I asked myself a simple question: Which intervention could have the greatest impact? It turns out I was asking the wrong question.
As a rookie nudger, I gave no consideration to whether or not the characteristics of the experiment—the ease of randomization, the simplicity of data collection, etc.—made it well-suited for a first-time experimentalist.
Now, having completed multiple RCTs, I have learned that there are six criteria first-time experimentalists must consider. Using the “TRIAL1 Checklist” I developed, first-time experimentalists can anticipate potential pitfalls an experiment might have and, in turn, more quickly consider better-suited opportunities for their first experiment.
1.) Transport: There is a reliable, preexisting vehicle that can transport the independent variable
Good example: Nudging a non-profit’s email subscribers to donate using the pre-existing weekly email. Simply create control (standard weekly email) and treatment (standard weekly email with nudge) conditions and send email subscribers the email condition they have been randomly assigned to.
Bad example: Nudging citizens to attend a town hall meeting. Testing this nudge would be challenging if it means creating a new transport vehicle (e.g., mail, email, SMS) to reach citizens.
2.) Randomly: The independent variable can be randomly assigned and easily recorded
Good example: Nudging students to pay fees using SMS messages. To randomize and record, use Excel’s random number generator and record which subjects end up in which conditions.
Bad example: Nudging students to volunteer using an in-person hand out form. This experiment introduces the possibility of human error because it relies on someone recording which subjects receive which form.
3.) Individual-level: The unit of observation for the dependent variable is the individual-level
Good example: Nudging pedestrians to not litter fliers when the nudge is randomly assigned to each pedestrian who is handed a flier. If 100 pedestrians get a flier each day, a 10-day experiment would produce 1,000 data points.
Bad example: Nudging pedestrians to not litter fliers when the nudge is randomly assigned each day to pedestrians who are handed a flier. If 100 pedestrians get a flier each day, a 10-day experiment would produce 10 data points.
4.) Acquirable: The dependent variable data is acquirable
Good example: Nudging to increase parent attendance at parent-teacher conferences. Likely, this data is already being collected.
Bad example: Nudging parents to read to their children at night. While the nudge might work, it would be difficult to collect accurate data, which occurs outside of school and can be misreported if researchers rely on self-reports.
5.) Low variance: The dependent variable has low variance
Good example: Nudging to reduce beverage calorie consumption at a cafeteria. This data set’s low variance (caused by it not having extreme outlying data points) enables a simpler experiment because less data is needed to detect the nudge’s impact.
Bad example: Nudging to reduce tax fraud. This data set’s high variance (caused by having extreme outlying data points) creates a more complicated experiment because more data is needed to detect the nudge’s impact.
6.) One-time decision: At the individual level, the dependent variable is a one-time or infrequent decision
Good example: Nudging to increase organ donation registration, voting, or getting the flu shot. Because the decisions related to these behaviors occur rarely, these experiments largely avoid novelty effect concerns.
Bad example: Nudging to increase healthy eating or exercising. Determining a nudge’s impact on these outcomes requires added complexity to differentiate the nudge’s novelty effect from its long-term effect.
Conclusion
Excitement about a potentially high impact nudge has led many people, including myself, to choose overly complex problems for their first experiment. This mistake can be avoided using the TRIAL1 Checklist. Through evaluating behavioral problems on these six criteria, rookie nudgers will be better positioned to identify a problem well-suited for their first experiment.
Robert Reynolds, HKS MPP ’15, founded Harvard’s Behavioral Insights Study Group.
Photo by Christy Zinn via Unsplash