Rapid Message Testing | Haley Bryant

Square One to Year Four

Scaling a SaaS Product between election cycles

Overview

As the first product hire at Grow Progress, I led the design team for four years, transforming a nascent survey product into a suite of powerful tools built to enable clients to understand their audiences better, test messaging strategies effectively, and drive impactful persuasion campaigns.

Along the way, I helped set up a design-centered engineering process, built a research-backed roadmapping process, and hired a team of designers and product managers to expand on the work I started.

This case study details a very specific component of my work maturing the results side of GP's flagship product, "Rapid Message Testing," from a minimally viable engineering prototype used by a handful of clients during the 2020 presidential cycle to a robust survey platform used by over 100 clients and hundreds of users during the 2024 cycle.

** a note on my role: I eventually hired two additional designers to work on this product. Later iterations of the UI were done by my Associate Designer, Cathy, but the work I present here is my own unless otherwise stated.

Designing infrastructure for the left — throughout and between election cycles

Grow Progress provides software and services for social good strategists, aiming to increase the impact and effectiveness of their persuasion campaigns. In an electoral context, social good persuasion means growing support for candidates on the left, but Grow Progress clients work on all sorts of campaigns, between and beyond election timelines. Besides getting out the vote, persuasion campaigns increase support for climate change mitigation, defend a woman’s right to choose, build up union membership, and uncover new ways to engage young people online.

Changing minds by connecting with the values that shape the way people think

Persuasion science combines principles from psychology, communications, behavioral economics, and other social sciences to shape an ethos and a methodology for connecting with people based on their values (what they care about and what motivates their decisions) instead of the way they look. Values-based persuasion is one of the main tenants of Grow Progress' work.

For social good strategists, developing a persuasion campaign has four main components.

understand

Understanding the values and motivations of a target audience

translate

Translating the values of a campaign into messages that match the values of the target audience

test

Testing the messages to determine which effectively engage or move a target audience

deploy

Sharing the best messages with a target audience in a place that they will reach them

Testing content using Randomized Controlled Trials

When I started at Grow Progress in January of 2021, they had entered the market with a bare-bones version of what would become Rapid Message Testing, a tool for running automated randomized controlled trials (the same concept as medical trials) to test the effectiveness of messages with specific target audiences.

Starting with a testing tool would prove the theory that supplying strategists with quick, affordable information about what messages work and with whom increases the impact of persuasion campaigns.

Screen Shot 2021-03-24 at 3.54 1 (1).png

The first thing I did at Grow Progress was establish a basic design system to create a unified, scaffolded user experience

Part 1: Generative Research

I have always been a mixed-methods researcher, allowing the environment, organizational maturity, and my positionality to determine which types of user research to employ for any given project. At Grow Progress, I had the opportunity to develop the research discipline from scratch. In the early days, much of my work focused on contextual inquiry and developing a deep understanding of clients' day-to-day experiences and challenges.

To deeply understand the workflows and pain points of executive directors at non-profits, advocacy organizations, and other social good strategists, I conducted generative research using a variety of user-centered methods:

Semi-structured user interviews to uncover pain points, mental models, and unmet needs.
Affinity mapping to identify key themes, shape product opportunities, and uncover usability improvements.
Personas and journey mapping to illustrate user motivations, behaviors, and friction points, ensuring product decisions align with real-world needs.

Meet the Executive Director of an Advocacy Organization

Caught Between Stakeholders

Knowledge workers operating between boots-on-the-ground organizers and campaign funders.

High-Bar for Proving out Impact

Mission-driven and place a lot of personal stake on their ability to move the needle in campaigns.

Data-Driven Generalists

Need sophisticated tools to prove out theories, but they aren't data scientists.

Time & Resource Crunched

The campaign environment is highly urgent and they don't have time to spend learning new software.

Resulting Design Philosophies

Resist The Dashboard

Avoid the modular dashboards typical of many SaaS tools, which can overwhelm users with unnecessary metrics and competing priorities. Instead, focus on simple scaffolded results to help users make decisions. Ensure first-time users can easily understand their results and identify actionable steps to improve future tests.

Narrative First

Center incremental storytelling, avoiding graphs "above the fold" and offering users insights in a newspaper-style format. Use headlines to highlight key results, with further information and data manipulations available as users scroll. This narrative-first approach simplifies complex concepts and allows users to absorb results step by step.

Prioritize Control

Emphasize user control by empowering clients who operate between diverse stakeholders. To alleviate anxieties about misinterpreted results, prioritize features that allow users to create custom visualizations, contextualize data, and tailor how results are exported and shared. This approach ensures users can present findings with clarity and confidence.

Part 2: Design

In the early days, my main priority in the development of Rapid Message Testing was to present results without using standard graphs. This meant designing easy-to-understand visualizations to communicate the important aspects of the traditional Randomized Controlled Trial without relying on charts.

Anatomy of an RCT: Laying the Groundwork for Narrative-Driven Results

Users went through the following steps to test messages and content using Rapid Message Testing:

Define Success Questions

Ex. How likely are you to vote in the 2024 election?

Generate Messages

Develop a set of messages with different frames, messengers, and tones.

Send Messages

In a survey platform, show each message to a randomized "treatment" group.

Measure Performance

Compare the percentage of respondents that respond positively to the success question to a placebo group.

Analyze Results

Apply analyses, weighting, and other data cleaning methods to determine error and statistical significance.

As a chart, the results looked something like this:

Collaborating with data science, I went through a process to create a single visualization to communicate message testing results in narrative terms.

I wanted users to be able to answer the following questions quickly:

Which of my messages performed the best?
What does it mean for a message to perform well?
Did any of my messages cause backlash?
What should I do next?

Part 3: Evaluative Research

The resulting prototype looked something like this. I added a few key elements to present results in narrative form:

Persuasion Categories: Defined five tiers using statistical significance, marking top messages as "persuasive" (green) using confidence intervals.
Headlines: Delivered clear, actionable takeaways.
Subgroup Results: Pinpointed key audiences for each message.
Next Steps: Suggested concrete actions to take.

"Describe the results of this message test to me in your own words..."

Using a few different prototypes with varying results, I conducted user tests with a product manager to test for understanding. I used a think-aloud protocol to measure whether or not users could explain the message testing results accurately, whether they knew how to navigate the interface, and if they had particular questions or difficulties.

Part 4: Results

Evaluating the Persuasion Project

Over time and many iterations, we adapted the persuasion rainbow from my original mocks and today it looks like this:

Measuring Success:

My ongoing philosophy for the design of technical and professional products is to continually examine the “sufficient capability” of the user base. Sufficient capability ensures that users can use tools to complete the tasks at hand without being overwhelmed by too many features or details. During my four years at Grow Progress, our user base grew by several degrees, and so, too, did the average user's understanding of randomized controlled trials and the value of message testing. As the average user's knowledge became more advanced, it allowed me to build out different lanes in the product, adding more advanced functionality to deepen longer-term users’ ability to grow their knowledge while keeping the primary pages accessible and narrative-driven. Because Rapid Message Testing was tied up in a web of democrat infrastructure, this method had the effect of “lifting all boats” as network effects gradually improved the knowledge and sufficient capability of the movement.

200%

Average by which clients increased their message effectiveness testing with Grow Progress

500%

User increase between the 2020 and 2024 presidential cycles

1,000+

Rapid Message Tests run in 2024 alone

3,000+

Pieces of content available in Persuasion Library for strategists on the left

"Whoever did the user experience here is 👌"

Advocacy User in 2022

"The design of the platform is so straightforward and elegant that anyone can use it."