A Company's Perspective on Experimentation

By Russ Frey
Director, Intelligence, Maritz Behavioral Science & Innovation

Editor's Note: We advocate for behavioral experimentation a lot, but often from the perspective of the scientist or an outside observer. What about the perspective – and goals, needs and concerns – of the organization? How does the process play out for it and what drives its stakeholders? Good questions. Here are some answers from one large company seeking greater agility.

Full disclosure: Maritz supports this website. It also champions applied behavioral science and isn’t just recommending a culture of informed experimentation to others, it’s embracing it itself… as Russ’ story shows.

Not long ago, Experient, a part of Maritz Global Events known for delivering successful meetings and events for hundreds of clients, wanted to try something new.

While sign-up rates for client events were routinely high, many who registered failed to book hotel accommodations – even with plenty of hotel options conveniently displayed on the same website. Why did most people pass on the opportunity to book a recognized brand featuring a convenient location and special pricing only to return later and find those deals had sold out?   

Puzzled, several Experient project managers wondered how they could nudge attendees to book a room during their registration visit, or soon after – while preferred hotels still had space available.

Thus began a journey sparked by questioning long-standing practices and fueled by behavioral science. This journey would ultimately shift Experient’s culture toward increased agility and innovation.

The first step: build excitement about behavioral science

The Start of a Journey

“We’ve always been driven by what our clients were asking for,” said Aaron Dorsey, senior director of strategic client technology at Experient. “Sometimes without considering the end user and what their experience would be like as a result. We sensed that it was time to shift our focus.”

That journey toward agility started when Maritz’ own Behavioral Science & Innovation team introduced Experient to Irrational Labs. Co-founded by behavioral economist Dan Ariely and his colleague Kristen Berman, Irrational Labs is a non-profit consultancy that applies behavioral science research to help people increase their health, financial wellbeing and life satisfaction.

With the support of Experient leadership, and guidance from Irrational Lab’s Managing Director Evelyn Gosnell, several event teams participated in two experiments designed to increase hotel bookings through event registration websites.

The first step: build excitement about the power of behavioral science through an immersive workshop facilitated by Evelyn and Kristen. Invitees included all functions and team members who would participate, regardless of when they would be involved. No one would be left out, even if their work didn’t start until much later in the event launch process.

"We also wanted everyone on each team to feel empowered to explore new tactics, even if they didn’t always turn out,” said Director of Project Management Angela Marini. “The workshop reminded us that the biggest obstacle to progress might be how you’re currently doing something – especially if it’s deeply ingrained in a business process.” 

Focusing on “Three Bs”

Central to the Irrational Labs approach was shifting team members’ perspectives from registration website functions and features to building a user-centered experience that helps people make informed choices quickly. To help accomplish this, Evelyn asked the teams to focus on “Three Bs”: 1) Identifying the key user behavior, 2) Reducing barriers to that behavior, and 3) Amplifying its benefits. This framework gave teams a fresh perspective on what users experience when navigating their websites. According to Angela, “We found ourselves saying: Wait a minute, let’s put ourselves in the attendees’ shoes, and consider if we’re unintentionally creating barriers for them based on what a client has asked us to do.” 

Shifting perspectives from website features to building a user-centered experience that helps people make informed choices quickly

With the Three Bs in-mind, the project teams quickly began looking for what Evelyn called “Just Dos” or simple, intuitive improvements that didn’t require extensive testing. “Just Dos are often about simplifying a process – removing many of the small barriers that are in the way of the key behavior,” said Evelyn. “These are often fairly easy to implement without complex development, which is why we start here,” she said. By carefully reviewing how their websites displayed information and prompted action, the teams made small refinements to enhance the user experience.

Now it was time for a deeper dive. While small in scale, the first round of experiments would test user preferences for how hotel choices were displayed and described. Besides a control group of users that that saw “typical” website content encouraging them to select and book a hotel, two other site versions were launched for distinct groups of randomly selected attendees. Treatment 1 displayed “Recommended for you” with hotels options presented as a single list, and Treatment 2 also displayed “Recommended for you” but with reasons for the recommendation, such as “best luxury” or “best value.” Within the small sample size, Treatment 2, which combined greater personalization and categorized choices performed best.

Before launching the next, more ambitious round of experiments, Evelyn reprised her behavioral science workshop for the three new event teams that would be participating. “We also agreed to set aggressive timelines for ourselves,” Evelyn said, “to make adjustments early in the process based on what we were learning.”

Expanding Team Capabilities

From the relative simplicity of the first experiments, three new teams upped the ante in the next round by testing both the effectiveness and timing of multiple message e-mail campaigns reminding people to return to the site and book their room. This approach required that each of the teams decide what variables would be measured for their event and how data would be captured.

To help the teams launch more personalized and time-sensitive messages, Experient moved up the implementation of a new communications management platform by six months. The new technology replaced a batched, one-size-fits-all approach with the ability to deliver content according to the time since each recipient had registered.

“Even though it was our busiest season, knowing that experiments would focus on message content and timing, we wanted to do everything we could to run them on the new technology,” said Marketing Specialist Cameron Korb.  

Following the launch of their respective event registration sites, each of the three teams started messaging a control group, along with two additional treatment groups that varied the timing and content of reminder messages. Messages would use loss or gain frames based on recipient status and the availability of rooms in the preferred hotel block. “The first reminder was triggered within a day of registration and framed as if the attendee had mistakenly overlooked an important part of the registration process,” noted Cameron. “Other messages focused on the dwindling availability of rooms and not missing out on the benefits offered by preferred hotels.”

While analysis continues, indications are that these experiments validated some current practices and revealed important new insights.

Beginning in late 2017 and continuing into 2018, all nine communications campaign treatments were executed, and data was collected across the three events. And while analysis continues, indications are that these experiments validated some current practices and revealed important new insights. “The last round of experiments surprised us. We weren’t that far off with what we were already doing,” said Angela. “But a big win was learning to be more agile – especially with digital messaging.”

Want to Use Behavioral Science to Boost Organizational Agility?

Here are some of Evelyn’s favorite practices that have been adopted by Experient:

  • Provide teams with a basic education about behavioral economics.
  • To increase buy in, involve all team members who will contribute from the beginning.
  • Identify key behaviors, barriers and the benefits of positive change.
  • Spell out what variables will be measured and what performance data will be captured in advance.
  • In any experiment, include a control group.
  • Randomize the assignment of people to the control and all treatment groups.
  • Look first for small wins or Just Dos that help identify where to experiment next.
  • Timelines matter – foster a sense of urgency and adjust to unexpected outcomes quickly.
  • Know that things don’t always turn out as anticipated – and that’s precisely why organizations should experiment before implementing at scale.

Reflecting on the progress Experient has made in the last six months, Aaron notes: “This is how we want to work ... to prove with measures and data, with our customers seeing an immediate benefit. Experimentation has helped us be more consultative – especially with clients interested in trying something new.”

By Russ Frey
Director, Intelligence, Maritz Behavioral Science & Innovation

Subscribe

Get the latest behavioral science insights.