A/B Split Testing

My Role

I provided sketches, wireframes and the visual design, collaborating with 1 product manager and the CEO to bring this feature to life. We worked with 2 developers on this update.

Opportunity

It can be difficult for marketers to figure out "What is a good push notification campaign?" and "Why do some push notification campaigns perform better than others?"

Sometimes, something as simple as the language used or the use of emojis in a push notification can make a huge difference in the success or failure of a campaign.

Keeping in mind that app users can be bombarded with notifications every day, we want to help marketers to create effective, eye-catching push notifications that get clicked, rather than ignored.

The ability to A/B split test campaigns has been around for years, often seen in email marketing tools, so it made sense for us to introduce this ability to our push notification campaign builder.

A/B split testing will help marketers experiment with different push notification variances, figuring out what factors are driving uplift in push notification engagement rates. This feature will empower marketers to learn from their push notification content, understanding how different customer segments respond to different content, continuously improving the customer experience.

Previous Push Notification Builder

This design displays the simpler version of the push notification builder, without the rich media and interaction abilities that were later added with the iOS 10 update.

iOS 10 Updates

Other A/B Split Testing Solutions

Since there are already plenty of tools out there that offer A/B split testing functionality, I researched some of these, including how our competitors designed this experience. I reviewed the usability of their designs, taking note of what I thought worked well and what could be improved.

Display the Variances in a Bar or as Tabs?

Considering the current layout of the push notification screen, I wanted to fluidly incorporate this new feature into the current UI, rather than reshaping the UI just to fit this update. This should provide an easier learning curve for marketers and not get in the way.

When creating the content for a push notification, it's important to help the marketer to focus on the content, 1 push notification variance at a time. This is why we opted for 2 potential solutions:

• The ability to create new variances and toggle between them could be displayed within a slim horizontal bar

• The sidebar could include tabs that would stick out from the sidebar, using different states to denote which variances is being viewed

Solution A: Sidebar Tabs

• Displaying up to 9 or 10 variances as these vertical tabs will require a lot of additional vertical screen space, potentially adding a lot of noise within the UI.

• The sidebar is already made up of vertical content that expands and contracts. Having these tabs positioned to the left of this expanded content might imply a direct relationship, when this is not the case.

Solution B: Slim Bar

• A simpler, more elegant solution, taking up less screen space as a slim bar.

• Provides an independent section in the UI for this feature's functionality, including the +Add New Variance and Variance Settings components.

• Will this additional bar make the UI too repetitive with the other bars on screen?

How to Determine the Winning Variant?

After multiple team discussions, design reviews and iterations, we decided that the winning variance (renamed to "variant" in later design stages), would be determined by the highest conversion goal rate. Every push notification campaign has a conversion goal event to be measured, to establish the success of each campaign. Automatically assigning this as the success metric will easily highlight the winner to the marketer, rather than letting them figure it out for themselves when reviewing the campaign report.

I've attached some of the iterations below, demonstrating the evolution of this feature.

Editing the Conversion Goal Event

When creating a push notification campaign, the first step the user is greeted with is the naming/conversion goal modal. The purpose is to focus the marketer on the intention of the campaign - what they're trying to achieve (such as to influence app users to click a specific button within a certain time period). This feature is woven into this update. From the A/B Settings modal, the marketer can still edit the conversion goal event.

Final Designs for the A/B Split Testing Feature

1. The New Variants Bar

2. Adding a New Variant (dropdown)

3. Multiple Variants Added

4. Maximum Variants Added with Control Group

5. Editing the A/B Settings

6. Decreasing the Dispatch Percentage

7. Hover State of the Control Group Tab

8. Validations - Showing Errors when Trying to Proceed to Next Step

Comparing the Performances of the Variants

A/B Push Campaign - Report Screen (Wireframes)

Normal Push Campaign - Report Screen (Wireframes)

Final Design - A/B Campaign Report

What I Learned From This Project

It's so beneficial to have a clear view of the upcoming roadmap to keep in the back of your mind when designing. This allows designers to produce design solutions that are somewhat future-proof. Otherwise, the team could invest a large amount of time and energy in solving one problem but that design solution doesn't fit nicely with the following feature requirements. Often when we're designing features, we consider how future features could fit in our proposed designs. In this case, we designed the conversion goal update first, which weaved smoothly into this feature update.

Interested in working together?

Or you can find me here: