7 winning A/B tests that feel illegal to know 🏴‍☠️

The secret test results of Apple, Peloton, and Spotify [revealed]

Today I’m sharing something special.

Since you all loved my previous A/B test breakdown that blew everyone’s mind a few weeks ago, I had to bring you more split-testing ideas!

My friend Casey Hill (CMO at DoWhatWorks) gave me a sneak peek at his company’s database of A/B test results from companies like Apple and Spotify. In this week’s newsletter, we get to see a few of their most powerful test recommendations.

Let’s go 💡

Subscribe now

Guest post by Casey Hill

A/B testing is mostly a waste of time.

Between 2020-2024, a shocking 89% of all tests failed to beat the control. That’s not a typo.

Why does it happen? 👀

Most teams are obsessed with running tons of tests instead of the right tests. They trust opinions over evidence. They guess instead of knowing.

Let’s fix that right now.

Here are 7 surprising A/B tests that reveal what actually works in 2025 ↓

1. Clarity beats simplicity ☀️

We’ve all heard “keep it simple” – but the data shows a different story:

Clarity wins.

Take this test from Peloton. The winning version doesn’t have a simpler design… It has a clearer one.

The winner has two specific CTAs instead of one generic one:

1. “See Details” (get specs, read reviews, etc)

2. “Quick Add” (add this exact item to cart)

Why it works: “Shop now” is ambiguous when you’re already on a product page. The winning version creates clear expectations about what happens when you click.

Subscribe now

2. Carousels actually work 🖼️

Contrary to popular belief, carousels in hero sections consistently outperform static images.

The New York Times saw this in action:

Turns out, carousels reduce cognitive load by showing one element at a time. They also create a sense of progression, guiding users on a journey that deepens engagement.

Both manual click-through and auto-toggle versions perform well (with auto-toggle potentially reducing friction on mobile).

3. “Benefit-first” CTAs actually lose 🕹️

This shocked me.

I’ve always told clients to focus on benefits in button text. I was wrong.

Look at Apple’s test:

“Start Listening” (the benefit) lost to “Try it Free” (the clear action).

Here’s why: You can’t directly start listening without a trial first. There’s a gap between expectation and reality.

The same pattern shows up across other tests. Buttons with benefit-focused text like “Elevate your marketing” or “Save time now” consistently lose to crystal-clear options:

• ✅ “Start Free Trial”

• ✅ “Book a Demo”

• ✅ “See Pricing”

The bottom line: CTAs should set clear expectations about what happens next, not sell benefits.

Subscribe now

4. Messaging is going back to basics ⏪

Remember when Mailchimp’s homepage said simply “Send better email”?

Around 2020, they shifted to complex messaging: “Do it all with Mailchimp - Bring your audience data, marketing channels, and insights together so you can reach your goals faster—all from a single platform”.

Oof.

This pattern is everywhere in SaaS: A company starts with a simple product that solves one problem well. Then they scale, add more features, expand globally, and their messaging gets muddier.

But in 2025, we’re seeing a major reversal:

Top companies are returning to simple language focused on their core value prop.

For Mailchimp, “Get down to business and grow sales” lost to the much clearer “Turn emails into revenue”.

5. Competitor comparisons kill conversion 🥊

Here’s a shocking test from Square:

They wanted to see if competitor comparisons on pricing pages would help conversion.

It tanked. 👎

The problem is that nobody trusts you when you talk about competitors. Those grids where you check every box and competitors miss key features don’t drive conversions.

Instead, try these alternatives:

1. Be strategic about placement (pricing pages are the wrong place)

2. Focus on your unique category position (like Klaviyo does with “the only CRM designed for B2C”)

3. Highlight ‘defensible’ features your competitors don’t have

4. If you must mention competitors - show the value of all tools in the category and work hard to make it a more objective take.

6. Customer logos actually hurt conversion 💰

This one is wild:

Nearly 70% of the top 100 SaaS brands include customer logos on their sites.

Yet when Asana tested head-to-head, the version without the logos won:

What works instead:

1. Specific references to press coverage (“As seen in TechCrunch”) with links to articles

2. Detailed testimonials with context (“Since implementing Webflow, our monthly spend on web dev has decreased 70%”)

3. Case studies organized by industry with detailed problem/solution breakdowns

7. Strikethrough pricing breaks trust 🙅🏻‍♂️

Strike prices ($99.99 $29.99) feel manipulative to modern consumers.

Spotify tested this extensively and found a clear winner:

Why? When you show you can slash prices by 70%, customers question the real value. If you can make it $30 today, why not $20 tomorrow? The moral of the story is: don’t show the original price.

If you enjoyed these testing insights (I know I did), follow Casey and his team!

See you next week ✌️

Tom

—If you enjoyed this guest article, please tap the Like button below ♥️ Thank you!