Push notification A/B tests and learnings
Push is a permission that can be revoked silently, so the tests that pay are about relevance and restraint rather than wording. Trigger quality, frequency and whether the notification leads somewhere useful decide retention and opt-out far more than the headline. Novelty effects are strong here, so re-run anything you plan to rely on.
- Every testable surface in a push program, from permission prompt to deep link
- Test ideas for each lifecycle stage, linked to the matching stage page
- Opt-in, throttling and platform constraints that shape what you can run
- How fast push learnings fade, and graded evidence from real programs
What is actually testable in push
Push has a short visible surface and a long invisible one. The invisible part is where the results are.
The permission prompt
When you ask, what you promise, and whether you pre-ask in context first. This decides the size and quality of the audience for every later test.
Trigger quality
Whether the notification is tied to something that happened to this person, or to a schedule. It is the strongest predictor of whether the permission survives.
Frequency and throttling
How many notifications a user can receive in a period, and which ones lose when several qualify at once.
Timing
Immediate against delayed, and whether send time follows the user's own activity pattern rather than a global schedule.
Title, body and media
Length before truncation, specificity, and whether an image or action button earns its place.
Destination
Where the tap lands. A notification that opens the home screen instead of the thing it described is the most common and most damaging defect in the channel.
The anatomy of a testable push notification
When it appears and what it promises.
Personal event or scheduled campaign.
Cap per period and priority when several qualify.
Specific claim, truncated on most devices.
One sentence, visible before expansion.
Image or buttons, only when they add.
Immediate, or matched to the user's active hours.
Deep link to the exact screen described.
Every slot above is a variable you can hold constant or change. A clean push test moves one of them at a time.
Push test ideas by lifecycle stage
Each stage below has its own page with the full set of levers, guard metrics and graded evidence.
Onboarding
See the onboarding page- Test asking for notification permission after the first useful moment rather than at launch.
- Test a contextual pre-prompt that explains what notifications will be used for.
Activation
See the activation page- Test a notification tied to a real in-product event against a scheduled nudge.
- Test deep linking straight into the action rather than the home screen.
- Test a lower frequency cap, read on active users rather than session count.
- Test personal signals against editorial campaigns for the same slot.
- Test a single push to dormant users against a series, with uninstall rate in the read.
- Test stopping push entirely for users dormant beyond a threshold.
Constraints that decide what you can test in push
Push runs on someone else's platform, which sets limits you cannot test your way around.
- Permission is opt-in on both major platforms and revoking it is a single tap, so the audience shrinks silently.
- Operating systems summarise, delay and group notifications, so a delivered notification is not a seen notification.
- Character limits before truncation differ by device and by lock screen state, so length tests need a device split.
- Uninstall is the ultimate opt-out and it is invisible in most reporting unless you deliberately measure it.
- Web push and app push behave differently enough that a learning from one should not be copied to the other without a re-run.
How fast push learnings fade
Push carries the strongest novelty effect of the lifecycle channels. A new notification type often performs well simply because it is new, and the same notification measured a quarter later usually reads much lower.
Learnings about trigger quality and destination are durable, because they describe whether the notification was useful. Learnings about tone, emoji and headline style are not.
Platform behaviour changes with operating system releases, so re-run anything load bearing at least once a year and check the source date before copying a result.
Push has the sharpest novelty curve of the three channels. Wins from a new notification type often shrink within weeks as people learn to ignore it, so recency matters more here than anywhere else.
Format learnings from the corpus
- Short copy with a single clear action is the most repeated winning shape.
- Timing and frequency caps show up more often than copy as the deciding lever.
How to run a push test
Randomise on the user, keep the eligibility and throttle rules identical across arms, and read on retained active users rather than on notification opens. Track uninstalls and permission revocations for the full period, not only during the send.
Set the sample size and the read date before the test ships, not after you have seen the first hour of data. If the audience cannot reach the sample you need inside a sensible window, change the test rather than the standard.
A test card template for Push
- Hypothesis
- Because users who [signal] return when reminded about [thing], sending [variant] instead of [control] will increase [retained action] within [window].
- Success criteria
- Retained active users or completed actions across the assigned population, not notification opens.
- Guard metric
- Permission revocation and uninstall rate over the full period.
Free to copy and use in your own program. Fill the brackets from a test you have read in the library.
Guard metrics: what a win must not cost
The cost of a bad push is invisible unless you measure it deliberately.
- Notification permission revocation rate.
- Uninstall rate over the test period and the weeks after it.
- Session quality after the tap, since a bad destination produces an immediate exit.
- Share of the audience still reachable at the end of the test.
Evidence from real Push programs
The library holds 11 graded tests run by Push teams: 11 report a win for the variant, 0 a loss, and 0 no clear difference.
- Push vs self-scheduled reminders for daily practice habit
Algorithmically-timed push beat self-scheduling for daily-habit adherence (an exact amount vsan exact amount), but the authors warn the crutch can undercut intrinsic habit and over-frequency fatigues users.
Grade AVariant won2026 - Personalized push (owner + pet name) lifts app engagement
Personalization compounds (owner plus pet name beat owner name alone), but layering in behavioral history on top of names added little, so identity cues do most of the work.
Grade AVariant won2026 - Time-varying push effect on same-day app engagement (mHealth MRT)
A push reliably lifts same-day engagement, but the lift shrinks the more habituated or already-active the user is, so blanket sending wastes the effect on people who did not need it.
Grade AVariant won2018 - Personalized app-notification promotions by engagement stage
Push works better when timed to where the user is in their engagement lifecycle rather than sent on a broadcast schedule, and the stage can be modeled.
Grade AVariant won2016 - Location-based mobile promotions drive 6-12x purchases
Location-triggered push drove large same-day and 12-day-delayed purchase lifts, and measuring only the immediate response undercounts the true effect by half.
Grade AVariant won - Duolingo notification optimization cuts daily churn
Notification optimization delivered a large churn reduction, but the discipline was testing carefully to avoid burning the push channel, which is the real constraint.
Grade BVariant wonDuolingo2026 - Notification optimization at Duolingo cuts churn
Sustained notification testing delivered a large engagement/churn win, with the discipline being to protect the push channel rather than maximize sends.
Grade BVariant wonDuolingo2026 - Push notifications for retention
Lifecycle push can nearly triple 90-day retention, but the same channel over-sent makesan exact amount of users kill notifications, so the guardrail is opt-out, not open rate.
Grade CVariant wonAggregate2026 - Day-3/7/14 push re-engagement timing windows
The recoverable window closes fast, so push re-engagement at day 3-14 beats a 30-day win-back that arrives after the app is mentally filed away.
Grade CVariant won2026 - Behavior-triggered omnichannel win-back at MPL
Whether a user won or lost their first session is a usable churn predictor, letting reactivation messaging fork before the user fully lapses.
Grade CVariant wonMobile Premier League2025 - Time-optimized push sends
Optimizing send time per user lifted push reaction an exact amount, a timing lever that costs nothing but data.
Grade CVariant wonAggregate2024
Direction, grade and source are free. Exact figures open up once you sign in.
What a winning push test looks like
A winning push test increases retained activity while permission revocations and uninstalls stay flat, and it still holds when you re-run it after the novelty has gone.
Wins built on trigger quality and destination survive. Wins built on tone or format should be treated as seasonal.
Frequently asked questions
Does Push messages still work in 2026?
The library holds 11 graded tests here, and 11 of them report a win for the variant against 0 losses and 0 with no clear difference. That is enough to say the direction still holds, and not enough to promise a number for your own program.
What is the most credible push test in the library?
Push vs self-scheduled reminders for daily practice habit. Algorithmically-timed push beat self-scheduling for daily-habit adherence (an exact amount vsan exact amount), but the authors warn the crutch can undercut intrinsic habit and over-frequency fatigues users.
What tends to fail in Push messages?
No losing test has been recorded in this cell yet, which is a gap rather than a signal. Treat the wins here as directional only.
How fresh are these push learnings?
The newest record in this cell is from 2026 and the oldest from 2016. 0 records have been reviewed and vouched for by the CacheMagpie editor.