A/B Testing Tools: What to Look For and Which to Consider

A/B testing tools are software that randomly split visitors between two or more versions of a page, feature or app screen and report whether the difference is real. Choose one by where the test has to run (browser, server, data warehouse, app or ad platform), then compare the stats engine, traffic needs, page speed impact and pricing unit.

This guide in the A/B testing hub covers only tools that run controlled experiments. Heatmaps, session replays and surveys belong to the wider stack described in the guide to choosing a conversion rate optimization tool. Vendor facts and prices are as of October 2026, checked on each vendor's own pages.

Can GA4 before-and-after comparisons replace an A/B testing tool?

GA4 before-and-after comparisons cannot replace an A/B testing tool, because seasonality, campaigns and traffic mix change between the two periods. An A/B testing tool assigns each visitor at random to a control or a variant, keeps that visitor in the same group and compares both groups over the same dates, so the change is the only difference.

Every split testing tool does four jobs, and vendors differ in how well they do each one:

• Assignment. The tool buckets each visitor or user into a group and keeps them there on return visits.
• Delivery. The tool shows the variant, either by changing the page in the browser or by choosing a code path on the server or in the app.
• Measurement. The tool records goal events such as purchases, leads or sign-ups for each group, or reads them from your analytics or data warehouse.
• Statistics. The tool decides whether the gap between groups is larger than chance would produce.

Split testing software is another name for the same category. A split URL test is a variant of the method that sends each group to a separate URL instead of changing one page.

Where should an experiment run: browser, server, warehouse or app?

An experiment can run in the visitor's browser, on your server, against your data warehouse, inside a mobile app or app store, or inside an ad platform, and each location suits a different team. Browser tools suit marketers who change page content, server-side tools suit developers, and warehouse native tools suit companies whose metrics already live in SQL.

Picking this location first is the run-location rule: where the change lives decides the architecture, and the architecture narrows the vendor list.

Where the test runs	How the variant is delivered	Who builds tests	Named options	Watch for
Browser (client-side)	A JavaScript snippet changes the page after it loads	Marketers, with a visual editor	Optimizely Web Experimentation, Wingify, Convert, Kameleoon	Flicker and script weight
Server or feature flags	An SDK picks the code path before the page or screen renders	Developers	Optimizely Feature Experimentation, GrowthBook, Statsig, PostHog	Flags left in code after tests
Data warehouse	An SDK assigns users; analysis runs as SQL on your tables	Data team	GrowthBook, Statsig Warehouse Native	Clean, documented event tables
Mobile app	Remote config values or SDK flags in the app	App developers	Firebase A/B Testing, Wingify Mobile App Testing	App release cycles
App store listing	The store shows alternate icons and screenshots	App marketers	Apple product page optimization, Google Play store listing experiments	Store assets only
Ad platform	The platform splits campaign traffic and budget	Paid media team	Google Ads custom experiments	Tests campaigns, not pages

"Statsig Warehouse Native is an experimentation platform that runs analysis directly in your data warehouse," per the Statsig docs, and GrowthBook connects to BigQuery, Snowflake, Databricks, Redshift and more. As of October 2026, Apple product page optimization tests up to three treatments for up to 90 days and Google Play up to two variants. Google Ads custom experiments share a campaign's traffic and budget; for incremental impact, see Google Ads conversion lift.

How much traffic does a split testing platform need before it pays off?

A split testing platform pays off only when the tested page collects enough conversions to finish a test in weeks, not months. The sample size depends on the baseline conversion rate and the smallest lift worth detecting, so calculate it for your most important page before you compare vendors or plans.

Evan Miller's 2010 rule of thumb estimates the sample per variant as 16 times the variance divided by the squared minimum effect, where the variance of a conversion rate is p × (1 − p).

Hypothetical worked example. A landing page converts at 3% and gets 20,000 visitors a month, split over two arms.

• Detecting a 10% relative lift (3.0% to 3.3%): 16 × 0.0291 / 0.003² ≈ 51,700 visitors per variant, or about 103,500 in total. That takes about five months.
• Detecting a 20% relative lift (3.0% to 3.6%): 16 × 0.0291 / 0.006² ≈ 12,900 visitors per variant, or about 25,900 in total. That takes under seven weeks.

Halving the detectable lift quadruples the required traffic. A page in the first situation needs bolder changes or more traffic, which no vendor fixes. Run your own numbers in the A/B test sample size calculator before you sign. Size any plan by the users the tool actually buckets into tests, because that is the unit most vendors bill.

Which stats engine should split testing software use?

Split testing software uses one of three statistical approaches: fixed-horizon frequentist tests, sequential tests that stay valid when you check results early, or Bayesian models that report a probability that the variant beats control. The right engine is the one that matches how often your team checks results and that the vendor documents openly.

Platform	Default approach	Other options documented
GrowthBook	Bayesian	Frequentist t-tests, sequential testing, CUPED
Optimizely	Sequential (Stats Engine)	False discovery rate control across metrics and variations
Statsig	Frequentist p-values and confidence intervals	CUPED, stratified sampling, switchback tests (Warehouse Native)
Wingify	Bayesian (SmartStats)	Applies to accounts not yet on the Wingify Data Platform
PostHog	Bayesian and frequentist both supported	Choice per experiment

GrowthBook says it defaults to Bayesian statistics "because they provide a more intuitive framework for decision making for most customers," per its statistics overview. Optimizely writes that its Stats Engine "uses sequential experimentation, not the fixed-horizon experiments you see on other platforms." Wingify calls SmartStats its "Bayesian-powered statistics engine," and Statsig reports p-values and confidence intervals.

The engine matters most when people peek. In Evan Miller's 2010 simulation, checking a fixed-horizon test after every observation pushed the false positive rate to 26.1% against a nominal 5%. For the underlying math, see how to calculate statistical significance.

How do website A/B testing tools affect page speed, GA4 and consent?

Website A/B testing tools that change pages in the browser add a script that must run before the visitor sees the variant, which forces a trade-off between flicker and load time. The same script sends experiment data to GA4 and stores the assigned variant on the device, so it touches page speed, measurement and consent at once.

Flicker is the original page showing briefly before the variant replaces it. Optimizely states that "Optimizely Web Experimentation uses a synchronous snippet to prevent flickering," so the browser waits for the snippet before rendering the rest of the page. Server-side tools avoid this by choosing the variant before the page leaves the server, and the GrowthBook docs say its SDKs "evaluate everything locally with no network requests."

GA4 has no built-in experiment feature, so the testing tool passes variant data in. Optimizely's GA4 integration sends variations to GA4 as audiences, allows one GA4 property per Web Experimentation project and asks you to place the Google tag near the snippet in the page head. Other vendors use events or a tag manager; check that the variant reaches the reports your team reads.

Client-side tools usually remember the variant in a cookie or local storage, so check that the tool follows your cookie consent banner and how it counts visitors who decline.

What changed in the A/B testing tool market after Google Optimize?

The A/B testing tool market has consolidated since Google Optimize shut down on September 30, 2023. Google named AB Tasty, Optimizely and VWO as integrations, two of those three announced a Wingify merger in 2026, and Statsig changed owners twice in eight months, so vendor lists older than a year are often wrong.

What is officially confirmed:

• Optimize sunset. Google Analytics Help states that "Google Optimize and Optimize 360 are no longer available as of September 30, 2023," and running experiments ended.
• Wingify. VWO and AB Tasty announced their merger on January 20, 2026. As of October 2026, the VWO pricing URL redirects to Wingify pricing, which lists Growth, Pro and Enterprise tiers behind "Schedule a Demo" and does not mention a free plan.
• Statsig. Statsig announced on September 2, 2025 that it had agreed to join OpenAI. An update on the same page says Amplitude acquired Statsig on May 5, 2026: the product, brand and customers moved to Amplitude, and the original team stayed at OpenAI.

What is reported: MarTech covered the deal on May 8, 2026 and quoted Optimizely CEO Alex Atzberger, a direct competitor: "Amplitude is getting Statsig's code without the talent, it is a race car without a driver." Whether Statsig's roadmap slows under Amplitude is speculation, so get roadmap and support terms in writing.

Which A/B testing tool fits your team and traffic?

The A/B testing tool that fits is the one whose architecture matches who builds the tests and where your conversion data lives, priced in a unit your traffic does not inflate. Apply the run-location rule first, then shortlist two vendors from the matching row below and trial both on the same page.

Your situation	Architecture	Shortlist	Pricing unit to check
Marketing team, landing pages, little developer time	Browser (client-side)	Convert, Kameleoon, Wingify	Monthly tested or tracked users
Product team shipping features every sprint	Server or feature flags	GrowthBook, PostHog, Optimizely Feature Experimentation	Seats, flag requests or quote
Metrics modeled in BigQuery, Snowflake or Databricks	Data warehouse	GrowthBook, Statsig Warehouse Native	Seats or quote, plus warehouse compute
Mobile app with frequent releases	Mobile app	Firebase A/B Testing, Wingify Mobile App Testing	Included in Firebase, quote for Wingify
Paid media team testing bids, keywords or ads	Ad platform	Google Ads custom experiments	No extra fee, uses campaign budget
Several teams, governance and SSO needed	Mixed	Optimizely, Wingify, Kameleoon Enterprise	Custom quote

Published entry prices as of October 2026:

• Convert: Growth is $399 per month, or $299 billed annually, for 100K monthly tested users (Convert pricing).
• Kameleoon: Starter is from $495 per month for 50,000 MTUs (Kameleoon plans).
• PostHog: experiments are "Billed with feature flags," with 1M free requests per month (PostHog pricing).
• GrowthBook: self-hosting is free; Cloud Pro is $40 per seat per month (GrowthBook pricing).
• Optimizely: "Every Optimizely plan is individually packaged" (Optimizely plans).

Common mistakes when choosing A/B testing tools

The common mistakes when choosing A/B testing tools come from buying by brand instead of architecture, trusting numbers nobody verified and paying for traffic the tests never use. Each mistake below costs money or produces test results the team cannot act on.

• Choosing a vendor before an architecture. A visual editor cannot test server logic, and a feature flag platform frustrates a team without developers. Apply the run-location rule first.
• Paying for traffic you do not test. Plans bill by tested users, tracked users or requests. Load the script only on the pages you test and size the plan to that volume.
• Skipping the A/A test. A tool that declares a winner between identical pages, or whose counts drift from GA4, will mislead every later test.
• Ignoring flicker. Visitors who see the original page flash before the variant are no longer in a clean test. Measure render time with and without the snippet.
• Leaving finished flags in the code. Server-side tests leave branches behind. Remove the losing path once a test ends, or the codebase fills with dead switches.
• Relying on old comparison lists. Many lists still include Google Optimize or separate VWO and AB Tasty plans. Check the vendor pricing page on the day you buy.