Why can I only test one variable at a time?
Testing multiple variables for the same product skews test data which makes it impossible to know which test variable helped or hurt the overall test performance. Instead, Optimal A/B requires that you only test one variable at a time for each product, leaving all other variables the same.
“In an experiment, scientists only test one variable at a time to ensure the results can be attributed to that variable. If more than one variable is changed, scientists cannot attribute the changes to one cause.”
-Google AI
Related Articles
Test Conflict/New Test Conflict?
If you received a test conflict error, this means that you are trying to schedule or run more than one test at a time for the same product. Running or scheduling more than one test for the same product can lead to multiple variable changes during the ...
Modifying product listings during a test, ruins data!
It is best practice to never modify a product listing outside of our app when you have a running test for that same product. This could lead to multiple variable changes during the duration of a test, which would make it impossible to know which ...
Selecting a start time?
syncX: A/B Testing changes your product listings on a scheduled interval. If your test starts or is scheduled to start at 8:30 PM for example, and you select a change frequency of 12 hours, the app will alternate between changing your product listing ...
What do the test analytics mean?
Avg Revenue Per View: The average revenue generated by a single view for a given test variant. This is the default metric used to determine the winning variation for Image, Title, and Description tests. Avg Profit Per View (only available for Price ...
How many test variants can I create per test?
“Free” plan subscribers can test up to 2 variants per test. “Basic” plan subscribers can test up to 3 variants per test. “Premium” & “Enterprise” plan subscribers can test up to 6 variants per test. See additional plan details here.