I've been running a test for 7 days now and the analytics keep saying that the conversion rate of the variant is identical to the baseline. I am just finding it very hard to believe that two variants can have an identical conversion rate every single day. I would expect there to be at least some variance.
Is this normal behavior? Has anyone else experienced this? Perhaps I need to wait longer?
Did you get an update on this? My baseline and update are showing as identical on Day One – which seems pretty unlikely when the difference is calculated with fraction of a percent precision.
Looking at your results, I'd say it's even more unlikely as that would require identical fluctuations each day.
What seems strange is that the table only shows a line for the test treatment, not the baseline treatment. If I were to guess, the dashboard is calculating the difference based on that same line item which, of course, will always show a zero difference.
Wow – that must have been incredibly frustrating. I've contacted support, also. This feature should be disabled in its current state. I'm also getting reports that existing users who update – not just new users – are receiving the Icon treatment under test.
With your results, It's crystal clear to see that you're having daily fluctuations on the graph you share. The chance that the Treatment and Baseline would have the exact same fluctuations day-to-day is next to impossible. Clearly, either the value displayed for either the Baseline or Treatment under test is wrong and they're comparing either the baseline-to-baseline, or treatment-to-treatment. I'm curious to hear Apple's justification that this 'works as expected'.
What was the result of your test, and how long did it take to start to show differences.