Version where bug was found:
6.5.0
Describe the bug
Metric data appears to persist after an experiment is deleted and recreated with the same experiment design.
After deleting an experiment and recreating it, the new experiment appears to show metric data that was logged for the previous experiment, even when no new metrics have been logged for the recreated experiment.
This seems unexpected because the recreated experiment should have a new experiment ID and should not have metric data associated with the previous experiment.
To Reproduce
Steps to reproduce the behavior:
- Create an experiment named
Exp1
- Start the experiment
- Enroll a user, for example
user1, using /init, /assign, and /mark
- Log metrics using
/log
- Delete the experiment
Exp1
- Recreate
Exp1 with the same experiment design
- Enroll the same user again using
/init, /assign, and /mark
- Do not log any new metrics
- View the Metrics Data using
/query/analyse
- Previously logged metric data is still displayed for the recreated experiment
Expected behavior
The recreated experiment should not show metric data from the deleted experiment.
Since the recreated experiment has a new experiment ID, its Metrics Data should be empty until new /log calls are made for that experiment.
Additional context
A related observation is that the stale metric data appears to be grouped under the user's newly assigned condition in the recreated experiment, not necessarily the condition from the original deleted experiment.
For example, the same student previously received control in the original experiment and logged metric data there. After deleting and recreating the experiment, the same student received variant. Without logging any new metrics, the old metric values appeared under variant in the Metrics Data view.
This could contaminate results if some clients fail to send new /log requests, for example due to a timeout or request failure. In that case, metric values generated under a previous experiment condition could be counted under a different condition in the recreated experiment.
Version where bug was found:
6.5.0
Describe the bug
Metric data appears to persist after an experiment is deleted and recreated with the same experiment design.
After deleting an experiment and recreating it, the new experiment appears to show metric data that was logged for the previous experiment, even when no new metrics have been logged for the recreated experiment.
This seems unexpected because the recreated experiment should have a new experiment ID and should not have metric data associated with the previous experiment.
To Reproduce
Steps to reproduce the behavior:
Exp1user1, using/init,/assign, and/mark/logExp1Exp1with the same experiment design/init,/assign, and/mark/query/analyseExpected behavior
The recreated experiment should not show metric data from the deleted experiment.
Since the recreated experiment has a new experiment ID, its Metrics Data should be empty until new
/logcalls are made for that experiment.Additional context
A related observation is that the stale metric data appears to be grouped under the user's newly assigned condition in the recreated experiment, not necessarily the condition from the original deleted experiment.
For example, the same student previously received
controlin the original experiment and logged metric data there. After deleting and recreating the experiment, the same student receivedvariant. Without logging any new metrics, the old metric values appeared undervariantin the Metrics Data view.This could contaminate results if some clients fail to send new
/logrequests, for example due to a timeout or request failure. In that case, metric values generated under a previous experiment condition could be counted under a different condition in the recreated experiment.