[Segmented Onboarding] Experiment setup and metrics - #9711
Conversation
d7ac72f to
a348288
Compare
b2aab91 to
6bdf2ef
Compare
|
Privacy Review task: https://app.asana.com/0/69071770703008/1218184374391087 |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6bdf2ef. Configure here.
902d575 to
d8139bf
Compare
d8139bf to
58a7ef0
Compare
| /** | ||
| * The branch the user picked on the download reason step. Only enumerates a subset of paths that can have side effects on contextual CTAs. | ||
| */ | ||
| enum class SegmentedOnboardingPath { |
There was a problem hiding this comment.
It's a bit confusing that we deleted this enum and we now use the DownloadReasonSelection instad. Not a blocker for this PR, just trying to understand the reason behind.
There was a problem hiding this comment.
Both of these enums represent the same information in the end. I initially introduced SegmentedOnboardingPath because it only handled a subset of paths, but ended up just folding them together now to avoid unnecessary mapping.
| onboardingBrandDesignUpdateToggles.configDrivenDialogs().isEnabled() && | ||
| onboardingBrandDesignUpdateToggles.brandDesignUpdate().isEnabled() && | ||
| onboardingBrandDesignUpdateToggles.configDrivenDialogs().isEnabled() && | ||
| !onboardingPromptsToggles.addToDockAndWidgetExperimentJul25().isEnabled() && |
There was a problem hiding this comment.
Nit: Do we need all these checks?
There was a problem hiding this comment.
In theory, yes. In practice we're unlikely to collide with things like the widget/dock experiment, but I guess it doesn't hurt to have the check in place.
| type = MetricType.NORMAL, | ||
| value = "", | ||
| toggle = toggles.onboardingFlowByDownloadReasonExperiment(), | ||
| conversionWindow = listOf(ConversionWindow(lowerWindow = 0, upperWindow = 0)), |
There was a problem hiding this comment.
In the previous experiment we also measured the onboarding completion on d0-14, to get a "total" completion rate, not just the one on d0. Should we add it here as well?
There was a problem hiding this comment.
d0-14 is not part of the experiment design in this case, wasn't on iOS either.
catalinradoiu
left a comment
catalinradoiu
left a comment
There was a problem hiding this comment.
Looks good and works as expected 👍
2 participants
Task/Issue URL: https://app.asana.com/1/137249556945/project/1208671518894266/task/1218095440012407?focus=true
Tech Design URL (if applicable):
API Proposals URL(s) (if applicable):
Description
Steps to test this PR
logs
message~:"onboardingFlowByDownloadReasonExperiment".regression
onboardingFlowByDownloadReasonExperimentpixels fire.control
privacy-config/privacy-config-internal/local-config-patches/review_patch.json:[ { "op": "add", "path": "/features/segmentedOnboarding", "value": { "state": "enabled", "features": { "onboardingFlowByDownloadReasonExperiment": { "state": "enabled", "cohorts": [ { "name": "control", "weight": 1 }, { "name": "treatment", "weight": 0 } ] } } } }, { "op": "replace", "path": "/features/onboardingPrompts/features/addToDockAndWidgetExperimentJul25/state", "value": "disabled" }, { "op": "add", "path": "/features/onboardingPrompts/hash", "value": "patched" }, { "op": "replace", "path": "/version", "value": "90000000000001" } ]privacy-config/privacy-config-internal/local.properties:experiment_enroll_onboardingFlowByDownloadReasonExperiment_controlpixel was sent.metric=onboarding_completedwas sent.treatment
privacy_config.jsonandreview_patch.json:controlto0treatmentto1experiment_enroll_onboardingFlowByDownloadReasonExperiment_treatmentpixel was sent.metric=download_reason_selected_<reason>was sentmetric=onboarding_completedwas sent.metric=duck_ai_new_chatmetric=duck_ai_prompt_sentmetric=searchmetric=download_reason_search_retention_<reason>returning user
onboardingFlowByDownloadReasonExperimentpixels fire.clean up
privacy-config/privacy-config-internal/local-config-patches/review_patch.jsonand revert changes inprivacy-config/privacy-config-internal/local.properties.Note
Medium Risk
Touches new-user onboarding branching, experiment enrollment gates, and a breaking analytics param rename (
variant_download_reason); incorrect persistence or enrollment could skew experiment data or CTA behavior on segmented paths.Overview
Adds native experiment support for
onboardingFlowByDownloadReasonExperiment(control/treatment) with metrics for completion, per-reason selection, and D5–7 search retention by download-reason segment.Telemetry rename: onboarding pixels now use shared param
onboardingDownloadReasonVariant(variant_download_reason) instead ofonboardingSegmentedVariant(variant_segmented).State model: replaces
SegmentedOnboardingPathwithDownloadReasonSelectionpersisted inOnboardingStore; download reason is written before the download-choice clicked pixel and cleared when a new linear onboarding run starts so attribution and retention metrics don’t leak from abandoned runs.Enrollment:
SegmentedOnboardingExperimentManageractually enrolls viaSegmentedOnboardingFeatureToggles, with prerequisites (brand design update, config-driven dialogs, no conflicting experiments, non-reinstall). Metrics fire on established stage, reason confirmation, and ATB search/Duck.ai retention refreshes.Reviewed by Cursor Bugbot for commit 58a7ef0. Bugbot is set up for automated code reviews on this repo. Configure here.