Carepatron synthetic onboarding test
Current onboarding vs three setup-checklist prototypes
Bias controls used: anonymous randomized target labels in-agent, screenshot-only interaction, same rubric and goals for every target, no prototype hypotheses in prompts, and prototype-aware scoring that excludes backend completeness while still judging visible goal support.
Run incomplete: 3 job(s) failed during this run. Missing cells below should be treated as run-failures, not product findings.
Reading the ranking: two scores per target. Flow quality (clarity/trust/motivation/ease) measures the visible UX independent of persona scope. Goal fit (value/concept-fit/likelihood-to-subscribe) measures how well the prototype serves THIS persona's specific job. A prototype with high flow quality but low goal fit is well-designed for one job and poorly-aligned for another — that's information, not a bug. In-scope-only columns recompute both scores using only the reviewers for whom this prototype's scope was relevant.
Ranking Table
| Rank | Target | Runs | Flow quality (all) | Goal fit (all) | Flow quality (in-scope) | Goal fit (in-scope) | In-scope runs | Subscribe | Goal completed | Attempt allowed | Median friction |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | v2 Personalized: profession-first wizard | 20 | 8.3 | 7.9 | 8.3 | 7.9 | 20 | 7.0 | 100.0% | 100.0% | 2.0 |
| 2 | v3 Focused: first booking | 20 | 8.3 | 7.9 | 8.3 | 7.9 | 20 | 7.0 | 100.0% | 100.0% | 2.0 |
| 3 | v1 Refined: closest to current | 20 | 8.2 | 7.8 | 8.2 | 7.8 | 20 | 6.8 | 100.0% | 100.0% | 2.0 |
| 4 | Control: current production onboarding | 1 | 3.0 | 3.3 | 3.0 | 3.3 | 1 | 1.0 | 0.0% | 100.0% | 2.0 |
Friction Heatmap
| Step / friction zone | Control: current production onboarding | v1 Refined: closest to current | v2 Personalized: profession-first wizard | v3 Focused: first booking |
|---|---|---|---|---|
| All 5 onboarding steps | 4 | |||
| All steps (global) | 5 | |||
| All steps — sidebar | 2 | |||
| All steps — sidebar nav | 3 | |||
| All steps — trust layer | 3 | |||
| Completion screen | 4 | 2 | ||
| Completion screen (T9) | 2 | |||
| Final completion screen | 3 | |||
| Pick a service (T2-T3) | 2 | |||
| Post-completion screen (T6–T16) | 3 | |||
| Practice type selection (T1–T2) | 1 | |||
| Practice type selection (T1–T4) | 3 | |||
| Set a location (T5-T6) | 2 | |||
| Set your hours - availability setup | 2 | |||
| Setup complete - celebration screen | 6 | |||
| Setup complete screen | 2 | |||
| Sidebar nav at completion | 2 | |||
| Step 1 - Select services | 2 | |||
| Step 1 – Select Services | 2 | |||
| Step 1 — Add services | 2 | |||
| Step 1 — Practice type selection | 1 | |||
| Step 1 — Practice type selection (T1–T2) | 2 | |||
| Step 3 - Location | 2 | |||
| Step 3 - Where you see clients | 2 | |||
| Step 3 completion label in checklist | 2 | |||
| Step 3 of 4 - Location type selection | 2 | |||
| Step 3 of 4 – Location setup (T6) | 2 | |||
| Step 4 - Send test booking | 3 | 2 | ||
| Step 4 - Test booking | 1 | |||
| Step 4 - Test booking email | 2 | |||
| Step 4 of 4 - Send test booking | 2 | |||
| Step 4 of 4 — Send test booking | 1 | |||
| Step 4 – Test booking | 2 | |||
| Step 4 — Send test booking (T6–T8) | 3 | |||
| Step 4 — Test booking | 2 | |||
| Step 4 — The moment (send test booking) | 2 | |||
| Step 5 – Test booking | 1 | |||
| Step 8 — Completion screen | 2 | |||
| T1 - Add services | 2 | |||
| T1 — Get Started header | 3 | |||
| T1 — Welcome / Get started screen | 5 | |||
| T1 → T2 (Practice type) | 2 | |||
| T1-T2 Practice type selection | 10 | |||
| T1–T2: Practice type selection | 2 | |||
| T2 - Practice type selection | 5 | |||
| T2 - Set your hours | 2 | |||
| T2 — Pick a Service (Quick Pick expanded) | 2 | |||
| T2–T3: Pick a service | 2 | |||
| T3 - Set your hours | 6 | |||
| T3 Service selection | 2 | |||
| T3 Set your hours — initial render | 2 | |||
| T3 location/modality step | 3 | |||
| T3 — Location step | 2 | |||
| T3 — Set your hours | 2 | |||
| T3 — Where you see clients (Step 3 of 4) | 2 | |||
| T3–T4: Service selection | 2 | |||
| T4 - Set a location | 4 | |||
| T4 - Set your hours | 2 | |||
| T4 - Step 4 email field | 1 | |||
| T4 Availability setup | 2 | |||
| T4 Set a location | 3 | |||
| T4 Set your hours | 3 | |||
| T4 — Set a location | 4 | |||
| T4 — Try a test booking | 2 | |||
| T4-T5 — Send test booking | 2 | |||
| T4-T5: Set a location | 2 | |||
| T4-T5: Test booking step 4 | 2 | |||
| T4/T5 — Test booking email field | 1 | |||
| T5 - Set a location | 2 | |||
| T5 - Step 3 Location setup | 3 | |||
| T5 Location setup | 2 | |||
| T5 Set a location | 2 | |||
| T5 — Set a location | 2 | |||
| T5 — Set a location (scroll required) | 2 | |||
| T5 — Setup complete screen | 2 | |||
| T5 — Step 3 of 4 Location setup | 2 | |||
| T5-T6 Location setup | 2 | |||
| T5-T6 Set a location | 2 | |||
| T6 - Completion screen | 2 | |||
| T6 - Set a location | 2 | |||
| T6 - Step 4 send test booking | 4 | |||
| T6 - Test your link | 6 | |||
| T6 Step 4 — test booking send | 2 | |||
| T6 completion screen — AI Notetaker feature card | 3 | |||
| T6 completion screen — Explore top features | 2 | |||
| T6 — 100% completion state | 4 | |||
| T6 — Completion screen | 2 | |||
| T6 — Completion state | 5 | |||
| T6 — Explore top features | 2 | |||
| T6 — Explore top features (0 of 6 done) | 3 | |||
| T6 — Reward banner | 2 | |||
| T6 — Test your link | 2 | |||
| T6-T7: Test your link | 2 | |||
| T6/T7 - Step 4 send test booking | 2 | |||
| T6: Location setup – Step 3 of 4 | 3 | |||
| T6: Set a location — needed to scroll after sele | 1 | |||
| T6: Setup complete screen | 2 | |||
| T6–T7 — Test your link | 3 | |||
| T7 - Step 4 of 4 Send test booking | 1 | |||
| T7 Test your link | 2 | |||
| T7 — Completion | 3 | |||
| T7 — Post-completion sidebar | 2 | |||
| T7 — Step 4 of 4 Test booking | 2 | |||
| T7-T9 Test your link | 2 | |||
| T7: Set a location — Virtual selected | 3 | |||
| T8 - Completion screen | 2 | |||
| T8 - Completion screen sidebar | 3 | |||
| T8 - Completion state | 2 | |||
| T8 - Setup complete | 2 | |||
| T8 - Step 4 Send test booking | 2 | |||
| T8 Setup complete celebration screen | 3 | |||
| T8-T12 Test your link | 3 | |||
| T8: Completion state | 3 | |||
| T9 - Completion screen | 3 | |||
| T9 — Completion state (sidebar nav) | 2 | |||
| Team size selection (T8–T21) | 5 | |||
| Test your link - booking link preview | 2 | |||
| Try a test booking — Step 4 of 4 | 5 | |||
| Where you see clients (T3) | 1 | |||
| Where you see clients — Step 3 of 4 | 3 | |||
| Workspace setup screen | 3 |
Persona x Prototype Matrix
| Persona | Control: current production onboarding | v1 Refined: closest to current | v2 Personalized: profession-first wizard | v3 Focused: first booking |
|---|---|---|---|---|
| Dr. Maya Chen, LCSW | Attempted | Completed | Completed | Completed |
| Sam Rivera, LMHC | Not run | Completed | Completed | Completed |
| Priya Shah | Not run | Completed | Completed | Completed |
| Dr. Tom Walsh | Not run | Completed | Completed | Completed |
| Lara Okeke, PT | Not run | Completed | Completed | Completed |
| Anna Bauer, OT | Not run | Completed | Completed | Completed |
| Chris Park | Not run | Completed | Completed | Completed |
| David Knight, DC | Not run | Completed | Completed | Completed |
| Nancy Hanisch | Not run | Completed | Completed | Completed |
| Jordan Lee | Not run | Completed | Completed | Completed |
| Dr. Elena Vargas | Not run | Completed | Completed | Completed |
| Mary McKinnon | Not run | Completed | Completed | Completed |
| Becky Tran | Not run | Completed | Completed | Completed |
| Marc Robinson | Not run | Completed | Completed | Completed |
| Liz Chen, SLP | Not run | Completed | Completed | Completed |
Quote Wall
Dr. Maya Chen, LCSW · Control: current production onboardingTeam size — Choose team size — This is required. Back. Continue.
David Knight, DC · v1 Refined: closest to currentThe note about each location having its own services and hours gave me exactly the signal I needed — but I nearly gave up before finishing because the Skip button just would not work.
Healthcare vertical expert · v2 Personalized: profession-first wizardThe vertical specificity earns immediate credibility, but a mental health practitioner won't trust their patient notes to a platform that never once mentions compliance.
Healthcare vertical expert · v1 Refined: closest to currentThe onboarding earns clinical credibility through sensible defaults and a smart activation sequence, but one missing word — HIPAA — will stall every compliance-gated buyer before they reach checkout.
Healthcare vertical expert · v3 Focused: first bookingThe funnel gets me to a live booking link in under 5 minutes, which is genuinely impressive, but I'm handing out a URL that touches patient data without a single compliance signal in the flow — that's going to be a hard stop for any practice manager who's been through a HIPAA audit.
Liz Chen, SLP · v2 Personalized: profession-first wizardThe fact that Allied health actually said 'speech' in the description made me feel like this was built with me in mind — and seeing telehealth as a location option in setup, not buried in settings, was exactly what I needed to see.
Marc Robinson · v2 Personalized: profession-first wizardSetup was fast and the defaults were spot-on for a mental health clinic, but I finished in four steps and still have no idea where I add my other 19 clinicians — that's the whole point for us.
David Knight, DC · v2 Personalized: profession-first wizardThe flow was smoother than I expected and the location-slug URL told me multi-site is built into the architecture, but I had to take it on faith that chiro lives under Allied health and that the extra sites are actually there waiting in settings.
Marc Robinson · v1 Refined: closest to currentFor a clinic at my scale, this feels credible and organized, not consumer-grade.
Dr. Tom Walsh · v3 Focused: first bookingThe Virtual option was right there and easy to pick, but I still had to stare at a pre-filled street address wondering if my telehealth clients were about to get sent to a building I don't work in.
Lara Okeke, PT · v2 Personalized: profession-first wizardEverything felt fast and built for someone like me until the location step handed me an in-person address form — I'd already told it I do home visits.
Dr. Tom Walsh · v1 Refined: closest to currentVirtual setup was easy to find and click — I just wish the address field had hidden itself once I chose Virtual, so I could be certain my clients won't see a physical address on their booking page.
Anna Bauer, OT · v1 Refined: closest to currentGetting set up was genuinely quick and logical, but the default services felt like they were built for a GP, not a pediatric OT.
David Knight, DC · v3 Focused: first bookingGot Brunswick Clinic live in under five minutes and the URL slug tells me each site probably gets its own booking page — but I'd need to see how I manage three locations from one dashboard before I commit.
Dr. Tom Walsh · v2 Personalized: profession-first wizardI was relieved to see Virtual as a proper first-class option in step 3, but the physical address still showing and my booking link reading 'brunswick-clinic' left me second-guessing whether the system actually registered me as a fully remote practice.
Lara Okeke, PT · v3 Focused: first bookingSetting up on my iPad took maybe eight minutes and the booking link was live at the end — the defaults were spot-on for a physio practice, which meant I barely had to think.
Dr. Maya Chen, LCSW · v1 Refined: closest to currentThat progression felt genuinely fast — I went from zero to fully configured without getting lost or confused.
UX heuristics expert · v3 Focused: first bookingThis onboarding nails the Nielsen fundamentals — system status, recognition over recall, error prevention through smart defaults, minimalist design — and the celebration state with the payment unlock lands at exactly the right emotional beat; the one real heuristic gap is the dual CTA in step 2, which breaks consistency and briefly leaves the user unsure which button owns their hours input.
Anna Bauer, OT · v2 Personalized: profession-first wizardAllied health with NDIS billing pre-loaded felt genuinely built for someone like me — I went from zero to booking-ready without hitting a single confusing step.
Becky Tran · v1 Refined: closest to currentThe setup was genuinely painless — I'd confidently tell any practice owner that they can be ready to take bookings in under five minutes.
Mary McKinnon · v2 Personalized: profession-first wizardI understood every step, nothing stumped me, and seeing that party popper at the end told me I'd actually finished.
Marc Robinson · v3 Focused: first bookingThe four-step structure gave me exactly the confidence I needed — I always knew where I was and what came next, which is what you want when you're evaluating a system for a real clinic.
Dr. Elena Vargas · v3 Focused: first bookingFour steps, everything pre-filled, booking link live in under five minutes — this is what onboarding should feel like.
Brand expert · v3 Focused: first bookingThe brand is doing real work here — this feels like a product worth paying for before I’ve even booked a client.
Top 3 Fix List
1. Make 'Open calendar' functional in the prototype so the reviewer can see a post-setup product state — even a static calendar view would break the 10-turn completion loop and let the reviewer feel what the product is actually like to use, which is the moment that converts setup satisfaction into trial continuation.
Seen in 1 run(s): v1 Refined: closest to current
Seen in 1 run(s): v1 Refined: closest to current
2. Hide or disable the physical address input when Virtual is selected in Step 3, and update the checklist completion label to reflect the specific mode chosen (e.g. 'Virtual only') so telehealth practitioners have unambiguous confirmation their preference was saved.
Seen in 1 run(s): v1 Refined: closest to current
Seen in 1 run(s): v1 Refined: closest to current
3. Remove or defer the 'Explore top features — 0 of 6 done' section from the immediate completion screen so the 100% celebration lands cleanly — introduce the next feature discovery phase only after the user has taken one post-setup action like opening their calendar or sharing their booking link.
Seen in 1 run(s): v1 Refined: closest to current
Seen in 1 run(s): v1 Refined: closest to current
Production Calibration Check
| Expected persona signal | Matched? | Evidence |
|---|---|---|
| tom_walsh | No / not run | |
| maya_chen | Yes | Team size selection (T8–T21): The team size dropdown or selector did not register any of 14 click attempts. The field remained stuck on 'Choose team size — This is required' throughout, blocking the Continue button and preventing any forward progress. | Workspace setup screen: Three required fields (team size, primary goal, insurance) are gated behind a single Continue button with no inline validation feedback explaining why the button stays inactive — compounding the confusion when one field silently fails. |
| priya_shah | No / not run | |
| chris_park | No / not run | |
| nancy_hanisch | No / not run | |
| david_knight | No / not run | |
| becky_tran | No / not run |
Visual Evidence Index
| Target | Reviewer | Status | Screenshots | Video | Run JSON |
|---|---|---|---|---|---|
| Control: current production onboarding | Dr. Maya Chen, LCSW | Scored | 0 | n/a | result.json |
| v1 Refined: closest to current | Anna Bauer, OT | Scored | 6 final | video | result.json |
| v1 Refined: closest to current | Becky Tran | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Behavioural psychology expert | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Brand expert | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Chris Park | Scored | 10 final | video | result.json |
| v1 Refined: closest to current | Conversion expert | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | David Knight, DC | Scored | 12 final | video | result.json |
| v1 Refined: closest to current | Dr. Elena Vargas | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Dr. Maya Chen, LCSW | Scored | 17 final | video | result.json |
| v1 Refined: closest to current | Dr. Tom Walsh | Scored | 8 final | video | result.json |
| v1 Refined: closest to current | Healthcare vertical expert | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Jordan Lee | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Lara Okeke, PT | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Liz Chen, SLP | Scored | 8 final | video | result.json |
| v1 Refined: closest to current | Marc Robinson | Scored | 8 final | video | result.json |
| v1 Refined: closest to current | Mary McKinnon | Scored | 6 final | video | result.json |
| v1 Refined: closest to current | Nancy Hanisch | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Priya Shah | Scored | 7 final | video | result.json |
| v1 Refined: closest to current | Sam Rivera, LMHC | Scored | 6 final | video | result.json |
| v1 Refined: closest to current | UX heuristics expert | Scored | 7 final | video | result.json |
| v2 Personalized: profession-first wizard | Anna Bauer, OT | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Becky Tran | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Behavioural psychology expert | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Brand expert | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Chris Park | Scored | 10 final | video | result.json |
| v2 Personalized: profession-first wizard | Conversion expert | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | David Knight, DC | Scored | 8 final | video | result.json |
| v2 Personalized: profession-first wizard | Dr. Elena Vargas | Scored | 8 final | video | result.json |
| v2 Personalized: profession-first wizard | Dr. Maya Chen, LCSW | Scored | 10 final | video | result.json |
| v2 Personalized: profession-first wizard | Dr. Tom Walsh | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Healthcare vertical expert | Scored | 10 final | video | result.json |
| v2 Personalized: profession-first wizard | Jordan Lee | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Lara Okeke, PT | Scored | 10 final | video | result.json |
| v2 Personalized: profession-first wizard | Liz Chen, SLP | Scored | 12 final | video | result.json |
| v2 Personalized: profession-first wizard | Marc Robinson | Scored | 10 final | video | result.json |
| v2 Personalized: profession-first wizard | Mary McKinnon | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Nancy Hanisch | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Priya Shah | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | Sam Rivera, LMHC | Scored | 9 final | video | result.json |
| v2 Personalized: profession-first wizard | UX heuristics expert | Scored | 10 final | video | result.json |
| v3 Focused: first booking | Anna Bauer, OT | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Becky Tran | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Behavioural psychology expert | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Brand expert | Scored | 10 final | video | result.json |
| v3 Focused: first booking | Chris Park | Scored | 8 final | video | result.json |
| v3 Focused: first booking | Conversion expert | Scored | 9 final | video | result.json |
| v3 Focused: first booking | David Knight, DC | Scored | 8 final | video | result.json |
| v3 Focused: first booking | Dr. Elena Vargas | Scored | 11 final | video | result.json |
| v3 Focused: first booking | Dr. Maya Chen, LCSW | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Dr. Tom Walsh | Scored | 11 final | video | result.json |
| v3 Focused: first booking | Healthcare vertical expert | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Jordan Lee | Scored | 11 final | video | result.json |
| v3 Focused: first booking | Lara Okeke, PT | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Liz Chen, SLP | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Marc Robinson | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Mary McKinnon | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Nancy Hanisch | Scored | 9 final | video | result.json |
| v3 Focused: first booking | Priya Shah | Scored | 10 final | video | result.json |
| v3 Focused: first booking | Sam Rivera, LMHC | Scored | 13 final | video | result.json |
| v3 Focused: first booking | UX heuristics expert | Scored | 9 final | video | result.json |
