Synthetic UX testing with Grok Bot

// why

Scripted tests said "works". People would have said "where are the results?"

Parking NL already had end-to-end tests, and they paid off. They caught a first load stuck on "Calculating prices…": two requests in one Worker isolate shared the same in-flight data promise, and the second one waited forever. The fix was to never share an in-flight load between requests, plus a regression test that fails if the page still says "Calculating prices" with no options listed. They also caught phone search losing focus: the vaul drawer didn't pass modal=false to the Radix Dialog underneath, so its focus trap stole focus from the search input.

Those are bugs with a clear yes/no. What scripted checks don't tell you is whether a commuter standing next to her car, phone in one hand, can actually compare prices and open a tariff. That's what this round was for: a cheap first pass at friction, before spending real people's time.

// 1 · personas

A persona is a test config with a goal

Each persona is a goal, a device, a language and constraints. The device and constraints become browser settings. The goal becomes the tasks.

A second round added a fourth persona, Henk, 68, on an iPhone with large text. More on that below.

// 2–3 · flows and evidence

Real taps on production, measured from the DOM

The flows are one Playwright script, e2e/ux.ts: 17 of them (6 + 6 + 5), run against parking.davidbroza.dev. Sanne runs in WebKit with the iPhone 14 profile, Dutch locale and a fake geolocation in Den Haag. Tom runs in Chromium as a Pixel 7 with geolocation not granted. Priya runs on desktop and calls the REST and MCP examples exactly as written on /developers. Taps are real touchscreen taps; typing is real keystrokes.

const c = await b.newContext({
  ...devices["iPhone 14"], locale: "nl-NL",
  timezoneId: "Europe/Amsterdam",
  geolocation: { latitude: 52.0799, longitude: 4.3113 },
  permissions: ["geolocation"],
  recordVideo: { dir: `${OUT}/video/p1` },
})

Every step records:

• a screenshot, plus a video per persona;
• time per step and per flow, and taps;
• off-screen swipes: how often a control the task needed was off-screen and had to be scrolled into view first. That's a cheap, honest friction metric;
• page errors, and observations pulled from the DOM (result counts, top results, which nav links exist);
• strict success criteria, also from the DOM. Example: the 8-hour task passes only if the tariff card actually opens, "navigate" passes only if an iPhone user is offered Apple Maps or Waze, and "share" passes only if a share control exists.

// 4 · think-aloud

The AI only gets to react to what was on screen

Then Grok Bot writes each persona's think-aloud, one step at a time. The rule: every quote must be based on that step's screenshot and DOM observations. No imagined screens, no features that weren't there. A few, before the fixes:

“The date, time and 7 duration chips eat the whole sheet. '8 uur' was off-screen to the right; I only found it by swiping the row. I see exactly one result under all those controls, and the list won't scroll unless I drag the sheet all the way up.”

Sanne · tomorrow 08:30 for 8 h · synthetic

“I said no to location and now it shows me … Utrecht? 'now: Utrecht centrum'. I'm not in Utrecht. Nothing tells me why or what to do.”

Tom · location denied · synthetic

“There's only a Google Maps button. I use Apple Maps and Waze in the car.”

Sanne · navigate · synthetic

“Limits say 'Anonymous: 20 tool calls/min', but the OpenAPI spec says 20 and the API terms say 30/min + 500/day. Which one is true?”

Priya · developer docs · synthetic

At the end of their flows, each persona also gets a 0–100 "SUS-style" ease score. That's an estimate from the simulated perspective. It is not a measured SUS, and I only use it to compare before with after.

// 5 · findings

Ranked by severity, each with evidence

// 6 · fix and rerun

Same scripts, same flows: 14/17 → 17/17

After deploying the fixes, all 17 flows ran again, unchanged, on production. The only change to the test was one extra tap on the new "when" summary button before using the pickers.

Tom passed all six flows even before the fixes, but only by the letter of the criteria. Being dropped in Utrecht passed the check and still confused him. That's exactly the kind of thing a pass/fail script misses and the think-aloud catches.

// the real report

Round 1 report (PDF, 10 pages)

This is the full report the test produced: personas, method, every flow before and after, the observed DOM facts, the think-aloud, the findings table and the before/after evidence. It's unedited, just compressed for the web.

Pages 1, 5 and 7. Click to open the full PDF.

// what we changed because of the test

Finding → shipped fix

All of these fixes shipped in one Parking NL commit, f1ca657 ("Synthetic persona UX test (report + PDF); fixes: list-first phone sheet, wrapping durations, location-off banner, Apple Maps/Waze, share + deep link, P+R explainer, consistent API limits"). The screenshots are the same Playwright step before and after that deploy.

F1 · Collapsed "when" button, list scrolls at every sheet height

The date, time and 7 duration chips used to fill the phone sheet, leaving about one result, and the list wouldn't scroll until the sheet was dragged to full height. Now arrival and duration are one summary button ("Morgen 08:30 · 8 uur ▾") that opens the pickers, and the list scrolls at every height. Sanne's 8-hour task went from ❌ to ✅, with 2 → 1 off-screen swipes.

F2 · Durations wrap, filter row fades at the edge

"8 uur" and "1 dag" were cut off at the right edge with no hint there was more. The durations now wrap inside the picker, so none are hidden, and the filter row gets a fade at the right edge. You can see the fade after "Terrein" in the after shot above.

F3 · Location-off banner

With location denied, Tom silently got Utrecht. Now a banner explains it, and the fallback is labelled as an example:

loc_off:     "Location is off. Search an address or place, or"
loc_try:     "try again"
example_loc: "example: Utrecht centrum"

F4 + F5 · Apple Maps / Waze row and a Share button

Navigation was Google Maps only, and there was no way to share a place. The main button now opens Apple Maps on Apple devices and Google elsewhere, with an "Open in: Google Maps · Apple Maps · Waze" row. A Share button opens the native share sheet or copies the link. Links now carry the place (a=<id>) and reopen its card. Sanne's navigate and share tasks both went from ❌ to ✅.

F6 · P+R explained in the card

Tom found P+R Duivendrecht at €10.76 but nothing explained what P+R is. P+R cards now carry a short explanation in Dutch and English. That text is in the card, below where the test screenshots end, so here's the shipped string:

pr_explain: "Park + Ride: park at the edge of town and continue
             by public transport. Often cheaper when …"

F7 · One set of API limits everywhere

The docs and OpenAPI said 20 per minute, but the terms and the actual limiter said 30 per minute plus 500 per day. The /developers page and the OpenAPI description now match the limiter:

- Anonymous: 20 tool calls / minute per IP …
+ Anonymous: 30 calls / minute and 500 / day per IP …
+ API key:   600 calls / minute and 50,000 / day …

The report left four minor items open. A later commit, a201def ("Landmark index, display names, blue zone explainer, …"), went after three of them: landmark search, clean display names and the blue-zone explainer. The draft legal pages still need a lawyer.

// round 2 · clarity

Four personas, one question: is it obvious?

Round 2 ran the same way, scripted Playwright sessions on production, but asked a different question. Can people tell what they're looking at? It added a fourth persona, Henk, 68: non-technical, on an iPhone with large text (the page is rendered at 130% text size to approximate iOS "Larger Text"). It also took screenshots at zoom 12, 14 and 16 in Den Haag, Amsterdam and Haarlem.

The before run had 28 tasks (7 tasks × 4 personas). The after run had 40, because three new tasks tested the new walking and door-to-door features. On the same 28 tasks, after the fixes, 27 passed.

Top confusions (before)

• No "best option": you had to read the whole list, and distances were straight-line metres.
• Restrictions used red and amber, colliding with the price colours (red meant both "> €20" and "no parking").
• Occupancy colours on garage badges reused the price ramp.
• On phones the legend was tall and always open (about 45% of the screen), with no explanation of "approximate" or "may have changed".
• Clutter at zoom 12–14: labels, dots and hatching everywhere.

What changed

• One visual language. Price is shown only as zone fill (colour-blind-safe ramp, ΔE checked). Occupancy is only a ring on a blue "P" badge. Restrictions are a subtle dark hatch, behind a toggle, and only appear from zoom 16.
• Collapsible "Legenda" chip, which takes 0% of the screen until you open it, plus "What do the labels mean?" help in the list and in every card.
• "Best option" card on top (best balance, cheapest within N min walk, or closest under €X), real walking minutes, and drive + walk door-to-door totals.
• Walking isochrones: dark rings with a white halo, labelled 5/10/15 min, with a one-tap "5·10·15" chip on the map.
• iOS-style blue location dot. The red pin now means the destination only. Plus a one-line hint on first visit.

The bug the screenshots caught: outlines only

The zone screenshots showed Scheveningen with outlines but no colour. MapLibre was rejecting the zones-fill layer because a zoom expression was nested inside a "case". The fix shipped with a style validation unit test and an e2e check (zones-shots.ts) that fails if no fill renders.

The two remaining fails

• Henk, large text, T8: with large text, the ranking chips sit below the fold at the drawer's default height. The script's click timed out after 30 s, and a real Henk would have to drag the drawer up first. Not fixed yet.
• Desktop, T2: "Best balance" ranked a free street zone with a 14-minute walk above a paid spot right at the door, so the "cheapest legal option ranked first" check failed. That's by design, and "closest" mode is available, but it's an honest disagreement between the ranking and the test.
• Also flagged: on Sunday, Den Haag's "Kernwinkelstraten" zone shows as the best free option. That's what RDW says, but the shopping streets may be car-free in practice.

Round 2 report, pages 2, 1 and 6. Click to open the full PDF.

// round 3 · information hierarchy

"It's super busy": a redesign driven by the tests

My brief after round 2 was: "It's super busy. Focus on information hierarchy, and use the usability test results as input." Round 3 added a fifth persona: Lisa, a parent driving from Pijnacker to an event at Ahoy, who needs a door-to-door answer. It ran the same 59 tasks on the shipped UI and then on the redesign.

The baseline had every feature but a crowded first view: 10–12 floating controls on a phone and 33 on desktop, 18 of our own map layers, and 2 taps to start navigating. The redesign uses three levels. What you always see: search, one best card, the map. One tap away: the list, sort and time. On demand: filters, walking rings, restrictions and the legend, all in one map menu.

One tap opens your default maps app from the card's navigate icon; choosing Apple Maps, Google Maps or Waze is still 2 taps, inside the detail card. The only failure left on phones was the strict "≤ 3 floating controls" check: the phone had 4 (search, ×, map menu, and the attribution ⓘ).

Click to open the full PDF.

Download Round 3 report (PDF, 0.6 MB) ↓

// round 4 · modes and "price here"

77/79, and a change in how we count

Round 4 kept the five personas and the round 3 script, and added four checks. Is the mode clear (my location / destination / map area)? Does the card show the price here first and then a cheaper alternative? Are there type icons? Does the sheet chevron work? It also checked that door-to-door only switches on when you ask for it. Result: 55/59 → 77/79 tasks passed. Time to first decision fell for four of the five personas (Henk: 1,357 → 938 ms; Lisa: 1,073 → 701 ms). Priya went from 1,341 to 1,413 ms.

To be honest about the measurement: between rounds 3 and 4 I changed how the floating-control check counts. It now leaves out the map attribution, which is legally required, and the desktop side panel. Part of the improvement in that check comes from that change, not from the UI. Even so, the check still fails twice: for Priya on desktop, and for Henk with large text.

Bugs found and fixed during the round:

• Switching to "Mijn locatie" from destination mode did nothing while it waited for a fresh GPS fix. It now uses the last known fix at once.
• Flying the map to your location flipped the mode to "map area". The app now ignores camera moves it makes itself for 3 s.
• The GPS dot was announced as a button. It's now an image and not focusable.
• Drive time appeared without a starting point. Door-to-door is now opt-in.

Click to open the full PDF.

Download Round 4 report (PDF, 0.6 MB) ↓

// accessibility audit

WCAG 2.2 AA: 71 serious issues → 0

Not synthetic: an automated axe-core run against the WCAG 2.2 AA rules. It covered the app (map, detail card, phone search) and ten subpages, in light and dark, on a Pixel 7 and on desktop. Serious or critical violations went from 71 to 0. Lighthouse mobile accessibility scored 98. The biggest fixes:

Also checked: keyboard order and visible focus, a skip link, 200% text at 320 px with no horizontal scroll, reduced motion, and a colour-blind simulation of the map palette (no confusable pairs). Screen-reader behaviour was only checked through the accessibility tree. No real VoiceOver or TalkBack session was run. The axe check is now one of the seven deploy checks, so any serious violation rolls the deploy back.

Click to open the full PDF.

Download Accessibility report (PDF, 0.1 MB) ↓

// limits

Simulated people are not users

• The personas never get tired, distracted or annoyed in surprising ways. They only notice what the script makes them look at. A real person will find problems no flow was written for.
• The quotes are written by a model. Grounding them in screenshots keeps them honest about what was there, not about how a real person would feel about it.
• The ease scores are estimates. They're useful for before/after direction, not as a usability benchmark.
• The run was on a Sunday morning, when many street zones in Den Haag are free. That's why the first screens show a lot of "Gratis"; the 8-hour and concert tasks deliberately set a weekday or an evening.

What it is good for: catching obvious friction fast and cheap. It took one morning and a handful of fixes to go from a core phone task that failed to one that works. The next step is the real one: about five real people per segment (commuters in Den Haag and Delft, event visitors, a fleet PM) before calling it done.

Try the result at parking.davidbroza.dev — or read what Parking NL is →

Note: this article was written by an AI from the test report and scripts. The test sessions and numbers are real; the personas, quotes and ease scores are synthetic. The prose isn't hand-written.