AI-assisted full regression automation for a cloud product
Turned existing manual test cases into Gherkin, Robot Framework and Playwright scripts. Every step keeps before/during/after screenshots and a recording; the product QA reviewed each item before taking over maintenance.
System architecture and data flow
Gherkin describes and cross-checks test specifications. Robot Framework scripts execute through three layers of shared keywords and use Browser Library to control Playwright.
- Manual test casesProduct requirements and existing checks
- Gherkin specificationsDescribe scenarios and map test cases
- Robot test scriptsAI-assisted authoring with human review and maintenance
- Test casesSuites, fixtures and setup / teardown
- Business keywordsUser, policy and sign-in operations
- Domain keywordsNavigation and shared product flows
- Technical keywordsLocators, waits, input and clicks
- Browser / PlaywrightOperate the lab website from a dedicated test host
- Run artifactsXML / HTML, screenshots and videos
- Result classificationCount passed, failed and skipped tests separately
- QA review and handoverReview evidence against cases before QA maintenance
- Specification scenarios, executable scripts and a single run are different counts. A Gherkin scenario does not establish execution.
- This flow runs on a dedicated test host and is not yet integrated into CI. AI agents assist development and maintenance; reports and artifacts establish the results.
My scope
Built, verified and handed over the suites, and coached QA to use AI agents.
Approach
- Background: three attempts in 2021–2023 (Squash, Selenium, Robot Framework with Appium) could not be maintained while support work came first and versions changed fast.
- In 2026 I rebuilt it with Claude Code on dedicated test hosts; self-healing locators absorb minor UI changes.
- To keep the agent honest, every test must leave screenshot and video evidence.
Verification
The product QA reviewed screenshots, recordings and HTML reports item by item.
Results
| Metric | Value | Source | Evidence |
|---|---|---|---|
| Cloud product full regression (2026-07-02) | 541 cases in 2 h 23 min | Robot Framework report | System record |
| Same scope by hand | about 2 weeks | Own estimate | Own estimate |
| Related hours | 280.5 h | Timesheets, 2026 | System record |
| Logged hours split: me / AI execution | me ≈82 h, AI execution ≈199 h | Own estimate; hours since 2025-02-22 split by the period’s ratio; company-logged hours only | Own estimate |
Limits and next steps
Cases that need physical phones, hardware or special environments were not part of this automated run. A full regression produces about 5 GB of screenshot and video evidence; it runs on a dedicated test host and is not yet in CI.
Tools
- Robot Framework
- Playwright
- Gherkin
- Claude Code