What actually works today

Three uses of AI are delivering real value right now:

  • Test generation: turning requirements, user stories or a URL into a first draft of structured test cases saves hours of blank-page work, with a human reviewing for intent.
  • Self-healing automation: when a locator changes, AI can adapt the script instead of failing the run, which cuts the maintenance that quietly kills most automation efforts.
  • Assisted analysis: triaging failures, spotting flaky tests and summarising results lets engineers spend their time on judgement, not janitorial work.

Where the hype outruns reality

'Fully autonomous testing' and 'no more testers' make good headlines and poor plans. Agents can execute and adapt, but they don't understand your business risk, your compliance obligations or what 'correct' means for your customers. Left unsupervised, they produce plausible tests that assert the wrong things, confidently. The teams winning with AI treat it as a fast, tireless assistant, not an unaccountable replacement.

The pattern is not new; generation just makes it faster. A typical example from our test case reviews: a test called 'Book payment' opens the customer account, enters an amount, clicks Submit and checks that the page title is 'Payment'. It passes, and the booking itself was never checked. Generated tests reproduce exactly this unless someone defines the expected business result. That decision (the test oracle) stays with people who know the business.

A new discipline: testing AI systems

The other half of the story is testing software that itself uses AI. That breaks a core assumption of traditional testing: determinism. The same input can yield different outputs, so 'expected result' becomes a range, not a single value. Testing here means evaluation sets and scoring rather than single assertions, checking for hallucination, bias and prompt-injection, and putting guardrails around what the model is allowed to do. ISTQB's syllabus for testing AI-based systems names the techniques: acceptance ranges instead of one expected result (the test oracle problem), metamorphic testing, which checks that related inputs produce consistently related outputs, and adversarial testing to find where a model breaks. It is a genuinely new skill set, and most teams are building it now.

How to adopt it pragmatically

  • Start where maintenance hurts. Self-healing and AI-assisted triage pay back fastest because they attack your biggest hidden cost.
  • Keep humans on strategy. Let AI draft and adapt; keep people on risk, coverage decisions and exploratory testing.
  • Measure, don't believe. Track escaped defects, maintenance effort and coverage before and after, and adopt what moves the numbers.

Our take

AI is the biggest shift in testing in a decade, and it rewards teams who are specific about where it helps. We use it where it earns its place (generation, self-healing, analysis), including agent frameworks that generate automated tests for the framework a team already uses (for example Selenium or Playwright) from the test cases in its test management, integrated into its CI/CD; more on the test automation page. And we help teams test their own AI features with the rigour those features now demand: see Testing AI systems. The goal isn't fewer testers; it's testers spending their time on the work only humans can do.