Testtalks | Automation Awesomeness | Helping You Succeed With Test Automation

I asked thirteen people inside AI testing one question: can AI test your software for you? Founders, CTOs, project leads, and quality leaders, many of them selling AI testing tools right now. Not one said yes. This episode is my session from the Testear.la 2026 conference, built entirely from twelve months of interviews on this show with the people actually building this technology, including Applitools, Maestro, Bright Security, QA.tech, Cucumber, Ito, and the lead of the Selenium project. In it you'll learn why one AI testing vendor deliberately removed AI from the part where tests run, the real productivity number teams are measuring versus the number in the case studies, the four incompatible architectural bets vendors are making and how to tell which one your tool has chosen for you, why AI generated code is producing vulnerabilities far faster than it produces code, and the one architecture rule that stops an agent from editing the test that was supposed to catch it. You'll also get the list. Every task

Is AI going to replace software testers, or make them more valuable than they have ever been? Jonathon Wright says both, and in this episode he does not back down from either position. Jonathon Wright has spent thirty years in test automation, going all the way back to WinRunner. He now runs thirty AI agents across twelve screens, he sat with Gartner for their research on agentic platforms, and he says the last six months changed the job more than the previous three decades combined. We get into what he calls confidence engineering, why he has not opened a browser in five years, and what happened when he generated thirty seven thousand test scenarios from space agency specifications and put them in front of NASA. He also tells the story of a testing leader who taught his agents everything he knew about accessibility work and then got let go. A few minutes later he tells you your career has another fifteen good years in it. I pushed him on the contradiction. Listen for how he answers it. What you will learn in

Everything you knew about performance testing changes when the system you're testing is non deterministic. In this episode of the TestGuild Automation Podcast, Joe Colantonio sits down with Kandasamy Selvaraj, Principal Architect and author of the free book Rethinking Performance Engineering for Agentic AI, to unpack what it really takes to move an AI agent from a working demo to an enterprise system handling millions of conversations per hour. Checkout his free book: https://leanpub.com/agentic-ai-performance Kandasamy shares the practical playbook he's built running agentic AI in production, including why the same request can take three seconds one run and eight seconds the next, how to use harnesses to bound tool calls, reasoning loops, and token budgets, and why your SRE dashboard can look perfectly healthy while your token costs quietly balloon to five times baseline. You'll learn how his team: Shifts performance gates left into every commit with JMeter Shifts right with synthetic monitors on blue green

Most testers are stuck arguing about whether AI is coming for their jobs. Swati Seela thinks that's a distraction from a much more useful question: when we say "AI testing," what do we actually mean? What do you think? Take our TestGuild State of Test Automation Survey now: https://testgld.link/27data In this episode, Swati walks through the AI Testing Landscape, a practical framework she presented at CAST 2026 that breaks AI testing into five overlapping areas: AI assisted testing, AI evaluation and trust, AI development quality, testing products that contain AI functionality, and testing the AI models themselves. Each one has a different test intent, a different technique, and a different toolset, which is exactly why lumping them together leaves new testers with no idea where to start. Along the way Swati gets specific about the failure modes she keeps hitting in real work. Why AI opens almost every response by agreeing with you, and why that false sense of correctness is the modern version of a green das

Manual testing never died. IoT is the proof. In this episode, Joe Colantonio talks with Oleksii Cherkashyn, a QA team lead, automation engineer, and IoT testing specialist who built a complete test architecture from scratch across hardware, firmware, mobile, web, and API. IoT breaks the usual automation playbook. You are not testing a shopping cart. You are connecting a real microcontroller, flashing a sketch, validating that a physical command produced the right change in a dashboard widget, and then doing the same thing again through the REST API and the mobile app. No marketplace framework covers that, so Oleksii built his own. You'll learn: Why manual QA remains essential in IoT and which cases can never be automated How Oleksii built a custom Node.js library to simulate up to 50,000 device connections for performance testing without buying the hardware Why he chose WebDriverIO over Playwright and the mobile automation reason behind it How he uses WebDriverIO MCP and cloud coding agents every day to crea

In this episode 600 of the TestGuild Automation Podcast, Joe Colantonio talks with Jason Arbon, founder of Testers.ai, Jank.AI and IcebergQA and author of the new book Testing AI: Engineering Confidence in Non-Deterministic Systems. Take Our 2027 Survey Now: https://testgld.link/27data Jason makes a case most testers have not heard yet. Coding is being absorbed by AI. Specification work is thinning out. Product, development, and test roles are converging into one. And when the music stops, the only seat left belongs to the person who can look at what the machine produced and make an evidence backed call on whether it ships. He calls that confidence engineering, and he argues it is not a rebrand of QA. It is what QA was always supposed to be. Along the way, Joe and Jason get into the containment problem and why alignment, not lockdown, is now the real safety goal. They dig into why testing cost scales quadratically, meaning ten times more generated code creates roughly a hundred times more testing demand. Jaso

Your developers just got supercharged by AI coding agents. Your test coverage did not. So how do you keep quality high when product code is shipping faster than any test team can follow? In this episode Andrew Knight, the Automation Panda and Senior Director of Product and Engineering at Cycle Labs, shares how his six person team is running the biggest release quarter in company history using AI coding agents, spec driven development, and Playwright. You will discover: How Playwright turned itself into an AI automation platform with the MCP server, the planner, generator, and healer agents, and the new CLI and skills approach that cuts your token usage way down. Why Andy uses Spec Kit to codify his testing strategy once, in markdown, so quality standards get baked into every single pull request instead of being caught in review. How to decide which AI generated tests are actually worth running when compute time and budget are finite. What AI slop looks like from a manager's seat, and how to build a team cult

Keith Klain has spent 25+ years leading enterprise quality programs in financial services and is one of the testing industry's most respected voices. In this episode, he joins Joe to discuss his BrowserStack talk, "The Death of Test Engineering," why testing isn't dying—but outdated testing roles are—and how AI is reshaping the future of software quality. In this episode: Why "The Death of Test Engineering" isn't what you think The difference between checking and testing Will AI replace software testers? Why confidence engineering matters How to communicate testing in business terms Where automation adds real value Why agentic AI testing is overhyped Notable Quotes "If you look at AI as a competitor, you're screwed. If you look at it as just another tool... it's not a threat." "Uncertainty in testing is a feature, not a bug." "If an organization doesn't value testing, you're doing it wrong." "You can be incredibly confident about something and incredibly wrong." One Thing to Try This Week Ask yourself

Nandini Srinivasan has spent 25 years in the quality industry and leads a global QA organization of over 150 engineers across the US, Canada, India, and Pakistan. In this episode, she breaks down exactly how she built a QA AI acceleration charter, ran a train-the-trainer model, and used a phased proof-of-concept approach to separate real AI from what she calls "powerful automation dressed up as AI.  We get into: How she frames quality metrics for executives Using language around revenue protection Risk mitigation Feature velocity instead of test coverage percentages. She talks about the four pillars she uses to present her team's value: quality, scalability, performance, and availability. Nandini also shares her take on the future of QA hiring, why the judgment layer will always require a human, and what skills testers need to stay relevant as AI agents take over the more mechanical parts of automation. She is also writing a five-part LinkedIn series called "The Voice of QA in the AI Era" if you want to fol

Matt Wynne, co-creator of Cucumber and BDD practitioner, joins Joe for the first time in over a decade to talk about what two years inside a Silicon Valley AI startup taught him about the future of software testing. Matt spent time at Mechanical Orchard working alongside experienced XP practitioners to modernize legacy COBOL mainframes using LLMs, and then spent a week with the team that coined the term "software factory," where the rule was simple: humans never write the code, never read the code. In this episode, Matt breaks down what harness engineering actually means, why shared understanding is still the real bottleneck even in an agentic world, and how testers can use multiple LLMs to review AI-generated pull requests without reading every line. He also gets honest about the grief that comes with realizing you can encode years of hard-won expertise into a Markdown file, and why that does not mean your skills are worthless. If you are working in a brownfield codebase, wondering how to handle the flood of

Amit Rawat is an agentic engineer who spent two decades in QA before shifting fully into building AI agents. He's the creator of PromptWright, a desktop tool that turns natural language prompts into automated Playwright browser tests, complete with screen recording, Gherkin scenario generation, and self-healing locators. In this episode, Amit and Joe get into what it actually takes to work with AI agents at a high level, starting with why the planning phase matters more than the prompt itself. Amit breaks down his own workflow, brainstorming with AI, building a detailed plan in HTML before ever executing, and why curiosity and technical depth still matter even as AI gets more capable. They also cover why Amit believes QA professionals, more than developers or DevOps engineers, are best positioned to thrive in the agentic era, how he tracks the ROI on his $200-a-month Claude subscription, the "Chief of Staff," "Chief Health Officer," and "Chief Financial Officer" AI agents he's built to help manage different a

Most API testing stops at the happy path. The problem is that the bugs that actually hurt you in production are sitting in everything many testers skip, like the boundary values, the oversized payloads, the missing tokens, the security headers, the inputs that make no sense at all. In this episode, Joe sits down with Liudas Jankauskas, who has spent almost twenty years breaking software and testing APIs since 2008. Liudas demonstrates Rentgen, his free and open-source API testing tool, live on screen. You'll watch him take a single request from a real app, map it in seconds, and generate dozens of tests covering security, boundaries, performance, and load—all from one click. You'll learn: How to discover APIs hiding under the hood of any application, even when there is zero documentation Why happy path testing leaves you exposed How to run a fast hygiene check before your real automation ever starts Liudas also explains why Rentgen runs completely locally with no server and no data leaving your machine, mak

Your AI code review tools read the diff. They stare at your code. But they never actually run it. So the bugs that only show up at runtime, the broken user flows, the bad query plan, the duplicate submission, sail right past review and land in front of your customers. In this episode, Joe Colantonio sits down with Evan Marshall, founder of Ito and a fifteen year engineer who spent five years in applied cryptography securing hundreds of millions of dollars for millions of people. Evan is taking that ship fast without breaking things discipline and pointing it straight at testing. Ito is an agentic QA platform that builds and runs your actual app on every pull request, navigates it like a real user, exercises the frontend and backend as one system, and brings back real runtime evidence: video replays, logs, the exact lines responsible, and steps to reproduce, posted right in your PR. You will learn: Why static code review misses the bugs that cause real production incidents How Ito spins up ephemeral environme

What happens to QA when AI is writing ten times more code than your team can test? That is the exact problem Ivan Barajas Vargas set out to solve with Amikoo, a purpose-built AI QA agent designed to help testers, SDETs, and even developers move faster without sacrificing coverage or quality. Ivan is no stranger to AI in testing. Before generative AI became mainstream, he co-founded MuukTest, a test automation platform built on symbolic reasoning and expert systems. After six years and thousands of customer conversations, he went back to first principles to build Amikoo from scratch, this time with a harness of 12 specialized agents and 43 tools trained specifically for testing workflows. In this episode, Ivan and Joe dig into the real-world gap between AI code generation and AI-powered testing, why the QA role is being elevated rather than replaced, how Amikoo uses Playwright and page object model patterns under the hood, and where human judgment still has to stay in the loop. Ivan also shares practical advic

Everyone is talking about AI replacing testers, writing tests, and transforming software quality. But what if we're asking the wrong question? In this solo episode, Joe Colantonio shares a growing concern he's seen while traveling across the country for TestGuild IRL events: a decline in testing fundamentals at the exact moment AI hype is reaching a fever pitch. Drawing insights from Carissa Véliz's book Prophecy: Prediction, Power, and the Fight for the Future, Wayne Roseberry's work on AI and meaning, and Tariq King's concept of Human Experience Testing, Joe explores why AI systems may be far less intelligent than many believe, and why human testers remain more important than ever. You'll discover: ✅ Why large language models generate plausible answers without understanding truth ✅ The difference between prediction, correlation, and genuine understanding ✅ Why AI can test software but cannot experience software ✅ What "Everything is tested, but nothing is experienced" really means ✅ How AI hype may be distr

AI coding tools promised to make development faster — and they delivered. But here's the problem nobody talks about enough: when you speed up coding, you don't eliminate the bottleneck in the SDLC. You just move it. And for most teams, it lands squarely in QA. In this episode, Joe sits down with Vilhelm von Ehrenheim, Co-founder and Chief AI Officer of QA.tech, to dig into how agentic AI is reshaping software testing from the ground up. Vilhelm brings serious ML credibility, he helped build Motherbrain, one of the earliest production LLM systems in venture capital, and he's now applying that experience to one of the hardest problems in software delivery: testing at AI development velocity. You'll learn how QA.tech's behavioral knowledge graph gives AI agents the context they need to actually understand your application, why validating user intent beats checking element identifiers every time, how autonomous agents can review PRs, reproduce bugs from Slack messages, and generate targeted tests without a single

What happens when AI agents can not only write mobile app code, but also validate their own work automatically? In this episode, I sit down with Maestro Co-founder and CEO Leland Takamine to explore one of the biggest shifts happening in software testing right now: agentic mobile testing. Leland shares how his team went from solving mobile performance testing challenges to building one of the fastest-growing mobile automation frameworks used by companies like Microsoft, Meta, Amazon, and DoorDash. We dive deep into: How AI coding agents are changing mobile testing workflows What "closing the agentic feedback loop" actually means Why deterministic testing still matters in the age of AI How Maestro MCP lets AI agents validate mobile app changes automatically Why mobile test maintenance may finally become manageable The future role of testers as AI-generated code explodes Leland also gives a live demo showing an AI agent building, validating, debugging, and generating a reusable mobile test completely autonomo

AI-powered testing tools are exploding across software engineering teams… but so are the hidden costs. In this episode, Joe sits down with Arthur Hicken to unpack the growing problem of runaway AI token usage, unexpected LLM billing, and the operational risks of deploying AI agents into testing and DevOps pipelines. Inspired by Arthur's article on the emerging "Token Tax," this conversation explores why many teams are underestimating the true cost of AI automation. You'll learn: Why AI-generated testing can create unexpected scaling costs How runaway AI agents and infinite loops happen Real-world examples of massive AI billing surprises Why deterministic problems shouldn't always use LLMs The hidden risks of "vibe testing" and autonomous AI remediation How QA teams can monitor, test, and control token usage Why performance testing and service virtualization matter more than ever in AI systems Practical strategies to avoid expensive AI deployment mistakes Whether you're a software tester, automation engine

AI-powered testing tools promise faster automation and less maintenance, but most require teams to abandon their existing frameworks. In this episode, we explore Alumnium, an opensource AI-native end-to-end testing solution created by Alex Rodionov, an engineer at Airbnb and a tech lead on the Selenium project. Instead of replacing tools like Playwright or Selenium, Alumnium adds an AI layer on top, helping teams: Reduce test maintenance by removing brittle locators Build more resilient, self-healing tests Write less code while increasing coverage Run tests across web and mobile using intent-based steps We also go beyond the hype and break down what actually matters for real teams: Why AI-driven tests can still become flaky The performance and cost tradeoffs of LLM-based execution What "context rot" is—and how it impacts long test runs How to safely introduce AI into existing test suites without rewriting everything Check it out now: https://testguild.me/alumAI

AI is changing how we build and test software, but most teams are still struggling to turn AI-generated tests into real production value. Use code TESTGUILD3 try for yourself free now for 3 months: https://links.testguild.com/Kobiton In this episode, we break down what actually works when it comes to AI-powered mobile test automation, especially when running tests on real devices not simulators from Claude. You'll learn: How teams are generating and running Appium tests using natural language in minutes Why AI-generated tests often fail—and how to avoid costly false positives The real impact of AI on test automation roles and responsibilities How real device testing exposes issues AI alone can't catch Practical ways to reduce test maintenance while increasing coverage We also explore a major shift happening right now: AI is making it easier to create tests—but dramatically increasing the volume of code and risk that needs to be validated. That means one thing: Testing isn't going away—it's becoming more cr