My Checklist Before I Let a New AI Coding Tool Anywhere Near a Client Project

There’s a new AI coding tool announced practically every week at this point — a new agent, a new IDE fork, a new CLI promising to make everything else obsolete. I used to try almost all of them, out of genuine curiosity and a nagging fear of missing whatever the actually-better tool turned out to be. That habit cost me more time than it saved, because switching tools mid-project has a cost that only shows up later: a different mental model for how context gets built, different conventions for where config lives, a workflow that doesn’t transfer cleanly. So now there’s an actual checklist I run before letting something new touch a client codebase.

Can it read my existing conventions without a rewrite?

Every project I maintain has some version of a CLAUDE.md or equivalent — architecture notes, naming conventions, the things a new contributor (human or agent) needs to know so they don’t immediately break the codebase’s patterns. If a new tool needs its own bespoke config format to get equivalent context, that’s not automatically disqualifying, but it’s a real cost I weigh. I’ve skipped adopting a tool purely because the cost of restating everything in a new format wasn’t worth what I’d gain.

Test it on a throwaway branch of a real project, never a toy repo

Demo videos and quickstart tutorials are built on codebases with no legacy weirdness, no half-migrated patterns, no fifty small decisions accumulated over two years. My actual projects have all of that. A tool that looks brilliant generating a fresh Next.js app from scratch tells me almost nothing about how it behaves when asked to add a feature to an existing 40,000-line codebase with its own quirks. So the test is always the same: pick a real, moderately annoying ticket, hand it to the new tool on a branch nobody’s depending on, and see what happens when the codebase pushes back a little.

What happens to my data, specifically — not what the privacy page says

For client work this isn’t optional. Some of my contracts have explicit clauses about where code can be sent, and “we use industry-standard security practices” on a landing page doesn’t answer whether a specific client’s proprietary logic gets used for further model training, retained past the session, or processed in a jurisdiction that matters for their compliance needs. I ask this directly, in writing, before code from a client repo ever touches a new tool. More than once the honest answer changed what I actually fed it — sanitized fixtures instead of the real thing.

Does it degrade gracefully, or fail silently?

This is the one I care about most, and the one demo videos never show, because demos don’t include the failure case. I deliberately give a new tool a task slightly outside what it’s obviously good at — an ambiguous requirement, a file with an unusual pattern — and watch what it does when it’s uncertain. Some tools ask a clarifying question or flag their own uncertainty. Others confidently produce something plausible-looking and wrong, with no signal they’re guessing. The second category costs me the most hours, because confident-wrong output is exactly the kind of bug that slips past a quick review.

Can I actually roll back?

Some tools want to own more of the workflow than I’m comfortable handing over on day one — their own branch management, their own commit conventions, their own way of structuring a PR description. I check: if I use this for two weeks and decide to walk away, is my repo in a state any other tool can pick up cleanly, or has it planted assumptions only it understands? A tool that’s genuinely good doesn’t need to lock me in to prove its value.

Cost, measured against what I’m already paying — not against zero

Every new tool’s pricing page compares itself to doing the work by hand, which isn’t the comparison that matters to me. I’m not choosing between this tool and no AI assistance — I’m choosing between this tool and the one I already use, with workflow habits already built around it. The bar for switching isn’t “better than nothing,” it’s “better enough to justify relearning my own habits.” That’s the biggest reason I’ve stuck with the same core toolset for months while trying plenty of others on the side.

Two tools this year looked genuinely impressive in their own demos and fell apart on step two — real codebase, real mess — within an hour of actual use. That failure wouldn’t have shown up if I’d judged them the way most people judge a new tool: watching someone else’s polished five-minute demo and extrapolating from there. The demo is the tool at its best, on a codebase built to make it look good. This checklist is just an attempt to see it closer to its worst, before betting real client hours on it.

None of this makes me anti-new-tools — I still try most of what gets released, because the field moves fast enough that standing still has its own cost. It just means trying isn’t the same as adopting, and the gap between those two used to cost me more time than I realized.