5 Questions to Ask Before You Buy an AI Tool

In the last post I covered the basics — what AI actually is, and the mental shift from traditional software to AI-native software. This one's the practical follow-up: the questions I tell every distributor and manufacturer to ask before they hand a vendor their data, and the red flags that mean you should walk.

This is drawn from the same West Coast LBMA session I did a couple weeks ago — the version I wish someone had handed me the first time a vendor sat across the table.

First: figure out if you're using, building, or buying

Before you evaluate any vendor, it's worth knowing which of three lanes the job actually falls into:

• Use an off-the-shelf model (Claude, ChatGPT, Gemini) for individual, one-off tasks where you're comfortable reviewing the output yourself. Cheap, instant, no setup.
• Build your own tool with AI, when it's something internal and lower-stakes — a calculator, a tracker, a dashboard. Describe what you need in plain English, get working software back. This lane barely existed two years ago and it's now genuinely useful — I built a sales commission calculator for my own team in about ten minutes, for zero dollars, just by describing what I wanted.
• Buy a vendor system when the job is recurring, mission-critical, and depends on your real data — orders, inventory, pricing. This is where the five questions below actually matter.

A lot of businesses skip straight to "buy" for things that could just be built internally in an afternoon, and skip "build" entirely because they don't realize it's an option now. Know which lane you're in before you start evaluating anyone.

Security, in plain language

Once you're in "buy" territory, security stops being an abstract concern and becomes the first thing you actually need answers on. A few concepts worth having straight:

• Is your data training someone else's model? Free consumer AI tools often use what you input to train their models. Paid tiers usually restrict this — but policies change, so don't assume; ask directly and re-check periodically.
• Zero data retention (ZDR). Some vendors route your request through a model that executes the task and then purges the memory entirely, as if it never happened. This is what you want for anything touching pricing, customer data, or supplier terms.
• Cloud vs. private deployment. Cloud means your data leaves your building and lives on someone else's servers (AWS, Azure, GCP) — which is fine, and how most modern software works, but it means the vendor's security practices are now your security practices too.
• SOC 2 compliance. This is close to an industry litmus test for whether a vendor takes data security seriously. If they can't produce a report, that's worth noting.

None of this means avoid AI, or avoid the cloud, or lock everything down. It means know what you're agreeing to, and set one internal policy so your team isn't each making that call individually. Mine is simple: don't paste customer data, pricing, or supplier terms into a free consumer AI tool. Pay the twenty dollars a month, and know what your enterprise tools already protect you on.

The five questions to ask every vendor

Print this list, keep it by the phone, use it every time someone pitches you:

1. Where is my data stored, and who has access to it?
2. Is my data used to train your model — or anyone else's?
3. How does your tool integrate with the systems I already run?
4. What does implementation actually look like — time, cost, disruption?
5. Show me a customer in distribution or manufacturing who's been live 12+ months.

That last one matters more than it sounds. A demo is a controlled environment. A reference customer running your messy, real-world data is a completely different thing.

Four red flags

• They can't answer the data questions clearly. Vague answers about where your data goes, or a "don't worry about it," aren't acceptable. Push for specifics.
• Suspiciously precise ROI promises. "You'll save 34% on labor" with no methodology behind it is a number someone made up.
• A long-term contract before any proof of value. A vendor confident in their product should be willing to prove it with a pilot first — 30 or 60 days, not a three-year lock-in on day one.
• A great demo, no live customers to reference. See above. A demo is staged. A reference is real life.

Adopt smartest, not fastest

The businesses that win with AI aren't the ones adopting it fastest — they're the ones adopting it smartest. That means empowering your people to do more, not trying to replace them; it means starting with the middle of the spectrum instead of chasing the overhyped end; and it means asking the questions above before you sign anything.

You don't have to move fast. You have to move informed.

Got AI questions of your own, or a vendor pitch you want a second opinion on? Drop them below, or reach out directly — happy to talk through what any of this looks like for your distribution or manufacturing business.

— Shawn