Skip to content
echogeco
AI in the operation · Tooling

Tooling literacy without vendor worship

Place a marketing tool in one of four capability classes, then judge it in a week on your own work: baseline, cost per usable output, failure sample.

Lesson · 7 min · 48 of 60
On this page

Nearly every marketing tool now claims AI, and every demo shows its best output on data the vendor chose. What decides a purchase is the tool's cost per usable output on your own work, against what that output costs today, and the way it fails. Products change faster than lessons, so this one names them only as instances of four capability classes, each linked to its vendor's documentation, and teaches the week-long evaluation that outlasts them.

Four capability classes

ClassWhat it producesThe unit to priceThe failure to sample
Generationcopy, images, ad variantsa usable asset after reviewa claim nobody can substantiate
Enrichmentfields on contact and company recordsa correctly filled recordno match, or a wrong match
Orchestrationsequences of actions fired by triggersa correct sendthe wrong records enrolled
Analysisscores, attributions, auditsa decision it changeda number from someone else's checklist

Generation. General-purpose models are one instance. Anthropic's guide Increase output consistency (opens in a new tab) tells developers to make Claude's output consistent by specifying the format precisely, constraining it with examples and grounding it in retrieval: the vendor's own documentation treating inconsistent output as something to manage. Ad platforms are another instance: Meta's Get Started with Advantage+ Creative (opens in a new tab) describes opt-in creative features, including one that generates alternative backgrounds for eligible product images. Whoever generates the asset, the review that catches an unsubstantiated claim stays yours.

Enrichment. HubSpot's Get started with data enrichment (opens in a new tab) says contact enrichment uses first name, last name and work email address, and company enrichment the domain. The failure to sample follows: a lead who signs up with a personal address gives it no work email. Measure the match rate on 100 of your own leads before trusting any coverage figure.

Orchestration. Klaviyo's Getting started with flows (opens in a new tab) defines a flow as a sequence of automated actions, such as sending messages, triggered by a behaviour or an event; HubSpot's Create workflows (opens in a new tab) describes enrollment triggers that add records automatically, followed by actions such as sending an email. A trigger that enrolls the wrong segment sends to all of it, and a sent email cannot be recalled, so the permission ladder from agents in marketing operations applies to every step.

Analysis. HubSpot's Determine likelihood to close with predictive lead scoring (opens in a new tab) defines its score as the percentage probability that a contact becomes a customer within 90 days, from a model it calls a black box. That is a calibration claim, and predictive scoring shows how to check one on your own history. Google Ads Help's About data-driven attribution (opens in a new tab) says the model compares the paths of customers who convert with those who do not, across interactions with Google's own Search, YouTube, Display and Demand Gen ads; it makes no claim about a podcast or a shared link. Semrush's Site Audit Overview Report (opens in a new tab) describes its Site Health score as errors and warnings found relative to the checks performed: a measure of the vendor's checklist, not a forecast of traffic.

The three model facts the course needs

Three facts about language models explain most of what goes wrong with these tools; each is taught elsewhere. A call costs in proportion to the text in and out, and an agent makes many calls per task, so a price per seat can hide a cost per output: where the money goes. The only way to know whether a tool is good at your task is a fixed set of your own examples with a stated way to mark them: evals as the unit test of LLM work. And the same input does not always give the same output, and a vendor's model update changes it with nobody on your side touching anything: sampling and determinism.

The mistake: choosing on the demo

The usual purchase runs on a demo and a feature grid. The demo is the vendor's best output on its chosen input; the grid counts features instead of pricing the one task the team needs done. Neither contains your baseline, so neither can say whether the tool saves anything.

Evaluating a tool in a week

Worked on a generation tool for paid-social ad variants, at the course's $100 marketing hour.

Day 1: the task and the baseline. Write the task as one sentence with its unit: "a headline-and-body variant for a paid-social ad, publishable after review." Then cost it today. The team writes 120 variants a month at 20 minutes each, 40 hours, $4,000: $33 a variant.

Days 2 to 4: the run. Put 50 of last month's real briefs through the tool, and time the person who would review them anyway. Assume the subscription comes to about $4 a variant at this volume. Review averages 6 minutes, $10 a variant. 35 variants publish after edits; 15 are rejected and rewritten by hand at the old 20 minutes.

Day 5: cost per usable output and the failure sample.

review       50 x 6 minutes          = 300 minutes   $500
rewrites     15 x 20 minutes         = 300 minutes   $500
tool         50 x $4                                 $200
total for 50 publishable variants                    $1,200   =  $24 a variant
by hand                                                          $33 a variant
saving       120 a month x $9                                    about $1,080 a month

A real saving, about a quarter, not the tenfold the demo implied. The tool keeps paying until about 58% of variants need a full rewrite: $10 of review + $4 of tool + 58% of $33 ≈ $33. Then sort the 15 rejects: 6 made a claim the company cannot substantiate, 5 missed the brand's tone, 4 got a product detail wrong. The six are the expensive class, because an unsubstantiated claim that reaches a live ad is a legal problem rather than a wasted 20 minutes; they are why review time never falls to zero, however good the tool becomes.

The week ends in two numbers, $24 against $33, and a sorted list of how the tool fails.
100%SVG (opens in a new tab)

Pinch to zoom · drag to pan

The decision, and its expiry. Buy, and keep the 50 briefs with their marks as the tool's evaluation set. Re-run them every quarter, and whenever the vendor announces a model change, because the $24 was measured on this quarter's model; past about 58% rewrites, the tool stops paying.

In practice

To a vendor: "Run it on 50 of our briefs; we will count the usable ones and time the review." One who insists on its own examples has told you what the trial would show. Ask whether model changes are announced, which actions the tool takes without a person, and whether your data and the action log can be exported. To the team: one page with the task sentence, the baseline, cost per usable output, the failure classes and the re-test date.

The mistake that costs money is buying on the demo. The demo implied $33 a variant falling to a few dollars; a week on the team's own briefs found $24, with six unsubstantiated claims in 50 that still need a person to catch them.

Check yourself

1. A second generation tool scores better in the vendor's demo and costs twice as much, about $8 a variant. In your 50-brief trial it gets 40 publishable after edits and 10 rewrites, with the same 6-minute review. Which tool do you buy?

Cost for 50: review $500, rewrites 10 × $33 ≈ $333, tool 50 × $8 = $400, total about $1,233, so about $25 a variant against the first tool's $24. Buy the cheaper one unless the second's rejects contain fewer unsubstantiated claims, since that is the failure that costs more than time.

2. An enrichment tool claims to fill company size on most records. In a sample of 100 of your leads it matches 62, and 30 of the 38 misses signed up with personal email addresses. Is the tool the problem?

Mostly not. Enrichment documented as working from name and work email has little to match on for a lead who gave only a personal address; of the 70 with a work email, the tool matched 62, about 89%. The cheaper fix is upstream: ask for a work email on the form, then re-measure the match rate before comparing tools.

3. Three months after purchase, the same 50 briefs produce 25 rewrites instead of 15. What do you do?

Re-cost it: $10 + $4 + 50% of $33 ≈ $31 a variant, still under $33 but most of the saving gone, heading for the 58% line. Ask the vendor what changed in the model, re-run next month, and keep the by-hand process ready.

Sources: each page read on 2026-10-18.