Can AI Replace Your Email Marketing Agency? Here’s What It Got Right

A founder booked a call with me last week and brought a list of about twenty questions. She hadn’t written them. Her AI had.

I’ll be honest, my first reaction was to be a bit prickly about it. My second reaction, about four questions in, was that this thing was asking better questions than most agencies ever get asked.

Quick answer: AI can already do a big chunk of what an email agency charges for. Campaign ideas, calendars, copy, subject lines, segmentation logic, even reading your Klaviyo exports. What it can’t do is decide what’s worth sending, spot the technical problem quietly costing you money, or be accountable for a number at the end of the month. The output got cheap. The judgement didn’t.

I’ve worked on Klaviyo accounts for over 125 brands, and I ran my own product-based store before that, so I’ve been the founder staring at a dashboard with no idea what’s actually working. Here’s what her AI got right, where it overreached, and the questions I think every founder should be asking before they hand over a retainer.

Was the AI actually right about what it can do?

Mostly, yes, and I’m not going to pretend otherwise.

Give a decent AI a good brief and it will hand you a campaign calendar, ten subject line options, a segmentation plan and a readable summary of your Klaviyo export in about a minute. Two years ago I charged for a good chunk of that.

Her AI wrote it plainly: a lot of what’s in a typical retainer can be produced with AI. It was right.

Where it stopped being right was the assumption that producing the work and running the account are the same job.

An account is not a content plan. It’s a live system with a Shopify integration behind it, deliverability that moves when you make a mistake, flows that skip people because a filter is set wrong, and a list that costs you money every month whether you send to it or not.

What was the AI’s sharpest question?

This one, and it’s the question I’d want asked of me every time:

“Will you measure incremental improvement, rather than simply reporting total Klaviyo-attributed revenue?”

That is a genuinely excellent question, and it’s the one that separates real reporting from a screenshot of a big number.

Attributed revenue is the revenue Klaviyo claims credit for when someone opens or clicks an email and then buys within a set window. Incremental revenue is the part that exists because of the work, measured against a baseline. They are not the same thing, and only one of them proves anything.

Total attributed revenue goes up on its own when your traffic goes up, when it’s your busy season, when you run a sale you were going to run anyway.

Here’s what that actually looks like in an account. If Klaviyo revenue drops 20% but your store sessions dropped 25% over the same period, email didn’t get worse. Your traffic did. Most of your flows are triggered by website behaviour, so when fewer people visit, fewer people enter the flows. That’s a Shopify story, not an email story, and any agency reporting to you should be able to tell you which one it is.

If your monthly report never mentions your store sessions, it isn’t a report. It’s a highlight reel.

Should you be honest about your attribution settings?

The founder on that call had already done something most brands haven’t. She’d removed bot clicks from her attribution.

Bot clicks are exactly what they sound like. Security scanners and spam filters click every link in an email before it reaches the inbox, and Klaviyo counts those as real human clicks unless you tell it not to.

Her attributed revenue fell from 25 to 30% of total revenue down to about 15% overnight.

Her words were that it was a bit sad. My answer was that nothing had actually changed. The 25 to 30% was never real. What she had now was a true baseline, which is worth more than a flattering one.

This matters if you’re comparing agencies. If one quotes you a benchmark built on inflated numbers and another quotes you one built on clean numbers, you’re not comparing the same thing at all.

The rule I work to is simple. Whatever the settings are, they don’t change month to month. Bot clicks in or bot clicks out, Apple opens counted or not, pick one and keep it, so every comparison is like for like. Changing the settings and then celebrating the lift is not reporting, it’s decorating.

Before your next call with any agency, open Klaviyo and write down what your attribution settings are actually set to. If you don’t know, that’s job one. A Mini Klaviyo Audit is $69 and will tell you where your numbers really sit, and what a Klaviyo audit actually covers if you’ve not had one before.

What did the AI get wrong about testing?

Her AI told her to expect more than subject line tests, and to be disappointed if “deeper testing” meant nothing else. Fair warning in general. Wrong for her account.

Her open rate is around 70%. Her click rate is under 1%.

When your open rate is already that high, testing subject lines is the least useful thing you could possibly do. The subject lines are working. The problem sits after the open.

So in an account like that, the testing goes to the click. What the button says. How long the text on it is. Whether it’s square or rounded. Where it sits in the email, up top or further down. Whether it’s a “buy now” or something that names the actual product.

That sounds trivial until you watch it move. I’ve got a client with a very bright palette where we tested three different button colours across her flows, and different flows wanted different colours. Same brand, same audience, different answer depending on the flow. You would never guess that. You’d only find it by testing it in that specific account.

Generic testing advice is easy. Knowing which test is worth running in your account this month is the actual skill.

Click rate is where the money is. For an account sitting under 1%, the first step is getting above it, then pushing towards 1.5%. Once it holds there, that’s when increasing send frequency makes sense, and not a moment before. Worth knowing where the wider market sits too: Klaviyo’s 2026 email benchmarks put the average campaign click rate at 1.69% and flows much higher, so a staged ladder is about getting your account moving, not about calling 1.5% the finish line.

Can AI handle segmentation for a seasonal business?

This is where her AI was genuinely useful, and where it also showed its limits.

It flagged that her customers don’t buy on a normal cycle. In her category, someone buys once for a specific occasion and may never buy that item again. Some customers buy annually. Some come back two years later for a completely different product and size.

Her AI was right that standard 30, 60 and 90 day engagement segments will misread that customer completely. Someone who buys annually looks lapsed at day 120. They aren’t. They’re right on schedule.

That’s a good catch. Here’s the part the AI couldn’t do.

RFM analysis (recency, frequency, monetary, the method that sorts customers into cohorts like loyal, at risk and lost) is where you find the real cycle. I pull it from Shopify rather than Klaviyo, for two reasons. Everyone has access to it in Shopify, and Shopify’s version accounts for discount codes where Klaviyo’s doesn’t.

On one account, that report showed customers were placing a first order, then leaving a very long gap, then orders three, four and five came quickly. The whole problem was the gap between one and two. So we built a flow specifically to accelerate that second purchase. You can see how that kind of lifecycle work plays out in a lifecycle flow build for an accessories brand.

An AI can tell you RFM exists. It can’t tell you your gap is between order one and order two.

What should you actually ask an agency?

Steal these. The best of her list, plus the ones I’d add.

• Will you report incremental improvement, or just total attributed revenue?
• What attribution settings do you use, and will they stay the same every month?
• What will you baseline before you start?
• What will you test beyond subject lines, and why that test first?
• Does “two campaigns a week” include strategy, segmentation, copy, design, build and scheduling? And do segmented versions of the same email count as extra campaigns? (They shouldn’t. Same email to five segments is one email.)
• Who actually does the work?
• If I stop, what stays in my account?
• Am I locked in?

That last pair matters more than people realise. Everything built in your Klaviyo account is yours. The flows keep running whether or not anyone is still on retainer. If an agency is cagey about that, you’ve learned something.

The numbers worth remembering

• Email should be doing 25 to 40% of your total revenue. Most accounts I open are sitting well below that.
• One founder’s attributed revenue dropped from 25 to 30% down to 15% the moment bot clicks were removed. The lower number was the honest one.
• Changing a pop-up offer from 10% to 15%, with nothing else altered, took the submission rate from 2.2% to 3.6%.
• A healthy pop-up submission rate is above 3%.
• Klaviyo’s 2026 data puts the average campaign click rate at 1.69%.

Quick glossary

• Attributed revenue: revenue Klaviyo claims credit for when someone opens or clicks an email and then buys within a set window.
• Incremental revenue: the revenue that exists because of the work, measured against a baseline you set before starting.
• Bot clicks: automated clicks from security scanners and spam filters, counted as human engagement unless you exclude them.
• RFM analysis: recency, frequency and monetary scoring that sorts your customers into cohorts such as loyal, at risk and lost.
• Flow: an automated email sequence triggered by behaviour, such as a welcome series or abandoned cart.

Key takeaways

• AI genuinely can produce campaign ideas, calendars, copy, subject lines and segmentation logic. That part of the work got cheaper and pretending otherwise is silly.
• Total attributed revenue proves nothing on its own. Ask for incremental improvement measured against a baseline, with store sessions in the picture.
• Clean up your attribution settings before you benchmark anything, then leave them alone so every month compares like for like.
• The right test depends on your account. High opens and low clicks means the subject line isn’t the problem.
• Seasonal and annual buyers break standard engagement segments. RFM analysis from Shopify shows you the real buying cycle.
• What you’re paying an agency for is the deciding, the implementing and the accountability, not the typing.