Why AI Agents Are Only as Effective as the Work You Give Them

Introduction

AI Agents are everywhere right now – in headlines, product demos, boardroom conversations. You understand why. The idea is intoxicating: software that doesn’t just answer your questions but can do the work for you.

And yet, depending on which headline you read, they’re either transforming business as we know it…or delivering absolutely zero return on investment.

Case in point: a recent MIT report found that a staggering 95% of generative AI pilot projects failed to deliver any measurable ROI at all. Ouch! But that stat (even if partially true) doesn’t mean the technology is broken. It means we’re still learning how – and when – to use it effectively. And in the meantime, the appetite for AI efficiency hasn’t abated. A recent PYMNTS survey found that 85% of executives now expect AI to meaningfully improve compliance operations. 

So, which AI agents headline do we embrace? Do agents bring disappointment…or exponential potential?

The truth is: it depends on what you’re asking the AI to do.

It’s Not Whether AI Works. It’s Whether It’s the Right Job for AI.

When people ask “Do AI Agents work?” what they’re really asking is, “Can I trust an Agent to do this job?” And the answer – as with any employee, tool, or intern – depends entirely on the task you’ve assigned it.

That’s where we’ve found some thinking by Cedric Chin and Vaughn Tan to be extremely helpful. Tan, whose Harvard Business School doctorate focused on organizational behavior and sociology, writes about “meaning-making” tasks as being (still!) a uniquely human endeavor. 

Between the two of them, we can create a conceptual distinction between two kinds of tasks:

• Meaning-reducing tasks: These are repetitive, well-understood processes. They require consistency, not creativity. The goal is already clear, and the work is about getting to that goal faster.
  
  
• Meaning-making tasks: These require context, nuance, and open-ended thinking. The goal might shift along the way, and human judgment is central to doing the work well.

What purpose does this distinction serve? Simple. By separating tasks into these two groups, we can identify which tasks to automate and which to leave to human judgment.

Some people might say, “does that distinction really matter?” And we would respond – if you’re working in financial compliance, we’d say it matters a lot.

It’s certainly mattered to us as we’ve built out Hummingbird’s AI features. 

Financial Compliance Is Full of Meaning-Reducing Work

Let’s take a typical case investigation. Before an analyst can determine whether a behavior is suspicious, they need to:

• Pull customer data from internal systems
• Search for negative news
• Check multiple watchlists
• Normalize data across vendors
• Aggregate and document results from all sources

This is all meaning-reducing work. The goal is clear – gather everything relevant so the analyst can make a decision. But it’s slow, repetitive, and error-prone. Perfect for an Agent.

Where Agents Still Need a Human Teammate

Now let’s look at what comes next.

Once this initial phase is complete, more “meaning-making” work begins: investigators must interpret the data, apply policy and context, and ultimately make a judgment call. Do they escalate the review? Does the subject’s story check out? Is this risk real or merely a false positive?

This kind of work isn’t something we’d want to outsource to AI Agents (at least not today). And why?

• It’s an area where agents can’t (because of data architecture) or shouldn’t (because of data privacy concerns) have full access to all data points
• It’s an area where agents can’t currently apply the same level of data analysis as a seasoned investigator
• It’s an area where an agent might struggle with subjective decision-making in the absence of clearly-defined procedural guidelines
• It’s an area where an agent could still struggle with long chains of reasoning, and it’s a practice area where small errors could compound.

Fortunately, this is exactly where human analysts shine! So if you hear us at Hummingbird say “human in the loop” AI compliance, this is what we’re talking about. This is the loop.

How to Think About AI Agent Deployment (Especially in Compliance)

So, what does a successful Agent deployment look like? In our experience, it comes down to a few simple rules of thumb:

✅ Use Agents When:

• The task is repetitive and decisions are made based on clear guidelines
• The output can be verified easily
• The task doesn’t require broad or shifting context
• Speed and consistency matter more than creativity

🚫 Avoid Agents When:

• The task involves nuanced judgment
• The goal is ambiguous or evolving
• Mistakes have high downstream consequences
• There’s no way to validate the Agent’s output

A Better Question Than “Do AI Agents Work?”

The conversation needs to shift. The question isn’t “Are Agents ready for prime time?” It’s “What kind of work are we asking them to do?

Because if you’re asking an Agent to replicate the judgment of a seasoned compliance analyst across a tangled web of policies and edge cases… you’re setting yourself up for disappointment (at least for now). 

But if you’re asking it to gather and structure the raw material for that analyst so that they can focus on actual decision-making – well, you’re going to see your time (and results) improve exponentially.

That’s where the ROI is hiding. Not in vague promises of AI transformation, but in specific, well-chosen use cases that honor what the technology can do – and what only humans still should.

The Path Forward

AI Agents are no more a cure-all than any other promising new technology.

Used intelligently, however, they’re an undeniable force multiplier.

At Hummingbird, we’re committed to deploying AI in ways that match both the technology’s strengths and the needs of compliance professionals. That means building Agents that speed up work without dumbing it down – through features that deliver clarity, not complexity.

If you’ve been navigating some of the questions we’ve been exploring in this essay and want to learn how to apply AI where it counts, we’d love to talk.

Let’s give Agents the right jobs – and watch what happens.