Why AI Compliance Agents Need a Harness, Not Just a Model With Peter Piatetsky, CEO of Castellum.AI

Peter Piatetsky is the CEO and co-founder of Castellum.AI, which started in 2019 as a primary source for sanctions, PEP and adverse media data and now builds AI agents that resolve compliance alerts for banks and fintechs. A former intelligence analyst, OFAC analyst and bank compliance head, he has seen the alert overload problem from every side. We talk about what actually happens when an alert fires, why the real question today is build and maintain versus buy, and what Castellum’s benchmark shows about LLMs with and without a harness.

What We Covered

• From intelligence analyst in Afghanistan to OFAC
• Becoming lead Russia analyst when Russia first invaded Ukraine
• Running US compliance for a major Asian bank
• Why Castellum chose to be a primary data source
• Going modular to meet clients where they are
• What happens when an alert hits the Arbiter agents
• Training an agent on a client’s policies and procedures in 10 business days
• Why the AI reliability debate is over
• Build and maintain versus buy
• Benchmarking nine models with and without a harness
• Running inside LexisNexis Bridger and Fircosoft
• How criminals are using AI at scale

Key Takeaways

• The harness matters more than the model. In Castellum’s testing of nine models, top LLMs on their own were accurate about 70% of the time when asked to read policies, screen, create and adjudicate an alert. With Castellum’s harness of specialized data, training, regression testing and QA/QC, accuracy rose to 99% and consistency got close to 100%.
• The question is no longer whether AI can do this. Within a year, buyers have moved from doubting AI agents to asking whether they have budget this year or next. The common objection now is “why not just use Claude,” which Piatetsky frames as build and maintain versus buy, including retesting every new model on hundreds or thousands of cases.
• Sit on top of what banks already have. Castellum’s agents run inside existing screening systems such as LexisNexis Bridger and Fircosoft, so clients avoid new model validations and tuning reports. His analogy: a water filter on the tap instead of ripping out all the pipes.
• L1 is going away, and that could be good news. Piatetsky expects L1 alert review to be fully automated and much of L2 to follow, freeing analysts to focus on alerts that matter and giving large institutions real visibility into what is happening inside them for the first time.

About Peter Piatetsky

Peter Piatetsky began his career as an intelligence analyst in Afghanistan, then joined the US Treasury Department, where he worked at OFAC on Iran and later became the lead Russia analyst. He went on to run US compliance for a major Asian bank before co-founding Castellum.AI in 2019. Castellum’s clients include a top four US bank, a top three global payments provider and a top five global crypto exchange.

Cleaned Transcript

Peter Piatetsky (00:10):
We tested nine different models, both with a Castellum harness and without, and we found that the top AI models, when they were asked to read policies and procedures, conduct screening, create an alert, and adjudicate an alert, they were accurate only seventy percent of the time without a harness versus ninety nine percent of the time with a harness. So in other words, kind of for a compliance program, if you’re using just Claude, you’re at high risk of getting three out of ten of your alerts wrong, which is kind of what gets any compliance officer fired immediately.

Peter Renton:
This is the Fintech One-on-One Podcast, the show for fintech enthusiasts looking to better understand the leaders shaping fintech and banking today. My name is Peter Renton, and since 2013, I’ve been conducting in-depth interviews with fintech founders and banking executives. Today on the show, I’m delighted to welcome Peter Piatetsky, the CEO and co-founder of Castellum.AI. Peter began his career as an intelligence analyst in Afghanistan.

He then moved to the US Treasury Department where he worked at OFAC on Iran and later became the lead Russia analyst. After that, he ran US compliance for a major Asian bank before founding Castellum in 2019. In our conversation, we talk about why Castellum chose to be a primary source for sanctions, PEP, and adverse media data, and how it grew into building AI agents that resolve compliance alerts.

Peter walks through what actually happens when an alert fires, explains why the real debate today is build and maintain versus buy, and shares benchmark results showing how much an LLM harness improves accuracy and consistency. We also discuss how to tell vendors apart, how criminals are using AI at scale, and his prediction for L2 and L1 compliance work. Now, let’s get on with the show.

Peter Renton (02:18):
Welcome to the podcast, Peter.

Peter Piatetsky:
Thank you, Peter. I’m gonna enjoy saying that.

Peter Renton:
Yes, indeed. It’s nice to have another Peter on the show. So let’s kick it off by giving listeners a little bit of background. I know you’ve had an interesting career, spent quite a bit of time in government before starting your own company here. So give us some of the highlights of your career to date.

Peter Piatetsky:
My first real job was as an intelligence analyst in Afghanistan.

Peter Renton:
That’s a heck of a first job.

Peter Piatetsky (02:47):
It really is. I think it confused me as to what most jobs actually are, but that was my first like W-2. And I really wanted to do national security and like fight the bad guys and went to Afghanistan and was supporting a special forces unit there. And it was unclear to me kind of what our, why we were really there. Nobody could really tell me. And I think you’ll hear a similar story from anyone that was in Afghanistan, at least from the US side.

And I read about sanctions and I was like, okay, you know what? This sounds great. You get to fight bad guys and it’s very focused. I mean, that was one of my challenges with our mission to Afghanistan was that I wasn’t sure, are we fighting terrorism? Are we building democracy? Are we training national security forces? Are we doing anti-drug trafficking? Like there was not one clear mission that we could win at. And so with sanctions, I thought, okay, here is something very clear. You know, there are let’s say narcotraffickers or proliferators or governments or terrorist financiers, they all either need money or use money. So they either need money or they want money. Like narcotraffickers, they do it for money. Terrorists generally don’t do terrorism for money, but they need it. And so having that really narrow scope of the work, I think was really energizing and motivating because I felt like I could measure success. Money frozen, aircraft impounded, like these are in many cases directly translates to lives saved.

So worked in Afghanistan, then got a job with the Treasury Department, first at OFAC, the sanctions office. And then Putin invaded, I was working on Iran, then Putin invaded Ukraine the first time. And they were like, hey, who here speaks Russian? It’s like me. So they were like, okay, you are now lead Russia analyst, and worked on Russia, eastern Ukraine, Russia-Ukraine, Eastern Europe, and then during the JCPOA back to Iran. But kind of in all of my government work, I found it very difficult to run investigations. And I found it kind of very backwards that the US and other governments are spending hundreds of millions of dollars to identify bad actors and sanction them and say, hey, here general public, here are these people that are very risky. But then actually finding that information was incredibly difficult, even for me while I was in the government. And so the origin of Castellum is very much so from my work in government and my frustrations there.

After, I went to banking, ran US compliance for a major Asian bank. And I think it’s kind of seeing how compliance technology was used across the private sector really convinced me that there was a lot of room to improve in terms of speed, in terms of accuracy, in terms of ease of usability. And you know, now here we are. Castellum is seven years old and we have a top four US bank, top three global payments provider, top five global crypto exchange. And we’re really excited to be solving financial crime problems for clients.

Peter Renton:
Right. So when you started, I mean you started in 2019, I believe. So that was before the AI revolution was upon us. And, you know, you’ve certainly embraced that. So tell us a little bit about what you were doing when you started and how that’s evolved as we now live in a world of AI agents.

Peter Piatetsky:
So we started as a data company and we initially started with sanctions data. And then we, you know, would go to clients and they would say, okay, this is cool, but we need politically exposed persons data. And we added politically exposed persons data. And then you know, we went back to clients and they said, okay, this is great, but we also need adverse media. So we went to add adverse media. And I think this initially felt like it was a mistake, but it’s now turning out to have been, you know, a very fortuitous move, is we did not outsource our data collection at any point. So we are a primary source. We get global sanctions ourselves, we build our own PEP database. We get media directly from publishers, and then we turn it into adverse media. So we separate, you know, let’s say Russian oligarch donates money to charity from Russian oligarch involved in corruption with construction, for example, and then identify who all the bad actors are in the company. So we are a primary data source.

And I think what we learned is that the compliance data market is actually really extremely difficult to work in, because we might go to folks and say, hey, our software and our data is going to lead to 80, 90% efficiency improvements with no loss of accuracy. And people would say that’s amazing. But then there’s change management costs. And so it’d be like, okay, well you just need to integrate and it’s this easy. And then that’s when people just start ghosting you. And so we decided to take the company in a modular direction. So instead of saying that, hey, you have to buy data, screening, case management, and agents, we separated out the modules, and that’s really been huge for the company in terms of meeting clients where they are, making it a lot easier to get the product.

And the product that is really taking off right now is AI agents for alert resolution. And I think it’s partially because we have narrowed down our focus. Like specifically to AI agents for sanctions, PEPs, adverse media alerts, you know, level one, which is resolving the alert with the information available, and then level two, which is kind of going outside of the alert and conducting additional due diligence. And it’s been frankly like really, really exciting. When we founded the company, we thought, okay, this is something that we could do and it’s possible, but now actually seeing our products do this is incredible. And I think there’s an actual sea change coming in the world of financial crime, because no large organization actually knows what is going on inside of it simply due to the volume.

And you have all of these systems that are designed to tee up alerts, but because organizations get fined for not enough alerts, not for too many alerts, when systems are configured, you know, program managers generally configure it to overgenerate alerts because they’re trying to not get fined. Oftentimes more so than, like, of course they want to catch financial crime, but really the biggest thing they’re trying to avoid is being yelled at by the regulator and the regulator telling them you’re not trying hard enough. And so as we all know, like investigators, analysts are drowning in mostly useless alerts. And so a lot of actual financial crime is not getting caught because of quite simple, like, volume issues. And I mean AI agents are going to, at least for our clients, are already allowing them to remove in some cases 95 plus percent of alerts accurately, which is really important, because inaccurately reducing workload is useless. But yeah, it’s really allowing them to spend a ton of time on alerts that really matter, and that is phenomenal.

Peter Renton:
So let’s dig into that just for a little bit. Can you take us through the process when an alert fires at, you know, one of your clients? What actually happens then and how is sort of the AI agent involved, and where is the human involved and when does that happen?

Peter Piatetsky:
Some of our clients use Castellum for data and screening, and then they use our agents. And other clients use different screening systems, like let’s say TRM or LexisNexis Bridger or Hummingbird, with integrations with those, and happy to talk about that more later. But let’s say an alert is generated either in our system or in another system. And so what happens is that alert then goes via API to our Arbiter endpoint. Arbiter is our family of AI agents. And the first thing is the alert gets routed to either a sanctions alert clearing agent, a PEP alert clearing agent, or an adverse media alert clearing agent. So because we started as a data and screening service company, we have a lot of IP and knowledge around how alerts are resolved. And you know, there are very different risk tolerances between the same alert that matches on a sanctions name, it’s gonna have a much higher threshold for clearing it than the same alert but that matches on adverse media.

And so before our agent reviews any alerts, our agent ingests the client’s policies and procedures. And sometimes these are 13 pages, other times they’re 145 pages. And usually before we even sign a contract with the client, we go through a full proof of concept, where our agent learns the client’s policies and procedures, then does an open book test, that’s alerts with adjudications. And then we can see, hey, is our agent hitting 100? And interestingly enough, our agent almost always identifies cases where the human analysts whose work we are kind of being graded against have made mistakes that don’t correspond with the policies and procedures. So even during kind of the POC phase, we’re identifying improvements. But our agent ingests the policies and procedures, our agent takes the open book test, and then our agent takes a closed book test, which is alerts without the adjudications. So we go through this training period, which we do in 10 business days, and then the alerts go to an already trained agent.

And so if it’s a sanctions, PEP or adverse media alert, it gets routed that way. And then each one of those agents has dozens of sub-agents. So there’s, for example, a text normalization agent that will clean up the alert if it’s kind of dirty and unreadable or might have a typo in it. Then there’s a name normalization agent. Sounds really silly, but for example, is Le Thong Wei written in Mandarin characters the same as Le Thong Wei written in English? Is Mohammed, let’s say, Bostrukayev in Russian the same as Mohammed Bostrukaev spelled differently in English? So kind of difficult transliteration, phonetic analysis. Location analysis, contextual analysis, severity analysis. So we do not take an alert and just send an alert to an LLM and then say, hey, what do you think? That leads to very low accuracy and very low consistency. And in our space that’s unacceptable. So that’s a short answer. I know that was like five minutes, but the agent learns the policies and procedures, the agent gets tested. And then when alerts get sent to the agent, we really break down the alert into as many different components and evaluate each one of those separately and then come back with an answer.

Peter Renton:
So as you’re talking, it seems to me that every single financial institution needs this technology, but I’m curious about who is your market? I mean, who are you focused on? Are you focused just, you know, on the sponsor banks, on the fintechs, on community banks, on everybody? What’s the market?

Peter Piatetsky:
We focus on US based banks and fintechs. We are primarily trying to work with mid-market banks and fintechs, but in practice, we have seen a fair amount of interest from tier ones. And so that is, we’re not gonna say no to, you know, a top ten global bank or top four US bank knocking on our door. So essentially in practice we work mid-market and up. But what we’ve seen is in one year, the space has gone from AI agents are not reliable enough to do this to I need AI agents to do this now. And the only question is, do I have budget for it this year or do I need budget for it next year? And we have seen this from folks that were AI skeptics. We have seen this from folks that have laid off people and rehired them. And we have seen this from people that have been kind of very bullish on AI use cases all along. And I think part of it is AI tools have gotten exponentially better. But also we’ve been able to really show, and some other companies in the market as well, we’ve been able to show that, hey, agents can clear alerts reliably.

Peter Renton (15:37):
Right, that is the key, because I think that was the knock early on on some of the AI technology, as you just said, and hallucinations with LLMs has made a lot of people think that there is a reliability issue. So when you’re talking with potential clients, is that reliability piece central or is that debate kind of falling by the wayside now?

Peter Piatetsky:
The reliability debate has actually fallen, is gone. We have very few conversations where people say, I wouldn’t trust AI to do this. And if someone says that they don’t trust AI to do this, we say that’s totally fine. We’re gonna go talk to the next person, because we have so much demand right now, which is, you know, a great place to be. And the question that we have much more often is, why should I not use Claude? And I think that’s a great question. And it’s a very reasonable and smart question. And I think, whether it’s Claude or Astor or Copilot, a lot of institutions have, you know, enterprise AI accounts with really large budgets. And they’re saying, look, why don’t we just build it? And so the question that we mostly have is what we used to call build versus buy, but what is much more accurate to be called build and maintain versus buy.

And so we have done a fair amount of really detailed, data-driven AI research showing what it is like to use an LLM and what it is like to use an LLM with a harness. A harness is a combination of infrastructure, data, regression testing, QA/QC, specialized instructions, and more that is put around an LLM to make it purpose-built. And for those that follow Formula One, there’s a couple of companies that are engine manufacturers. Like Mercedes manufactures engines for itself, but also for other teams. But then the teams that buy the engines from Mercedes, they still have varying results from Mercedes. And they will put on different tires, they have a different driver, different wings, brakes, et cetera. So using, you know, Anthropic or OpenAI as kind of the engine, everyone should be using an engine, but then the harness around it is actually what makes the difference.

And so for financial crime compliance, that harness includes specialized data, specialized training, specialized regression testing, built-in QA/QC, and the results are enormously different. So we tested nine different models, both with a Castellum harness and without, and we found that the top AI models, when they were asked to read policies and procedures, conduct screening, create an alert, and adjudicate an alert, they were accurate only 70% of the time without a harness versus ninety-nine percent of the time with a harness. So in other words, kind of for a compliance program, if you’re using just Claude, you’re at high risk of getting three out of ten of your alerts wrong, which is kind of what gets any compliance officer fired immediately.

That’s an unacceptable number. Like three out of a thousand would get someone in trouble, let alone three out of ten. And so we found that the harness greatly improved accuracy, but it also really made consistency close to a hundred percent. So if you give an agent the same alert three times, is it going to adjudicate it the same way? And if we’re just using an LLM by itself, the answer was no, it would not. So LLMs are incredibly smart and really powerful, but they absolutely cannot be trusted to just be given a financial crime compliance problem and to solve it. They need to understand the policies and procedures of each institution. They need to understand the risk metrics. And the harness is really what does that.

And so we never say, like, you know, we work with some very large fintechs that have engineering teams bigger than our whole company. So I would never say to those people you couldn’t build it. Of course they could. It’s a question of, do they want to build it, maintain it? Are they going to staff, you know, their agent with dedicated compliance engineers and dedicated compliance subject matter experts? Are they going to do the model validation? Are they going to do the regression testing? Every time a new model comes out, you know, whatever is going to power the agent on the back end, it has to be retested on hundreds or thousands of cases and alerts. And so we’ve actually found really interesting kind of allies amongst the general counsel’s office, where they’ll say, like, look, we know my team at, you know, name a big fintech, can build this, but do we want to own the risk? Also, it’s way more expensive for us to build it and maintain. So really the question is build and maintain or buy. And that’s kind of the debate we’re having more often now.

Peter Renton:
Everyone has existing sanctions screening and a whole range of compliance tools at their disposal today, that they’ve had for decades in many cases. When someone’s deploying your technology, what kind of improvements can they expect? And I know, you know, it’s gonna be different for every system out there, but give us some sense of, when Castellum comes in, what happens as far as improvements to their systems typically.

Peter Piatetsky:
I’ll talk more so about our partnership motion. So our agents are integrated into a number of different systems, including LexisNexis Bridger, LexisNexis Fircosoft. We just announced an integration with TRM and we have an integration with Hummingbird. And there are some others. So when someone adds our agents over the top of an existing screening service, the first thing that they can expect is not to change their existing screening service. That’s great from a change management perspective.

And the way I like to think about this is, you know, if someone comes to you and says you have lead in the pipes in your house, you can’t drink that water anymore. And there’s a number of solutions. And one of the solutions is you rip out all the pipes and you replace them. That’s very expensive. You’re gonna have to live in a hotel for a little bit. And the Castellum approach is, hey, just buy this water filter and put this water filter onto your tap. It’ll filter everything out. You can drink clean water. So that is kind of our agent approach with existing screening systems. Our agent runs inside of Bridger and inside of Fircosoft. So what that means is clients can continue to use Bridger, Fircosoft, or other systems that they have in exactly the same way they were. No new model validations, no new tuning reports. And then just alerts are handed over to the agent.

And this is kind of an approach of just saying, let’s be realistic about what can be done, about budgets, about change management appetite. This is a way to focus on results first. Like we’re not gonna say that, hey, we’re gonna do this massive reimagining of a whole compliance program. Usually whoever starts that tends to leave before it’s finished. Those are like three to five year projects at a minimum. But if someone, let’s say, has Fircosoft or Bridger and they wanna get immediate results, cool. Our agents are already inside. It’s like turning it on. And that’s really our focus right now, is on supporting clients through our channels.

Peter Renton:
As we were saying before we hit record, I was at Money20/20 this week, walking the halls of the show there. There was some, you know, AI companies talking about sanctions screening and different alerts that their agents can do. So you’re not the only game in town, and it’s difficult I think for people to know who’s better than who and who’s really not. So if someone’s listening to this and sort of saying, well, how do I evaluate these companies that all say they’re doing something similar, how do we tell you guys apart?

Peter Piatetsky (23:54):
You know, last year I had, at Fintech Meetup, a bank CEO telling me literally that. I don’t know how to tell you guys apart, with some words in there. And we have a couple of resources, which we would love if you’re able to direct your listeners to. We have an agentic screening market map. So we have done a bunch of analysis and put together a number of vendors, I think over a dozen, that really focus on this. So we’re not including, you know, a legacy company that bolts on agents, or somebody that is just white labeling, like, ChatGPT and sending alerts directly to a public unencrypted LLM service. It’s who is really focusing on agents. And so we encourage everyone to take a look at the Castellum agentic screening market map. That’s one that helps people understand who’s in the space. You know, obviously someone could say that we’re biased, but we do put a bunch of our competitors up there and say good things about them.

And I would also say I don’t view other agentic companies as really competing with us. I see it as we’re all going after the same problems, and we’re really competing against, like, BPOs and consulting firms that are just used to outsource this work somewhere. So I mean the agentic screening market map is one. And then we have a FinCrime Agent Benchmark, which I mentioned earlier, where we actually show research. Like, look, this is what our agents have done. This is how an LLM without a harness will perform. And I think, Peter, you’re right. The hardest thing right now to do in the space is to stand out.

Peter Renton:
I’ll make sure we do link to that in the show notes here. Okay, so I want to talk about the other side, the financial criminal side of things, because they’re using AI at scale like never before. They don’t have to deal with any compliance rules or anything like that. What are you seeing on the offensive side that was not there, say two or three years ago, where the criminals are taking advantage of the latest tools?

Peter Piatetsky (26:00):
So it’s horrifying. And I mean that’s actually one of the reasons that regulators have really come around to AI in fin crime, is because they are seeing criminals use it extremely well. And so regulators basically initially had a lot of discomfort around the hallucinations and is AI reliable, and now it’s just more so like there is a deluge of AI-powered financial crime, and so the only way to really fight back is to use AI, because AI, done well, allows you to scale up your fin crime operations, or anti-fin crime operations, but also allows criminals to scale up their fin crime operations.

So in terms of what we’re seeing, the first is scale. So any scam that was occurring beforehand, or crime that was occurring beforehand, is now occurring at, you know, like 10 to 20X in terms of volume. I think we’ve all heard about scam farms, again, especially in Southeast Asia, where people are kidnapped, passports taken away, they’re told they’re going to work somewhere, and then they’re put into this call center where they have to commit elder fraud. And those centers don’t need as many people anymore. They’re able to run the same scams with fewer people. So I think one is just volume.

Two is fake IDs and fake personas. So it’s just become that much easier to create fake documentation. People can create a deepfake online. Like, you know, unless you’re touching someone, you may not know that the person that is there is the person who is there. And that creates a lot of issues for account opening, wire transfers, theft of funds. And so I mean those are the two primary attack vectors that we’re seeing. One is, and it’s not so much an attack vector, but like all the existing scams are now superpowered in terms of volume. And then there’s a lot in terms of account opening that is being done in the financial crime world, because people are creating synthetic personas, complete synthetic personas, using them to open accounts and then commit money laundering, sanctions evasion, or just fraud.

Peter Renton (28:17):
Okay, so last question then. As you think about the future and how AI agents are gonna continue to reshape compliance programs, take us through where you think, you know, two to three years out, what the landscape will look like, and where does Castellum fit into that landscape?

Peter Piatetsky:
Two to three years out, Castellum continues to be the leading provider of AI agents to banks and fintechs in the US, knock on wood. But I think where we see the space going is that all work that is known right now as L1 has been fully automated. So L1 is, an alert is generated and then an analyst looks at the alert, but they’re only looking at, like, what is in the alert. They are not doing external research. They’re just like, Peter Piatetsky matched with Peter Pizatsky, false positive, name mismatch. And the reason, again, those alerts are generated in the first place is because compliance program owners tend to bias towards, let’s generate more alerts so we don’t get fined. Because if we miss something, that’s when we really get in trouble. And so with agents, you can continue to generate a fair number of alerts, but reviewing them becomes much less expensive. So I think L1 as a job does not really exist anymore.

And most of those analysts are now going to be working on L2 alerts. So things where you need to do some external research and analysis. Now I think a lot of L2 work is also going to be automated, because that is basically taking an L1 alert where there is lack of clarity and then going out to certain external sources to do research, come back and help make a decision. So that is something that agents are very good at doing, like gathering data, creating a decision point.

But I think what this means also is that, I do believe, depending on how this technology is implemented, we could see a real true global drop in financial crime, because you would actually see some of the compliance shops at large banks and fintechs become very effective. And for someone to be able to say, hey, I actually know what’s going on in my bank fully, that’s something that’s never been possible for a large bank. Like that’s only been possible for a private bank where maybe the bankers all know all the clients. Similarly to much more what you would see at a credit union. But even credit unions are trying to grow now. And you know, almost all have online account opening and don’t know all the members. So long story short, Peter, I think we are going to see, you know, near full automation of L1 and L2 roles in the space, but also a corresponding rise in effectiveness in fighting financial crime.

Peter Renton:
That is an optimistic and a positive future. Great place to end. Peter, really appreciate you coming on the show. I hope that your prediction comes true and everyone is much more effective at fighting financial crime. So best of luck and thanks again.

Peter Piatetsky:
Thank you, Peter. Pleasure to be here.

Peter Renton (31:34):
I loved Peter’s framing of the impact of compliance AI when it comes to large financial institutions. He said today that none of them truly know what is going on inside their company. Only a small private bank where the bankers know every client has ever had that kind of visibility. Peter believes AI agents could finally give a top-tier bank the same understanding, and that this could lead to a real drop in financial crime.

That is a bold prediction, but it changes how we should think about AI and compliance. This is not just about cutting the cost of L1 reviews. If he is right, compliance teams will finally get to do the job they were always meant to do. Anyway, that’s it for today’s show. If you enjoy these episodes, please go ahead and subscribe, tell a friend, or leave a review. And thanks so much for listening.