0:15
Hello, my name is Kenny Pyatt. I am the founder of DevOcho and the host of the DevOcho podcast. Today we're going to talk about AI agents. And I I know AI agents are kind of the hot topic right now. That's part of why we're talking about it. We get calls like every week, honestly, probably every day from somebody wanting us to create a bunch of AI agents for them. So, I wanted to talk as a person that really understands this. I want to explain what is an agent, what are the different ways to use agents or deploy agents, what are good places that that agents can fit in a company right now. I want to be honest and tell you what they can and can't do. And I want to be honest and tell you what they really are. Um, because there's a lot of confusion about what an agent is because there's a lot of hype. I also in the next 20 minutes or so we'll be talking about how we at DevOcho build agents since that's what that's a big part of our job is machine learning and AI and and that kind of work. So I'm going to go and jump straight in. If if you're technical I will touch a few of the technical points but I'm going to try to stay kind of high level and just give you keywords that you could jump into Google for and key technologies that you could you know deploy inside of your company. So we're going to start with like just what is an AI agent? And if this is something you know, you can skip ahead a little bit. But honestly, like all of this is is hyped to a point where there's a lot of confusion about what an agent actually is. And they like to say like the marketing teams and these companies like to be like, well, we had chat, but now we have agents. But all of it works in the same technology. There's a a large language model or LLM that drives decisions. And there's different types of LLMs and what they do. And there's thinking and non-thinking models. There's the idea of like a, a mixture of experts set of models where there's multiple framings around the same model. So, so an, an agent is really honestly like to oversimplify a little bit, it's a set of tools that the LLM has access to. It's the LLM itself and it's a set of instructions, the prompt. And I guess like throw all that together and we can add this idea of a conductor. And all the conductor is how often does it trigger this thing running and
Understanding AI Agents and Their Functionality
2:23
how often does it actually cause the the system to do its job as an agent. So let's let's use an example something in the real world like we do a marketing bot here. I'm going to give away a secret of what we're doing with our own stuff. In the marketing bot we have the um it has a a set of connections to some outside data sources. So, we sat down with the marketing team and we came up with our ideal customer profile. We actually have a couple and we tell the agent, we want you to build lists of these people. We want you to go to a few different data sources and pull information and then we do a score for everyone and we build a list. If your score is higher, in our case, we do a scale of 1 to 100. If the score is higher than 85, you're likely a good target for us. So, we want to do marketing to you. So, in the agent that we've built, we're kind of crazy people, so we build some things custom. In this case, we built some custom Python to do our agent, but everything I'm about to say, you could do in Open Claw. You can do with Conductor if you're on a Mac. There there's there's tools that will allow you to do what I'm about to explain. Notebook LLM is pretty good for this moment as well, but we we did Python because we're programmers. So, basically, I gave it a prompt. I explained what the ICP was so the agent would know what's going on. I explained to it where it was going to get data and then I gave it access to a set of credentials. You got to be careful with this and I'll explain how you do it safely. And I I basically turned it loose. So in marketing almost everybody is going to know there's a couple companies but we use a company called Apollo where we can access data and it's company information and contact information. And I apologize. I understand that marketing is super creepy and that we're doing super creepy things to monitor you. We try to be good guys, but it is still a little creepy that we can look up your phone number and your email address and your work history and all those kinds of things. So, we gave the agent instructions and we said, "We want to build a list and and we'll do a 1,000 to 2,000 people at a time. We build a list and we pay for credits with Apollo and we actually I'm going to give you another trick as I explain this. We use a frontier model.
4:31
In our case, we use anthropics. So, we use Claude. And in the the Claude model, it's making decisions, but before we go to the Claude model, we do things to save ourselves money. So, we have local models. We have an AI server in the office. It's got three different graphics cards in it. And we will do the initial work on our local card. I'm going to go way more in the weeds than I I probably should. We
Real-World Applications of AI Agents
4:51
go to Apollo. We pull whichever ICP we're working on. We pull a list. We take that list and then we run it through a local model to decide, is this worth spending our credits on or not? If it's not worth spending our credits, then we drop them out. If it is, then we'll go back.
5:08
We'll enrich the record from Apollo. That's when we pay Apollo money. Apollo will then give us a a much more vibrant record. And then we go to Claude, the agent. Um, we'll will run a much more sophisticated grading that I'm not going to share. And it will come back and give us a list of these are the right companies, these are the right contacts that you should target your marketing at.
5:31
And from that list, you can go, if you've ever done marketing, you can upload those lists to um you know, LinkedIn, Tik Tok, X.com, wherever you're going to spend money on marketing and they will help you find people that are like those people in your list. And it's it's really powerful and it works. So that's one of the AI agents we have actually can just start it up and I can give it instructions in a chat interface and say, I want you to go find more contacts for ICPA. I hit enter. A couple hours later, I have a file. And that's a really cool use for an agent, but that's an agent that I trigger. It's not working for me all the time. That agent is just a Python script that we run, a prompt that explains each of our different ideal customer profiles and a set of instructions for how to talk to Apollo, the Apollo API, and and it's got budgets and other things like that, all like laid out in that prompt, so it knows what to do. It has a set of credentials, but the way we do credentials because we never want those credentials sent to the model. We don't want Anthropic to have our Apollo credentials. We put those in environment variables so that the system can access those as it's making its move, but they're never sent to the LLM. And that's very important if you're doing agents in environments. For example, we do a lot of work in healthcare. So, if you're if you got, you know, PHI, protected health information, patient name, date, dates of birth, social security numbers, anything like that, insurance information, you never want to send those out to the model unless you've you've reached an enterprise agreement and they promise, but even then, I don't trust them. I try to never send PHI to those companies um because they might make a mistake. They may accidentally in the next round of training their model, they may include the PHI. That is terrifying. So we just don't want to mess with that or or deal with that.
7:20
So an AI agent is exactly that. It's a, it's a set of models. It's a set of tools and then there's a set of ways that those models can interact with those tools. So regular old APIs that have been out for basically as long as I've been programming for more than 25 years. You also have something that's called a model context protocol or MCP server which gives the model a specific way to call like make requests and get back specific types of data and it can send in parameters that
Technical Insights on Building AI Agents
7:50
it can clarify. So one example of that we have a in our office we have an ERP system that we built for ourselves that helps us track our projects and our staff and what we're doing. We built a model context protocol that works beside that. This is actually really new. We're in the process of doing this right now. So, I'm speaking like it's done, but we're in the process. But, it allows AI agents to hook in and get information out of the the system. So, we can have conversations like, hey, we've got a new project coming in. It looks like it's going to need six people to to do the work. I need a scrum master, a DevOps. I need two developers, QA, and a machine learning person.
8:25
Help me plan that out. Right? and it'll come back and say the following people in each of those roles are available because it has a a function in that server that it can pull that information and it allows us to to do some pretty neat things where we can also have an agent that runs on a schedule. So let's say like every night and it can look at the utilization of all of our staff and it can make a report for us and say these people are not running. Now the old way for my entire career almost up until the last couple years I would do that in a data warehouse. I would go and build a report in a data warehouse. I'd have a full ETL process and we still do that, but I would have an ETL process. It would generate a utilization report. But when you need ad hoc reporting, you buy a tool like Tableau or PowerBI or or something that you set on top of your data warehouse where you can go build custom reports. But now I can just talk to a model. It has an MCP server, which is effectively the ETL portion. It's just real time. And it'll go pull that information, bring it back. But what do you do about hallucinations? So this is a real issue.
9:29
In the absence of information, models will simply make up stuff and you don't want them to do that. So the way that you handle that is a couple things and it depends on your exact circumstances for the techniques, but I'll give some generals. If there's any numbers that are coming back, you can do sanity checks on those numbers. So if the model is doing math, obviously you can double check the math in regular programming and say like it's supposed to be adding up the time for this project for these six people and somehow it ended up with enough time for eight people. Well, that can't be right, right? You can also do things where as you're doing prompts, you can prompt it to say like this is where you'll find this information. If you don't
Challenges and Limitations of AI Agents
10:07
find this information, put this value instead. And it's not perfect. The models sometimes have a a will of their own, but they will frequently if you say I always if you don't know a number, always put a dash. A lot of times you can actually get the model to do what you want there. But then you go back later and you check all the output. This where I'm going to get a little technical. If you're building an agent, it's best if you have the agent speak in a language that computers and programming languages understand. So we do a lot of our work back in comma-sepparated values, CSV files or JSON. JSON is a a structured format that um you a lot of APIs speak in JSON and with those two we can validate the model gave us something real. We can check each individual value like if it's a CSV file you think about an Excel spreadsheet column A number 17 should always be a dollar amount and if it's not you know you've got a problem and that's a really good way to protect against hallucinations as well. As we're running through this a little bit scattered I need to pull my notes out. I want to talk about like the truth of all of this is there's a lot of like hype and as I mentioned earlier, it's really easy to get yourself in trouble when you hear about this one guy and 23 AI agents built this giant billion-dollar company and and I'm going to tell you a lot of that stuff is not real. We build these things. We're always in there. They make a ton of mistakes. It's actually quite difficult to get one dialed in and correct.
11:33
It's really easy to make one do a specific task. So, for example, I can create a demo right now with my ERP software and that exact report that I just explained where I can have it go in and tell me who's available for this project. Super simple demo. And I can get that demo to work every time. But what if I ask it a different question? What if I ask it something that it it doesn't have access to the data on? Like, as you start to make it more and more complex, it gets harder and harder to make it work. So the whole move fast and break things issue is is where you're seeing headlines around people. They deploy an AI agent very quickly. They deployed it to their customers very quickly or the general public and 15 minutes later all their data was compromised. They've been hacked. Everything's terrible. That is a real risk and you've really got to be careful. The LLMs, this is something I wish like I wish there was a way that I could come up with a good analogy here.
12:22
LLMs are not sentient. They're not humans. They don't actually have the same capabilities yet to think the way that we think. So an LLM doesn't know if it's right or wrong. It just knows. It doesn't even know. It's truthfully a statistical engine. It's got a whole bunch of weights in it.
12:40
And I like I don't think everybody's going to understand this. So maybe I can get Montse (editor) to throw a picture in the edits here, but there's this game that's super popular in Asia. It's called I think
The Future of AI Agents and Memory
12:50
it's pachinko ball machines. And there's these little pins everywhere. And and the way it works, it's kind of like pinball. Pinball lays on its side and this is vertical, right? And so you've got this thing and you start pulling this thing and launching thousands of balls into the machine. You got flappers. You're trying to keep the balls up in the air. As the ball goes into that machine, it hits pins and it can go oversimplified. It can either go right or left when it hits a pin. Now, if it's got some momentum, obviously it could bounce and go a couple pins over. But as it goes down, it hits each pin and it's going to roll right or left. And as it rolls right or left, it's moving or navigating through that machine. That's kind of exactly how the the neural networks work inside these models. They hit a point and in that point there's weights that tell it under these circumstances it's better to go this way or this way and it and the the ball, the thought, the token is making that move through the machine and it falls out the bottom. And what it spits out the bottom is a response token. And it's not thinking while it's doing it. It's just doing math. Math on its way down. Now, I understand the philosophical argument for, well, what are you doing? you've got neurons and things like that. Reality is I have no idea. I have no idea how my brain works beyond what we sort of kind of understand, but we have no idea. But we do know how these models work. We can trace them. Almost every model, you can eventually build a system where you can do traceability on it and understand how it works. When you do a trillion parameter model, which is these frontier models right now, it's it's a lot harder to trace them. It takes a lot of computer power to understand all the weights and everything that's going on in the neural network, but it's still possible. And the reality is like when you're asking it questions or giving it tokens, it was trained on this enormous corpus of data and it's spitting out the most likely answer to your question or your set of instructions, but it's not thinking. So, please don't think it is. Don't think I can spin up an AI agent and and make it an administrative assistant and it's going to do all the things right for me all the time. They don't. They're really good at specific tasks and they're getting better at at broader tasks, but they're still really good at specific tasks. The more prompts you give it, the more things you give it, the the more it seems. The more skills that you develop for the model, which is just another type of prompting um or another set of instructions, the better it gets at doing more tasks. So, a as we talk about like things that are like the
Ethical Considerations and Data Privacy
15:10
models, how they work and things like that, the model is the foundation of everything. I want to talk a little bit more about just what's cool, the good stuff, and and where I think this is going. I have been, you know, 25 plus years writing software and about, I don't know, 8 to 10 years, I guess, doing AI and machine learning. I've got friends that make jokes about me and they'll say like Kenny was doing machine learning before it was cool. And that's kind of true, but to me, it's always been cool, so I don't understand. But when you think about like the future or where we're going and how these things work, I feel like what we just did with LLMs, so like I was working for a company at the time when Google released BERT and they released the white paper that kind of kicked this whole thing off, which is the attention is all you need uh white paper. They realized that like training a model on more than one language at the same time, larger data sets improved its abilities, but nobody really at the time understood why.
16:02
Even now, we're not 100% sure why. But we've got some pretty good ideas now. But I feel like what happened is we figured out one technique. And in that one technique, we were like, "This is awesome. Let's just go bigger. Let's do more. And let's just wear this thing out." And what we've done is is we've it's almost like inside the human brain, there's I don't I don't know 16 20 areas.
16:24
Forgive me for you that do know the human brain, but you've got like emotions, you've got memory, you've got your ability to like logically think, you you've got the visual processing, audio processing, you got all these things in your brain and it's almost like we figured out one of them really really well. We said this is amazing. All in. This is AI. We did it finally and we started beating on it and then we started realizing like memory is important and LLM actually don't remember anything. We do a lot of things around them to give them that context back so they'll remember again. And so as I think about like where we're going in the future, how that's going to work, there's some really cool research right now around how to actually help the LLM have a a both short-term and long-term memory. And as I say that, my friends that are in data science are going to be like, "We've been doing long and short-term memory for a long time, LSTM, right, Kenny?" Yeah. Yeah. But that's for an individual node. And I'm talking about like the whole of the model having a true memory. Not just being able to take context and and add to your context and shove that down the chain again, but actually like as it's going down the chain, it's like remembering things in the same way the human brain seems to be able to do that interjection. You have a memory and you seem to be able to recall that and it adjusts your opinion or your position. And right now the LLMs do not work that way. But there is some pretty there's some pretty cool research going on there. So, I think I believe to go to the next level on these, we're going to we're going to learn that we need a few new techniques that we still don't have, a few new capabilities that we still don't have. And what we've done with agents is we give it a memory. And a memory is it finishes doing a process, a round of running, and then it will write a file that's just text to say like don't forget this, don't forget this. And then the next time it runs, it pulls those memories in. And and the sophisticated agents will actually take those memories and say, "This one doesn't matter. This one doesn't matter. This one, this one matters."
18:20
And it'll pull that memory in. And it just puts it into your prompt. You just don't see it cuz the agent's doing it under the hood. It sends your prompt plus a bunch of other things. And that's not really memory. It's starting again from fresh. The agent starts over from fresh with the larger set of text. And that text is its memory. And it's just not real. But I think I think in the next 12 to 18 months we're going to figure that out. Like there's a lot of really smart people working on it. I'm enjoying reading the research that's coming out of that area. And once we figure that out like true memory in an LLM, I think it's going to like double the capabilities of what they have now in the model. I also think on the future the we use a lot of what was called mixture of experts models now. So you'll have a like a a central model that will be able to decide which agents or models it needs to talk to inside of the the whole and it it's I'm going to try to simplify this.
19:17
Basically you you can think of it like when you talk to Gemini everybody Google doesn't say but we're pretty confident it's a mixture of experts model. So you talk to Gemini the core model then goes out to specific fine-tuned models to get back information on a specific set of topics.
19:32
So if I'm asking something about technology, it might have an expert in technology that it goes and it it takes its knowledge and the knowledge of that expert brings them back together and from the the group it comes back with an answer. If I ask about a business question in technology, it might have a business expert and a technology expert and pull back responses from both of those. It's really cool, but it's it's it's not it's almost like if I had a board for DevOcho that was helping me make decisions and know when to take risk or not take risk, know when to invest in this versus this. That's the mixture of experts. It's really valuable. And then there's this one model that's deciding which experts are right and which experts sound crazy. Like that's a cool feature. But where I think we're going in that idea is that core model will be an expert at multiple things, not specialized. This is kind of a prediction of the future and those are terrible and I'm not always right. But I think it's better if that mixture of experts, if you have like 16 20 experts in a group, if each of those experts knows like 10 things, your total capability is higher. And I think as we think through the neural networks, it's going to be possible to build pathways into each type of expertise in a lot of the same way that our brain works. There's a crazy thing about the human brain where if I if I learn to juggle, there is a physical characteristic on the surface of my brain that on my autopsy, if the person knows this, they'll be able to tell that I could juggle. Same with playing guitar. Same with playing piano. there's actual like wrinkles and structures that form in your brain when you learn that skill. That spot on your brain is where that skill lives. And so when you think about that neural network, if each expert had more than one expertise, I think we'd be in a really good spot. So I think that's where we're going. I also think we're going to learn to be more efficient with the parameters in a model. So right now, like I said, we have trillion parameter models. We're just throwing everything in the kitchen sink into that thing and hoping it comes out good. I think we're going to learn that there's a lot of like useless stuff that we're pushing into those models that we don't need and we're going to learn how to create more efficient better models. We're also like Moore's law computers are getting crazy better all the time. So my forward-looking idea there is like 2 years from now my phone in my pocket is going to be twice as powerful and servers are going to be twice as powerful and maybe more than twice as powerful. So a model that right now is impossible to run locally will be trivial to run locally in in 3 to 5 years. That's just how this has always worked, right? Like the computers that you needed to to play video games back in the day were like, "Wow, it's so powerful." And now you laugh at how terrible they are. And that cycle's always been that same 18 months to 36 months for cutting edge tech and then 5 years to 7 years for the stabilized tech.
22:21
And I think that's the future as well. That's what's going to happen. So that's my theory. I love to discuss this. If you want to hit me up on social media or LinkedIn, I'm happy to to talk about it. Okay. Before we wrap up my cautionary tale and how to protect against it, I was talking earlier about PHI and data and what not to send to the the frontier models like you know ChatGPT, Gemini, Claude or rock or any of them. There is a lot of advancements and capabilities in local models. Local models run on your hardware. The data never leaves your environment but they're not as good. And I don't want to try to act like they are. I would say trying to think of a good analogy here. If like Claude is currently the the top dog in the fight as of June of 2026, I expect Google to have a new model soon that based on their typical cadence and I guarantee you they're going to make sure that they're in competition. Um, and X is constantly changing theirs. I have no idea where it's at right now, but Grock is constantly iterating and of course ChatGPT also constant improvements. So today, Claude, as you're you're thinking about a local model and you think about a specific task, you want to size a model's parameters effectively to how complicated the task is. So if you have a task and let's say the task is I'm going to present you with five products and I want you to pick the best product to present to a person to try to do an upsell, right?
23:50
And so you need a model that's really good at understanding the options that it has to present, making a choice based on some kind of information it's learned about the customer. Maybe the the customer has a a list of products they've been looking at. Maybe there's some kind of history, you know, about this customer. And so you're going to pull all that together. You're going to give it to a model and it's going to produce a result. Now, an LLM is actually not the best use case for that. But let's just talk about it like it is because it it'll help push my analogy with the agent. If you go with a really low parameter model and and low still sounds pretty big when you say like 2 billion or 4 billion parameter model, it's going to be okay at making that decision. But if you step that up and you give it all the context of everything you know about the customer and you pump that into a larger model that's got like 20 or 30 billion parameters, it's going to make a better decision because it's better at processing that context. It knows more.
24:40
It's got more information inside the model. I actually have a really fun theory for why bigger parameter models perform better. And it's not my theory. It's one that I found. But basically, the idea is if you build a model that has a trillion parameters, it's almost like if you take the SAT and you you come up with every possible answer to every possible question they've ever asked and you throw all that in the model and then you get an SAT question, you already have the answer somewhere in that model. The model just has to find it and give it back. And so we build these trillion parameter models which basically puts all of human knowledge in it and somewhere in there is actually the answer. We just have to find it. And so that's the lottery theory that like if you throw enough if you play enough tickets in the lottery, you're going to win. If you make enough parameters in your model, you're going to have the right answer. So I think that's actually there's some truth to this. But um anyway, with our example, you want to rightsize the model to your task. So you want to pick the smallest model in a local setting that still does a good job performing the task because it's cheaper to run. You need a lower power GPU the smaller down you can move in that model. So I'll give an example. We were doing a project for a customer recently and they are huge. They have a huge volume of data and they wanted to do um AI decisions or AI suggestions on a bunch of attributes in um in the system. And so we started with a 70 billion parameter model that was a couple like I guess it was one generation old but when we started it was the the the newest thing. And we just went back recently and took a new generation model that only has I think we went 27 billion parameters. It's got fewer parameters, but it's a neuro generation. So, it's got a thinking capability. And the thinking capability is a clever way to just say it runs itself back through its own stack of a statistical engine. And as it's making that loop, it does a better job. It's just simply it's better because it's only half the number of parameters. It needs half the memory to run and load. It needs more like processing power, but half the memory. Memory is the expensive thing in data centers right now for video memory. So we were able to like basically double the throughput of this system that we had built by switching to a newer more capable model with lower parameters. And that's what I mean by right sizing like you try to figure out what is the right capability or what is the the right size for the accuracy the precision recall is what we say there for our task we're trying to solve. All right, my caution. I don't trust So maybe I'm skeptical. Maybe I'm just like paranoid. I don't trust a lot of these these large companies with my data. I don't believe them when they say we won't use it. It's just too tempting. Like if you're sending it like if you're a healthcare company and they're trying to improve their model on performance in healthcare and you're sending it data that it's giving you decisions back on and then you've got a human like human in the loop idea where they're saying yes, no, and they're giving more valuable information. They want that so bad. that is so valuable to them and I just I can't believe the temptation wouldn't be there to pull that and still include in the model. But the risk is if you think about that giant set of statistical nodes, those pins in that pachinko ball machine, your chances of your PHI or social security numbers or credit card numbers or whatever sensitive data you're storing ending up in that model are just really high. And I know it's hard to retrieve specifics out of it, but go ask a model about a celebrity and ask about an obscure fact about a celebrity and it's going to give you an answer that's almost always right. How does it know that I'm going to make this up that Keanu Reeves doesn't like broccoli? Like I he might, I don't know, but the model would know that because it read it somewhere on some page in some paragraph and it's got that in its data set.
28:20
If you send it the cancer diagnosis to a patient, it's going to know that Sally May has cancer and it's this type and it's this stage and it's going to know all that information and the model could return that information should somebody ask it. So to me, I never trust like personally identifiable information, PHI or any kind of sensitive numbers or information there. And I know models hallucinate. So I'll give you one more scary example. So, as I was starting our business, I went to Grock and I asked Grock if it had any information on other companies like mine that were the same size, basically trying to get an idea of how they structured things and their financials, like how do they budget, how do they approach things. And Grock may have lied. It may have hallucinated to me and just made stuff up. But it came back and it told me, and again, it probably lied, but it told me, I have 27 companies that have been sharing financial information with me that are similar nearshore custom software companies. here are the budgets that they've shared with me and and it gave me a summary of a table of a bunch of different uh budgets. It wouldn't give me their names, but I feel like it could have I feel like there was a prompt that was preventing that and if I kept trying it would I got a crazy story there.
29:28
There was a guy that told an AI model that if it didn't give up some credentials to an online service that a little kid was actually going to die. and it basically it guilted the model into deciding to give that set of credentials up. So that's terrifying. So anyway, I'm I'm going off on the rails, but I would recommend if you've got crazy sensitive data, you remove the sensitive pieces before you hit the the model and do that. Um to to sum up, to wrap up everything, if you have specific questions, reach out to me on LinkedIn. Of course, we would love to help you build AI agents in your company. So, if that's something that you need help with, we're there to help you. I also would would say if you are in a situation where you need to put some protections around or or safeguards around AI or just do some basic consulting around that, we're there for you.
30:18
We can help you guys do that. If you've made it this far in the video, it takes you just a second. It's free to hit the like and subscribe, but it really does send a signal to YouTube that this was a worthwhile uh watch and it it helps our channel. Thank you guys very much. Have a great day. I'm
DevOcho Youtube Outro Final.mp4
30:49
done.
