0:01
[music] Hello, I am Kenny Pyatt. I'm the founder and CEO of DevOcho and I am uh also the host of the DevOcho Way Podcast. So today we're going to talk about a few different topics all around the theme of AI as it's always the hottest topic of everything that we discuss and it's what everybody talks about right now. Anyway, um I I feel like I have um the authority to speak on this subject because I have
The Challenge of Hiring Engineers in the Age of AI
0:40
more than 20 years as a software developer and I have more than 10 years building uh writing and developing custom AI machine learning models and things of that nature. So this is an area that I know really really well. So we're going to jump in. The first area I kind of want to talk about is what it looks like right now to hire specifically engineers or software developers or technical workers. So so the ways that we used to do that is is we would go in and we would either give you a take-home test or a quick questionnaire and you'd fill that out and that was kind of our screening. And if if that didn't work, if you didn't do a good job, we didn't interview you. It was wonderful. But now with coding assistance, that's useless. The other thing that we would do um is bring you into the office and have you do a whiteboard exercise or something like that. But with remote work and hybrid work, that was really hard to do. So, we're in a spot today right now where your resume, people are just making up stuff on their resume. AI is writing resumes. You can't really trust the resume like not like you could. And then on the other side, you can't do the take-home tests anymore because the AI makes that free. They can they can always solve it. So So what do you do?
1:54
How do you actually have an interview? And I we we literally This is crazy. I we were interviewing a candidate a couple weeks ago and he was wearing glasses and we could see the reflection
AI-Generated Resumes and Interview Responses
2:04
of his glasses because we've got a giant TV in our conference room. We could see this this this computer screen and he had AI up and you could see it as I would ask a question. It was typing what I said and then it would give him the answer to say to me and bro was straight reading the responses from the AI and I'm ornery like I'm kind of a ornery person and so I was like trying to think of something to say to the AI so that he would be reading the response back. I didn't really have a good one and I I'm going to come up with the one that if I catch this moment again I'm going to come up with a good response. I did ask a question basically like why would I hire you instead of just using AI and he got really uncomfortable. So I thought that was at least a little bit of karma in that moment. The the way we're handling this and the way that we're we're working we use AI coding assistance in our company. Almost all of our customers will let us do that. We do have an old school customer that told us we couldn't, but everybody pretty much
The Shift from Writing Code to Reading and Reviewing Code
3:10
everybody's using coding assistants and now you read code a lot more than you write code. So the test the the skills that we're looking for are that. So if you go back I think it was like 19 like 90 mid 80s like 1990 I can't remember the Python programming language is being developed and it must have been later like in the 90s but like Guido van Rossum is the creator of Python and Guido said code is read six times more often than it's written and so for the longest time in interviews the skill that we were looking for was your ability to write code. Now, the skill we're looking for is more the ability to read and correct code because the coding assistants are writing a lot of the code. You just need to be able to get in there and and read it and make sure that the the code makes sense, that the AI is not doing something crazy. So, the way we handle interviews now is we put code on the screen for people to read and find errors with, find mistakes with. We do ask some super basic programs. We do still do interviews in person. Um and we'll ask like uh really basic questions around just like looping really basic data structures enough to get the idea this person actually is a programmer and then we swing straight into logic problems and code finding code issues.
DevOcho's New Approach to Technical Interviews
4:31
You just can't believe everybody. We will have people put technologies on their resume or or things that they can have and and and I'll ask a simple question about the technology they put on their resume and they don't have have an answer for it because the AI told them they were supposed to have that um to get the job.
4:50
Next thing in this that's actually kind of fun to me. We also do a um test drive with people. So we'll bring them into the office. We'll have them, normally we try to bring somebody in, eat lunch with them, and then have an afternoon and we'll give them an assignment and let them build something with their favorite AI, their favorite IDE, their favorite programming language, and at the end of the day, they show us their work. So, it's kind of a short in the office, use the tools, prove to us that you can build it. But but basically what we're looking for is can you explain what happened? What did the AI build? How
The Challenge of Hiring and Training Junior Developers
5:31
does it work? Explain it to me. And that's a place that has been really interesting. We have a lot of candidates that cannot even explain like we had a candidate, this was a few months ago, he could explain the backend portions of the code, but he couldn't explain nothing from the front-end portions of the code. He didn't know what the system did um or how it was even built. So, he hadn't even looked at the front end at all. The AI built the whole thing and he just was like, "All right, ship it." Well, that's super dangerous. If you can't explain what's going on, if you're not reading what's going on, the AI is not doing it right. It's making mistakes and you've got to be involved in that.
6:10
The other thing that's really interesting in the the hiring right now is a lot of companies, this is always a challenge, but it's getting even more of a challenge right now. People don't want to hire junior engineers. And it's partly because I guess there's a like a semi-legitimate lack of trust. If the junior can build code fast that looks to be of good quality, but they don't have the experience, they haven't made the mistakes to understand that how it could be a problem. It's kind of dangerous.
6:40
So, the last 6 months there's been a real like almost like nobody's hiring junior developers. And I think in two years from now that'll be a huge issue for the industry. This is not really new. We've had this issue for the whole time I've been in in leadership and management. Um you go into a new project, you hire three senior guys and you turn them loose and they build something for you. That's a
Using AI to Screen Job Applications
7:06
lot easier than oh man, let's hire a senior and a mid and two juniors and let's go build something. And there's also an argument that it can be cheaper. three seniors versus five or even six people. But the reality is if we don't have juniors, eventually we don't have programmers. So we have to we have to do that. And so what what we do is we have a training program. So we bring in the interns or we bring in junior programmers. We we everybody goes to training program but the juniors really like have to do it.
7:42
And for those, we literally tell them not to use AI. Um, and if we find they're using AI, I have a real issue with it because the goal of that is to teach them how to think logically through the process so that they're able to to do the work of, you know, using AI and building things.
8:02
And if they haven't known themselves, they're never going to be able to do that. So, it's kind of it's kind of the funny thing here. Like AI on one hand is making it harder for us to interview and find people and at the same time AI on the other hand is making it easier to screen applications and do things like that. And I'll give an example. So, we opened 10 positions recently in our company. We're hiring a couple different skill sets across the board. We're hiring a couple of of people for our HR team, an accountant, an administrative assistant for me. We're hiring a handful of technical people as well. And in our country, in the Dominican Republic, there is 3 million people that live in the capital. There's a million people that live in Santiago, the second largest city, and then there's the rest of the country. And so you've got 11 million people here. So 30ish% of the people live in the capital. And for some reason the people in the capital just consistently apply to our jobs in Santiago were about two hours away something like that with with traffic maybe a little more. They're not going to commute to work every day and they're just not paying attention. I it's almost like the thought is that well of course this job is in the capital. It's a tech job but it's not. And so I I've got a application of AI where it takes a resume, extracts their address, figures out they're not in our city, and helps me not have to review those, and we save the resumes just maybe someday we will open an office in the capital. But that that's a good use of AI for us. What I don't like AI doing, u what I I've seen in the workday scandal and all those things is using AI to kind of grade or make a judgment about the the resume.
9:53
That's really dangerous. Again, I build custom AI. I build machine learning models. Like I do these things all the time and I I still don't think AI is
When AI Is Not the Right Solution
10:02
quite ready to decide. Now, I don't have a problem with prioritizing. Like if you're using AI to say like, hey, I think this resume might be better. Um, that makes sense to me. And I do also appreciate AI is allowing these people to apply to dozens or hundreds of jobs at a time and are doing it. We got on 10 positions. We have in just a few days more than 500 applicants.
10:27
It's crazy. Trying to process 500 résumés is impossible when we're a small company, right? We have right now we have one HR person. We've just extended today offers for two more. But like three women trying to to read 500 resumes, make phone calls, set up interviews, like process those people, like that's impossible. So you have to do something to prioritize. I understand that. But don't don't trust the AI.
10:55
Double check what's going on with the models that you're using or things that you're doing. Um, if you're curious about the prioritization, I'll be a geek for just a second. I uh we have an AI server here in the office with three GPUs and I used a GEMMA4 I think it was a 12 billion parameter model and a really simple prompt that was find the address in this resume and give it to me in a JSON object and then from there we just put the address on a we have a a Kanban board we put the address on the board and that helps us see capital or not capital and then I've got a button that I click that if they're in the capital and it saves a resume and it sends them a message that this job is currently in Santiago.
11:37
All right, so that's around AI and hiring. Now, I'd like to to switch to my my second idea of the three ideas I want to cover with. Everything doesn't need AI. I guess I I should explain. AI is not always the best solution. And I love AI. I absolutely am constantly
Rules-Based vs. AI-Powered Systems
12:02
finding new and interesting uses for AI and I've been doing it my whole almost my whole career since the I mean basically since like 2006 2005 2006 I've been doing things with statistical based machine learning. I love AI but where AI like fits really well is in any place where the data is not clear. you you've got some like unstructured stuff you need to work with. Like these are really good use cases for AI. And any place that the data is structured, it is clear like decisions are easy to make. Don't use AI to make those decisions. It's such a waste of time, resources, money. And I' I've got a a bunch of examples I put together, but basically like if you can write rules around the thing that you want to build, you probably don't need AI for that. If you need it to be auditable, you probably don't need AI for that. If you always want the same input going into the system giving the same output, that is that is a place that AI doesn't make sense. where AI does make sense, you have unstructured data that you need to process. So like medical notes is kind of a famous example there. Doctors just start dictating things. They're writing whatever and you need to process that and actually extract some some information from it. If if variance is tolerable, meaning if running the same thing multiple times can give different results and that's okay. That is a really good use for AI. Also, anytime a human is going to review the outcome, probably okay to use AI. Where do you where do you need to put a human with AI like kind of blend the human in the loop so to speak? For me, this anywhere that it's really like a regulated industry or it's very high consequence. Let the model kind of make a suggestion and a human approved that suggestion. Does this have to produce an identical answer every time? If yes, just write code deterministic. That's what you want. You don't have to ask any more questions.
14:14
Does someone have to defend this decision later to a regulator, an auditor, or a customer? If if yes, then then you should have a human involved in the process. Is the
AI, Healthcare, and Human Oversight
14:27
input messy and the the tolerance for variance acceptable? That's AI. you can just use AI to solve those problems. Um I'm I'm going to give a couple examples here. We um we've got a couple different uh product uh projects in healthcare and when you're trying to make any kind of healthcare decision, it's better to let a human be the one that makes that decision. So we will read we'll use AI and we will read um you know medical notes we'll read pathology radiology like like the labs and and we'll do a suggestion from AI and present that suggestion to a human we'll make our cases for why we've made the decision or why the AI has made the decision and at the bottom a human's going like yep that looks right or no detailing this and in each case you want to if you can capture notes, explain why the AI was right or wrong, and then you can use that to improve it. The that that's a that's a really safe use case. It's something that like with our resume problem, we have 500 resumes.
15:39
I can't read 500 resumes for 10 positions. I I can't do 80 phone screens. I can't do um 25. I guess we're we will typically do three in-person interviews per position. Three to four. Um, so you're you're looking at 40 in-person interviews that we're going to do. So you extrapolate that out. How do you process 500 interviews? Well, using AI to kind of help sort and prioritize those is a valid way to do that. I don't think using AI to reject them is valid at all.
16:13
So that kind of explains my position on on that part of this subject. I'll give another example with the some software that we're doing for a customer right now. They have it's something like 400,000 patients in a database and they need to find patients that match specific um like specific diseases, specific stages, specific um conditions, uh specific health markers. and a human reading, you know, nearly half a million patient records is impossible. So using something that can go through that and and in a lot of the cases, unstructured data, this is the perfect use case for AI. AI can make those those reads. It it's going to get it right a lot of the time, not all of the time. Um, my last segment, the last thing I want to talk to you about today is just kind of around AI and and human judgment and specifically engineering judgment, our business. So, this is fun. So, let's go back last year.
17:20
We officially opened the doors of our business in April. We started hiring people last year earlier in the year. And as we're hiring people, all these coding assistants are starting to come out. And the AI is starting to get better and better at making code. And and everybody keeps telling me, "Oh, Kenny, this is the absolute wrong time to start a custom software business. AI is about to replace all the software engineers in the planet." And I was sitting I was sitting in my apartment here in in Santiago, Dominican Republic. And I was really philosophizing on this. And I was like, my whole life I've been into computers and technology since I was 8 years old. My dad brought home a a personal computer, a trash 80. And um I
Why Engineering Judgment Still Matters
18:08
started playing with it and getting a chance to see this thing. And I've been that way my whole life. And when I look at society, most people don't like computers at that level. Most people don't like to play with or work on computers past just a simple piece of software that they're using. They they might be really good at Excel, but they don't care to make Excel. They might be really good at playing a video game, but they don't really want to understand their BIOS inside the processor or the balance between RAM and swap space inside of a server. Like they don't care about that stuff and I always have. And when I look at the gamut of people in society, there's somewhere around 2 or 3% of people that like legitimately care about technology. And so I was like, is AI going to change that? Does AI bring more people into technology? Well, probably yes. There are people that wanted to be programmers, but programming's hard and they couldn't they just couldn't handle it or they just didn't like fighting with the semicolon that was missing. Like they just didn't want to deal with that stuff. And so with coding assistance, it it opens up more people to have access, but it's still programming. like it's still like understanding the difference between swap space and and memory usage and and it's still like installing software and setting up data. like it's still all the deep things like none of that changed and last year I was sitting there going none of that's going to be different but the demand is going to go up because it's going to be faster and cheaper for the same people to build software than it was and that is exactly what's happened and we're hiring as fast as we can we're growing as fast as we can but we're trying to make sure we always do a good job so AI didn't remove the need for engineering judgment.
AI, Software Development, and the Cost of Bad Code
20:12
It just it it just raised the price of not having it. If you have people building code in your organizations, using AI without an engineer watching it, looking at it, seeing it, you're really asking for it. And I'm not trying to be like a doomsayer or oh well you need to use a professional like no I think there are super smart people that can build software without engineers. I don't think that's the norm I guess is a better way to say that. There are probably tens of thousands of people that can do that. But what if you pick one of the people that isn't in that group having an engineer that can like look at things like like I'll give an example coding assistance the large language models learned from code on Stack Overflow. Stack Overflow is full of tens of thousands of examples of incorrect code where people couldn't solve the problem with other people below that coming up with solutions to the problem.
21:11
So the the models were trained on both the problems and the the best practices, but also in that set of stuff on Stack Overflow are tens of thousands of horribly wrong examples of code. And so the AI had to kind of as they were building their neural networks, they had to kind of label good code, bad code, good code, bad code. And they didn't get it right. And they're getting better and it's constantly getting better. But they also went to GitHub. And on GitHub, they downloaded probably all of the open source.
21:44
Anything that wasn't protected, they downloaded it. There's probably some stuff that was protected that still got downloaded. I don't know. But in there is examples of incredibly good code and there's examples of incredibly bad code. For example, about I don't know 50 times a day right now, you find out about a vibe-coded application. Vibe-coded meaning a nonprofessional engineer built it, and they're getting hacked. It is constant right now. It gives me so much anxiety and I would rather I think it's okay if you vibe code something and you run it on your computer and you use it. I think that's awesome. I love that. I think the second you decide you want to put it online, it's almost like going to build your own house. Are you really good at building houses? Then do it, man. Live it up. Are you an architect or somebody that knows how to build a house? Do it. Phenomenal.
22:48
Are you this guy that just kind of sort of thought you had a good idea one day for building a house? Yeah, you probably shouldn't do that, right? You probably ought to hire somebody to help you do it the right way. Now, let's go past a house. What if you're trying to build like a five-story building? What if there's going to be a parking garage underneath that five-story building? How are you going to do the foundation?
23:09
Like, how does this all work? Man, that's a place that you want an engineer, right? You want somebody that knows what they're doing, that's done it a couple times. This is the same case. Like, the coding assistants are going to make it sound like they know everything all the time.
23:25
I spend most of my time now fixing, correcting, coercing these agents to do things the right way. So, I I'll kind of back up and wrap up all of my my three thoughts here. With hiring people, AI has made that harder because people on the technical side can kind of fake their skills. It's really hard to know what's real, what's not. We solve that by doing in-person interviews when we can. Quick dry erase board exercises just to get sanity checks. Nothing crazy. And then we we roll to an actual go build something. Um we use the same exercise for everybody.
Choosing the Right Technical Approach
24:06
We change it kind of every couple months. We have to change it because people tell people what we do and so then they people have already practiced stuff or they show up and it's mysteriously done in like 15 minutes.
24:17
But anyway, um, so we we rotate our assignments, but it's never anything that we're going to use for the company. We're not trying to get free work out of anybody. We're literally just trying to understand, can you really do this right on the to use AI or not to use AI spectrum, it's cheaper not to use AI. So I tend to tell people more often than not, we don't need AI here. A lot of people are really surprised when I say that. I'll give an example. In the the the data as a service project that we built, our original idea of what would be the best solution was just a vector database and then a model to like make a choice from the results of the vector database. So we did experiments there and it worked but it wasn't awesome and we weren't not hitting that 100% recall for sure. So then the idea was what if we use the vector database to classify K nearest neighbor if you're want to understand what I'm talking about there.
25:18
So you figure out the closest vectors that's your group the most out of that group of the same classification is the crossation that worked incredibly well for this data set these data sets. It was in super impressive how often it was right like near 100% of the time it was correct. Um, again, millions of records, it's easy to build a good setup, right?
25:38
So, that's a place where we're using a model to make a a vector. We're embedding that vector in a database. Vector is just a fancy math term for um where something is positioned and in the direction it's pointed in a 3D space.
25:53
From there, the next level down was actually deterministic. Um, so we used a a machine learning technique to classify
The Risks of AI-Generated Software
26:02
and bucket and figure out the area we're playing in. But then after that, the actual match, the actual moment of doing the match, we've got all these beautiful attributes. We sometimes have a product ID, uh, we've got a manufacturer most of the time. Like in that moment, it's really a lot easier to use a set of rules to make a really like this must be this because it follows these rules.
26:26
That is not a best practiced use case for AI. And if I was dumb, I would have been like, you know what, that first one's close enough. Let's ship it. But we wanted to do the right thing for our customer and our customers customers. So we did experiments and we did tons of experiments until we really figured out this is actually on this type of data. This is the right way to approach this.
26:52
Those experiments are not cheap. They take a lot of time. They take expertise and again the AI models will tell you this is the right way. We got it and they're like well I need a different way and they're like nope that was the right way. So having an engineer that can coax it out is really valuable. All right there's the saying bad code is cheaper to produce than ever. So the cost of not recognizing it went up. So get a junior developer, throw them into the mix, they AI generate a bunch of code and they don't notice any problems with it and they ship it and then it causes huge problems. These are stories that really hurt my soul, but I I'll tell one example. I have a friend that works for a company um in the United States. How's that for vague? And um they just got a bill for like a little over $120,000 um because they had integrated Claude into one of their systems and they didn't really have any kind of throttling in place or caching in place and the models went nuts and processed a whole lot of extra data that they didn't need to process. And so part of having an an engineering team
Three Practical Takeaways for Using AI
28:07
and a good group of people testing things is to stop that $120,000 random overage on top of what they normally would spend. All right, I got three quick takeaways that I'll give you guys. Swap your take-home test for code reviews. Like have people like read code and look for issues in code. That's what they're doing with AI now. take take any kind of AI feature you're planning and ask those three questions I gave you like can it be rules-based is it okay if there's variance in the output um is the data structured or unstructured and then the last one is if you've already shipped an AI uh feature how are you evaluating it and how are you monitoring it anytime that you release AI like we we try to like it's not perfect always but when we can we calculate something called the F1 score that's the blend of precision and recall. Um, a lot of people call that accuracy and I've got friends that are data scientists that the word accuracy like makes them super uncomfortable. But the general idea is like do you have the the precision and recall set up the correct balance for always gets it right or it's okay to get it wrong sometimes but find a larger group or a larger sample. You should be measuring and looking at your your metrics around your AI all the time. You shouldn't just release it and hope. Um hope is not a strategy as they say.
29:38
All right. Thank you guys very much. Um I love questions. I do get questions occasionally emailed to me. You can
Final Thoughts and Closing
29:46
email me at [email protected] . I need to tell you because my marketing person is right off the camera. Please subscribe. Please like. Those are really powerful signals for us. They help us a lot. If you've made it all to the end of the podcast, come visit us. We'll host you. We love it when technology people, business people come down. Um we've got a really cool company here. Um I'm I'm really busy right now, so maybe shooting an email first to make sure is a good time. But but really honestly, we love it when people come down here. Um, it's great due diligence. It's an excellent write off and we are just a little over an hour from one of the best beaches in the world. So, if if you come and visit us for a couple days and then just, you know, your flight's a couple days later and you want to go visit the beach, we'll we'll help you find a good resort or a good Airbnb and and get you settled. Um, thank you guys again for watching. Again, the DevOcho Way, devocho.com if you want to learn more about our business. Thank you.
30:49
[music]
