The Blueprint for Autonomous Work Agents | Gavriel Cohen, NanoClaw

You're here at the AI Engineer with Kovid Goyal, owner of Nano Clock. How has it been? >> It's been amazing. This has been such a great event. Yeah. Congrats for putting it on. >> Yeah, it's it's great to see you in person. I could see what you on online as well. And it sounds like you had a very eventful trip to Singapore. You met You met the Minister of Foreign Affairs. Yeah, this is like what did you do? What happened? They tell the story of like the Singapore trip. >> Yeah, so this really goes back maybe about a month now. I was actually on vacation like taking my first weekend off since putting Nano Clock out there. And finally took two days off at a nice hotel with my wife and then I I think it was Friday night. I was just scrolling on X and I see this somebody reposting a Facebook post from Minister of Foreign Affairs of Singapore where he did this whole write-up of how he's using Nano Clock, his whole setup, memory system, indexing, deployment, running it on a Raspberry Pi, and he called me out by name, which is like I was like, "Okay, that's My name isn't even on the repost. He's clearly nerding out and he went deep." And then I went and I found this post and he had done a Facebook post. And he posted this whole GitHub gist of his whole setup or whole write-up of his whole Nano Clock setup. And he's got this kind of second brain type of setup where he's got a Karpathy LLM Wiki with embeddings and some memory system. >> Which did you have that in your original design? No, right? >> No, not at all. That's all stuff that he's brought in. The memory system that he's used, Nemon, which has kind of gotten a little started to get a little bit popular now that he's posted about it. >> Find that. It sounds like is the 141 GitHub stars. I was like, "Let me check it." I was like, "What's the Where is this from?" Like >> When I first checked it, I think it had like 43 GitHub stars. So and that was after he started using it. So, he found this system, he liked it, he found Nano claw, and he put it all together himself. He he codes, um, and he he created this setup. He did a whole write-up. So, apparently he was migrating from the first version of Nano claw to the second version, and he created this whole, uh, write-up of his setup as part of the migration, and then just decided to post it. >> Yeah. >> And it started to go a bit viral on X, and I did a retweet of it, and then he retweeted my retweet, and we started to talk. >> Now you're best bros. >> Best bros, and and he said, you know, when if you're ever in, uh, Singapore, we should, uh, you know, I'd I'd love to, I don't know, have you over and and host you. And then, I think it was two, three days later you reached out, and you said, "We're doing this event." And I'm like, it just it's it's all perfect. I checked that he'd be around, and and he said, "Yes." And then you managed to get him to come down and and talk here, which is, wow. >> See, uh, I think what it did help that you were here. So, you wanted to be. And I think also, no, my orientation is a particular in the sense that we just want to promote the the industry in Singapore. I'm Singaporean. Uh, I live in I live in the US, but like, I think the government wants to support the we basically we basically we basically this. Um, I'm curious, we guys talk for 2 hours. Anything you can share about like surprising, uh, use cases of our whatever. It's like you talk about the tech stack. Like just like and how does he think about it that maybe open your mind is sort of, oh, you know, you should can you learn about? >> I think for me, his use case really helped for me to kind of crystallize what direction we want to build in. Going back, I would say a month ago, there were these two overlapping, but kind of distinct directions when we're thinking about adoption of claw type of agents, autonomous agents, in a business setting in a company. Uh, so there's the one where it's the team manages agents. So, it's agents that you build an agent factory or or agents that uh automate workflows. I gave a talk about that, the agent factory that we built, and we had started to build that already for ourselves. And we've been building it for a while. It's still kind of under construction. So, that's one to see factory. It's the team as a team working together to build the agents and managing them as a team. And then you've got the other side, which is personal agents in in a work setting, in a business setting, but individual people who have their agent, their assistant that's helping them do their job, and it's more one-to-one. And we were going working on both of those use cases both internally. We were using agents in both ways within our team. Uh and started to work with design partners that were interested in both use cases. So, we were working with a team where they wanted to give each person in their legal team their own personal assistant. And then we're working in with another uh design partner that wanted to give build agents that automate a workflow for their legal team. So, automate drafting first contracts. With the foreign minister or the Ministry of Foreign Affai- Ministry of Foreign Affairs uh use case, it really helped to crystallize that from my perspective, the way to start in a business intro- introducing agents to a company is to give each person their own agent. Uh because there's a certain knowledge and expertise and learning curve on how do you work with agents. What are they good at? What are they not good at? How do you prompt them? How do you uh build the instruction skills? Adjust those over time. Manage the uh context window. And it takes some time to learn that. And initially, one of the biggest mistakes that people make is that they just want to throw something in an agent and then walk away and expect to get a finished result at the end. You got to you got to put in the effort, right? It's the dream that we all have, and and that's the natural instinct, right? It's good to be to kind of be ambitious, but you have to work on it. You have to fine-tune not fine-tune a model, but but fine-tune the output by adjusting instructions, adjusting skills, adjusting what you put in, and and also iterating with the agent. From my perspective now, especially seeing his use case, the killer use case for Claude type of agents, autonomous agents today, is the second brain use case, where you're just kind of dumping in information, and you're not expecting it to give you ready-made output, but it's just collecting that information, building up its internal memory or knowledge graph or wiki wiki LM wiki Wikipedia of you, and then able to give you useful outputs at the end. >> Yeah, I've been using town assistant for that. My last in AI Europe, I talked about how I'm using Devin from LHR, a town assistant, even a little bit of uh so Victor as well. These are all like some of the the alternative personal assistants. I use open call personally, but I I think like there's this general field of like knowledge management that is being kept those that I think you guys had the very same effort. And like I think you have the simplicity and the privacy focus that that drew him, right? Like I think that's he mentioned that he looked at over call first. Uh did he talk about the the security side or like what, you know, the something over call a person? >> Yeah, the security side was the key thing that that that brought him in. There are a lot of people like him. I actually started to use open call, and we were using it for our business at the time. I was building a AI native marketing agency. We had some customers we were ramping up, and then we set up open call, and it started to manage our sales process. And very quickly, within like two days, it was doing the work of an employee, of a sales manager, just managing a whole pipeline. But then I started to dig in, and I started to see the size of the code base, and the number of dependencies, and some other things like logging all messages in plain text, that just made me a a bit apprehensive to use it for like production use cases, to build a business on it. So I felt fine using it personally, but when I was saying I'm going to connect it to my customer's data, and I'm going to start to build all these workflows that are running, and I'm trying to build my whole company on top of it. I didn't feel like it was stable enough for me. So, the key changes, first of all, made a really minimal code base with Nano Claw. So, I didn't clone Open Claw and change it. Just started from the ground up from scratch. Used a lot more components out of the box. So, use Agent SDK instead instead of building our own agent. Like Open Claw uses Pi, so then you got to build a lot of session management and compaction and a lot of other things. It's Pi is really minimal. Instead, we use Agent SDK, which comes with a lot of that stuff out of the box. Um only supporting one model initially, one agent initially, so saved a lot on that. Used integrated with things for connections to messaging channels. Like we integrated with Vercel's Chat SDK library that just gives you all of the messaging different messaging apps out of the box. So, really minimal, fewer dependencies. Uh but then the biggest thing is the isolation model. So, first of all, you run your whole Nano Claw in a VM or on its own Mac mini, and that's great, but you still need to isolate the agent in its own agent runtime and separate that from the messaging bridge that's connecting to Slack or Discord or whatever and the router and the other pieces that decide what gets pushed into the agent and what gets done with the agent's output. So, you need to have isolation of that. So, we run each agent in its own container. And then make sure that the agent doesn't have any credentials in its environment. And that's really big. If it's got credentials and it's touching unsanitized input, which for most useful use cases, it's going to be it's going to be, you know, reviewing a PR, it's it's got the whole PR input. Anybody could open a pull request to an open source repo. So, making sure there are no credentials in the agent's environment, so even if it gets prompt injected, it can't leak credentials, it can't leak API keys. And then the third key thing is separating, so the agent can use credentials. All requests coming out of the agent's environment get proxied through a vault and the vault adds credentials if the agent is supposed to have access and then the third thing is having access policies and access control including human in the loop approval. So, you can give your agent access to emails, maybe set a policy that it can read emails without any approval process, but if it wants to send an email, you get a message in whatever channel you've connected, so in Slack and you've got buttons approve, reject, you can see what it's trying to send and you decide if you should let it >> Yeah. I feel like he may not have like the fine brains understanding that you have, but like it the the messaging does come across really very your your DOS. That is what it is. Um yeah, I mean like what else have you found in this trip to Singapore? Like you met other users, like have you met other builders that inspired interesting conversations? >> I met a lot of really interesting people out here. Um you know, I I in my talk I put my agent out there to book book coffee chats with them. >> I think there's some kind of VPN proxy thing. It didn't think I was in Singapore because I I have my US hallway. Uh so it was doing something weird where like it didn't let me >> And the talks are streamed live on YouTube, right? >> Yeah. >> So, yeah, I just wanted to make sure that people who were here would be able to access and it didn't get taken down by people, you know, massive people connecting around the set. So, I did set it that it's geo-blocked to Singapore. Uh sorry about that. I'll open it up for you later. So, yeah, so I I I put my kind of agent out there and people chatted with it and then booked coffee chats and chatted with really interesting people building in especially memory students out here who are building really interesting things. >> Actually, I did want to ask do you have your own memory solution that you use? >> Personally, my what I'm using personally is is kind of LLM Wiki a of solution. I just point my agents at Karpathy's uh post and I say, "Read this. Look at this. Let's chat about it for a minute and then uh But Nana has built in kind of simplified version of that where it just has some basic instructions to the agent saying, "Create markdown files. Anything any anything substantive that the user shares with you, make sure to save it somewhere. Either it goes in your cloud.md or it goes in a markdown file or it goes in some other file, but it's got to be saved somewhere." >> Yeah, I I think my my probably both days is just that even using Obelus, it would create a lot of duplicate files, which is like not know that it already has a similar thing and then you just create another duplicate set of files. And now I have like two sources of truth, right? >> Yeah, you got to have other instructions on top of that saying, "Create an index of all your files and save that index in your in your instructions. See what files you have, then go and save it." And then you do need to have like a background process that runs daily or or every few days looking over your your memory files and finding the duplicates and flagging them. Uh yeah, there's definitely a lot more to do there that we're not doing yet. I know some of the other kind of claws are >> Anything off the shelf, sort of say. Sounds like you just want your own. >> I'm not sure that like retrieval based solutions are are ideal for like personal assistant because it's very specific and it isn't necessarily uh something that retrieval is really good at. Like if you ask your assistant, "Um what are the most important things I should be focusing on this week?" There's no retrieval based search, semantic search, keyword matching search that's going to be able to give your agent that information. But if you have a good LM Wiki that has like here are the projects you're working on, here's the timeline, here are the you know, here's a log of the different calls you've had this week, it can go through a few different files and collect the things and give you that list. >> Yeah, amazing. Um yeah, it's really cool. I I think last version is by your company, right? Uh when I first messaged you, you I still we still had a the marketing stuff. But now you've started Nanocloud Co. Uh what is Nanocloud Co? What's the vision? Uh is it going to be a VC style startup or sounds about right? >> Nanocloud is we shifted gears. We were when I first spoke to you, when I first created Open Claw, we were building a AI native marketing agency. We had some customers. We were ramping up. It was going well. I built uh Nanocloud just for my own use as a side project on the weekend, launched it on uh Hacker News, MIT license. >> That's where I first found it. I tweeted about it. >> Yeah, yeah, or immediately within like hours, I think after it came out, I >> Okay, like I can't read this code base. I cannot read Open Claw code base. >> That was the idea behind it cuz I said I'm going to build something that I have an idea of the right right way to structure it to make it kind of sane and secure, but I I don't quite trust myself with it, so I want to make it readable and let other people look at it and validate it. And since then I've had a lot of security experts review it and and critique it and give feedback and point out small things, but what's come out of all of that is that the core approach seems to be sound because nobody's pointed out an issue with uh general approach. So, we were building that agency. I built Nanocloud for myself. It started to get a ton of uh traction and attention. And then Karpathy uh tweeted about it and it went to just a whole 'nother level. And for a little while we were debating if we should build the agency or build Nanocloud. And then after Karpathy, it just became clear like there's such a big community, so much energy behind this. So, we went in all in on it. Um we we have a company now. We've got 10 people on the team. Uh and what happened was the early adopters, I think not just of Nanocloud, but Open Claw, Hermes, every all the claws have been a lot of VC CEOs, executives who are using it themselves and get really excited about it. And we've had tons of CEOs and other executives approaching us and saying, "Hey, I built out this great setup for myself. I've got these three different agents, this memory system." Just like uh the the Minister of Foreign Affairs here. Uh and they say, "I want to roll this out to everybody in my team. But I don't want to become the IT guy who's now like fixing their agents and debugging the memory issues. Can you guys come and help me set this up for my team?" >> Yeah, so now you're now an agent lab like you're in willing it all of this hosted mental. >> Yeah, so we're going to do deployments uh and for a lot of companies >> Things I said in my talk that would be right right over beer. >> I'm going to go back and catch up on some of the talks. >> Skilled at the dev in journey but like more focused on knowledge work rather than coding. >> Yeah, I'm going to get catch up on your talk and a few others. So, yeah, the going to do the deployments uh for a lot of companies it's going to mean deploying on their infrastructure to their cloud. >> Taking notes from my team. It is exactly those like people are going to tell you about like SSO and like uh uh BBC peering and like on-prem deployments, all that stuff. >> Yeah, we're going to have to hook up to their credential management system like >> Getting their credentials from your their vault or yeah, on AI is like actually a lot of it is not yet. >> It's not. But uh you have to understand the AI. I think that's what's missing for a lot of these companies. They have good engineers, they've got good devops. I spoke to a lot of these companies. Um they've got good security teams. But if you don't have the piece of the AI engineering, it's hard to put together the different pieces and to have confidence that what you built is secure, it works, and and it's it's the right setup. So, having I I I think of it more as a partnership between AI engineers coming in and working with the their devops, their security team, their IT department, and helping them get set up the initial setup, and then afterwards helping them manage it over time. Uh initially, you got to connect it to their credential management, observability, their other security systems, um and then help them get set up with their initial integration with their internal data sources. Um but then over time, their IT team, their security team can pick up those pieces, and when there's a new request for integrating with a new data source, they can take that on their own. But there's still management to do in terms of upgrading and maintaining it. Agents are different than normal software in that normal enterprise software, you can deploy it, put it on some server, and let it run for like 5 years, and as long as you never touch it, it just works. Uh agents don't really work that way. The core thing that you're building on is constantly changing over time. You can't just run 4.6 for the next 3 years and just leave it. You got to upgrade to 4.8, 4.9, 5. And every one of those upgrades changes things. Uh and and the labs are putting out new features, new capabilities, uh memory is getting baked into the LLM, getting baked into the agents, uh and you have to be constantly updating in order to just stay kind of at the at the front. So, that's what we're going to be doing. We started to do our first deployments. Uh we've got over 100 companies that have approached us interested in rolling out agents. Um and yeah, big announcement's coming up. >> Two philosophical questions. One, uh for your agent factory, do you think that Git and GitHub will last? >> I think it's going to have to last for us because it's we're an open source project, and you've got the culture in the community. >> Uh I'm trying to see what comes after GitHub. >> Yeah. >> have any idea. >> There is a weirdness there, right? Because the whole agent factory should be on GitHub and not happening in Slack. But because it's an open source project, like if it was closed source, we would just have the agents leaving responses, doing the reviews, test results, it would just all be in the in the pull requests on GitHub. Because it's open source, I don't want to have an internal discussion back and forth, leaving feedback for my agent meta, you know, feedback on the factory itself happening in a public uh GitHub thread. It could be interesting, but it seems a bit off. Uh but if this was internal, I would want it to be on GitHub, and then I think we would be running into the issues where GitHub probably still isn't quite the right format of that. >> My my people do all their stuff in Slack, and then it's they pay me to stamp the the PR in GitHub. But like, why do I need to Why do I need GitHub at all, you know? Like, I should just be in the Slack. >> Okay, so maybe our approach is the right way. Wait. So, we have the whole thing in Slack, and then >> That's how I'm running IE. >> And all the way till, you know, merge, and then you hit approve, and it gets merged. Um and I think there's a lot to do. Slack has good pretty good UI. You can you can build out some good uh user experiences just in Slack. >> So, Slack over what? Can you use this WhatsApp? You use Slack. There's like a lot of Telegram. I I like I like Telegram's like there's so much usage that I'm bought to Telegram. Yeah. But you can't do you know that. >> I can do my uh my tiers of our messaging apps. So, I think uh S tier would be uh Slack. Uh and then Discord, Teams. And then goes like Telegram, and then WhatsApp. They're very very limited in what what you can do, and it's hard. Like, you have to have your own phone number if you want like a decent experience. Uh chatting putting the bot on your own number, it makes sense if you wanted to just see your messages read-only, but to chat with the bot on your own number, it's I mess you up. >> I haven't experimented with it at all. >> Oh my god, it's uh terrible. Uh you can't I I can't message myself because I needed a separate number to to receive it. So, I I just gave up. >> Okay, so that's below WhatsApp on that tier list. We've filled out our whole tier list. >> See what worse it is. Obviously, you have to buy you have to buy your own number. But like, it's annoying. I I think there's some services that do that. Anyway, uh last last question. What can people help you out? What are you looking for help? What are you looking at like What is a question you want answered? >> Managing open source projects today is is a huge challenge. So like I would say I won't even say for for for AI engineers, I won't even say contribute a pull request. It's like >> I mean >> It's part of the problem. Yeah, it's like triage. Help us figure out how to manage a open source project in a better way. Um it's a big open challenge. It's like this it's an arms race where coding agents have just made it exponentially easier for people to open pull requests. And then it's so hard to to triage, to review, to understand that it's aligned with >> The same as what you saying saying, right? No more pull requests, only prompt requests. Like don't give me your code, give me your what what your use cases. And then something that comes to my mind after having this discussion is you have all these like oh you know, my bug here, my feature request here, whatever. It should just come in all go into a wiki of future developments. And then you pull from the wiki. So like the wiki is kind of a buffer. All the changes come in as a continuous stream stuff. But it becomes a wiki and then you pull from the wiki as you as you eat. >> Yeah. We have started to build a wiki for our agent factory. >> Yeah. >> So as we're developing it, we're adding to the wiki kind of on each merge, on each commit. And then I think that is going to become a standard part of building open source project. So what you guys are doing with is it Deep Wiki? That's awesome. I think though that it's going to become a standard for teams to build out the wiki as they're doing the development in real time. And then yeah, the the the future development is in the wiki linked to the different parts of the project that it's related to and the bugs are part of that wiki. And then as you're developing it, each time you start working, you pull information from the wiki to get context and then you finish the PR, finish commit and you push back to the wiki some more context. >> Yeah, great, great. Well, thanks for coming. I'm so glad to meet you. Uh sure this is not the last time but we'll see you. All right, but hopefully this has been a nice trip so far. >> It's been awesome. Yeah. >> Great to be here. >> Thanks.