From fork() to Fleet: Designing an Agent Sandbox Cloud — Abhishek Bhardwaj, OpenAI

[music] Welcome everyone. Can you guys hear me? Okay. >> Yeah. >> I've been standing here for 15 minutes without saying anything. So, we can start now. My name is Abishek. I'm on the RL and agent infrastructure team at OpenAI. Um what that means is we work on the infra for reinforcement learning specifically and on the product side we also develop infra that helps run untrusted code as part of chat GPT codeex web securely and reliably at scale. This talk is called from folk to fleet designing an agent sandbox cloud. Um I'll be very clear that there are a lot of words in the title that have OS and infra concepts but this is the first principles talk. Uh so we'll cover what sandboxes are and why they are needed from first principles. We will also try to cover design intuitions around designing an agent sandbox uh cloud to run sandboxes securely and reliably at scale. So if some of these words don't mean anything, don't be worried. We'll explain from first principles and and go from there. Um last year I gave a t talk called uh how to build an AI sandbox from scratch. uh if if you're interested you can uh look at that talk as well. Think of this as a spiritual sequel uh to that talk. Okay. So now let's forget about sandboxes or clouds for a second. Let's just go back in time. Chad GBT came out. It's a very large pre-trained model. Uh people ask all sorts of questions and it responds really really well. Really really humanlike answers. But when people ask questions like what is 3 plus 3 or what how many hours in strawberry? Sometimes it works and sometimes it doesn't. It answers questions like what is 3+ 3 quite well because it's trained on the entire internet and apparently people have written 3 + 3 equal to 6 many many times on the internet. So it gets it right. Uh but people haven't asked how many hours in strawberry enough on the internet and so it gets it wrong. And so it's obvious that for anything code or math related uh or any problem which has a verifiable reward which means that it can be tested that it's true or false uh the model needs something more and the key unlock was that given the model uh tool calling capability or a way to execute code the model gets these verifiable reward questions around code and math correctly. And thus from first principles came that if you give the models the ability to execute code it can hill climb and be very very good at uh math code and other domains that have verifiable rewards like cut to 2026 and we are seeing the the consequences of doing that at scale. Uh so now how can it answer what is 3 + 3 and how can it answer how many hours in strawberry? Well, it can write code to do this. So if you see the diagram like uh we have a training loop and the training loop gives it task or questions and then the training and the harness then parses the response of the model and the response might say hey execute code on my behalf. Uh the the harness is responsible for executing the code and then a grader judges whether the answer is correct or not and then the training loop backrops and changes the weights and that's how we train it to do two things. We train it to call code execution or tools on certain classes of problems and secondly we ensure that the code it executes actually solves the problem. So this is why like pool calling is important uh on the training side. Now let's talk about the product side. In the previous slide we showed how the models are trained to emit uh code in order to solve certain tasks. The agent executes the code and we verify the reward. Well, all of this is useless if we don't support this model on the product side. So, it's basically the same slide as before, but we don't have a training loop. And now the harness is parsing the response and calling uh the tools and executing the code somewhere, right? So, where is this code or tool being executed? Right? This could be your laptop like with codeex or any agent you want or it could be a node on the cloud with codeex web or charge GPT. While we can expect the model to generate non-malicious code but good security practices and hygiene mean that we want to protect our environment whether it's a laptop or the cloud node. Attacks can be like intentional or unintentional and we want to protect against those and it could be like trying to get root on your system or trying to exploit a kernel variability. The models are getting really really big and they might try to help you in an overzealous fashion and try to get root to do so. So we want to try and avoid and make sure like it doesn't attack the node or where it's running. Thus this is where the sandbox comes in. We need the sandbox to run this untrusted code and ensure that it can do its work but it shouldn't be able to exploit and revulnerabilities and get root on your system in the cloud. Other sandboxes with other users data must might be running as well and we don't want it attacking and getting data of other users out. So a sandbox is a place uh an environment in which you can run these tool calls and execute code on behalf of the model securely and it could be on your laptop or it could be on the cloud. Uh I I think that like everyone's probably used OpenClaw or any agent like that. Um I think it's a very very very big peak into what's to come. A lot of these agents are running locally on your laptops. And I think it's it's kind of a slap on the face for 20 years of cloud computing that everyone's running this locally on on their laptops. And if you see the the image, I don't know if this is an actual product, but it's very funny because the lid of the laptop is open. It's because you don't want your agents to sleep. Well, we have a whole cloud. There are like industries built on this thing. So, I think like the future is like us running your agents in the cloud. Like they're persistent, long running. And I really really hope that moving forward this is not something you see uh anywhere. But people rented like VPS's they ran open claws on Hexner or Mac minis in the cloud. So I think like OpenClaw was a very very good like peak into like what might be for like sandbox clouds in the future. We've discussed why sandboxes uh are important in both research and product uh but they have slightly different needs. Um on research we want to optimize for throughput. We want to run many many training loops at scale uh and have many many rollouts. Um a rollout is one version of a task. So what is 3 plus 3 and you might have five five answers to it and one answer is a roll out. Um we want to take many many shots on goal and parallel u and so throughput is very very important uh in research in product latency is very important. I think any any successful product in the last 20 years has been super fast. So if I if you don't start a sandbox in time and you don't execute code fast enough, people will churn uh from your product. Um reliability is important on both both sides. Like if you fail constantly, you've wasted like GPU tokens uh on both sides and GPU is like gold right now. So you want to just make sure what the tokens you're getting uh are useful. Um and similarly on like if if you're not if your agents aren't reliable on the product side like it's game over right like people will churn from your product. Uh security is also important um on the research side like uh we are training on open AI infrastructure. So if a model gets root and it's not aligned it can try to attack open AI infrastructure. it can exfiltrate and release our model weights or whatever and then on the product side it can excfiltrate other users data attack the infrastructure as well. So security is important for like both both uh research and product. So today we'll focus on these three pillars. There are many many parts of a sandbox cloud but we'll specifically focus on runtime. So how can we run a sandbox on one node securely? Secondly we'll focus on persistence. Uh, I think compute was the first unlock. People realized you give sandboxes a Linux computer and they do crazy things because they're pre-trained on so much Linux data. But I think now if you help them, if you give them a computer with an actual disk that can be saved, then they become a true knowledge worker. And the last part is orchestration and how to run these at scale for many many users at CH GPT uh and codec scale. Yeah. So before we start, this is a first principles talk. So let's discuss like how Linux executes code on your machine, right? So on Linux, a thread is the smallest unit of execution. U the kernel is the thing that provides privileged access to a thread via something called system calls or IOTS. So whenever whenever the user space program wants to access some hardware or some privileged resource, it needs to talk to the kernel and call this like operation that switches the hardware context uh to a more privileged context. And so let's look at what that looks like. So if you see here, your processor has like different rings of execution. So based on which ring you're executing in, you get different privileges. So the kernel mode is executing in ring zero. It has the highest privilege and anything in user mode is uh running in ring three. Whenever we want to access privilege resources, we call system call instructions that change the context of the CPU. And so there are two attack vectors in a Linux system, right? First is getting root. In getting root, you're still in ring three, but you're the highest privileged user on the system. So you can actually pretty much do anything on the system. You can read your SSH keys, encryption data, etc. And then the second version is actually running a kernel mode exploit. So running code in ring zero. This is terrible. You can actually dump processes, memories, and like like I don't even want to say what all can happen. So you can get root and you can have kernel exploits. And if you get kernel exploit, it's like it's a it's a New York Times article waiting to happen. So, so these are the two attack vectors on a Linux uh system. So, with that background, let's design the simplest way to execute tools on a Linux system. Uh for example, we can literally have an API server that your harness is calling and it for every tool call, it can fork a process, exec the tool that the model needs and just have a fork and exec model. Right? So now there are a couple of problems uh with this uh model right a as we discussed the execution model the forked process can now directly talk to the kernel and so the model can try to attack the kernel get root or try to get a kernel exploit right the second thing is imagine you have a while loop and just forking processes in one tool call right so now you've kind of become a bad neighbor or a noisy neighbor and you've kind of brought down the node code and no other tool calls can run. Right? So, fork exec is the simplest thing you can do. It has one thing going for it. It's the most performant solution uh because there's it's it's as fast as just forking something. It's native performance. But everything else is very bad about this solution. Yeah. Uh so like we saw that fork in execing was bad. There was a noisy neighbor issue and there's a security issue. We now turn to something called containers on Linux. uh you must have heard of Docker and other things associated with the containers but remember this is the first principles talk so let's go what in deep into what containers are in their very raw form containers rely on two concepts on Linux uh namely namespaces and croups uh namespaces are for resource isolation and croups are for controlling the amount of resources a set of processes uh can consume right so let's see in depth how they can help our problems So in this diagram you can see that there are different type of name spaces a set of processes can have. So on the left you can see uh this container has a pit name space. So inside inside that process name space it looks like we have a process hierarchy of pids 1 2 and three. But if you look at the pids from outside the container they just regular processes with other pids. Right? So you've abstracted one resource which is the process inside this container. Another example is a mount name space. So if you see the bottom half of the diagram, you can mount any file system on top of uh any mount inside the container. But from outside the original mount is still visible, right? So this is a nice way of like isolating different resources in a container and there are many name spaces like pit, network, mount etc. Uh so moving on to the second principle of containers right like croups. So now we have a way to isolate resources within containers via namespaces. But remember I gave the example of uh someone doing a while with fork, right? Like how can we control that? Well, we can have croups and control how much CPU and memory a container can uh uh consume. And that way like we can like uh make sure that one container doesn't bring down the entire node. So name spaces and croups provide some amount of isolation uh and control on the uh system resources to not bring it down in case it's malicious or bad. Um if you've seen the previous diagrams uh there's the the fundamental problem with containers is that there's still native processes uh running on the host. So even if we abstract resources and two containers can't attack each other because their resources are isolated but a process in a container can still exploit the kernel boundary and try to get root or a kernel exploit right and once they can get root they can attack other people's data from other sandboxes exfiltrate data and try to get control of the node like we discussed before. So security is a spectrum like you can still get a decent amount of protection from containers. Uh one example to do this is something called sec comp. So you can actually have a filter on the amount of system calls uh your container can call and you can also control the arguments each system call can take. So you can say that you're reducing the attack surface of the kernel uh for for this like container and like kind of have some amount of sanity on how it can attack the kernel. The problem with this is that many times you don't know beforehand what system calls some container might call, right? So now you're blocking requests for users and later on like you have to like change the setcom filter to allow something or not allow something. So the feedback loop is like pretty bad for a product like open claw or any agent that wants to do like crazy things, right? Like you want to make magical experiences happen in the sandbox and you don't want to restrict them, right? So yeah, this is the f like to sum it up like containers interact with the same host kernel. So they do they do have some protections but at the end it's the same host kernel they're trying to attack, right? And they can get root and kernel exploits etc. Right? So we went from like forking to containers like raw fork was obviously very very bad for all the reasons. Uh containers are better than fork because they still have some protection and you can still have seccom for reducing the attack surface. Can we do better? Can we reduce the host kernel exposure uh a little bit more? Yeah. So the second thing we can discuss is uh to solve the attack surface problem is something called G Visor. The crux of the security boundary is the kernel API from before. So G Visor uses this point as its key security story. U so G Visor implements a lot of sys calls and user space. You can think of it as an application kernel. Its sentry is a user space kernel written in go and the file system is accessed by another damon called the gopher. So the sentry uh implements the Linux API itself including process management, workload management and whenever you call a system call, it's intercepted and it's like serviced by this user space uh program. So then you can argue that you're not implementing and attacking code that's running in ring zero or kernel mode. You're actually running in user space and ring 3 and any exploit you have is still in user space. So it's better than you exploiting the kernel directly. But the fundamental problem is that the sentry and the gopher like just going back the sentry and the gopher are still on top of the host kernel. So if there is an exploit on them, you can still have a chained two-step exploit. So you first exploit a problem in the sentry or the gopher and then you exploit from the gopher to the kernel, right? You can still get to the host kernel eventually. And with with these models uh of like 5.6 six and other types like you can have like like them like figuring out bug reports and these things and like trying to chain exploits, right? So you can still get to the host kernel, right? So yeah, the chain is harder here because it's a two-step chain compared to the others, but it's still reachable. Like can we do better than this? Yeah. Yeah, that's the same slide. It's saying that in all of these uh solutions you can eventually get to the host kernel whether it's folk g visor or containers. Yeah. So the key question is can we have a way in which untrusted or malicious code can have exploits but they can never or find it very very hard to exploit the host right so even if I can get root even I if I can exploit the kernel I still want my host to be protected right like that's the final goal we have so do we have something that we can use for this so if you've used if you've like known some of my background you'll know where I'm going with this uh so turns out Linux provides a very very nice thing called virtualization uh which is hardware powered at the CPU level and it provides an abstraction at the hardware level. So it which means that even if you can get uh root or execute in like ring zero in the guest your host is still protected. And this happens because the guest kernel runs in ring zero but in a separate processor context called VMX nonroot while the host kernel and the hi hypervisor run in ring zero in something called VMX root mode. So ring zero gives the guest kernel full control inside the guest but no control on the host. So you can exploit the guest all you want but the host is still protected. uh the processor has to switch between the guest and the host whenever the guest wants to access some privileged uh resource. We'll see how that works. But that is a key trade-off here. There's a performance penalty you pay every time the CPU is switching back and forth between these two modes. Let's see with a diagram what I mean. Um so you can see in in the in the old diagram we showed ring zero and ring three for executing user user space and kernel code. But here the guest kernel and the guest user space uh are running in separate like CPU context. So this is guaranteed at a hardware level. And so if you can even if you can get to ring zero in the guest like your host is uh like protected uh from it. Yeah. Now let's talk about what what does para virtualize and hardwarebased virtualization mean? This is the key of all the VM sandboxes you might have seen on hack and news or Reddit. So we'll we'll go from step by step. Right. So on the right hand side you see something called the VMM. Um can can any can everyone see that on the right hand side? It the VMM stands for virtual machine monitor. Um this software is called a virtual machine monitor and you might have heard of QMU when you search search for Linux virtualization. QMU and other VMMs their sole job is to talk to devk. So you see that arrow going down like the devk is the hypervisor API of the Linux kernel. All QMU and other VMMs do is set up the kernel root FS set up some allocate some memory for this guest virtual machine and call like into the dev KVM API as discussed inside when the when the guest runs on the left hand side it's running as a completely different Linux system it doesn't know what's happening on the host side but it has these devices like a block device or a net device coming up as regular Linux devices But when it actually tries to access the devices, they exit out to the host context. And if you see the block back end and the network back end, they're act there's actually code running emulating the devices in the host. Uh so whenever the guest exits out, it's being serviced by the VMM process and these device processes. So what do we mean by para virtualization? Uh so para virtualization means that we want good performance while we want to access hardware in the guest. So the guest drivers when you when the user space talks to them the guest drivers are aware that they're running in a virtual machine and so they talk via something called vert.io which is a more efficient way of the guest and the host talking to each other. Uh and so inside there are just PCI devices like any other hardware and when you talk to the PCI devices you magically exit out onto the host. Uh so 15 20 years ago a bunch of Linux wizards made this thing happen and made this performant and it it it truly is magical like how it works reliably and performantly across different uh hardware processors. And so from the host's point of view, when this is running, you just see a block thread. Like if you do a ps, you'll see a block thread, but that's running on the guest context. And when the thread exits out, the thread finally wakes up, it wakes up in this VMM process, and the process runs and it's running inside. Yeah. Yeah. So we had this seismic shift in 2023 where a bunch of new VMMs came. For a long time it was just QMU that was used for running virtual machines. But QMU has a lot of craft. It supports many many architectures. It has many devices and it's written in C. And historically many many escape attacks were attacking the devices written in C. So the first the first Rust based VMM was something called crossvm which our team at Google wrote when I was there to support Linux virtual machines on top of Chromebooks. And the key point here was that we don't need all the craft of QM QMU and we can use Rust to be a memory safe implementation of this tricky system software. And secondly like we have this emulated devices that we can jail. So if you attack the block device, we've only given it permission to access block resources. So you can't access network resources. Similarly, if you attack the net device, we we've jailed it and we don't give it access to block resources. So then you can still have a second gate of security. So both rust based safety like being memory safe and this jailing at a more granular level at the device like provides better safety than just QMU. Yeah. So you must have heard this word microVMs like everywhere and like no one really answers like uh why the word micro comes. Uh so turns out it has nothing to do with what's running inside the guest. It's it's everything to do with the VMM itself. So all these new age rust based VMMs uh they have a much smaller memory footprint because they don't support as many devices and they also boot much faster because they don't have as much croft. So a mixture of like just less blue less bloat and just booting up faster is why the industry has called them uh microvms. Yeah. One second. Uh yeah sorry. Uh and then you must have seen firecracker and cloud hypervisor. I there's a lot of confusion around what who came before. CrossvM was the first like rust based VMM that came before and then firecracker fked crossvm and it's used on Amazon for their lambda and serverless load and cloud hypervisor is a more general like VMM that many many companies like contribute to. And when you historically see microVMs on the internet like it's powered by one of these VMMs basically. Yeah. So it may seem complicated but at the end everything is APIs. Uh microVMs are no different. Um so here's a small view of how you can actually start a microVM. So your harness actually folks a cloud hypervisor binary process. When the process starts it exposes an API over a Unix domain socket. And if you can see in step two we call the create API give it like the root FS kernel CPU and memory. And then we finally call start. And when we call start, you can see from previous diagrams that the VMM literally calls into devk just like before. And it starts these uh like guest uh microvm. And then once the guest is running the like in this case the agent sandbox is running it might be that I want to talk to something inside the sandbox to say that hey save your state or I'm attaching some devices or I'm doing some xyz operation. So generally in agent sandboxes you have a pid one that exposes an API server and your harness or something on outside is talking to this uh uh API uh server. Uh and then this is the way how you can use like microVMs to be uh agent sandboxes uh that that can run like more securely than the other primitives uh that we showed. uh in this case we are using VSOC which is uh a socket that can help you communicate between the guest and the host or you can use the IP stack on the node itself and so there are no free lunches and systems uh and so there are some trade-offs so obviously you get really really good isolation using microVMs at a hardware level uh you can still attack the host but it's much harder you have to attack the KVM stack and then you have to attack the device the chain is much much harder to do but it has been seen and I'm sure it will be seen more with these models. Um you can jail the devices granularly so you can use sec and other security hardening things for the devices. So even if you get compromised on one device your whole system cannot be brought down. Um and like I said there is a performance overhead like we like when you are exiting and and entering the host and the guest context it's a very very heavy uh operation and you pay in like performance. Um memory sharing is not as easy. There's something called a balloon driver and so you have to actually ask the guest to give back memory and reclaim it. So it's always a like reactive thing. you can't immediately like reclaim and claim memory. Um, and then a lot of sandboxes these days have GPU access presumably for auto research or some sort of like ML research type of agents. This is not as easy with microVMs. Uh there's something called vert.io GPU but that provides high level graphic library type uh access. U for metal access direct metal access there's something called VFIO but it can only be shared by one sandbox at a time. It cannot have multi-tenant things. But given all of this, my view is that security like system tricks can cover performance issues, but they cannot hide security breaches. And as a company, you can lose trust ones and it's like very hard to regain. So I always prefer the more secure solution and try to make make up with system tricks for performance issues. And in my history of like working on sandboxes, I've seen there are like I would call it like the seven stages of grief, the seven stages of sandboxing. Like in the end, everyone always wants a VM because they tried everything. They tried containers, geiser, v8s and then they realize, oh, I want a whole Linux box because I don't want to like end up without XYZ functionality. And if I want a whole Linux box, I want to obviously be secure. So if you're a startup or a founder like in this space like let me save you the story and two years of grief just please use microVMs from the start and then if it doesn't work tell me and then we can tick talk about other things. So now we've discussed how to run untrusted code on one node and I think now like people have woken up to the fact that these models are very very good drivers of Linux boxes like so if you give them a computer they can just pretty much do magical things as we've seen with open claw they just pre-train on a lot of Linux right however like imagine if I gave you a computer without a disk every time you close the laptop like your data and your work goes away right like that's not a fun world to live in and and somehow the agents are some in the cloud at least are in this sort of world right now right so we want to give them durable storage and so this this part of the presentation is specifically working on disk storage not memory persistence but disk persistence so let's see why it's important and how we can do it so as shown in previous diagrams your microVMs have discks attached to them uh there might be just files or other dis devices on the node that you pass through to the VM. Um, and so from a product perspective, the task the task that the users are doing in these sandboxes are becoming much more complicated and much more longer horizon. So people are making like presentations and entire GitHub repos are being created inside the sandbox. Now, now imagine if like the node dies on the cloud or the model has a flake and you created this like presentation and like you just lost it, right? It's it's a bad for us because we wasted a bunch of GPU tokens. It's obviously bad for the user because you did a lot of this work and and you lost it. So, not just from a product perspective, uh but just from like a good experience and like utilization perspective, we need to have some way to save the disk state of the sandbox. And let's go into like three big use cases uh on what persistence can unlock, right? So counterintuitively persistence actually helps reliability and scale. Uh they might seem like orthogonal concepts but but they're very much related. So for instance you have a long running task in a sandbox that has many packages installed and you have created like GitHub repos and presentations and things right if you keep checkpointing it periodically and if the node fails or the cluster fails you can now restore the sandbox in the exact checkpoint state on another node. You can also do it intentionally if you want to upgrade a cluster or do some AB testing on nodes. The persistence has now let you reliably run sandboxes across your fleet, right? So, it's a very good way to scale and be reliable. Like I said, like how many of you have used gold mode in codeex or know what it is? Yeah. Amazing. Right. So, if you if you run gold mode now, it's like like my I think three days is my record for running uh uh something. like now people are doing longer and longer tasks and I think this trend will continue in the cloud as well. So to support this we obviously like need to have checkpointing so that the model can save state restore it on another node and like you can keep going forward and forward right like uh and so you're resilient to any failures uh across in the in the infrastructure. This is the most interesting part actually, right? Like so if your hardness wants to explore multiple like solutions or sample spaces, it can actually checkpoint the sandbox state and it can like do a Monte Carlo like research and like go ahead and like backtrack checkpoint again. So this way it can actually do rollouts over many many days and come back with the actual like uh solution, right? And I think my sincere hope is that if if we make this infrastructure correct, we can help like solve diseases like find new drugs because the model can just keep going on for longer and longer, right? But it doesn't happen till we have really really rocksolid primitives to do this. Yeah. And so given the needs and the the the pillars we want to support, here are some of the things that this snapshotting solution should support, right? uh first at chat GPT or codec scale we want to do incremental snapshotting and what that means is like if you call snapshot twice I'm just snapshotting the diff between the two snapshots otherwise if I have to save gigabytes of data at every turn like like I'm going to bankrupt the company and uh like it's just a slow experience regardless right um the snapshotting uh API itself should be very very cheap and fast so the model and the harness can keep snapshotting and exploring ing like very fast. U and similarly like just like creation should be fast for products like restoring is just nothing but just re recreation from a snapshot and so restoring should also be very very fast for a good product experience. And then we have two paradigms we'll discuss. One is always saving uh in which the harness doesn't have to explicitly call a save API versus explicit explicit saving where the harness is calling save save. We'll see how we can implement both. So I'll give a reference solution. Uh here uh and then there are some more design choices with disk snapshotting. Right? So first like I said on the left hand side do you want incremental or full snapshots? I argue I think at our scale we want incremental snapshotting. Secondly like do you want to uh snapshot the entire root fs or do you want to have certain folders like workspace or mount something something that you want to uh you want to keep that like configurable and lastly like you we can see how you can snapshot at a file system level or at a block. So at in Linux discs are nothing but block devices and each file maps to different blocks. we'll see after this. And so you can do very very efficient like snapshotting by just zipping up the blocks that have changed for a file or you can do entire files that have changed right for more right amplification. Uh and the high level flow of snapshotting is basically you figure out what's changed, you zip it up, pull it to the cloud and then when you restore you pull it down and and you restore the microVM, right? And so this is a first principles talk and so I just wanted to go over how Linux storage works from first principles. So Linux represents disks as block devices. So you see on the right like it thinks of a block device as having logical blocks from zero to end. U file system maps directories and files into something called an iode data structure and the iode says okay offset zero in file f maps to this block logical block in disk D and then on the disk itself there's firmware running which says oh logical block one is like sector 100 or sector 500 or page xyz. So the hierarchy is on these logical blocks up to the file system and we leverage that for blockbased snapshotting. And then within a microVM there are two ways of accessing storage. On the left hand side you can think of this as sharing a folder like Google Drive but it's very very inefficient because you're the VM is doing file system operations exiting at every file system operation which is very inefficient. uh on the right hand side you actually give a disk like abstraction at a block so you give a block device to the uh microVM and this is way more efficient because you can use the caches inside uh the guest and you don't have to exit out as much you only exit out when you truly need to access the block device and the host has to service you. This is the highlevel uh diagram of how you will have always on persistence uh in disk snapshotting. So if you have two microvms uh they have a block device which they will see this dot image as a block device inside and as they are writing to the block device we are writing through to the cloud. So there's some sort of like distributed file system that we mount inside that's giving this uh always on persistence and like I mentioned for explicit persistence uh we have an actual API called the save API calling the save API from the harness figures out what figures out what's changed between the last snapshot. It bundles up this div into an artifact and returns a snapshot ID. Later on, you can give us this snapshot ID. We figure out the lineage of snapshots that make this snapshot ID and then we download them one by one and apply it on the node and you get a microVM restored with this thing. Uh so let's see like how how we can implement this, right? Give me one second. I want to see how we're doing on time. I think we have five minutes left so we'll go fast. Yeah. Yeah. So, uh we can do explicit persistence uh using something called copy on write. So, Linux has these XFS file systems uh XFS like file system. So, you can do pretty pretty much have zero latency copies because you don't change any blocks when you copy. But when you actually change the blocks on a file, that's when you pay the penalty. And so in this in this design we have a base image which might be the codeex base image or whatever like the chat GPT base image we create a zero copy on top of it which is a writable layer and then when you write to it now you're changing the blocks in this layer and then when you want to snapshot I use something called fie map which tells me what blocks have changed and what ranges I zip that up and store it to the cloud. And I can do something very nifty here. I can actually lie to you while I'm uploading to the cloud. So the snapshot can happen return very fast as I'm uploading in the background. I don't have to wait till I'm back uh till I'm uploading. And then on the other side like I have this diff I download this like artifact. I figure out what extents have changed and then I apply it back on top of the base image and I start the microVM again. So now we have a restored sandbox with the exact same state at a block level. right now how can we do always on persistence right like uh so this is one way of doing this I know there's NFS also and other like distributed file systems but NFS for instance isn't as performant and it's not pix compliant and I think our models are just very good at anything pix compliant and standard so you can write a you can write a file system actually on top of a GCS or S3 or durable block storage and you can use something called NBD so within the sandbox you'll actually see a block device but inside it will have literally a tiered cache of blocks that are persisted first to an in-cluster cache and then the in-cluster cache is like writing back to the uh uh block storage the object storage so you have this like nice global tiered architecture where you're actually like caching things at the block level and finally inside the microVM and get a very performant uh like file system inside yeah so that's the persistent part of the presentation. uh uh and so the one takeaway I want you guys to think about is I think storage is the next unlock here as you're working on sandboxes think of what what all you can snapshot and restore fast to give like this new paradigm to harnesses so they can recover from failures and explore do Monte Carlo like searches u so we've discussed running things on one node and of course if our one node dies we are done so we want to be able to run across many many nodes across the world so I'm not going to mention about kubernetes or other like operation things. This is a first principles talk. So we'll lightly hint about the challenges here. So ideally we want multiple machines to support the runtime we discussed. So we can group nodes into clusters and spread the clusters across the regions. A top level control plane chooses a cluster using region, load and other factors. And it's not very different from the orchestrators you are familiar with. Um these like it ideally chooses a cluster close to your chat GBT cluster. So you can like have fast like uh access to the harness inside the cluster. There's a scheduleuler also and the scheduleuler tells you which node to pick based on the load and other factors right so if nodes are dying or failing it won't choose that it will intelligently like route the sandbox to the thing and again low latency and reliability remain very very key northstars for this architecture and so here like we can use some microVM features to support low latency creation ideas. So a lot of systems cheat for low latency. They've pre they pre-warm like sandboxes and they pick one which is great. Another way to do this is you can actually take a memory snapshot of a microVM and just in time start it start it in milliseconds as the request comes and so you can leverage this like nice microVM property that you can save the guest memory and start from that. And the third one is a hybrid solution. So you can have a warm pool but as it's growing you can like grow it from the memory snapshot. So you can get best of both worlds. Like the trade-off for a warm pool is that you you're consuming CPU and memory in idle state. Ideally, you don't want to do that, right? So there's a trade-off between one and three and two basically. And so here is a way where we can use snapshot restore for better orchestration. So remember we discussed that a snapshot can have a lineage of many many layers. So once you want to restore from a snapshot and you find out you have like let's say four layers that you want to pull down and that makes the lineage you can actually like smartly route you to a node which has to download the least amount of stuff. So in this diagram you can see node A has some layers node C has some layers but node B has all the layers that you need. So the scheduleuler then routes you and it it gives it the highest score and routes it because it has all the snapshot layers. So you can use like snapshot with orchestration to just have faster like uh uh creates and even just more reliable uh like orchestration. Yeah, this is the talk and I hopefully it gives you some design intuition around like sandboxes, why they're important, what's the next unlock and I I want to see more of you guys using it in a secure way. Thank you.