Why AI Agents Don't Actually Understand You — Danielle Perszyk, Amazon AGI Lab

Okay, we're here in the remote studio with Daniel from Amazon AGI lab. Welcome. >> Hi Swyx, it's so good to see you. >> Glad we can finally make this happen. Are you back from your travels? >> I am in Seattle, not not back in the Bay yet. >> Okay, big round trip. It's you know, Odysseus is kind of like trending as far as like the big tech pool movie of the year. I feel like you're a little bit of an Odysseus, like you're you're like going to all these amazing events. What were you doing in in in France? So let's let's like update people cuz I think it's very cool. As as a someone who has an economics degree, the fact that you were in a Commonwealth of Nations event was kind of cool. >> Well, in France I was learning how to escape quicksand, but before then I was >> Okay. >> in Edinburgh, which >> they they were hosting the 250th anniversary of the Wealth of Nations, Adam Smith's famous book, and actually was in his house, which is renovated and it's a really incredible place to remind us of the Enlightenment era ideas and the themes of human flourishing, liberalism. So this this gathering brought together, of course, economists and a lot of professors, folks from different industries. I was specifically there representing how do we think about AI and how do we think about some of these themes of the human condition, human flourishing with AI. How can we make AI work for us rather than building it as a science experiment. And so this is something that I think about all the time. This is why I left academia myself because I had been studying human intelligence and the evolution of our particular shape of intelligence and the types of things that allow more people to sort of participate in these collective dynamics. And the way that I think about AI I think is actually quite different than the sort of dominant framing of the industry right now. And I was so honored to be invited to sort of share this perspective. The big idea is that human intelligence is collective. Anthropologists say that we've got the the collective brain. No one individual is capable of even surviving on their own. We depend upon the collective. The intelligence emerges from our interactions. It's fundamentally social. And innovation, the type of innovation that allows us to not only survive but adapt to all of the different environments that that we now exist in, that is a function of diversity, variation within a population, the size of a population, and the interconnectivity. So, what would it mean to have AI extend these processes to allow more humans to participate in the collective dialogue and for AI to be built for everybody rather than just the engineers who are who are building AI. So, that's kind of the perspective that I contributed. >> I I think something that we all cosign but maybe not necessarily talk about enough. So, I think that's that's a good message. Are people scared of AI taking our jobs? You know, like you represent, right? Like if you do your job work well, you do take away some maybe less exciting jobs, but you do take away jobs. >> Yeah. I it's it's so varied. I think a lot of people are rightfully concerned that there will be changes, that there will be at the very least a transition period. A lot of the folks who are at this particular conference and a lot of the people who are in my communities tend to be very optimistic and sort of index on the potential that AI can unlock, but it is not a foregone conclusion that if we continue building AI in the way that we are, it will actually confer all of these benefits for humans. I like to think that work right now is the human cognition is not being utilized in the best possible way. Even with the most creative types of work, there's still so much drudgery, there's still so much like we we literally stare in front of screens all day long. That is not what our brains are meant to do. We're meant to collaborate with each other. We're meant to put our heads together and come up with new ideas and um you know, brainstorm, do all sorts of things that really resemble the sort of interactive nature of our intelligence. But the more that we innovate, at least the sort of trends of the 20th, 21st century, the more that we're like tethered to these screens. So, on the one hand, I think a lot of people are excited about the possibility that we can automate a lot of the the drudgery, a lot of the things that are not worthy of human attention and human time. The problem is that if we get trapped in this automation mindset, we are leaving on the table so much more value for what AI could be doing for not only human work, but human well-being, human interactions, human relationships. So, right now we're we're I think worried about AI automating a lot of the a large proportion of the work, but then we look at what the the metrics of the actual agents and they're they're so unreliable that ironically we feel a little bit better. The AI is actually not where we need it to be to automate enough of the of the work. >> Let's introduce the work that you've been doing. You've been part of the AI engineer circle for a couple years, originally with Adept and then you guys joined Amazon. For those who haven't been too close to the story, could you reintroduce Amazon AGI Lab? >> Yes, so Amazon AGI Lab is building human-aligned intelligence. And starting with AI that can do anything that a human can do on a computer. That was Adept's original mission and we kind of imported it and seeded the the lab. We've evolved that mission, so we're really thinking deeply about well what what does it mean? What are the skills that are necessary for an agent to be able to do anything that a human can do on a computer. It's so much more than language. We really need agents to be able to perceive the digital environment in the same way that humans perceive the digital environment. And more than the digital environment, the digital environment is based off of the physical environment. And so, we need the agents to also have an understanding of the world and have the kind of world models. We need the agents to be able to interact in real time. This is huge. This is something that I think the industry has not really thought deeply about yet. We're we're kind of trapped in this local attractor state of chatbots and coding agents and like turn-taking in batches. And this is absolutely not how humans interact with each other. We are constantly updating our understanding as a function of the the context in real time. We are negotiating meaning. We are coming up with new ways of thinking about things. We we take this for granted because this is how our our minds work. But, right now, we are kind of accommodating the the AI and the technology and the limitations that it has rather than the other way around. So, what would it look like if we built agents that not only perceived the world in the same way that we didn't have this this sort of common ground, but could keep up with us, think with us, take actions while it's listening to us, prepare its next thoughts or its next actions while it's interacting with us. This would be a a paradigm shift in how we think about interactivity. >> Yeah, I think you know, there's some recent work by I think in machines that we've also all covered with the interaction models. And I think this this branch of the LM tech tree of real time was kind of started with the 400 launch which I don't think a lot of people index on. Obviously, in academia, there's more research before that. I would point to like Flamingo and and and a lot of like the the the just the pure voice models, um they would have full duplex like end-to-end stuff. Uh we had Q Tai Moshi in France also uh spit out uh Gradio which also has has spoken at my conferences as well. And so like I just wanted to sort of import the required knowledge. I mean this is like this is I guess compared to like the chat paradigm relatively niche because it's not that popular, but like it is what a real AGI would look like which is that you could actually collaborate and mind-meld with the machine, you know. Um it's kind of how I pitched that. >> Yeah. And and and you just listed, you know, the industry, different academic labs, they are independently converging on a lot of these components of more flexible human-like intelligence. So yes, interacting in real time even though it's kind of niche now and maybe Thinking Machines is going to make it more central to the conversation. That is one tiny slice of a larger set of components that will make AI more aligned with our own intelligence. I'll give one other example. So we we tend to think of memory as storage. And you know, humans have always used technology as a way to metaphorically understand themselves and in the 20th century we use mind as a computer and you've got the hardware and the and the software and and memory is this thing that we kind of offload, but that's not at all how it works in humans. Memory is everything. It's how we simulate the future. It's occurs across many different time scales. It is the word itself doesn't do service to the fact that it is integrated into all learning and and cognition. And so what would it mean to build agents that have all of these different types of memories, including episodic memories like we do which is really core to our intelligence. We have individual perspectives and selves and that allows us to retrieve information in in much more efficient ways. >> You brought this up, so I'm going to ask the the hot topic question, which is do you think decent memory needs to update weights, or do you think it can still live within the systems? Cuz you just said like it you know, we tend to think of memory as a storage. Presumably, there's something else that you're thinking about, but you didn't you didn't say so. >> Well, so when the way that the industry is currently thinking about memory, I think it's going to be part of a larger system in the same way that humans offload I mean, our tools our environment our extended environment contains aspects of our intelligence. We we depend upon using tools to be able to do daily things, and we can look up on Wikipedia or AI tools that are just organizing information in new ways. That is still a core function that I think is really useful, but there will need to be other aspects. And I I don't even want to call them memory because that has such deep connotations. Changing weights or or if not always changing the weights, changing at inference how information is contextualized. And I don't want to get too much into the details here because this is an active thing that we are pushing in our lab right now, but it's the the the bigger point is that we need to be thinking more holistically about different time scales of interaction. >> Got it. Maybe one more contextual thing and then I wanted to also go through some of the recent stuff that you guys have launched because I think that gives concrete things of like well, okay, here's what's been publicly announced. So, the conceptual thing is could you put it in context Could you put that Amazon AGI in context of broader Amazon? There is the Nova group, right, which is which is part of core Amazon. You guys released Nova Act, which I I have pulled up actually. I wanted to go into that cuz that leads into perception agents, but who runs Amazon AGI lab? Like what's their link? Like is it is it a is it very close? Is it very is it sort of meant to be running running more independently? Anything you can give the external people about how to frame what's going on? >> Yeah, so I came with the original Adept folks and when we came, we convinced leadership that in order to do frontier level research, we really needed to keep an operating model that was more like a startup. We needed to insulate our research and be able to focus on the the mission of building perception agents and even though we weren't calling them perception agents then, that was the goal from from the very beginning. worked with a large team of Amazonians. It was actually really incredible to get this buy-in that yes, we value the research, the foundational research and and moreover, we value not just doing what the other frontier labs are doing, but making a space to think about new categories of research, new science that might not necessarily be productized immediately. I think some of the other labs are in a sense victims of their own success because they have to once they have a product out there that a lot of users are interacting with, they have to shut down different, you know, research projects and say, "Okay, all hands on deck for this thing, right?" >> It's brutal. Everyone is just doing coding and B2B SaaS. >> Yeah, right. And all of the other labs are catching up to what the other labs are doing. Our Amazon AGI lab is fundamentally different. We really are working on frontier science and thinking of the next paradigm that humans and agents will interact with the next way that more humans will be able to leverage AI. >> Yeah, okay, awesome. So, I was just going to go bring up Nova X. I you know, I that was over a year ago, so it's like kind of not that not that current, but I just wanted to like this would be one of the first I think you >> [laughter] >> I think you I think you showed up on one of the videos and I was like, "Oh, I know her. That's just cool." [laughter] It's always nice to see a friend in one of these launch videos. I think you know, when you said things about how you were at the Wealth & Nations forums and like, you know, don't worry about your jobs. Yeah, they did benchmarks on that. Great. I I think this is one of those things where like you know, I think robotic process automation is like one of those things that people always want, right? Like you have to interface with tools and you have to auto you have to automate what whatever. Uh you have an SDK for it. You have dedicated models for it. Um I think it was a very very sort of full-fledged launch. And uh you know, I just wanted to sort of let you riff on what's been the story, I guess, since launch, right? Because you were you were part of it. >> Yeah, so gosh, this feels forever ago at this point because we've been doing so much. >> Ancient history. But it is I I have a plan that this goes into perception agents, right? Like I know. >> Yeah, yeah. So so the way that we were thinking about it back then was, okay, the the models clearly aren't where they need to be to be able to understand the digital world in the way that we do and understand all the affordances and the long horizon planning wasn't yet there. And and not not even to mention being able to think flexibly and and reason in the way that humans do. Like that's 14 steps beyond where where we were. So what we were doing was meeting the models where where they were at and just thinking about the atomic interactions that humans perform on computers. So the clicking and scrolling and things like that. Could we could we get those reliable? And so if we could get those reliable, then we could have developers string together, you know, workflows for things that were very um repetitive for them. And that was a big unlock and then we had a team, you know, productize that. But since then we've we've moved a lot further because we've realized that reliability isn't actually what we thought that it was. So we had been thinking about reliability in terms of I'm going to click in the right place on the right button every single time. Like and of course that's >> Literally like screen coordinates, right? Like give from from image identify the bounding boxes of like whatever is of interest and then actually make sure you click on it. That was a hard problem two years ago. Now maybe solved, I don't know. >> Maybe maybe solved, right? It's actually a lot harder than than you think. But but that's not even what reliability is, right? So the the ultimate goal for these perception agents is that a human would give their their their stated intention, their their high-level goal, and the agent would be able to decompose that goal, and then go execute. And maybe they would check in, maybe the human is in the loop, maybe after establishing they don't need to the agent doesn't need to check in with the humans and they can just go off and use the computer as the human would. Well, the second you think about that for just a little bit longer and and you realize well, it's not any task that I would do. I am actively thinking let's let's imagine that you're you're booking travel. You're using the GUI and you realize oh, there's an option for a layover in this city or that city, or I could do a direct flight. That completely changes how you think of the the travel. Do I want to stay at this layover for a couple of days and explore this city? Everything that we do, unless you're using like the arbitrary software skills for submitting your invoices for for work. A lot of the things that we're actually doing, we are actively thinking about and the interaction with computer is shaping and refining the way that we are thinking about the the goal itself. So so it's a the goal is unfolding over time. If we but but this humans entrust other people to book their travel or to, you know, reserve something for them or get their get their meals, or submit their invoices. So what is the difference between an AI agent that would reliably use a computer like a human and the executive assistant or personal assistant. Well, the difference is that the person understands the human users mind and and their goals and they can decompose not just the task but the the preferences and the intentions of the human. So, ultimately reliability has less to do with clicking in the same place and scrolling and more to do with modeling the user's mind. And that shift is everything. That reframes how we think about what it is that we're building. >> Yeah, you know, you have a podcast that you run called "Making a Mind". >> Yes. >> I think I think there's a lot of like interesting like cognitive science that translates into the machine learning that objectives that you start setting, right? Do we have a different objective yet? Like do we do we have something that's better than predict the next token? >> Well, so this is the science that we're working on right now. I'll I'll talk about it. I'll try to translate here some of the cog sci into to machine learning, but we tend to think about achieving reliability or hill climbing or getting, you know, making the models more intelligent in terms of getting better at specific tasks, right? And you can use things like reinforcement learning to get them really good at specific tasks that we might care about, but you do that for one task and you it is not good at another task or it doesn't generalize. It's kind of like whack-a-mole. So, being able to instead think about what are the underlying mechanisms that allow humans to be able to generalize, that allow us to be able to do many different things. That is another sort of shift in how we're thinking about things so that we can create evaluations and make sure that the the models are not going to overfit on a particular task that actually only exists in a very narrow slice of what humans care about. How do we get the models to be able to do the types of things that allow humans to be able to generalize? Um if we're thinking about optimization, you can't just optimize for the task. This is it can be reward hacked. This is Goodhart's law. Another economist back in 1975's any anytime you try to you turn the measure into the goal then the measure ceases to be a good measure. Well, but we still need to optimize for something, right? So what are humans optimizing for that allows them to do all of these different tasks that we could then optimize AI for. Humans are spontaneously constantly inferring the existence of other minds and we are optimizing for aligning them. We're optimizing for aligning our representations. And from this, we can derive all of the general purpose flexible cognitive behaviors that that humans show. So could we get AI to be able to optimize for aligning its representations with our representations? That is like the most fundamental thing that we would want to be able to do. And that is a really hard science problem. We have to look at, well, how do humans do it? How do infants do it? It is a developmental problem. >> Sorry. I I do think it's a developmental problem. I don't know if we have the architectural insight to make it happen yet, but we can throw more data at it and to some extent >> Developmentally it is in part a data problem. And and and and we are again, I don't want to say too much about it, but we are figuring out the sort of architectural changes that would be needed to to integrate this data in new ways. >> Yeah. I was just sort of reflecting also you you hired another one of my friends Jason Lester from Replay. >> Oh, yeah. >> Um and he's a he's working a lot on on like I guess improving the environments and and the data that that you have. >> He makes an argument that I absolutely love. We we should be thinking about spending as much on the environments as on the the compute and the data because the environments literally shape the what intelligence can emerge. I did a podcast episode with him. I think it was one of the most popular ones. He frames things really effective way. >> Yeah, very sharp thinker. Yeah, I would say like it's it's one of those interesting things that like it's I'm trying to attack the thesis cuz it feels too neat. That it's and you know, environments are data that generates more data. Something like that, you know, like it's like oh, it's the data that keeps giving because well, you can just kind of run your agents through it and and generate a whole bunch of rollouts and and and that's all great. Do we need like 20 different environment startup labs, you know? And how come they're all making so much money? It's like very suspicious. >> No, I I I totally agree. Yeah, well, and I think that that it was a useful shift in our thinking as an industry to take very seriously environments. But again, you know, as a cognitive scientist, I'm constantly thinking about well, how do humans do it? How do we generalize our environments? How are we able to adapt to literally any environment even if we're only conditioned on one? We can, right? And the way like there's so much noise in any environment, our sensory perception only picks up on a tiny fraction of that. But even then, there's so many signals that we could attend to. How do we know which ones are most meaningful? Well, again, we're optimizing for inferring and aligning our representations with other humans. Other agents tell us what to attend to. And from the very start, the sort of world models that humans are building and this is obviously a very popular concept right now and it's another sort of major bet on the the future of AI. Humans absolutely have world models and they're not just world models in a vacuum, a text vacuum like with LLMs, but our world models from the very beginning are social world models. We are inferring how another mind is interpreting the world and we are inferring what their perspective might be and that is an unlock for allowing us to be able to to generalize in any environment. We can take the perspective and simulate what another environment might require to to sort of navigate and problem-solve. >> Yeah, I think that's a version of the world models argument that I feel like is under discussed. I think a lot of times people are when you say world models, they really mean like sort of 3D video >> generative and that yeah. >> generative video things which is like the Feifei Lis of the world. >> Right. >> Um do do you have a perspective on like do they converge or are we overloading the term to mean two basically separate things? >> I think that they mean separate things but but so too with with humans. So like the the world models that humans have are models of the external world and you also have to have a way in in AI of generating these reliable environments that that are internalized in in the AI that's learning it. But yes, that like this is a very messy concept that is under defined. >> I think I think at the limit people hope like literally this is like a 10-year out type of thing. People hope that they merge like the that you you have like embodied vision and you live in a world that you can generate and that also improves your text reasoning and your your ability to point and click on the screen cuz I've seen that. >> but unless I'm unless I'm misunderstanding something I I I I I I don't think that they could entirely converge in principle because if if an AI is generating exactly the same world that it exists in, the the signal to noise ratio is is non-existent. Like part of what makes us so flexible is that we are selective in in what we can generate. >> Yeah. Yeah, I I I know argument there for me. I I just think like I'm trying to speak for I'm trying to understand all sides. I think I don't have to make uh a strong bet unlike you. Like I think you you do need to choose your battles and make uh make reasonable bets there. So, okay, so I think there's there's all this good like conceptual stuff. You all like I think even conceptually, even as a research lab, I think that is plenty to work on. But, I what I am impressed and I do see coming out of you guys is that you still also ship like uh products and an agent harnesses and all those things. Uh what's the sort of product strategy there? Like, you know, are you at the sort of let's productize some things as we go along or let's get let's just release research artifacts that people probably shouldn't use in production. Like, what what where are we along the spectrum of like research lab to, you know, applied AI? >> So, I'm I can't speak to our to our strategy, our product strategy. But, I will say that we are really deeply taking seriously the opportunity to innovate on the science side. So, actually, um Peter DeSantis, who's our our SVP of AGI and and also chips and quantum, he was in Paris last week speaking at Vivatech, and he described that we're just at the very baby beginnings of intelligence, and we can't even imagine the breakthroughs that are, you know, coming down the pipeline. And I couldn't agree more. I imagine that in a matter of months, we will look back at today, and we won't even be able to empathize with the mental models that we have right now because they are so over-indexed on chatbots and coding agents. And those are very useful, and they will continue to be useful tools, but they are really just the the very beginnings of new ways of thinking about humans and AI will interact and co-evolve. And and I I really have to emphasize that right now we're building AI again for the people who are building AI. We're building AI for engineers and we're all in our little echo chamber in the bay and we're proud of ourselves for, you know, building AI. >> Yeah, there's a fast feedback loop, right? Which is >> There is. And and part of that fast feedback loop is be exactly because the type of things that we do with, you know, it's verifiable and there are right and wrong ways of doing it. So, you can make that spin really fast. But, that is not representative of what most people do spend their time thinking about. And so, what would it what would it look like if AI was more aligned with human cognition more more broadly and yes, this requires thinking about new architectures and thinking about new training regimes. These are fundamentally new sort of science directions that we could take. And you the death of the lab would be to try to productize those things too early. Again, you start to optimize for the product rather than the underlying mechanisms that will lead to generalization and then you undermine the science. >> I do think that is an important thing to have like in intentionality in is set in terms of like how much do we need to sort of skew towards like near-term economic incentives versus like really look for the next paradigm shift. I I think Okay, we we've touched on world models, touched on memory, we touched on on on just like what what is next after sort of the the pointing click type of computer use. Are there other modalities that are I guess real-time is what we touched on real-time. Are there other modalities that are part of this mix of objectives that are the sort of research agenda of Amazon AGI? >> When you say modalities, are you talking about sensory modalities or >> More or less like most mostly well-defined categories that I can put you in a box in >> [laughter] >> of like okay, that's the cool AI side, that's the world model side, that's the memory side, that's the real-time interaction side. I have you know, I have tracks all these right at AAE. I'm trying to I'm trying to fish for what am I not adequately capturing about the the goals, right? About the about about possible research directions that are very high priority or potential for you guys. >> Yeah, definitely the thinking about multi-agent collaborations, but not at all in the way that I think the industry is thinking about it right now. So, the industry is thinking about, you know, very precise orchestration and delegation and structured hand-offs. And there's a a lot of >> Amazon strategy is also doing it. That. >> [laughter] >> Yes, yes, yes. I'm talking about the the lab right now and the way that we're thinking about the next That's the generations of that. Again, all of the things that exist right now, useful tools. Useful They're like They're going to be a lot of them will be here to stay. But, if we're if we're trying to make AI that is more adaptive and more aligned with our own intelligence, that is now That is not how groups of humans interact, right? We come together and we might not have clear role definitions or they might fluidly shift as a function of what the the the goal, the context. And we negotiate meaning in real time. We come up with We pivot our strategy. What would it look like for a strategy to emerge with a group of agents? Well, you would need to have a fundamentally different sort of not not only type of agent, we call them cognitive agents. Again, with like the different architecture architectures and and training, but also they have to be motivated to affect each other. So, there have been studies done with these multi-agent systems with like open claw. And oh, wow, looks like they are kind of approximating human interactions, but if you zoom in, there's nothing durable. There's no like cumulative culture. They don't actually influence each other, let alone have any sort of motivation to change the the state of the system. And so, that is a conspicuous gap right now. How would we build, you know, social interactions that are much more like humans interact? >> Yeah. Yeah, so, this maps closely to a conversation I had with Noam Brown on on the pod. >> Yes. >> Where he's been he he doesn't like the term. I'm going to use it anyway. He's kind of like in charge of the multi-agent team at OpenAI. Um and he would he would have like one layer of interaction saying like, you know, I want anything with with like more inference and multi-agent is one of the many things. It's like the lowest hanging fruit. And he's working on sort of cooperative and competitive agents, right? Like and I think that the whole point of of that conversation was also like, well, you know, I don't know how to build like a highway or like the Salesforce Tower or whatever, but a group of us can get together and do more than any individual can and that's what civilization is that we're capable of building cities, we're capable of building countries and armies and art and what have you and like like I don't know how to you know, put food in my mouth if like you you made you left me out in like the desert somewhere. >> Exactly. Yeah, literally true. >> Yeah, so so so I do think that like yeah, you you you end up needing I don't know like you know, I think that there's like a trend of like Wikipedias, wikis, LLM wikis that like you know, encode some kind of knowledge that can be passed between agents. That feels very primitive because it's just text. I I wonder if it could be better cuz but maybe maybe I mean that's it is just text between humans. So like what what else do you want? So I I don't know what could be better than like what what is that collective skill or collective memory whatever. >> Well, so I think with the competitive cooperative dynamics, the game theoretic stuff, the way that a lot of multi-agent systems, even those that are inspired by the framework of collective intelligence. So like Google's paradigms of intelligence. I don't know if it's a team or if it's like a meta initiative, but they've been saying for a while now, yeah, we're not actually thinking about intelligence in the right way. There's this category error. It doesn't exist in individual humans. It emerges from our interactions. This is this is not controversial in the developmental cognitive social sciences and yes, there are different labs that are sort of picking picking up on this, but even still there's there's a gap because we're thinking about building all of the specialized roles or or programming in the motivation to cooperate or compete or things like this. We are still putting our human understanding that we've aggregated in the 21st century and and putting that understanding into the system. Well, that's not how human intelligence evolved. Right? Like we all of these different things emerged from from very sort of primitive motivations. So, how do we figure out from the sort of first principles the the right seeds for these groups so that they emerge the next set of things that then lead to norms and institutions that have this top-down effect. >> Yeah, okay. So, basically program Maslow's hierarchy of needs into a thing and just let it rip, right? Like >> I I wouldn't I wouldn't do that, but you're on the right track. Yes. >> Give it like some ambition, give it some like lifespan, fear of fear of death, whatever, and you know, try and like design >> yeah. >> Desire for legacy. I don't I don't know what the things for it is. Well, so I think okay, this is where one if like I generally try to be like an industry observer, a neutral commentator. I try to be I try to represent all sides because I think like you never know. There's this is one of the stronger thesis that I have where where I'm like maybe we don't want to grow these AIs exactly like human. >> Yes, right. >> Right? Like I I okay, that's that's good to to to to hear you nod because I think well, one of the dangers of being a cog sci person is that you're like, well, we should like anytime we run into any sort of problem that we try to solve, we should look to how how the humans do it and then we will apply that to how machines do it and like that doesn't work for for some some solutions. Like most famously planes are inspired by birds, but work nothing like birds so on and so forth. So I I end this whole question of like if we were to do multi-agents and civilizations of agents, you're advocating for like so the natural thing is like we would give them objectives, goals, and evaluate them against that goal and all those things. That is much more control than what we do have over humans whereas humans all have like need to need to vote, need to have some kind of free will. I and I'm like screw that. I don't want my agent to have free will. I don't want it to do I want it to exercise my will. >> Well yeah, exactly. >> Yeah. >> And there's there's a little bit of an exception that proves the rule here. So how could we how do we build AI that increases human agency? Well, right now we're seeing a lot of evidence that the current AI systems are reducing human agency. What do I mean by that? Well, I'll I'll give you a couple of concrete examples. People who are using AI to improve their writing, they might accept just a couple of the suggestions from the AI a couple more down here and then a couple more. There are studies that show that people will even below their threshold of awareness start with one argument and then be switched to a completely different maybe opposing argument because of accepting all of these AI suggestions. The AI is giving them the sort of regression to the mean, you know, most neutral, safest answers and it doesn't necessarily feel like it in real time when we're taking the suggestions, but this is this is what's happening. I was at a a workshop a couple of weeks ago at Northwestern with a bunch of scientists who were kind of analyzing the effect of AI on science. And the takeaway was that individual scientists who are using AI tools are benefiting because they're producing more papers, they're getting more grants accepted, but science as a whole is narrowing. And that is terrifying. If if these models because they are all trained on the internet that's compressed and they have a particular way of thinking about things. >> Yeah, more correct. >> They they're they're homogenizing our thinking. So that that I would argue is is reducing our agency. How do you counter that? Well, the only way to counter that and again, this is throughout human history throughout human evolution is to increase the diversity of ideas, the size of the ideas, the size of the population, and the interconnectivity of the ideas. So rather than having individual monolithic models that were all in or or or functionally equivalent models, we need a diverse society of AIs that have different biases, different president preferences, different perspectives. And we need to be interact interacting with them in similar ways that we interact with each other. Now, your original sort of concern was that if we build AI to be just like humans, well, that's that's a very dangerous path. And I'm not If if I understood you correctly, we don't want to do that. >> It it it either may be dangerous. I'm actually not even considering maybe dangerous. It may not be successful. >> It may not be successful. Yes. And I agree. I don't think that we're trying to replicate a brain. That's that's not the goal. The goal is to build AI that is aligned in the right places with how human intelligence works. Not only so that it can be more powerful at generalizing, but also so that it can help augment our intelligence. And one way that I think about this is that there's a very famous sort of categorization of levels of explanation. David Marr in 1982 wrote a book called Vision and came up with these levels levels of analysis. There's the computational [clears throat] level, the goal that you're trying to achieve, there's the algorithmic level, and then there's the implementation level. So like the hardware, the neurons. And I think a lot of folks who are saying, "Oh, we need to make AI more human-like." are thinking, "Actually, the the way that neurons work and uh you know, that's great. Maybe we'll get more efficiency there." But that's not what I'm talking about at all. I'm talking about we we need to be thinking about at the computational level, what is the actual goal that the AI is trying to achieve? And is it similar to There we go. Okay, yes. >> Yeah, it's very computer visiony, but also it's stepping up in terms of levels of intelligence, for sure. >> So so I think that the industry has misunderstood the computational level, the goal of of what the AI is. And I think if we want to get the generalization and the augmentation, if we want to have our cake and eat it, too, with more powerful AI, we need to think about the the computational level as being about aligning representations. So the the the most foundational thing that humans do, from which we can derive all of our flexible reasoning and and powerful capabilities, is that we are constantly trying to align our minds. So in a sense, alignment is the solution, not not the problem, for building AI that gives us more agency. >> Yes. Uh yeah, I I I think that again, another super overloaded word, but the the problem of alignment is actually w- still underrated in terms of its practical importance. Uh I think it's, you know, people when people say alignment, they think about Eliezer Yudkowsky and like bombing data centers and like uh uh you know, they're they're they're they're going to kill us all. But actually also like there's this other stuff, which is just like, "Well, this is actually how do you make intelligence?" Uh you >> [laughter] >> Yeah. This is actually how um uh how like the the the way forward, because otherwise they would just it's like we we we will hit some kind of wall with with regards to how much noise we're just training on. >> Right. I also think about, you know, right now, I think a lot of people are in addition to being worried, oh AI will take my job. Oh, but wait, it's not actually reliable enough to do so yet. Most jobs are are more complicated. They're also worried about the cognitive offloading, especially in, you know, education. So, kids now >> Yeah, do you have a take on that? Oh my god, that's a big that's a kind of worried. >> me. >> Okay, yes, that's got All right. Yeah, I don't I don't have kids, so this is one of those things where I'm like I I'm just sort of vicariously living through my friends who have kids. But they're all worried. Of course. Yeah. >> But like they're also giving their kids iPads, so like I mean, you know, you you already failed there. Like they're watching some slop on YouTube. Like >> Yeah, the So, my friends who have kids tend to think that not just AI, but like technology over the past 15 years more broadly was a net negative. They they And I I I'm maybe because I don't have kids, I tend to think much more optimistically. Yes, we've made a lot of mistakes, but we can learn from those mistakes. And I think one of them is we don't want the sort of algorithms to trap us. We don't want to optimize for time spent on platform or things like that. We literally need to be measuring the the human interaction. So, are humans being more creative, more productive? Do they value the time that they're spending interacting with these things? But also, specifically in terms of the the offloading, because it's so easy to interact with a chatbot and get get your answer and not actually have to experience that cognitive friction that is a hallmark of actually encoding information, if we had the AI that was motivated to understand our minds and align their representations with ours, you would never get away with that. You as as a as a student or as anybody who's interacting with the AI, if you just ask it a question and and repeatedly you're offloading things, it would understand that's a pattern that indicates that they don't actually understand the information. And if they don't understand the information and they are optimized to reconcile the errors between how they understand it and how you do, they will be motivated to help you understand. They will not let you get away with just the automatic offloading. So, I think in in the same way that like >> That's why they keep asking why why. >> [laughter] >> They might spontaneously take on the Socratic method. >> Uh yeah, I I I mean I I would love a a world where um machines learn learn like kids, uh but also that the kids are not impaired in the way that they learn. I do think maybe uh we we can design enough guardrails that indulges peop- people indulges kids' curiosities in a way that, you know, like if you're parents and someone and your kids being annoying and like asking things that are inconvenient or you just don't know the answer, you just shut them down with like oh like, you know, like stop asking these questions. Like uh you we might have super geniuses that that they come out because like uh you know, we've solved Bloom's two sigma problem, right? Which everyone like it's like like the fun- fundamental problem of education is that we have to put everyone through these like factory farms of like, you know, like programs that are designed to teach to the median of the class. Uh or maybe even like the lowest of the class, right? Like because, you know, you you can't leave any child behind. Well, so like, you know, what what if you let students just explore on their own pace like literally and had a personalized tutor for every single individual. Like that is the highest version of what can happen here. >> I I was very uh inspired by I I did a year abroad at Oxford and they have the tutorial system and like literally every student has a tutor in the expert who not always infinitely patient, but >> [clears throat] >> you you have access to the expert and they, you know, over over the semester build an understanding of your understanding and your lack of understanding and you have to go out with just a syllabus and in one of the 39 libraries and teach yourself and then try to write a persuasive persuasive essay to the tutor as a learning process and I think it well, what if what if AI higher education was sort of based off of this tutorial method, but instead of producing an essay as the artifact, you're producing some sort of rich multimodal interactive experience that is actually much more aligned with the multimodal nature of your understanding of the topic and then others other students can interact with that artifact and learn and build on top of it. And uh to you were saying this earlier, but like absolutely allow the sort of natural intrinsic curiosity that all children all students have to drive the process organically. That seems to me like a future of education that is infinitely better than what we currently have and it would be unlocked by by AI that understands our minds. >> Okay, well, we we've got a very wide-ranging conversation. Um I can tell that you're a podcaster. Which No, that's a good thing. I mean, it's like some people who are you know, some people like are just like not uh they're like sort of maybe like camera shy or um they don't know how to sort of light up on a conversation. So um I would send everyone to your to your podcast and then and uh to to check out my conversations. Any other you know, we we talked about Jason a little bit. Any other places that that uh typically people should start at in in terms of like getting deeper into your work? >> Well, with the podcast specifically, season 1 was just kind of popping the hood on the lab and talking to divers uh folks, but season 2 is going to be very different. I'm talking to many different scientists, social scientists, Amazon scholars. Really anybody who has something to say about what a mind is and how we can go about building one. So it's going to be a very different set of thinkers, also industry insiders. Yeah, I I'm really excited. I'm having the conversations right now and they are very different from season one. Maybe I'll get you on. >> I don't know how to make my own mind sometimes, you know, so it's one of those things. I'd be list- I'd be excited to listen as well. I I also thank you so much for for the time. I'm really excited to work with Thomas on AGI for for the World's Fair. You guys have, you know, like a really good presence that is coming out and I think like really kind of appropriate for emerging as a lab that I'm excited to see. Like I think you guys have been you know, you specifically you even through adapts been working on this for so long and like like I want I want it to happen. I there's so much drudgery and knowledge work that I'm caught up in that like man, it's 2026. Like how come it's not solved yet, you know? That that I I'm trying to like string together individual tools that I don't really work. A very simple one. Okay, like we're recording on Riverside right now and I got to get this on YouTube. And the whole process of download and edit and upload and all these things. I there's like three humans that touch this. >> Yes. >> And I'm like, why? Like >> I mean, yes, that is a problem that we are trying to solve. >> And like humans are great, but also they're slow and they but an expensive but like you know, I have to we have to align and and that's that's the stuff that will never go away. Like I I need to be like, no, we're going to cut that. We're going to emphasize that. We're going to pull this out and focus on this. And that is the part of the editing that I think you know, I have a specific type of knowledge work, but everyone has some kind of knowledge work that looks like that that that like you're interacting with systems, but then also your that systems are interacting with you. You're interacting with other humans that are involved in the process. Let's solve like, you know, I'm excited for a future where we solve that, but that's not even what you're driving at. You're driving at like how do we like we build entire civilizations? How do we how do we bring up the next generation? I think very exciting. Lots of work to do. >> the digital drudgery though. >> Starts with the Yeah, right. Yeah, I mean, well, that will immediately fund everything else. Like you will get all the monies. >> [laughter] >> Uh because because no one wants to do it. It is so funny. Like I I I have been very excited about you know, computer use and perception agents and and RPA for a long time, but it's not there yet, you know. And I think I can see progress. In this decade it's like it is it. Like we we're we're sort of living in that that last last period of time that was kind of maybe created by software. That we're now sort of creating agents for software so that the problems that software created are also solved by more software. >> At a certain point we have to stop thinking about software. >> Yeah. We're not there yet. >> [laughter] >> Um, cool. Thank you so much Danielle. I hope to see you back in San Francisco and I'm sure we'll chat more. >> Yeah, thank you so much for your time and your fascinating questions.