Introducing the Gemini Omni Flash API

Okay, so today Google has released Gemini Omni flash in a wider release than what they had previously including the API. In this video, I'm going to cover the four key areas where this model really shines and how it's different than VEO etc. We're also going to go through the API and look at how you can use this with the new interactions API. All right, so there are four key areas that Omni is different than VEO and other video generation models. So, I'm going to go through these quickly so you can sort of see what they are. The first one and probably the most powerful one is the whole idea of conversational editing. So, you can see here I've asked it for a wide shot of a man walking down a street in Tokyo with a black cat walking behind him. And sure enough, if we come in here and look at this, we can see we've got the man, we've got the cat, we've got Tokyo there. And it's pulled out prompt in reasonably well. And it considering the prompt was not super specific in here. All right, what we can do next is we can ask it change the cat to be a ginger cat. And you can see now we've got the exact same man, we've got the exact same street, and we've even got the same blocking of the shots etc. But we've conversationally changed the cat from being a black cat in the first one to being a ginger cat. And this is exactly what is meant by the conversational editing. We can change specific things in the video conversationally without changing other things that are going on. So, not only can I change the cat, I can ask it to change the time of day. And you can see sure enough, we get same cat, same guy, same location, different time of day. We get a few more people perhaps walking there in the daytime which makes sense. And then we can even do things like where we make multiple conversational changes at the same time. So, change it to be a woman wearing a red dress, the cat back to a black cat, and the time back to night. And you see, sure enough, we go back to pretty much what we had at the start, but just now we've got our black cats, and we've got a woman in a red dress now. So, this fundamentally is the power of conversational editing. You can swap characters, you can relight scenes, you can alter sort of angles, you can change whole qualities of the video. Not just video that's generated, but even video that you upload, as long as it's only 10 seconds at the moment, and get it to make changes to those. Now, you need to have the rights to that video to be able to do that, but once you've got that, you can pretty much get it to do all those kind of changes, whether you're prompting it in one of the Google interfaces, or as later on I'll show in the demo, you can do the same thing with code. Okay, so the next example is where we can have multimodal inputs. So, in this case, I've basically taken the video that we had. I've then combined it with a new picture for a new location, and then I wanted to put my own cat in here. So, you can see that the prompt that I used is in the video, change the cat to be the cat in this picture, and the street to be the one in the western town above, keep the woman in red the same. So, I'm going to turn off the audio, but you can see, sure enough, now we've got the same set of shots. We've got my cat in there, pretty good representation of even things like his tail, maybe not exactly right, but it certainly got his coloring and stuff like that. And you can see we've got that same shot of where they sort of walk down, we've got a shot of the legs, and we've got the character walking off to the left in there. Now, the perspective might be a little bit off in some of the shots, so you could certainly come back and change this again, but now we've got this idea of adding multimodal references in there. And if we had audio, we could put that in there. If we have multiple images and stuff like that, we can put that in there. We can put in other videos, etc. The one thing that is kind of limited at the moment, and understandably so, is that you can't put just like a picture of someone's face and then have a recording of them saying something and get them to lip-sync it. Google's been quite careful about the sort of deepfake uses of this kind of thing. You can do things though where you give it a recording of you talking and ask it to translate it to another language and it can do that kind of thing for you. Okay, so the next area that they've focused on a lot is the idea of sort of world model and simulations. So, we saw this a little bit already in the previous examples, but I've taken the video of the woman walking in red. I've basically asked it to add some light rain and more puddles so that we can see the reflections of the characters. And you can see, sure enough, we've got things now where we can see the reflections of the shoes, the reflections of the cat coming out quite nicely. And this is one of the goals that this model is trying to do is that in many ways it's trying to create a model of the world that will have similar physical properties to the real world would have. So, you can do that to some degree by basically getting it to emulate gravity and do things like that. Now, it's not perfect, but it is pretty good at doing this kind of thing. And you can sort of see, the one thing that has changed is put a close-up on her face so that we can see the rain a bit more, but we can certainly see the rain coming down. We can see the reflection of the cat in there quite clearly. Okay, so the fourth key area that the model can do is adding in text here. So, you can see we've got that video. I now basically tell it change the signs in yellow to be for the store GoGo Curry and have them in English. And you can see, sure enough, it's changed a lot of the signs, including some of the ones that perhaps weren't yellow before. And we've got tracking of most of them, although we've got some coming up in Japanese and some coming up in English. And we can actually do the same if we've actually got a logo. GoGo Curry is a brand. It's got a gorilla as their sort of logo. And so now if [snorts] I basically ask it, "Hey, add in the GoGo Curry logo and font for their designs in here." And you can see that sure enough, it's basically put in the logo. It hasn't done a perfect job. We do have the gorilla logo appearing in there. Now, we could use the conversational editing to go back and be very specific. I could have made my prompt more specific about the location that I wanted the text to be, etc. So, these are sort of the four main strengths of the Omni models versus the VEO model. So, let's jump into the code and have a look at how you could actually do that now. So, if we come in and look at a notebook of using this. One of the key things here is that this is using the new interactions API. So, this is an example of using it. And it kind of makes more sense here because we're not necessarily trying to have a normal sort of chat conversation here. We're actually doing tasks where we're getting video back. We're also even the ones where we've got multi-turn stuff going on. You'll see that each time we're getting videos back going through this. All right. So, there are a number of key things that this model can do. do text to video. That's the same as something like VEO or something like that. It can do image to video. That was certain versions of VEO can do that as well. But it can also do things like taking multiple references to video. And they don't just have to be images. They can be other things as well. And like I talked about earlier in the video, the whole goal here, while this first one is just video out, The goal here is that eventually the model can produce lots of different modalities out in here. So, first off, we're just basically getting API key. There's some helpful functions in here for just displaying images, passing images around, that kind of thing. And then first off, we start off with a text-to-video kind of example. So, this is basically a hyperrealistic close-up of a fully and completely pitch-black puma drinking a large cup of tea. So, the video that we get out is this. >> [snorts] >> Okay, so you can see that the video both produced video and audio. That's a common thing with this model. And you can see it's done a pretty good job, although it's maybe not completely pitch-black in the example of it. Now, to do that, we pass in the standard aspect ratio there of 16 by 9. But if we want to, we can change the aspect ratio. And there are a set of different aspect ratios that we can do. So, if you wanted to make something that Instagram shorts or on some kind of like portrait stuff, you can certainly do that in here as well. Now, you can set the duration, right? One of the arguments that you pass in is duration. Now, currently, it's maxing out, I think, at 10 seconds. They may have plans to make that longer, I'm sure they do. Look out for that perhaps in the near future. But you can sort of see that this is basically just an argument that we pass into the video configuration in here. That gets passed into the response format that we're going to have back from this. And from that, it's able to then produce out I've got a new prompt in there of a slow camera pan across a tranquil rainforest jungle during autumn, and we're tracking the puma from left to right. So, let's look at that. And you see it's done a very nice job there of being able to do this tracking shot. It's tracked it left right like we asked. It's gotten the rainforest. I don't know if rainforest would necessarily in autumn have the sort of autumn colors, but it's done its interpretation of this here. And you can see sure enough this is 10 seconds out, right? This is exactly 10 seconds out. If we change that duration, we can get something that's going to be shorter if we want that. All right, next up we've got image to video. So here we can use Nano Banana to just make an image. So now we've got a Japanese cherry blossom garden with our puma in it. So this is just a simple Nano Banana image. If we come in and look at this, we can see that okay, we can pass in which image model we want to actually use whether we want to use Nano Banana or Nano Banana Pro. That's something that we could do cuz we're just generating out an image, right? And then we're just showing that image there. Now, if we want to turn that image into a video, we can just pass this in as a reference. So we're still going to have a text prompt, right? That is going to the video is going to condition on. In this case, it's cherry blossom petals flow gently while a breeze ripples through the garden pond. The camera zooms back and follows the puma as it walks across the scene, right? So we can sort of see what we're going for here with the image before we've even made the video. So this is something you definitely want to do if you're trying to save money and get things that you actually want. We can see that from this, we then basically pass in for the input both the image data being our reference image and the text prompt going in there. And we're going to get back our video response back going through this. And sure enough, it goes off, and it's done this, and it basically renders off the image. And you can see here, we've got this basically the same as our starting picture there. If we play this, we've got the zooming back, we've got the tracking shot, we've got the puma deciding to turn around. Now, that's not something I necessarily asked for. So, if I was planning to have it sort of walk off screen or something like that, I'd probably want to put that in the text prompt in there. All right, we can do multiple references to video as well. So, we've got a black cat with a purple collar with a name tag which says "Nero" on it. So, this is basically now just using now banana pro to make two images. There are the two images that we've got. And now we can pass in multiple images as references in here. So, you can see, sure enough, we've got our video prompt, which is going to be the text. This is going to be The cat is playing playfully batting at the ball of yarn on the wooden floor in one long unbroken scene. Now, if you look at the way that we've done this, we've basically stored the sort of interactions that we've had before for making the different images. So, the first one is R1, and the second one is R2, right? Reference one, reference two. So, the way we can do this here is we basically just go for the previous interaction of R1, and then now pass in the yarn, pass in the video prompt, and then pass in this time the response format that we want is going to be a video, right? Type video aspect ratio in there. If we look at the ones up here, the response format was an image format, right? So, this is part of the interactions API here. We then render this new interaction, which is our video with R1 and R2 in it and it comes back with this. So, you can see sure enough we've got that cat, we've got the ball. Both of these are reference images, but now we've got it being guided by the actual text prompt in there. Okay, so we've got our playful cat there. Now, we've got that interaction there, right? This is our last interaction in there. So, what we can do is we can come in and do sort of multi-turn on the video. So, where I can say right, okay, I like that video, but hey, I want to change something in it. So, this is what you would see, for example, if you're using the Gemini app and you're just chatting with it as it makes the video. This is how you do that in code. So, we come in here, we can see that, okay, we've got that last interaction video, and we're now going to basically pass that in as the previous interaction. That means it's got the video, right? It knows the details about that. It even knows the images that were references that are coming in because we've had this sequence now of multi-turn. And then we're now passing in an edit prompt in this case. We're just passing in the input. It's just going to be the text input in here. And we're telling it turn the cat into a fully black puma kitten in here. So, if we go back to the start of the one before, we can see that that was the pose that it was starting. So, we're not changing telling it to change the pose or anything. We're just telling it to change the cat to be a puma kitten. And everything else should be basically the same. And you can see sure enough, it looks like it's done something which pretty much is the same sort of thing. It falls over in the same way, it plays with the ball in the same way, etc. So, that's how we would do editing. And you can see that it's not that difficult to sort of string a bunch of these together and then do something. Now, we can also keep this going with our last interaction and now we can take one of the prompts to make it look like a game. So, now we're just saying, "Okay, make the animation look like an old 8-bit retro video game." And you can see that okay, it looks like, you know, if we look at this, we've got a wooden floor, which is the same we had with the first one. We've got some kind of fireplace or logs but not not a fireplace but logs being stored back there. If we go to this cartoon sort of 8-bit retro video game look, it's going to play around with it. So, the floors, I guess they are still wooden floors. They're just like you would see. Our logs now are kind of become a fireplace a bit but we've now got our cat in the same pose. We've got the string there and you can see sure enough if we play with this, we've got it going through, falling over in a similar kind of way as it plays around with it. And you can do sort of multi turn editing. And that's one of the strengths of this entire model is that if you get it where you'd like something there or you like most of it there but you want to change one thing, you can actually come in here quite easily and get it to change. You can do things like if we make another sort of reference image. So, this is a watercolor, again made with Nano Banana and we're going to restyle it now with that one and this is again we're passing in that last interaction that we had up here after making it this sort of retro 8-bit sort of style in there. We can now basically restyle this video to match the watercolor painting on the style of the reference. You can see sure enough it's able to do that. If we had done this with the first one, we probably would have had the better logs and not this kind of fireplace. That's an artifact that came in the retro 8-bit. So, if we wanted to, we would store the different versions and be able to go back and edit one of those. That's something sort of key there. Now, finally, what we can do is we could also do things where we actually take something like that painting and pass that in as an image now to actually use in the video. And you can see sure enough this time it actually puts that image in there as we go through it. Now, we've gone back to a bit more like the 8-bit sort of style in there and then it flips. So, this one is kind of weird, right? And it shows you some of the challenges that you will have where it's kind of confused that okay, it's in the 8-bit kind of style a little bit. Actually not definitely not 8-bit, maybe 16-bit colors or something like that there. But then you can see at one point it flips to just be like a normal cat and goes back to sort of our normal cat kind of thing in there. So, the model's not perfect. You need to be aware of this. We can see we've still got the sort of blocky stuff with our fire in the background there, but we've gone back to this kind of wood in here. So, that's something that you want to be aware of. Another really cool thing that you can do, too, is you can actually edit your own videos. So, if you've got your own video and it's 10 seconds or under, you can do something like this. So, this is the example that I made that got featured in one of the launch videos for this. And the idea here was I'd seen some people do some things like this. I wanted to try it out. So, I basically just took a video with my phone of me filming a picture of my cat on the screen. So, if you look at this video here, you can see that I just filmed my hand going in and going to touch the screen and then pulling back. All right? Nothing super special in there. But that then becomes the reference video which we're going to edit. And then to that I add in the text prompt. So, I pass that in and I pass in the text prompt saying a hand reaches in to pat the cat. The screen ripples as the hand touches it and it crawls out of the computer screen. And I played around with this a few times, actually. I think the original version was actually a bit of a shorter prompt, but you can see that it works out that, okay, it's the cat that's going to crawl out of the computer screen, and we end up with something like this. And yes, it sure enough the cat comes out of the computer screen and gets on the desktop. So, it's pretty cool that it's able to sort of work things out in here. And it can do different versions of that. So, I was playing around before with it some other versions of where I didn't tell the cat to go to the right. And you can see on this one it actually just steps straight out and and straight off to the left, right? So, the more that you describe what it is that you want, the better the results you'll get out of it. But, the cool thing is you can actually do sort of full-on video edits. And there is some really nice videos that came out around IO of people being interviewed and animals just sort of walking in. And as it cuts from one close-up to another close-up, there's like different animals now with those people and stuff like that. That's done in this way, right? Where they break it down into sort of 10-second chunks. You work out like, okay, we're going to use this part. And then you just edit them all together to get the final result out. It's a lot of fun, actually, if you start playing with it and seeing what you can do with it. So, anyway, this is modified from one of the example notebooks, but I'll put this one in the description if you want to play with it yourself and go through it. If you are looking to do anything with video, this model is certainly useful for those tasks. It would be nice if it could do things longer than this. My guess is that that will come over time. And I think over time we'll also see that you can add in a lot more things of being able to do better conditioning on things like audio etc. As of the time I'm recording this, Google's been quite intentional that they don't want you to be able to sort of put up an image and an audio and make the person say what the audio says or things like that. That's something that obviously they don't want people making deep fakes with this. But if you are looking to do anything with the video generation, this is certainly worth checking out, especially if you've got video that you want to edit or you want to change specific things in this. Whether that's adding special effects, whether that's doing editing etc. Anyway, let me know in the comments if you've got any cool uses of this. I'd love to hear from people what they're actually interested to use this with for the API etc. And as always, if you found the video useful, please click like and subscribe and I will talk to you in the next video. Bye for now.