Meta Muse Spark 1.1 First Test – Is THIS a Frontier Model?

Is that a pigeon? I think it might have been. Today, we're going to be testing Meta's new updated Muse Spark, which is Muse Spark 1.1. Now, I tested the original one when it came out, and I do have to say I was actually rather impressed with it. It wasn't the most potent model on the block, but it was rather fun to play with, and it did end up making some enjoyable games. Right now, I am using this from within Open Code. And I have given this a prompt that I ran with Fable 5 to create a game called Castle Time, but there's a lot of additional niche detail that this needs to include, and we'll get into that a bit later, but I do just want to start out by taking a quick look at the introduction post. Of course, this does seem like a significant leap in capability over its predecessor based off of what we see in these benchmark charts right here. Now, for the purpose of this, I'm going to for this moment ignore the other models here. But look at the base one and look at the jumping capability on Deep S. SWE 1.1. It went from a 10 to a 53.3. Sweetbench Pro went up. Not as much of a significant leap, but still this does seem like a proper jump over its predecessor. And for that, I am excited. Now, because this is compared to other models like 3.1 Pro High, which is old now, Opus 4.8, ate and GPT55. I'm going to test this properly as I would were I testing any one of these. This will be getting no brakes or anything of the sort. This is a new entry into the Frontier state-of-the-art model space. It will be very interesting as well to see how it compares to Gro 4.5. That is not included in the benchmark charts here, but doesn't matter. We can tell on Vibes as well and things of the sort. So, before we go further, please do feel free to subscribe as I do want that 100K plaque. And let's just take a quick look at some of the additional information in this announcement blog post. Although to be honest, it's a fairly light blog post in terms of the model announcement. It doesn't tell us a huge amount about it. It just basically talks a little about it. Shows some demonstration and some benchmarks. So the feel for this will definitely be coming mostly from the testing. They do have this meta internal coding bench where this is showing like just under Opus 4.8. Now, let's take a look at some of the interesting things for at least development or API use of this model. It is price-wise very competitive. Though, I do need to say that with the caveat that I don't know where this sits in terms of what models it's competitive with from OpenAI or from Anthropic. So, these prices may change depending on whether or not it's comparable to like Sonnet or Opus or something. So, it's $1.25 per million in and $4.25 per million output tokens. They have some additional info here just pertaining to like using this as a developer. However, something interesting is the context window for this is listed at a bit over a million tokens. I don't have any specific tests that will really push its context window to that degree and see how well it does. But just having that over a million is pretty exciting, I would say. All right. So, from within the web playground, I am going to be doing the browser OS just so we can run something a little more intricate at the same time in open code, which is that Fable 5 game test that I have a result from Fable 5 and it was a lot of fun to play. All right, so in just under 15 minutes, Castle Time has been created. It is running apparently as well. So, we can just pop right to there. Port 8087. I'm excited. All right, so let's check out the Muse Spark 1.1 Castle Time. Okay, I want to make sure that we do have sound. The speaker is on. It should be giving us some sound, right? >> Wake up, Krono. I found it. The scroll your thesis was about. The one they say knows how to fold light. >> Okay, so we have to continue. Let's look around in the bedroom. Okay, not bad. It has those vertical weird like LEDs that you sometimes see. All right, I'm just going to Ae to continue. It's designed to work with a little Bluetooth controller, but um keyboard fallback as well. So like it doesn't seem like the keyboard fallback is working. That's nice. Why is it investigate the buzzing device on your desk? So we need to go over to it. keyboard E is supposed to work. >> Wake up, Chrono. I found it. The scroll your thesis was about how to fold light. >> Where did they get this voice? All right, I'm hitting E. you. Okay, I have a few choice words. Actually, I'm just gonna I'm gonna be calm. I'm just giving it some followup. Like the keyboard fallback is not working control-wise. I also like what what is that voice? Take a look at some of these assets. Okay. Wow. It's genuinely a single HTML file. The other I the prompt doesn't say it needs to be. The other models have done this in significantly more files. So, we'll see. All right. Supposedly, it fixed the keyboard not working properly. So, >> wake up, Chrono. I found it. The scroll your thesis was the one they say knows how to fold light. Okay, just please let me like E. Okay. Why can't I interact with anything? A E click space. I even have the controller hooked in and and it's not doing anything. Okay. Like like why do the keyboard? Okay. You know what? We're going to come back to this. In the meantime, let's do our browser OS test. It's not funny. I'm just All right. So, here's our browser OS test. Okay, it did it in 549 lines of code and it started out by spoiling the GTA game, which is infuriating. Though, I'm going to say the rest of this is quite acceptable. Clock is correct. I won't spend as much time on these. I know they're a little outdated. It didn't include a right click, which is surprising. Breach off or on? That's probably a special feature, so we'll save that for last. All right, let's check our start bar. Okay, Sunset Dune, Deep Void, Living City Live. That's a sick live desktop background. It's like a top- down view of a city or matrix code, which is just um I don't know. Okay, it's a fourthwall brake engine. The special feature. Very creative. Let's just look through the apps. I'm going to change the desktop background because it is a little distracting here. We'll just I like deep void. I think that's cool. Files. Breach protocol.exe. Time shards. Drag a file to GTA window to spawn object. Okay. And these don't open, but interesting interactivity. I think we'll go back to that. It seems very very focused on the special feature here. I like that. It's like one off on the bottom, but I like it. Nonetheless, let's try pixel paint. Can we resize it and still have the canvas change? Good. Not bad. No way to save it, but it was acceptable. Block City GTA. We'll save that for last. Now, Nova Drift. Well, I can't actually like I'm spam clicking the the key to move and it it's impossible to not hit these. Space is to break. Okay, W is boost. Okay, we did speed up the This is a cool game. It's just it's it was impossible to avoid the objects. Okay. E is to enter or exit a car. Drop a file to spawn ramp. This is where it seemed like the file system being open here. So, we can actually kind of sick. Good. Good. Um arrow keys. Oh, shift for turbo. The problem is we can't actually go forward when in the car. Now the keyboard and mouse have stopped um affecting the movement. This thing seems to have a difficult time just mapping keys to game events, which is curious. Okay. Can we All right. Good. Shift is for turbo. Good. There are mesh colliders. All right. Um, games can now leak to desktop. Okay. Maybe that's cool. Look, when we actually fail in the game, it affects the desktop. That's That is neat. I like that. It's very creative. And the way it actually changes the colors is cool as well. All right, that's a very interesting result. All right, now we have to go back and see what exactly is stopping the keyboard. I'm going to clear cash. We'll try again. >> Wake up, Chrono. I found it. The scroll your thesis was about the one they say knows how to fold light. All right. So, it's it's not working and all right. So, I'm telling it to just fix the code. Is it going to say all s master? I had accidentally closed the open code session. It may have been a Freudian close, but I just We'll just see cuz it was trying to like run it again and check it and I had Okay. Okay, it said the line. Good. So now, hypothetically, if we go back in here and then just run the same thing again and then go to the URL, it will hopefully have proper keyboard. And I have to listen to this voice. >> Wake up, Chrono. I found it. The scroll your thesis was about the one they say knows how to fold life. >> We'll just pretend like this is our first time. Like, oh, okay. Very cool. So now E. There's something like I've spent I've spent I've literally spent like $6 to just get like like kicked I need to ask Claude if I'm if I'm all right. So, we're having Claude fix this game because I gave the meta model a few chances and Claude gave us actually a valuable assessment of the issue here. So any click permanently jams the laser on. Okay, that's fine. Dialogue advanced is wired four times over plus an 80 millisecond polling loop. So the intent is clearly there and it does fire. Continue skips lines unpredictably. Here's I think what we're facing. E to interact only does something when you're basically on top of the object. So you need to be within 1.8 units and I'm unable to seemingly get within there. So, I'm just asking Claude to just fix this and change it. So, we don't need to be that close and then we'll take a look at it. Just keeping that in mind. The game is still Muse Spark. So, >> wake up Krono. I found it. The scroll. Your thesis was about the one they say knows how to fold light. >> Right. So, we should be able to Thank you. interact. Okay. So, our character isn't speaking, but that's okay. It could be a mismatch with the voice or something. It was on October 10th, 1283. Karvin Castle, North Wales 53.1393NUS 4.2768 WA Da Vinci type polymath was imprisoned in the dungeon, he added. But tonight, a trebuchet misfire will set the south wall ablaze. Everything burns by midnight. If you want it, you leave now. You didn't wash your laundry, though. Good luck. >> A bit of humor. Okay. Your Motorola time port on the desk is buzzing. It has a temporal antenna. Now interact with it. Nice. Good. Good. Good. Okay. Finally. Oh, initiate jump. It's interactive. 743 years. Don't think about socks. You forgot socks. >> Demon. His clothes shine like devil skin. Get him. It didn't. It didn't. >> Wake up, Chrono. >> But tonight, a trebuchet misfire will set the South Wall ablaze. Everything burns by midnight. 743 years. >> Are you kidding me? >> Forgot socks. You forgot socks. It didn't. >> Demon, his clothes shine like >> All right. You know what? I'm sorry. This test is a little too much for this model. And that's okay. It's a test that I've only run so far with Fable. We'll just keep going. All right, so I'm giving it the self-contained C++ skateboard sim test, the California Boardwalk aesthetic version, and we'll see how it does. All right, cool. So, it's just checking what specific things are available on this system for it to build this with. And it does not seem like it's going to try to use Ray, so we won't have to redirect it. Some folks say, why not ray? It makes it simpler for the model to perform. It has allowed for significantly smaller models like Gemma 412B or something, but for larger state-of-the-art models that cost a bit of money to use, I like to see their more performant capability without using Railip. All right, the code's been written. It's right at 1,600 lines, so it's going to verify it builds and then package the final thing. So, we may see our first glimpse of it depending on what it does here for QA. Okay, no, we got some errors when it was trying to build it. Only a few. So, it should likely fix these, which it's doing now. And then it will rerun to see if it builds. All right. And we have our result. So, let's take a look at it. And it is done. I know folks are saying, "Let it finish." It It was just um doing like That's not half bad. That's really not half bad, especially considering the issues we noticed with the game that like took years off my life to try to get working. Not bad. The shirt changes colors when we're grinding on the rail. I noticed the shirt changing colors is a common thing in this test. I have no idea why, but I noticed it a lot with open weight models. All right, kickflip animations. Not bad. All right, I think we hit the NPC and it actually made us bail, which was that was cool. Good. I like that. All right, what else do we have? A lot of tricks here. B is a 360. Oh, just like that. All right. All right. Let's take a look at the environment. Let's go back. We do have the palm trees are actually pretty nicely drawn. I'm going to say the water effect is acceptable. The storefronts are acceptable as well. Nice, dude. Watch it. And it even put like clever little things there. I'm actually I'm okay with this. It's not like state-of-the-art Frontier. This is definitely closer to like something you'd get with GLM 52 Kimmy K 2.7 than any of the high-end GPTs, but nonetheless, it's it's solid. So, from the web chat here, I'm giving it the beautiful front-end test. And this is the one where it needs to create a 3D model of a luxury watch, put it in the hero section of this website with a nice cinematic camera panning shot, and then the pricing cards for two models of watches should also have 3D models that it creates. So, this should just make a high-end front end with a nice 3D modeling and camera panning thing. So, here's our front-end design test, the watch website in specific. Okay, not quite. I actually don't mind the font it's used and the watch co section. It's just the models, unfortunately, not up to par with what we've seen from other models. Mainly the missing hands and things like that. Maybe it's an LED watch if we were to give it the benefit of the doubt. Okay, so I'm a bit troubled, but these watches are significantly cheaper than ones we've seen on other sites. So maybe that's a reflection of the rendering here. Let's give it some reinforcement. No, I I I don't want that. Stop it. Okay, so I I responded alert expecting it to be like I'm sorry. Like what do you mean? It just went ahead and thought that meant add an alert bar for the summer 26 drop. So I had to stop it and say sorry. I meant garbage alert. And we'll see now if it if it tries to improve the result. Got it. You hit the garbage alert. Let's take it out. Nice. The last one was too low poly and too much wood. All right. Keyshot grade now. I like it. It understood intent. So, here's the second try of our watch website. Okay, it's still loading. It just it didn't make much of a difference. It's slightly actually different. It's just unfortunately not really. Next up, I'm going to try a multimodal coding test just to see how it does. I see the ZX Spectrum photos. How dare you? That is my own specific design. Thank you very much. So, it's tasked with creating a 3D replica of this laptop that can be opened and explored. We also need to have a functional keyboard where we can press things in the keyboard that it 3D models and they will appear on screen. So, this is more of a test of its capability of seeing something and then transposing it into 3D based off of a decent amount of reference photos. All right, let's take a look at our photo to 3D model/interactive website test. Okay, you know what? That's that's not bad. It's not the best one I've seen, but often times models will invert the keyboard for some reason. This hasn't done that. Let's check the open close capability. Okay, this actually it's not perfect, but it's definitely better than some of the open weight models I've seen do in this task. Toggle RGB. Cool. Reset cam. Now we should be able to type on the screen. The problem is the screen is either hidden behind something or it's just not working. But the key press events don't look bad. I will say those are decently solid effects there. The close animation is not bad either. Interestingly, no model really has gotten the hinge to be in the correct location, which would be over here. There's definitely some issues here, but lid open60. Oh, that's just frames per second like from this scene. It's passable. So, next up, I'm going to give this a 3D model test where it needs to create a 3D model of a V8 engine for my remote control car. I have a 280 sized motor that I'd like to put inside of this engine model. So, I wanted to make it something that looks realistic and also is something that can be 3D printed. We'll see how this does. Interesting. It just wanted to find out about the system and it's looking up motor specifications. Good. Using standard 280 specs. So, my camera wasn't recording for the 3D model test, which you know, it does happen sometimes. Basically, overall, we'll look at this once more. It actually did a solid job of making a replication of a V8 engine. Some of the things it included were actually very interesting and levels of detail that I've not seen in a lot of other models when doing this prompt. Unfortunately, the big issue, as we're going to see here if we open the SCAD file, is that the hole for this engine is not deep enough to fit a 280 motor. And even if it wasn't intended to go all the way in there, the problem is this base here just cuts into the cylinder. So, it would be physically impossible to install that in here. I did also tell it to optimize this so it could actually be 3D printed. And unfortunately, there's just a bit too much odd geometry everywhere. And if we open this in a 3D slicer program, we'll be able to see a bit better in reference to that. So, this would be very difficult to print just with like gaps like that, you would need I mean, it wouldn't be horrible to print, but it would need a lot of support. And then we have an issue like the air cleaner here on top of what would be the carb is just floating. The valve covers are also kind of floating. The connection's not great to the V's. But the things that it did do that were good that are just weird is the valve covers actually have like individual hex bolts that show like them being bolted on. And they're really quite well done. Now that I looked more at that though, yeah, I mean that's cool. And then the other thing it did, it put an oil cap on one side of the valve cover, which is kind of how it would be in real life. So just weird detail that it included like that that's not often seen in other models. And then unfortunately it just wasn't altogether competently. But it does show some level of promise. And on first glance, just from like zoomed out, it's like, wo, this is really quite good. So, that was my initial reaction that I failed to capture on camera because I was not recording. Nonetheless, we're going to move on now to a different task, and that is going to be one that I unfortunately did not get. I tried doing this in my GPT56 test, and it was the last thing we ran, and it just froze up, and I was never able to play it. So, I've just taken a screenshot of that prompt. I had meta OCR it from this web chat interface, but I will have it be built through open code. I'm going to start it in plan mode here cuz why not? And we'll see how it differs if it differs. Got it. Street eat. You are big fist McGee in a pastel low poly city block. Okay, that name was not included in the prompt. So, it came up came up with that on its own. I did see some folks talk about how this could be like funny to talk to. We'll try that later. Okay, that was a good plan. Open questions. Now, the original Muse Spark, when I tested it, it made a very fun Steve the PC repairman office game where you just go around and smash PCs and beat up NPCs. So, it'll be interesting to see how this does. All right, let's check out Street Yeet. So far, I like what I see. One fist to yeet them all. Hold mouse for a mega charge. Look for golden objects. That's a bonus heat. Try to get the buses. I dare you. Again, we can we can punch pigeons, which was included in the GPT56 game. It's not included in the original prompt. This This is not bad. Well, let's uh Oh, yeah. All right. The pedestrians are not quite right, but hey, why can't I? Okay. Um, the cartoon style and stuff. All right. I It said try to heat the bus. So, let's just let's just try this. It didn't go as far. Okay, now that's actually it has sound effects when we hit it. Cool. We're supposed to collect these, but I don't see how. Let's just yeet this car. Mega delete. All right. I can't yeet the pedestrians. Oh. Oh, I wish they looked more humanoid. Is that a pigeon? I think it might have been. Well, birds are meant to fly. I can eat everything here. LM yeetus. Why is it models interesting? Okay, the model's fun. The Muse Spark, even the first one, it's good at making fun or funny games. All right, let's try some multimodal coding. Also, I'm going to turn off this just rhythmic pounding noise is questionable. Yeah, I see it. You're 108 deep, 13,000 chaos, 141 19 m far. Nice. I also see some issues. Okay, I wanted to include that the humanoid, like the NPCs are also garbage, but nonetheless, we're going to just let it ride. All right, let's check out our hypothetically improved Street Eat game. Let's just try some creative writing. So, I'm giving it this photo, which you may be familiar with if you watch the channel. So, thank you. So, I'm saying this is the cover for an upcoming psycho thriller romance novel. Give me the title and back cover summary of the book, and we'll see what it says. All right. All right. All right. Title: The Way Heolds Me. Let's zoom in a little. Tagline on the front. He'd do anything to keep her smiling. Here's the back cover summary. Everyone said Anna was so lucky when Mark asked her to leave the city and move into his beautiful crumbling Victorian home three hours from anyone she knows. It was romantic when he deleted her social media because you don't need anyone else. It was devoted. When he holds her for photos, her his smile is perfect. Hers never is. In the picture on their mantle, he is beaming his arm around her shoulder. To their friends, to their neighbors, to the camera, it's love. Anna knows better. It's a warning. Stay still. Smile. Don't pull away. She has been planning to leave for months. She has a bag packed in the attic and a bus ticket hidden in a loose floorboard. But Mark has been planning too. He knows she's unhappy. He knows she's been lying. And he has no intention of letting his wife ruin their perfect love story. A claustrophobic slowburn psychological thriller about performative love control and what happens when one person who says they'll never leave you means it. All right, that was that was good. So, I'm asking as a follow-up, give me a TLDDR of the whole story. Anna marries Mark, the perfect you're so lucky guy, and moves into his isolated home. Very quickly, the romance turns coercive control. He isolates her. Okay, we already kind of did this. She secretly plans to leave. While packing, she finds out Mark's first girlfriend didn't just leave him and move away like he said. She disappeared and the police never found her. On the night Anna finally tries to run, Mark catches her with her go bag and forces her to sit for one last photo, the photo from the cover. He tells her no one will believe her if she does leave. She pretends to give in. That night, she uses all the things he's taught her about how he watches and controls everything to turn it against him. Records his confession, locks him in the old root cellar, and calls the police. It ends with Anna back in the city a year later, living alone. She gets her mail, and there's a new framed photo inside of her sleeping in her new apartment taken from her outside window. All right, this definitely was something. So, next up, we're going to be doing the 3D city block that is shown in different time periods. We'll see how it does. This is one that I've more recently introduced as a test, and it's always kind of fun to see what the models decide to include, especially for the future of 2015. So, we'll see how it does. All right, in 4 minutes and 8 seconds, we've received the result for the city timeline test. Very, very quick. Keep in mind, none of these are instructing it that it needs to do these in single scripts. Okay. It started us in 1985. That's definitely um notice sunset VHS. All right. We have a a button there. I'm not 100% sure what these do as of this second, but that's okay. Oh, it froze. All right. We'll just refresh. Um Okay, cool. The cars are moving. There's a car in the store. It's possible there was an accident. Let's go to 1945. Kind of cool. Look at it got more grainy. And the folks like the It's almost grainy to a fault, but the people's clothing is changed to reflect. Even the car that's driven into the building looks a little older. The street lights have changed. Let's do 1965. Much happier. Very, very mid-century modern looking colors on the buildings. And the car does have fins. Good. Good. Good. Okay, this one's going backwards. That's all right. The people's outfits are more colorful and happy. Drink Atomic Mod Boutique. And the street lights have changed to like mid-century modern shapes as well. Kind of cool. All right. This is still like it's basic. This doesn't have the level of depth and intricacy that the other state-of-the-art models do. Not at all. But it's still okay. Bland like corporate 2005. Everything's a little more toned down. 2025. Okay. We have a sound. It's still like um solar panels on the roof. It's got some understanding. It's just not at the level of the other models currently that it's being benched against. All right, we have hover vehicles. Everything's kind of neon and glowing, which I personally like. It's just interesting how a lot of models converge into this almost kind of like 85 inspired but in 2055. So some sounds chronoblock Manhattan after light present tense analog sunset neon decade dream of tomorrow and ashes and victory. Interesting. All right, the last test I'm going to do is from the web chat interface right here. It is using this model. This is the suggested temperature from the docs on XH high. This is going to be the unified beautiful subway scene as well as turn it into an FPS just to see how it does on one final common game test. All right, let's check out our subway FPS result. I'm giving it a follow-up here that it will hopefully understand to fix this. All right. So, here's our hypothetically fixed subway. The same problems occur. It just changed the theme to like light mode from from dark mode and Oh, wow. Okay. All right. Third time's the charm. Good. Very good. This is a big station. Nice. Another big test. No, but we get some effects like of the ammunition hitting. So, I'll take that. This is one of the few ones that actually shows like the the subway tunnel instead of just being blocked off, which is curious. Flatline. All right. Well, that's going to conclude our first look and test of Muse Spark 1.1. I have to be totally honest that I don't think this model is ready for prime time. They gave me $20 in free credits. They give everyone that if you put a billing thing in. The entirety of this testing used $11.84. It's um again, I don't think it's ready for prime time. It made a lot of just the results just weren't that good. Some things showed promise and things that it did that included that like I don't see commonly in these tests. like the subway actually had a view beyond where the subway car would go. Most models just block it off. The 3D model test, actually having individual bolts for the valve covers, the oil cap and the correct side, but I have to say for the pricing of this model, I don't really see the value in this. I think composer 2.5 is cheaper and better. And yeah, it's a attempt at improvement from a player who hasn't been involved in the game for a while. And that's good to see. I would be interested to see how this progresses maybe over the summer, maybe into fall. But as of right now, I have to be honest that I am disappointed with this model, especially considering the price. Make of that what you will. Overall, I wanted to test this because I was very excited to test it and it's again, it's an early entry or improvement. So, with that, if you have any questions, please feel free to leave them in the comments and thanks for watching.