LLM benchmark

The same open-world assignment, different games.

Each model starts with the same demanding assignment: build a browser-based 3D open-world city inspired by Miami, with cars, pedestrians, police, missions, and synthesized audio. The strongest results are playable, but even those have bugs; lower-ranked outputs are incomplete or unplayable.

Explore the top result · Claude Opus 5.5

Updated September 30, 2026Reading time 3 min

Results: 16
Playable results: 13
Demos to try: 16
Method: Shared base + tuning

Claude Opus 5.5

Claude Fable 5.1

GPT-6 Astra

GLM 5.3 Flash

In brief

Every model begins with the same demanding assignment to create a 3D city open-world game in the browser.
Results are judged for functionality, playability, fun, and how thoroughly the game is developed.
Claude Opus 5.5 currently leads with an extensive open-world game.
Follow-up tuning is allowed only within the original assignment and is documented for each run.

Full ranking

Results and playable demos

The default ranking reflects the practical result. Filters keep the ranking intact and only hide runs that do not match.

Filters

Model

GTA benchmark ranking Subjective ranking by practical result in this benchmark
Model	Type	Parameter count / requirements**	Result	Bugs	Note	Demo
1Claude Opus 5.5	Closed	—	Matched the assignment Playable	Many bugs	No corrective implementation prompts; only guidance on concurrent agent count. The full run from the first prompt took 35 h 49 min; it used 130% of a USD 100 subscription allowance including one reset, hit the 5-hour limit several times, and the main agent reached 693.9K context.Show moreShow less No corrective implementation prompts; only guidance on concurrent agent count. The full run from the first prompt took 35 h 49 min; it used 130% of a USD 100 subscription allowance including one reset, hit the 5-hour limit several times, and the main agent reached 693.9K context.	Demo
2Claude Fable 5.1	Closed	—	Matched the assignment Playable	Many bugs	Prompts after the original brief The complete list in its original order, translated into English. Follow-up prompt 12026-09-05 16:34 Try again Follow-up prompt 22026-09-05 18:20 There are bugs. Moving the mouse upward tilts the character downward; you have it reversed. Pressing A moves the character to the right, and pressing D also moves the character to the right; it should be the other way around. When I enter a car, it remains stationary. When riding a bicycle, the camera looks backward instead of forward. I also cannot get off the bicycle. Drivers sit on the left instead of the right. The cars look strange; look at how they appear and improve them a little. Follow-up prompt 32026-09-05 18:41 · 2 screenshots There are still many bugs: 1. The camera is too low when driving a car or bicycle; see the first screenshot. 2. The cars are still broken; see the second screenshot. 3. I cannot get out of cars because I immediately get back in. Follow-up prompt 42026-09-05 18:54 · 1 screenshot Look at the screenshot. There are no asphalt roads except at the intersection. I noticed this throughout the map. Follow-up prompt 52026-09-05 19:01 · 1 screenshot Situations like this occur: pedestrians stand in the middle of an intersection while cars stop and honk. At least add a pedestrian reaction so that, when they are standing in the road and see a car or hear one honking at them, they move off the road onto the sidewalk. Follow-up prompt 62026-09-05 19:16 · 1 screenshot When I am on the bicycle, I hear an engine, which makes no sense. I am not pedaling either. Follow-up prompt 72026-09-05 19:22 · 1 screenshot There does not need to be a hole above the tunnel; see the first screenshot. I also have a question: do all cars currently have the same controls and the same driving characteristics? Follow-up prompt 82026-09-05 19:52 · 2 screenshots Roofless cars, as in the first screenshot, should also have a designed interior: seats, a dashboard, and a steering wheel. Near the tunnel entrance, or above the tunnel, as shown in the second screenshot, I hit what feels like an unidentified object and cannot continue. I do not know whether it is caused by the location or by the cars next to mine. The same thing happened while riding a bicycle: I passed another car at a distance of about 1–1.5 metres and could not continue. I would also like to accelerate on the bicycle and pedal faster when I hold Shift, which I also use for running. The bicycle should coast instead of stopping immediately, slowing smoothly depending on whether it is on level ground or going uphill, and accelerating downhill—like a real bicycle, even after I release the forward key. Follow-up prompt 92026-09-05 22:34 · 10 screenshots There are still problems with the tunnel entrances. In the first screenshot, there is a wall inside the tunnel, and there is also a wall at the tunnel exit in the fourth screenshot. In the second screenshot the tunnel entrance looks correct, but after a few metres I can see through the wall into the entrance, visible on the left side of the third screenshot. I also keep hearing alarm-like sounds even when I have not stolen a car. I still hear them after I am killed or arrested and restart. Cars are parked incorrectly: they ignore the white lines and are rotated 90 degrees the wrong way. I am also unsure whether you differentiated the driving characteristics of individual cars, because they all seem to lean the same amount, including low sports cars. Reconsider this. Tall, heavy cars should lean the most; low sports cars should lean little or not at all; medium-low cars should also lean relatively little—make it behave like real vehicles. They also turn far too sharply, so reconsider steering sensitivity. Motorcycles are the most extreme and are almost impossible to control. Is entering vehicles with F standard? I instinctively press E. When it gets dark, the street lamps look switched on but cast no visible light underneath them, as shown in the sixth screenshot. The sky is green after sunset in the seventh screenshot; should it not have different colours, or is that intentional? The stars are also very large pixels—is that intentional? My car's lights work, which is great, but other cars have no lights; fix that. I also noticed that the minimap in the bottom-left corner appears upside down, and I think the player arrow on it is rotated too. The highway entrance is narrow and begins with a bump above the ground. It does not merge smoothly into the highway. It should first form a parallel lane that continues for 20–50 metres beside the highway, connects directly to it, and then gradually narrows into the highway, like a real entrance ramp. The highway also has holes immediately after the entrance ramp that the car falls through; test it and you will see. It only has entrance ramps on one side, but it should have them on both. There are no cars or lane markings on the highway, and no concrete median divider. Another bug is the palm trees. One more bug: after the police arrested me, my hands remained raised when walking after the restart; I think they were raised during the arrest too, so check that. Adjust cycling as well. It does not look as though Shift increases the speed, or at least the pedaling does not become faster. The bicycle also seems to coast too far when I release the accelerator; it should slow down within about 20–30 metres at normal speed on level ground, rather than travelling 50–100 metres. I cannot slowly reverse the bicycle. When stationary, one foot does not touch the ground. When pedaling, the legs move exactly opposite to the pedals. I also saw the bizarre sight of a plane travelling along the ground through the city and its buildings. Finally, look at how the sun appears. Follow-up prompt 102026-09-06 00:07 · 11 screenshots Read TODO.md again to be safe. The character should pedal at least twice as fast while sprinting on the bicycle as during normal movement, and the bicycle should travel two to three times farther after the player stops pedaling by releasing the forward key. Cars are still parked at the wrong angle; see the first screenshot. Parked cars have light textures that look illuminated even though the lights are not actually on; the texture should reflect whether the car is running and has a driver. Car steering sensitivity is now too low. Increase it a little for now. The motorcycle is about right, though you can increase it very slightly—less than for cars. Buildings now flicker strangely while I drive through the city at night. Perform a visual check and fix it. The tops of palm trees are not connected to their trunks; see the second screenshot. Some people lie or sit strangely, as shown by the character on the right in the third screenshot and the fourth screenshot. I also think the traffic lights always show the same colours even though traffic seems to be controlled. When I hit a character, they often run sideways while fleeing instead of facing the direction in which they move; check and fix that. In the fifth screenshot there is a building with squares floating in front of it, as if windows were drawn in empty space. Some streets also have overlapping road textures; see the sixth screenshot. The highway entrance ramps are still not correct, as shown in screenshots seven and eight. They must be wider and merge smoothly by creating their own lane parallel to the highway for about 30–50 metres before gradually joining it, like real highway ramps. The highway does not have a concrete median or guardrails along its full length, as shown in the eighth screenshot. Where entrance ramps join, there should be no barrier on the outside edge because the ramp needs an opening. I noticed you had started working on ramps but did not finish them; see the ninth screenshot. Apply the same approach where one highway connects to another. The interchange between two highways does not have ramps connecting every direction; inspect it carefully. Also inspect the highway ends in the tenth screenshot, because they do not connect nicely to the road. More generally, review 90-degree bends on both four-lane and two-lane roads because they appear to be handled incorrectly. Follow-up prompt 112026-09-06 01:09 · 6 screenshots I am testing it and there are several things to fix. Cars are still cumbersome to turn. I told you to increase steering sensitivity, but you did not. Do the same for the bicycle. When I sprint on the bicycle it should accelerate; it still does not. Reduce the sprint pedaling speed to about 1.8 times normal pedaling. When leaning a bicycle or motorcycle, the character does not lean with it; check and fix that. The bridge in Sandbar Heights has a gap through which the water is visible; see the first screenshot. The highway connections are still incorrect; see the second and third screenshots. Street lamps only light up when I approach them. Is this a performance optimisation? If so, at least expose it as an optional setting. Do the same for cars, with a single setting controlling the distance at which their lights appear. When I am underneath the highway interchange, I can see the sky through the highway, as in the fourth screenshot. The interchange ramps have improved but still contain errors; see the fifth screenshot. Concrete median barriers must run along the entire highway without exception—there must be no gap at intersections or highway exits. Check every highway bridge and ramp bridge to ensure that roads underneath have enough clearance, so they do not end up blocked by concrete as in the sixth screenshot. Follow-up prompt 122026-09-06 03:31 I hit my usage limit while you were working, but it has reset now. Please continue from where you left off. Follow-up prompt 132026-09-06 11:56 I do not know what you changed, but compared with yesterday it stutters terribly. It uses 100% of the GPU, which I do not mind, but it runs at only about two or three frames per second. Yesterday I was getting roughly 10–20 FPS. Follow-up prompt 142026-09-06 12:34 · 10 screenshots Look at the tunnel entrance: a long concrete beam protrudes from the top, as shown in the first screenshot. What is the person above the bar doing in the second screenshot? I cannot get drunk; pressing E does nothing, as shown in the third screenshot. I also noticed that when other people get drunk and start running—for example because of a fight—they run just as fast as sober people even though they stagger. Should they, and my drunk character, be slower? I am only asking and leave the decision to you. Cars begin smoking after only one or two crashes and catch fire after a few more. Is that too strict? Tall vehicles such as SUVs lean far too unnaturally. Reduce their leaning somewhat, perhaps by half for all cars. When night began, I could still see lights turning on only as I approached: nearby cars and lamps are lit while distant ones are dark in screenshots four and five. Is that deliberate for performance reasons? Could distant objects use cheaper, lower-quality lighting and switch to the current lighting as I approach? At present they are completely off in the distance and visibly switch on. The arrow on the large map points in exactly the opposite direction, while the minimap arrow is correct. The highway entrance now looks better, but the barrier still shows through on its right side in the sixth screenshot. The end of the entrance ramp is abruptly cut off and lacks a concrete barrier in the seventh screenshot. A few metres farther along the highway, the correct ramp ending exists as a separate, clipped segment, shown in the eighth screenshot; inspect the first 50–100 metres after entering the highway. The highway ends at the runway. Should it avoid that area or go underground, as shown in the ninth screenshot? The connection from the main three-lane highway to the two-lane highway is still wrong; build it as shown in the tenth screenshot. Finally, add hidden cheats, including switching between day and night. They must not appear in any visible window and should activate only when I type “cheats”. Follow-up prompt 152026-09-06 13:03 So you did not fix the lights? Follow-up prompt 162026-09-06 13:17 · 1 screenshot The lights still appear to switch on only when I approach them—it literally looks as though they suddenly light up. I also noticed that cars cast a strange single beam down the middle; see the first screenshot. Add cheats for teleporting between several exact locations as well. Follow-up prompt 172026-09-06 13:32 When I open the cheats window, let me close it with Escape, for example, and also close it after entering a cheat. Apply the smooth light fade-in and fade-out to cars too. Cars, especially on the highway, seem to illuminate only the centre with one beam, and their headlight textures are not white or bright. Could you increase the range at which the expensive lights fade in by two or three times, or was that the cause of the severe stuttering? Could you enable them only for the viewport—or slightly beyond it—so that turning a few degrees does not make a distant light disappear even though its illumination still falls on visible textures? This could combine a nearby area, perhaps 20–30 metres, where expensive lights always stay on, with more distant lights rendered only within an expanded viewport. What do you think? A few more details: I sometimes see pedestrians on the highway, which must not happen. People also sunbathe and stay on the beach at night. Handle that without making them disappear if I stand there and wait for sunset: they should get up and leave, while people should arrive after it has been daylight for a while. Beaches are usually empty in the early morning, I assume. Follow-up prompt 182026-09-06 14:07 · 5 screenshots People sometimes cluster like this. I do not know what it means; they seem to keep trying to move but push against each other, as shown in the first two screenshots. Characters sometimes cannot sit on benches; see the third screenshot. You wrote: “I implemented your proposal exactly: within 28 m, real lights are always active; farther away, the budget of 14 lights is used only inside an expanded field of view (camera field of view plus 25° on each side, so a small turn does not switch a light off). Measured result: previously a 55 m radius in every direction; now about 100 m forward and nothing behind the player except the nearest lights.” Did you do this for street lamps as well? Other cars have crossed headlight beams; fix them. I can barely hear my own car—did you remove its sound? In the fourth screenshot, especially the upper-right part of the parking lot, there are rectangular graphical artefacts separating lit and unlit areas. The lighting behaves strangely on the texture and flickers continuously when I move the camera. It is visible at night in parking lots under street lamps. A motorcycle should have one headlight, correct? A bicycle should have a headlight two to three times weaker than a car's. The fifth screenshot shows a pedestrian on a highway exit ramp, and the ramp itself also has the defect visible there. Regarding sound, I constantly hear very fast beeping or what sounds like an active car alarm. What is it? I want to control both vehicles and the character with the arrow keys as well as WASD. Finally, cars are still cumbersome to steer at moderately low speeds. I cannot make a normal turn at an intersection without nearly stopping. You do not need to multiply sensitivity by two or three; about 1.5–1.8 times would be enough. Steering at high speed is fine. Solve low-speed steering sensibly without creating two separate control systems. In the cheats, I would also like to select a mission. Follow-up prompt 192026-09-06 14:42 · 1 screenshot Once I encountered a bug where the character kept moving forward or sideways and I could not stop it. To be safe, reset the controls when Escape is pressed. Cyclists now have only a single-beam light, shown in the first screenshot. It is too weak and looks like a beam painted on the surface rather than spreading sideways. Cars have the same problem, only with two such beams. My car's lights work correctly; can you do the same for other cars, or would that be too computationally expensive? I have also been wondering because I do not see women. Are there women? If so, give them slightly more pronounced hips or breasts. Vans are oddly tilted forward and have a strange rectangle above the roof that is not connected to the bonnet. I am also unsure whether weapons are scattered around the map. If not, add them in somewhat hidden places, as in GTA. I do not know what weapon variants exist—are there a submachine gun, rocket launcher, grenades, and a sword? Add health kits too if they are missing, again in side alleys rather than traffic lanes: put weapons and health kits in spaces between buildings or hidden parking lots set back from the street so they are not immediately visible. Pay attention to how weapons are held, what they look like, how they fire, and how they are activated. When I enter a car, my legs must not clip through its floor. If the car is enclosed, my head must not clip through either; you can hide the character after entering. Improve the entering animation a little, including when pulling another person out. You said: “The ‘car alarm’ was music from the bar, a two-tone melody four times per second. The bar groove is now slower and softer.” I still hear it. Can you give the bar a different sound? It does not resemble bar music at all; it sounds like an alarm. And Shift still does not make the bicycle accelerate! Follow-up prompt 202026-09-06 14:57 The vans have no visible wheels. Improve their design. You managed to model the cars well, so why not the vans? Buses have a cuboid above the roof and I cannot see their wheels either; inspect them. Weapons cannot simply be scattered on the street. They must be hidden and displayed as weapons rotating around the Z axis. Health kits must be hidden too. The character holds weapons incorrectly—the barrel points upward while firing. I told you to take care with both their handling and their design. They may have the Minecraft-like style, but they should still resemble real weapons. Add cheats for every weapon so I can test them. Follow-up prompt 212026-09-06 15:06 · 1 screenshot Some buildings in the city centre intersect the asphalt road; see the screenshot. Follow-up prompt 222026-09-06 15:35 When I throw grenades and Molotov cocktails, their flight should match where they actually land. I do not think this is handled at all right now, and the throwing animation seems to be a default one. A Molotov cocktail should look like a Molotov: a bottle made from two cylinders with a burning rag at the end. A grenade should not be just a sphere; it should resemble a grenade when thrown. When I start shooting without aiming, the character initially looks toward the ground, so the shots are aimed into the ground. This needs to be reconsidered. Women now appear to have breasts, but you did not account for exposed upper-body clothing: their breasts must remain covered while their stomach can be exposed, if you understand what I mean. Give them breasts of different sizes and shapes. Some men can have a belly, and some characters can be more obese, with wider legs, arms, torso, and a large stomach; they should run more slowly. The bar music is great now. You can make it longer and louder. When I get drunk, the character should not start running around by itself. Follow-up prompt 232026-09-06 15:44 One more thing: when I am killed or arrested, I should lose my weapons. Follow-up prompt 242026-09-06 15:47 And one last thing: how many missions are in the game, and what are they? Follow-up prompt 252026-09-06 15:48 What does each mission involve? Follow-up prompt 262026-09-06 15:50 Do the missions have any kind of intro? I am just asking. Follow-up prompt 272026-09-06 15:52 Would you be able to create something like short cinematic scenes before the missions? I am just asking. Follow-up prompt 282026-09-06 15:54 By the way, you still have not fixed aiming when shooting without aim mode. Do you know what I mean? Follow-up prompt 292026-09-06 15:59 The aiming dot probably does not match where I am shooting when I fire without aiming, does it? There is a fence near the beach that people regularly walk through because it has no collision. Add collision and plenty of entrances to it. Follow-up prompt 302026-09-06 16:05 Automatic aiming should not target every character. It should target only enemies, people currently in conflict with me, and the police. Follow-up prompt 312026-09-06 16:12 Do not automatically aim at vehicles either unless they are enemy vehicles or police vehicles. The same should apply to motorcycles and bicycles. Follow-up prompt 322026-09-06 16:18 When I switch to aim mode, the aim points downward. Move it upward by the same amount as when I shoot without aiming, if you understand what I mean. Also let me shoot with a keyboard key such as Ctrl, Command, or Alt. Follow-up prompt 332026-09-06 16:21 Okay, aiming now works and is in the same position, but when I stop aiming, restore the previous camera position—specifically the downward viewing angle I had before. Follow-up prompt 342026-09-06 16:25 People still cannot sit on benches. Regarding shooting, Command is probably unsuitable because it closes the browser. Look at how the baseball bat is held. I assume there is a sword too; it is held and swung incorrectly. Follow-up prompt 352026-09-06 16:29 When I am drunk, make me sway slightly even while standing still. Follow-up prompt 362026-09-06 16:36 · 1 screenshot People still cannot sit on benches; see the screenshot. I cannot see a rising progress bar while the city loads. This game will be on the web and will take some time to load, so I need to see how long loading is taking. Follow-up prompt 372026-09-06 16:41 When I stand near people who are talking to each other, I hear nothing—no voices or sounds resembling speech. Add something like Simlish, or at least some vocal sounds. Follow-up prompt 382026-09-06 16:47 Running people have a strange coloured rectangle underneath them; look at it. The voices of people talking are still too quiet. Follow-up prompt 392026-09-06 17:04 Now let us discuss the levels and their intros. Follow-up prompt 402026-09-06 17:07 Actually, forget it for now. This was only supposed to be a benchmark and I have already prompted it far too much. I do want the option to start any mission, however, and after the player finishes the game, they should be able to replay a mission and continue from it. Follow-up prompt 412026-09-06 17:43 Improve the drivers' AI. Instead of avoiding obstacles, they either crash into them or slow down, which makes them very easy opponents. Follow-up prompt 422026-09-06 18:07 · 1 screenshot When I start the game, I appear here at night. Is this the default state? I used to start during the day in a similar place, but facing the opposite direction. Follow-up prompt 432026-09-06 18:24 Is the save data stored in cookies? Follow-up prompt 442026-09-06 18:25 Multiple users will play this from my website. Will each user have their own settings? Follow-up prompt 452026-09-06 18:28 At startup, add the option to continue from the last position or start from the beginning. Also add Settings, which should also be accessible from the map. It should let me adjust several parameters, including resolution. I think resolution should be expressed as the number of screen pixels per game pixel rather than width × height—what do you think? Include settings for how far away objects, lights, and similar elements are displayed, so someone whose computer is struggling can adjust them. Controls could be configurable as well. Follow-up prompt 462026-09-06 18:39 Add a Load option showing the last saved positions, perhaps the most recent ten. Save the game whenever I complete a mission. Follow-up prompt 472026-09-06 18:56 I am looking at your structure and cannot see that its individual sections are connected to each other. Follow-up prompt 482026-09-06 19:03 · 2 screenshots These and the other containers must not collide with fences or roads; see the first screenshot. The water should not end a few metres from the shore. It should extend far into the distance, as shown in the second screenshot. Follow-up prompt 492026-09-06 19:21 · 1 screenshot The headlight lenses on cars do not turn white when the lights are switched on. Follow-up prompt 502026-09-06 19:31 Create an AGENTS.md file in which you record everything important about this project, so that after a restart you or another agent can continue. I will leave its exact structure and contents to you. Follow-up prompt 512026-09-06 19:44 One final thing: this was originally intended to be a one-prompt benchmark, with at most some control fixes and critical bug fixes. The first-prompt result was fantastic, however, and I got carried away, gradually fixing much more than I had originally planned. Can you compile every prompt I wrote in this conversation and save them in a dedicated Markdown file that I can give to the agent managing the benchmarks, so it can add this list as a special note? Follow-up prompt 522026-09-07 · post-benchmark fullscreen and mobile UI maintenance Can you add fullscreen to Settings with surgical precision and the smallest possible change? I do not want the game to be compromised. This must be a very small, precise, minimal change to the game. The same post-benchmark UI change added Close buttons to the map and missions so they can be closed on mobile, plus a wrapper option for opening the map, without changing game logic. About 1 h 50 min after the first prompt · approximately USD 50 · 623K context.	Final demo One-prompt demo
3Claude Fable 5	Closed	—	Matched the assignment Playable	Many bugs	1 corrective prompt. About 1 h 20 min generation after the first prompt; nearly 400K context.	Demo
4GPT-6 Astra	Closed	—	Partially matched Playable	Many bugs	3 prompts for minor control fixes, especially missing mouse support. Generation after the first prompt took 35 min.Show moreShow less 3 prompts for minor control fixes, especially missing mouse support. Generation after the first prompt took 35 min.	Demo
5Claude Opus 5	Closed	—	Partially matched Playable	Very many bugs	1 corrective prompt for controls. The city has impressive textures, but its buildings overlap.	Demo
6GLM 5.3 Flash	Open-weight	320B ≈196.6 GB	Partially matched Playable	Many serious bugs	Required 2 corrective prompts for controls. About 1 h 4 min generation after the first prompt; the connection dropped several times.Show moreShow less Required 2 corrective prompts for controls. About 1 h 4 min generation after the first prompt; the connection dropped several times.	Demo
7Kimi K3	Open-weight	2.8T 1.56 TB ≈147.9 GB / 100K	Partially matched Playable	Very many critical bugs	1 corrective prompt for controls. About 4 h generation after the first prompt; USD 89.	Demo
8GPT-5.6 Sol	Closed	—	Partially matched Playable	Very many bugs	Required 2 corrective prompts for controls.	Demo
9GPT-5.5	Closed	—	Partially matched Playable	Very many bugs	-	Demo
10Qwen 3.8 Flash Q2	Open-weight	180B ≈110.1 GB	Partially matched Playable	Very many serious bugs	The game rendered nothing after the first prompt. One corrective prompt made it display; controls and broken buildings required at least 10 more prompts. A total of 21 corrective prompts were used. Q2 quantization; generation after the first prompt took about 4 h 13 min.Show moreShow less The game rendered nothing after the first prompt. One corrective prompt made it display; controls and broken buildings required at least 10 more prompts. A total of 21 corrective prompts were used. Q2 quantization; generation after the first prompt took about 4 h 13 min.	Demo Two-prompt demo
11Gemini 3.8 Flash	Closed	—	Partially matched Playable	Many bugs	2 corrective prompts for controls. About 13 min generation after the first prompt; USD 1.79.	Demo
12Qwen 3.8 27B	Open-weight	27B 19 GB ≈6.71 GB / 100K	Partially matched Playable	Very many serious bugs	6 corrective prompts covering spawn and camera, cars, bridges, vehicle entry, chase camera, mouse attack, and E-key debounce. Generation after the first prompt took 3 h 52 min.Show moreShow less 6 corrective prompts covering spawn and camera, cars, bridges, vehicle entry, chase camera, mouse attack, and E-key debounce. Generation after the first prompt took 3 h 52 min.	Demo
13DeepSeek V4 Flash 0731 DS4 Q2	Open-weight, Q2	284B 164.6 GB ≈1.22 GB / 100K	Partially matched Playable	Very many bugs	Required many corrective prompts for controls, especially to map the A and D keys correctly.	Demo
14Gemini 3.5 Flash	Closed	—	Partially matched Unplayable	Very many bugs	-	Demo
15GLM 5.2	Open-weight (OpenRouter)	744B 464.8 GB ≈9.52 GB / 100K	Partially matched Unplayable	Very many bugs	-	Demo
16Qwen 3.6 27B ThinkingCap	Open-weight	27B 16.8 GB ≈6.71 GB / 100K	Did not match the assignment Did not work	Contains fatal bugs	No visible world. About 1 h 17 min generation after the first prompt.	Demo

**Shown when available: parameter count, Q4 weight size at 0K, and cache/state at 100K. We use a concrete Q4 checkpoint or a labeled Q4_K_M estimate; 100K assumes one sequence in the documented runtime. Weight quantization does not change the cache while the runtime keeps the same cache layout and data types. Hybrid models can differ substantially in another runtime. Unverifiable values are omitted, and a lone dash means no value is known. Runtime buffers, vision components, and offload are excluded; this is not a universal minimum RAM/VRAM requirement.

On mobile or tablet, Demo opens a website-owned wrapper with touch controls. The original benchmark code remains unchanged.

Download benchmark results as JSON

Method

Assignment used for the comparison

Every run starts with the same original prompt. Follow-up implementation prompts may only fix bugs and runtime failures; tune controls, playability, performance, and reliability; or polish graphics, animation, and audio within the original assignment. They may not change the core brief or add a new primary gameplay system. Every follow-up intervention is disclosed and the run is labeled multi-prompt; questions, recovery after interrupted work, and documentation messages are tracked separately from implementation prompts.

Prompt used for the assignment

Create a 3D open-world action game inspired by classic GTA-style gameplay. The game should have a stylized visual style close to GTA: San Andreas, but with improved graphics, better lighting, more detailed environments, and smoother animations. Do not use copyrighted GTA assets, names, characters, maps, or exact copies; create an original game with a similar open-world crime-action feeling.

The game should use a third-person camera. The player must be able to walk, run, sprint, fight in melee combat, shoot different types of weapons, and interact with vehicles. Weapons should be placed in various harder-to-reach locations across the map, or the player should be able to obtain them from armed gangsters or police officers after defeating them.

The player must be able to enter cars, steal cars, pull drivers out of vehicles, and drive around the city. The city should be inspired by Miami, with a tropical coastal atmosphere, boats, the sea, canals, bridges, beaches, hotels, highways, and dense urban districts.

The map should be fairly large and should not be a simple square grid. The road layout should include streets of different shapes, curves, alleys, intersections, multi-level intersections, bridges, tunnels, and a highway network with exits leading into city streets. The city should contain different districts with different architecture, atmosphere, culture, population density, and types of people, similar to real American cities.

Include districts such as:

* a Chinatown-style district,
* a Hispanic district,
* a district with a larger African-American community,
* a wealthy residential district,
* a downtown district with high-rise buildings,
* office districts with business buildings,
* coastal hotel areas with balconies, pools, bars, tourists, and people relaxing,
* dangerous gang-controlled districts,
* and one district occupied by homeless people living in tents.

The beach areas should be full of life. There should be people swimming, sunbathing, playing beach volleyball, walking, talking, skating, rollerblading, cycling, and relaxing near the sea. Streets in different districts should have different population densities, and pedestrians should visually and behaviorally match the district they are in.

Some dangerous districts should be controlled by gangs. Gangs should be visually recognizable by their clothing, colors, behavior, and preferred weapons. Gang members should patrol their areas, react aggressively to threats, and sometimes attack the player if provoked.

Create a custom map-generation format and tool for building the city. Think of it like a simplified SimCity-style internal format that describes roads, buildings, parks, parking lots, beaches, canals, bridges, highways, districts, spawn zones, and other city components. Use this format to generate repeated but varied patterns of buildings, roads, parks, parking lots, sidewalks, props, and environmental details. The city should feel handcrafted but should be generated from reusable rules and templates.

On the edge of the city, include an airport where planes can land and take off. The airport should have runways, hangars, parked vehicles, security areas, and moving aircraft.

The city should include civilians, gang members, police officers, police cars, police motorcycles, and two police helicopters. Police officers should appear as pedestrians, motorcycle officers, and drivers of police vehicles. The police should only start chasing the player when they witness the player committing a crime, such as shooting in the street, attacking civilians, stealing cars, or causing serious destruction. Police should not be everywhere; they should be distributed naturally so that the player is not immediately chased after the first small incident unless police are nearby and actually notice it.

Civilians should have different personalities and reactions. Most people should panic and run away when the player starts shooting, attacks someone, or tries to steal a car. Some drivers should panic and quickly drive away if the player tries to steal their vehicle but is too slow. A smaller number of civilians should be brave or aggressive and may fight back if the player attacks them, damages their car, or tries to steal their vehicle. However, if the player starts shooting, even these braver civilians should try to escape.

In dangerous districts, include dealers offering strange substances that can temporarily cause altered visual effects, distorted controls, hallucination-like effects, or other weird states for the player. These districts may also include prostitutes, addicts, and other street characters to make the atmosphere feel darker and more dangerous. Also include bars where the player can get drunk and where regular customers hang out, talk, argue, or react to the player.

The game should include classic GTA-style missions, but they should not feel repetitive. Each mission should be slightly different. Add around 20–30 missions in total. Mission types can include:

* delivering cargo from point A to point B,
* assassinating a gang leader,
* stealing a car full of money,
* winning illegal street races,
* escaping police during a race,
* finding and eliminating a rat cooperating with the police,
* protecting a character,
* attacking a gang hideout,
* stealing weapons,
* escaping by boat through canals,
* chasing a target through traffic,
* and other open-world crime-action mission types.

The game should include a mission system with objectives, markers, failure conditions, success conditions, rewards, and mission progression. Missions should use different vehicles, locations, enemies, and situations so that they do not feel copied from each other.

Add sounds for everything important:

* car engines,
* horns,
* brakes,
* crashes,
* tires,
* boats,
* helicopters,
* airplanes,
* footsteps,
* punches,
* gunfire,
* explosions,
* people talking,
* frightened civilians screaming,
* angry civilians shouting,
* police sirens,
* gang threats,
* bar ambience,
* beach ambience,
* city traffic,
* ocean waves,
* and general city noise.

Because this is a large and demanding task, plan it carefully before implementation. First create a dedicated TODO.md file. In TODO.md, break the whole project into clear milestones and smaller tasks. Then complete the work step by step, following the TODO.md file point by point. Keep updating TODO.md as tasks are completed. Do not skip details. Build the game gradually, starting with the core playable systems, then the city, traffic, pedestrians, police, gangs, missions, sound, UI, and final polish.

Show full prompt Hide prompt

FAQ

GTA benchmark FAQ

What does the GTA benchmark test?

It tests how AI models create a playable 3D open-world city with traffic, pedestrians, police, missions, and audio.

Can the results be played?

Yes. Playable outputs include a direct demo link in their result row.

Are follow-up prompts allowed?

Yes, but only for fixes and tuning within the original assignment. Every intervention is documented and the run is labeled multi-prompt.

How is the ranking decided?

It is a practical, subjective ranking based on functionality, playability, fun, scope, polish, and the bugs in each output.

Which result is currently first?

Claude Opus 5.5's extensive open-world game currently holds first place.