Gemini 4 Carbon vs Claude Opus 5.5: Benchmarks and Speed Test

By Beau Johnson··8 min read

Quick answer

Gemini 4 Carbon tied Claude Opus 5.5 on both of my hidden-test builds (12/12 and 28/32 each), but in my runs through OpenRouter it took up to 7.6 times longer and twice saved no file. Carbon costs $2 in / $10 out until November 30, then the same as Opus.

Gemini 4 Carbon opened to the public on November 10, and in this video I run it against Claude Opus 5.5 on my four-test bench the same day. I also cover what Carbon is, the independent numbers, the cyber story, and what people built in the first hours. I use Opus every day, so this is for anyone who builds with it and wants to know if it's time to switch.

What I cover in this video

  • 0:45 What Gemini 4 Carbon is, who can use it, and the price
  • 2:17 The independent numbers: intelligence, cost, speed, and hallucination rate
  • 4:34 The cyber story and why defenders got it first
  • 6:53 What people built on day one, plus my black hole and basketball court remakes
  • 9:08 How I tested, and test 1: the farm game
  • 10:40 Test 2: a real client job scored with 12 hidden tests
  • 13:45 Test 4: the bug hunt with 16 planted bugs
  • 15:16 The verdict and who should use Carbon

What Gemini 4 Carbon is and what it costs

Carbon is Google DeepMind's new top model, and Google's docs say it's based on Gemini 4 Argon. Google pitches it as the model you bring in for your hardest problems. It has a 1 million token context window.

You can't try it for free. Developers get it on the paid tier of the Gemini API in AI Studio and Android Studio. Google AI Ultra subscribers get it in the Gemini app, AI Mode in Search, and Antigravity. It's also on OpenRouter, which is how I ran it.

The price is $2 in and $10 out per million tokens through November 30, 2026. On December 1 it goes to $4 in and $20 out, the same as Opus 5.5. Thinking tokens are billed as output, and Carbon thinks a lot.

Why it took two weeks to open

Google released Carbon to cyber security defenders first through its Fairwind program, then strengthened its misuse safeguards before the public launch. The public model is not the one the defenders got. Fairwind members get a version called Carbon for Cyber, and Google's big cyber scores are for that version. In my bug hunt, the public model found, fixed, and explained a security hole that let one customer read another customer's invoices, with no refusal.

What the independent numbers say

These are scoreboards where somebody other than Google runs the test.

  • Artificial Analysis Intelligence Index: Carbon 60, Opus 5.5 on max effort 59. Claude Fable 5.2 dropped the day before and sits at 65, so Carbon is number two.
  • Cost per index task: $7.76 for Carbon vs $10.04 for Opus on max, at Carbon's full list price. That's about 23% cheaper.
  • Speed: Time per task is a wash at 25.6 minutes vs 25.1. Opus is faster per token, about 115 a second vs 80, but Carbon starts answering sooner: 67 seconds to its first answer token vs 370 for Opus on max.
  • Hallucination rate: Carbon 11.4%, Opus 50.3%. Opus answers more questions overall. Carbon just guesses less.
  • Coding Agent Index: Opus in Claude Code 62, Carbon in the Antigravity CLI 59. Carbon's row is from a pre-release run.
  • Vals and Arena: Vals has them tied, 70.45 to 70.14. On Arena's WebDev board, Opus leads 1818 to 1768.

How I tested and what happened

Both models got the same prompt in a fresh empty folder, with high effort, web search off, no follow-ups, and a 45 minute limit (60 for the bug hunt). Opus ran in Claude Code on my subscription. Carbon ran in Claude Code pointed at OpenRouter, billed per token, because my Google account doesn't have Carbon in Antigravity. It's one run each, so treat this as a vibe check and not a lab verdict.

Test 1: the farm game

Opus built a playable farming game in 22:17. Carbon never saved a file. It tried to write the whole game as one giant file, and Claude Code going through OpenRouter stalled. I retried once in Codex through OpenRouter and Google's servers sent back an error after about 11 minutes. I'm counting that against my setup, not the model.

Test 2: the client job

The models had to build a booking app for a dog groomer from a messy email thread and write the reply email. Both scored 12 out of 12 on hidden tests. On the email, two outside judge models gave Opus a 10 and a 10, and Carbon a 9 and a 10. Opus finished in 4:22. Carbon took 25:37 and wrote about 83,000 output tokens to Opus's 29,000.

Test 3: Google's songbird demo, redone

Both were still polishing at 45 minutes, and both had saved a working page. Carbon's looks more like Google's original. Opus's reads more like a science book. This was the closest result in the bench.

Test 4: the bug hunt

I planted 16 bugs and 11 decoys in an invoicing backend of about 3,000 lines across 42 files. Both models fixed the same 14 of 16, with zero false reports and zero regressions, for 28 out of 32 each. Opus took 6:04. Carbon took 45:54.

The demo remakes

On the black hole, Opus finished in 7:15 and Carbon was still polishing at the limit, but its page works and has its own brighter, more dramatic look. On the basketball court, Opus took 37:41 and Carbon hit the same stall and saved no file.

Who should use Carbon

If you're on a budget and you can wait, the intro price is a great deal for a model that ties Opus on real work. If you care about a model saying it doesn't know instead of making something up, the hallucination number is a real reason to use it. I'd try Carbon for asking questions about the data from my resale business while it's half price.

For building, Opus 5.5 in Claude Code is still my daily driver. I got the same score in a fraction of the wait, and on December 1 the price is the same anyway. Part of the slowness could be my setup going through OpenRouter, so I still want to see Carbon in Antigravity before I make that call for good.

Key takeaways

  • On the two tests with a hidden answer key, Carbon and Opus 5.5 tied: 12 out of 12 and 28 out of 32 each.
  • In my runs, Carbon took up to 7.6 times longer, and twice it saved no file through OpenRouter.
  • Carbon is $2 in and $10 out per million tokens through November 30, then $4 and $20, the same as Opus 5.5.
  • Carbon's hallucination rate on Artificial Analysis is 11.4% vs 50.3% for Opus 5.5 on max.
  • Opus still leads on the coding boards: 62 to 59 on the Coding Agent Index and 1818 to 1768 on Arena WebDev.

FAQ

Is Gemini 4 Carbon better than Claude Opus 5.5?

In my tests they tied on both builds with a hidden answer key, 12 out of 12 and 28 out of 32 each. Carbon is one point ahead on the Artificial Analysis Intelligence Index (60 vs 59), while Opus leads the Coding Agent Index 62 to 59 and finished much faster in my runs.

How much does Gemini 4 Carbon cost?

It costs $2 in and $10 out per million tokens through November 30, 2026. On December 1 it goes to $4 in and $20 out, the same price as Opus 5.5, and thinking tokens are billed as output.

Can I use Gemini 4 Carbon for free?

No. It's available on the paid tier of the Gemini API in AI Studio and Android Studio, to Google AI Ultra subscribers in the Gemini app, AI Mode in Search, and Antigravity, and on OpenRouter.

Why was Gemini 4 Carbon so slow in the test?

Carbon thinks a lot. On the client job it wrote about 83,000 output tokens compared to about 29,000 for Opus. Part of it could also be my setup going through OpenRouter, since Artificial Analysis shows the two at about the same time per task.

Chapters

  1. 0:00Carbon vs Opus 5.5: dead even, but way slower
  2. 0:34What Gemini 4 Carbon is (and who can use it)
  3. 1:57Join Shipping Skool
  4. 2:16The independent numbers
  5. 4:46The cyber story: why defenders went first
  6. 6:25What people built on day one (and our remakes)
  7. 8:41How we tested
  8. 9:34Test 1: the farm game
  9. 10:54Test 2: a real client job (hidden tests)
  10. 12:28Test 3: Google's songbird demo, redone
  11. 13:26Test 4: the bug hunt (16 hidden bugs)
  12. 15:01The verdict

Transcript

Read the full transcript

0:00Gemini 4 Carbon opened to the public this morning. It's one point ahead of Claude Opus 5.5 on the biggest independent leaderboard and until November 30th, it costs half as much. I ran both through our four test bench and the scores came out dead even. But Carbon took up to 7 and 1/2 times longer and twice it never saved a file. In this video, I'm covering what Carbon is, the numbers, the cyber story, today's demos, and all four tests. I use Opus every day, so I had to know if it's time to switch. Let's jump right into it. So, what is Gemini 4 Carbon exactly? It's Google DeepMind's new top model, and Google's own docs say it's based on Gemini 4 Argon. Google's pitch is

0:45simple. If Argon is great for complex tasks, Carbon is the model you bring in for your hardest problems. We covered it back on October 28th when you couldn't use it yet, and I promised you we'd bench it the day it opened. Well, today's the day. And the timing is kind of funny. Argon opened 13 days after it was announced. Carbon also 13 days. Same playbook to the day. So, who can use it? Developers on the paid tier of the Gemini API in AI Studio and Android Studio, not the free tier. So, if you were hoping to try it for free, you can't. You have to pay. Google AI Ultra subscribers get it in the Gemini app, AI mode in search, and anti-gravity. and it's on open router, which is how we ran it. The price is the part you want to

1:30write down. $2 in, $10 out per million tokens, but only through November 30th. On December 1st, it goes to $4 in, $20 out, the exact same price as Opus 5.5. So, you get about 20 days of half price and then it's a straight fight. It has a 1 million token context window and thinking tokens are build as output and carbon thinks a lot. Keep that in your back pocket because it comes back in our tests. Real quick, 263 builders are already inside shipping school. In premium, I show you how I make every video on this channel with Claude Code. New course material drops every single day, plus live calls every week, over 500 hours of replays, and a full course library. Join free link below. So, let's

2:17go to the referees. These are the scoreboards where somebody other than Google runs the test. artificial analysis intelligence index. Carbon 60, opus 5.5 on max effort 59. So carbon's a nose ahead, which is basically a tie. And that's a 10point jump over argon, which is insane for one version, but the leaderboard moved under it. Yesterday, Anthropic dropped Claude Fable 5.2 and it's sitting at 65. So Carbon held the top spot for about 12 days. Now it's number two. Rough timing, Google. Now cost to run one task on that index. Carbon costs $776. Opus on max costs $104. So about 23% cheaper per task. And

3:02that's at Carbon's full price, not the intro price. And the time per task is a wash. 25.6 minutes versus 25.1. Speed is where it gets weird. Per token, Opus is actually faster, about 115 a second versus 80 for Carbon. But Carbon starts answering way sooner. 67 seconds to its first answer token. Opus on max 370 seconds. 6 minutes. Remember, opus is called slopus. It has been slow since the dawn of time. Now, this is the part I didn't expect. The hallucination rate. That's how often a model makes something up when it doesn't know instead of just saying, "I don't know." Carbon 11.4%. Opus 50.3. That's more than four times

3:47better. But to be fair, Opus answers more questions, right? Overall, 65 1.5% versus 58. Carbon just guesses less. And then the one I actually care about, the coding agent index, where the model runs inside a real coding agent. Opus in Claude Code 62. Carbon in Google's anti-gravity CLI 59. Carbon was cheaper and faster per task there, but Opus still wins on the work builders actually do. And to be fair, Carbon's row on that one is from a pre-release run. Val's AI has them in a tie, 70.45 to 70.14, inside the margin of error. And on Arena's webdev board, where real people vote on websites, Opus leads 1818 to 1768. So on paper, it's close

4:34everywhere. Carbon's cheaper than Opus on Max, and it guesses less. Opus is better at coding, which is exactly why we're running our own tests. But first, the cyber story, because that's why we had to wait. So, why did Google hold this thing back for two weeks? It comes down to cyber. A vulnerability is a mistake in code that lets somebody do something they shouldn't, like read another customer's data. And in security, a model does three jobs. Find the hole, prove you can get through it, and write the patch. The same skill that finds the hole can be used to break in. That's the whole problem. So, Google went defenders first. In their words, "Two weeks ago, we released carbon to a set of cyber security defenders via our Fairwind program, and today they said

5:20they've strengthened our misuse safeguards to help ensure stronger defensive reasoning does not translate into offensive risk." The model you get is not the one the defenders got. Fairwind members get something called carbon for cyber with its own setup, and Google's big cyber scores are for that version. So, don't expect the public model to hit those numbers. Google also says its own red team agent got through the public model safeguards88% of the time per attempt. Argon was 3.1. That's Google's own testing. So take that for what you will. Now flip it around. Our bug hunt, which you'll see later in the video, has real security bugs in it. One of them lets one customer read another customer's invoices. Carbon found it, fixed it, and explained it. No refusal, no lecture.

6:07So, for normal defensive work, the safeguards didn't get in the way. And to be clear, Google never said it was too dangerous to release. Their Fair Wind page just says it straight. In the wrong hands, the same capabilities can become an equally powerful threat. So, security teams went first. Makes sense to me. So, what are people actually building with it? Now, it's only been out since 8 this morning Pacific. So, this is the first few hours, not the first full day. And a lot of the best clips come from people who had early access. The biggest one by far is Google's own. A song bird, an interactive model of how a bird actually sings with the vocal organ cut open, six real species, a synthesizer mode, and a live spectrogram. Over 120,000 views.

6:53Google DeepMind says it read research papers on how song birds sing and tuned the audio against real bird recordings. Hold on to that one because it's our third test. Leon Lynn built a 3D basketball court with early access. No pre-made 3D assets. Chainlink fence, autumn trees, brick buildings, parked cars. He says Carbon improved a lot in anything that involves taste like 3D websites and motion graphics. Pankage built a cinematic black hole and he compared it to Opus 5.5 himself. He said it's very close to Opus in visual quality, but Carbon has its own design taste. and StatsWire built a V12 racing game in one HTML file with the engine sound made in code sea tunnel and all.

7:38Very cool. So, I wanted to see that for myself. We gave both models the same prompt for the black hole and the basketball court. None of these people shared their prompts, so the prompts are ours. Black hole first. You can see Opus finished in about 7 minutes with lensing over the top of the disc. Carbon was still polishing when it hit the 45minute limit, but the page it saved works. And honestly, Pontage is right. It has its own look. Brighter, hotter, more dramatic. I'd call that one a matchup of styles. The basketball court is where it got rough. Opus took almost 38 minutes and built the whole block. Buildings, cars, the hoop closeup. Carbon never saved a file. And that one's on our setup. And I'll explain why in the game test. But not everybody's hyped. Early

8:23testers are already talking about wait times. One early access tester said before launch that testing got botched by inference times of up to five minutes for a step. And Google's own model card mentions occasional slowness or timeout issues. Remember that because slowness turns out to be the story of our bench. So here's how our bench works. Same prompt for both models. A fresh empty folder, high effort, web search off, no follow-ups, and a 45minute limit. 60 for the bug hunt. And every model runs as a coding agent, so it can write files, run them, and fix them. Opus ran in claude code on my subscription. Carbon ran in cloud code 2 pointed at open router build per token. Now to be fair to Google, their own agent is anti-gravity,

9:08but carbon in anti-gravity needs the ultra plan, and the Google account we use only had Gemini Flash. So Carbon ran through Open Router, and I'll tell you every time that mattered. four tests, a game, a real client job with hidden tests, Google's Songbird redone, and an extreme bug hunt with hidden tests and two judge models that aren't in the matchup, one run each. This is a vibe check, not a verdict from a lab. Test one, a cozy farming game. Think Stardew Valley. Walk around, ho the dirt, plant seeds, water them, watch the day clock tick. Opus took 22 minutes. And look at this. A farmhouse with a red roof, autumn trees, a fenced field, a scarecrow, crops actually growing, a

9:54coin counter. We pressed the exact same keys you see here, and everything responded. That is a legit little farm game. My kids would play that. Carbon never saved a file. Here's what actually happened. Carbon tried to write the whole game in one giant file in one go. and claude code going through open router stalled while that huge file was coming in. It retried and it ran out the 45 minutes. We've seen this exact stall with Argon before. So I retried it once in a different agent codeex also through open router. This time Google's own servers sent back an error after about 11 minutes. Still no file. So that's zero for carbon. But I'm not calling that the model. That's our setup. Something about big single file builds

10:40through open router breaks with Google's models. If you've got Carbon running cleanly in anti-gravity, tell me how it did on big builds. Comment down below. And honestly, that's the easy part of the bench. The next test is the one that would actually make me money. Test two is a real freelance job. A messy email thread from a dog groomer who wants online booking. And at the very end of the thread, her husband Tony jumps in and scraps it. The deposit goes from 20% of the price to a flat $25 and nothing on mail trims plus one vague line. Quote, "Don't let people book way out, like more than a couple months." The model has to build the booking app and write the email back to the client. And then we score it with 12 hidden tests the models never saw. Prices, hours, the

11:26change deposit, cancellations, all of it. Opus 12 out of 12. Carbon 12 out of 12. Both caught the deposit change. Both handled the couple months thing. You can see both apps doing the same walkthrough here, and they both just work. The reply emails were scored by two thirdparty models, Quen and GM, against a hidden rubric. Opus got 10 from both. Carbon got a 10 and a nine. The nine was for being more wordy than a client email should be, which fair, but look at the clock. Opus finished in 4 minutes and 22 seconds. Carbon took 25 and a half minutes. Same score, almost six times longer. And there's that thinking again. Carbon wrote about 83,000 output tokens on this job. Opus about 29,000. Now, to

12:13be fair, part of that could be our setup going through Open Router. On Artificial Analysis's own runs, Carbon's time per task is about the same as Opus. So, take our clock with a grain of salt. But if I'm sitting there waiting on a client job, I'm picking the 4-minute one. Test three. Google's own viral demo. The songb bird. Google never shared its prompt, so we wrote ours from the video. Six species, the vocal organ cut open with labels, a synth mode with a keyboard, the beak and throat you can drag, and a live spectrogram. And this one was too big for both of them. Both models were still polishing when the 45 minutes ran out. But both had already saved a working page. So what you see is what they had at the bell. On the left, carbon. The bird's body is cut open

12:59right where the organ sits. A lot like Google's original. The species cycle through, the colors change, and the spectrogram moves along the bottom. On the right, Opus. It went with a magnifying circle labeled the searings enlarged with the air sacks drawn on the body and a cleaner, busier spectrogram. Honestly, this is the closest result in the whole bench. Carbons looks more like Google's demo. Opus' reads more like a science book. I mean, which one looks better to you? Tell me down below. And now the hard one. Test four, the bug hunt. This is a real codebase, an invoicing backend for a small business. About 3,000 lines across 42 files. We planted 16 bugs in it. Most of them start in one file and only show up in another. A double charge when two

13:45payments land at the same time. Taxes rounded wrong. A date that shifts depending on where the server is. And that security hole where one customer can read another's invoices. plus 11 decoys that look like bugs but aren't. And the models weren't told how many bugs there are. Just find and fix every real one and explain it. False reports cost points and breaking something that worked costs more. Hidden tests decide what's actually fixed. Then two judges that aren't in the matchup. GPT 6.1 Soul and Deepseek decide what was explained. Opus fixed 14 out of 16. Carbon fixed 14 out of 16. the exact same 14 and both miss the same two. One is a cash that

14:31gets filled at the same moment it's being cleared. The other is a refund that should turn payment reminders back on and doesn't. Those two only break over time and they trip up almost every model we've run through this test. No false reports from either one. Zero regressions. Both judges agreed. 28 out of 32 each. A dead tie. And the clock again. Opus 6 minutes. Carbon almost 46 minutes. Same answer, more than seven times longer. So, who wins on the two tests with a hidden answer key? It's a tie. 12 out of 12 and 28 out of 32. On the songbird, it's a matter of taste. On the game and the basketball court, Carbon never delivered a file in our setup, and I'm counting that against our

15:16setup, not the model. But the pattern was the same every single time. When both finished, Carbon matched Opus. It just took a lot longer to get there in our runs. So, here's who should actually use it. If you're on a budget and you can wait, the next 20 days at $2 in and 10 out is a great deal for a model that ties opus on real work. If you care about a model saying I don't know, instead of making stuff up, that hallucination number is a real reason to use it. And if you're on a security team, Fairwind is where the real cyber version lives. So, here's how I'd use it in my resale business. We sell live on whatnot and Tik Tok shop and we're analyzing a ton of data from our streams products and SKUs. When I ask questions about that data, I'd rather have the model that guesses less. So, I'd

16:02actually try carbon there while it's half price. But for building Opus 5.5 in Claude Code is still my daily driver. Same score, a fraction of the weight, and on December 1st, the price is the same anyway. And I'll be honest, I still want to see carbon in anti-gravity before I make that call for good. So, where do you land? Would you wait up to seven and a half times longer to pay half the price until November 30th for the same result? Comment down below. I want to know. And if the bench helped you decide, a like goes a long way. Have a blessed day.

Ready to start building with AI?

Join the Shipping Skool community free. Paid members get live bootcamp calls every week and weekly-updated courses.

Join Shipping Skool

Keep reading