Kimi K3.1 Pro vs Opus 5.5 and Sonnet 5.5: Benchmarks, Cost, Tests
Prefer to watch? Watch on YouTube · Subscribe for new builds · Read the transcript
In this video I put Moonshot AI's new Kimi K3.1 Pro up against Opus 5.5 and Sonnet 5.5. Each model got the same prompt, one shot, no help. I also go through the independent benchmark numbers, the odd choice Moonshot made to cut hacking data out before release, what people built in the first 24 hours, and who should actually think about switching. If you build with AI and want to know whether this model can save you money, this one's for you.
What I cover in this video
- 0:47: What Kimi K3.1 Pro is, what it costs, and when the weights are due
- 3:03: What Artificial Analysis found when it tested the model on launch day
- 5:20: How Moonshot cut the hacking data out, and whether that worked
- 6:51: What people built in the first 24 hours, and why the mood turned the next day
- 9:08: How my tests are set up, plus test 1: a Minecraft game
- 10:39: Test 2: the viral low-poly horse
- 12:55: Test 3: the bug hunt results
- 13:42: My verdict on who should switch
What Kimi K3.1 Pro is and what it costs
Kimi K3.1 Pro comes from Moonshot AI, the Chinese lab behind Kimi, and it came out on October 23. Moonshot says it has 2.8 trillion parameters in total, with 104 billion active at any one time. It has a 1 million token context window, and thinking is always on.
It costs $2 in and $10 out per million tokens. That's the same as Sonnet 5.5 and half of Opus 5.5, which is $4 in and $20 out. It's also a third cheaper than Kimi K3 was in July. Moonshot says the full weights will be out by November 4, so you can't download the model yet. For now you use it through the Kimi app, Kimi Code, the Kimi API, or OpenRouter. With K3, Moonshot promised the weights by July 27 and shipped them on July 27.
The chart everyone shared was Terminal-Bench 4.0, which tests real coding agent work inside a terminal. On it, Opus 5.5 scored 66.4 and Kimi scored 65.7. Moonshot also says Kimi beats Opus on long coding jobs, on Humanity's Last Exam with tools, and on office work. It loses on computer use and CAD, and it falls well behind on Terminal-Bench Science (38 vs 63 for Opus). Keep in mind that Moonshot reported all of these numbers itself.
The independent check: Artificial Analysis
Artificial Analysis ran its own tests on launch day. Kimi K3.1 Pro scored 55 on the Intelligence Index, which puts it 4th out of 215 models. Ahead of it are Opus 5.5 at 58, Sonnet 5.5 at 56, and Meta's Muse at 56. That's close but not a win. It's also a big jump from Kimi K3, which scored 44 in July. And Opus on high effort (not max) scores 54, so Kimi edges it out at that setting.
Cost is the part that matters for your wallet. Running the whole index took Kimi 150 million tokens, Opus 260 million, and Sonnet 420 million. That works out to about $2.11 per task for Kimi, $5.46 for Sonnet, and $5.98 for Opus. So Kimi has the same price per token as Sonnet but costs about 2.5 times less to get the work done.
There's one catch. Kimi is the slowest of the three per token, at about 73 tokens a second compared with 96 for Opus and 133 for Sonnet. It finishes sooner because it writes much less and starts answering sooner: about 30 seconds before its first word, compared with 562 seconds for Opus.
The hacking cut
Moonshot says it took cyber and exploit data out of both mid-training and reinforcement learning, but kept all the normal coding data. Its rule is that it won't release open weights that push the frontier of hacking ability. It wants its models to stay behind closed models that have been out for at least three months.
On Moonshot's internal hacking test, K3 scored 33%. The uncut in-house version of K3.1 Pro scored 90.5%, and the version they actually released scored 81%. On ExploitBench, Moonshot's table shows Opus at 91, GPT-6 Astra at 100, and Kimi at 74.7. Again, Moonshot is grading its own homework here, and nobody independent has tested K3.1 Pro's hacking yet. I think cutting the data was the responsible move. But once the weights are out, people can train whatever they want back in, so I'd call it a speed bump, not a wall.
My head-to-head tests
I ran every model through OpenRouter on high effort, with the same 120K token limit. Each model got one run per test, so treat this as a vibe check, not a verdict.
Test 1: Minecraft
The prompt asked for a simple, beautiful Minecraft game in one HTML file. It needed grass, dirt, stone, sand, water, trees, breaking and placing blocks, a hotbar, and WASD to walk around. Kimi took 4:28 and cost $0.15. Sonnet took 7:57 and cost $0.67, and it spawned me inside a tree. Opus took 13:19 and cost $1.81, but it added flowers, tall grass, and clouds. Kimi wrote about 15K tokens, Sonnet 67K, and Opus 90K.
Test 2: The horse
I wrote my own version of the viral prompt: a low-poly horse galloping across an African savannah at golden hour, with a cinematic camera following it. Kimi finished in 4:39 for $0.15. Opus thought for over 19 minutes and hit the limit partway through writing the file. Sonnet spent all 120K tokens thinking and never wrote a word of an answer. The model's thinking and its answer come out of the same token budget, so a model can think itself out of room to answer. I tried again at 128K, which is the most OpenRouter allows for Claude, and both got cut off again. That was $7.44 of Claude and zero horses.
Test 3: The bug hunt
I gave each model a small to-do app with five planted bugs. Tasks didn't save on reload, delete removed the wrong task, the done filter showed the wrong tasks, typing code into a task could run it, and the counter never went down. Each bug found and fixed was worth 2 points, and each made-up bug cost 1 point. Claude scored the answers against a hidden answer key, and every answer is shown on screen so you can check them. All three models scored 10/10 for under three cents each. Sonnet was the fastest and Kimi the slowest. This test was too easy, so I'll need a harder one.
Key takeaways
- If you run big jobs through the API and cost matters, Kimi K3.1 Pro is the real deal. It finished every one of my tests for 32 cents total.
- If you want the best model, it's still Opus 5.5. It's three points ahead on Artificial Analysis, it's my daily driver, and I'm not switching.
- I'd use Kimi for big, boring batch jobs, like tagging listings or sorting data, where cheap matters more than perfect.
- Be careful with the Kimi subscription for now, because people are burning through the 5-hour limit in minutes. Use the API or OpenRouter instead.
- The weights are the date to watch. If they come out on November 4, Artificial Analysis says this becomes the best open-weights model in the world.
If you're picking a model for vibe coding or comparing options, see my best vibe coding tools page.
FAQ
How much does Kimi K3.1 Pro cost?
It costs $2 per million input tokens and $10 per million output tokens, the same as Sonnet 5.5 and half the price of Opus 5.5. Artificial Analysis found it costs about $2.11 per index task, compared with $5.46 for Sonnet and $5.98 for Opus, because it writes far fewer tokens.
Is Kimi K3.1 Pro better than Opus 5.5?
Not overall. On the Artificial Analysis Intelligence Index, Opus 5.5 scores 58 and Kimi scores 55. Kimi is much cheaper per task, and it finished my horse test when both Claude models ran out of token budget.
Can I download the Kimi K3.1 Pro weights?
Not yet. Moonshot says the full weights will be out by November 4. Until then, you can use it through the Kimi app, Kimi Code, the Kimi API, or OpenRouter.
Why did Opus and Sonnet fail the horse test?
A model's thinking and its answer come out of the same token budget. Both Claude models spent almost all of the 120K (and later 128K) limit thinking, so they got cut off before finishing the file.
Chapters
Transcript
Read the full transcript
0:00So, this is the same exact prompt given to three different AI models. Build me a Minecraft game. One shot, no help. On the left is Kimmy K 3.1 Pro, the brand new model out of China. In the middle, Opus 5.5. On the right, Sonnet 5.5. Kimmy built its game in 4.5 minutes for 15. Opus took over 13 minutes and $180. Slopus, come on now. And then on the next test, something weird happened. Kimmy finished in under five minutes. And both Claude models thought for so long, they blew through our 120,000 token limit before they ever finished the file. So, Moonshot is saying this model is less than one point behind Opus on coding at the exact same price as
0:47Sonnet. And in this video, we're going to be breaking it all down. what the independent numbers actually say, the weird thing Moonshot did to its hacking skills before releasing it, what people built with it in the first 24 hours, and our own head-to-head tests, and if it's even worth using. Let's jump right into it. So, what is Kimmy K 3.1 Pro exactly? It's from Moonshot AI, the Chinese lab behind Kimmy, and it dropped October 23rd. Moonshot says it's 2.8 8 trillion parameters total with 104 billion active at a time. A 1 million token context window. And thinking is always on. Now the price. $2 in, $10 out per million tokens, which is the exact same price as
1:32Sonnet 5.5 and half of Opus 5.5, which is $4 in, $20 out. It's also a third cheaper than Kimmy K3 was back in July. And here's the big one, the weights. Moonshot says the full model weights come out by November 4th. So right now you can't download it. You use it through the Kimmy app, Kimmy code, their API or open router. And to be fair, last time with K3 they promised the weights by July 27th and they shipped them on July 27th. So they were on time last time. So here's the chart everyone was sharing. Terminal Bench 4.0, which is basically can this thing actually do real coding agent work inside a terminal? Opus 5.5 66.4, Kimmy K 3.1 Pro
2:1865.7, less than one point behind and ahead of GPT6 Astra and Fable 5.1. They also say it beats Opus on long coding jobs, on humanity's last exam with tools and on office work, but it loses on computer use. It loses on CAD and it gets crushed on terminal bench science 38 versus Opus at 63. And these are all Moonshots numbers, vendor reported. Take that for what you will. So before I show you who actually checked them, real quick, every video I make is built in code with Claude Code inside Shipping School Premium. I show you how step by step to turn your product into videos that get views, plus new courses every day, 500 plus hours of replays, live calls every
3:03week, and a Q&A with me every Saturday. Join free link below. So, Artificial Analysis ran their own tests on launch day, and on their intelligence index, Kimmy K 3.1 Pro scored a 55. Fourth place out of 215 models. Ahead of it, Opus 5.5 at 58, Sonnet 5.5 at 56, and Meta's Muse Wonderful at 56. So, it's three points behind Opus and one behind Sonnet. Close. Not a knockout, but look at the jump. Kimmy K3 scored a 44 in July. That's 11 points in about 3 months, which is insane. And here's a fun one. Opus on high effort, not max, scores a 54. So Kimmy actually edges out
3:50Opus on high. And on their own Terminal Bench run, Kimmy got 67% and Sonnet got 64. Artificial analysis says that's the highest score of any model on Terminal Bench 4.0 in their testing. It's a different setup from Moonshots, so don't mix those two numbers together. Now, here's the part that actually matters for your wallet. Same price per token as Sonnet, right? But to run that whole index, Kimmy used 150 million tokens. Opus used 260 million and Sonnet used 420 million, almost three times as many. So, per task, Kimmy costs about $2.11. Sonnet 5.46, 46 Opus 598. Same sticker price as Sonnet, but about 2 and 1/2
4:35times cheaper to actually get the work done. Look at that cost comparison. Here's something to be wary about, though. Kimmy is not faster per token. It's actually the slowest of the three. About 73 tokens a second versus 96 for Opus and 133 for Sonnet. It finishes sooner because it writes way less and it starts talking sooner. On their numbers, Kimmy starts its answer in about 30 seconds. Opus, 562 seconds. That's over 9 minutes of thinking before a single word. So remember that Kimmy writes less. That's going to come up again in our tests. So here's the weirdest part of this launch. Moonshot cut the hacking out of it on purpose. Here's how it works. A model
5:20learns in stages. First mid-training where it reads a mountain of text and code. Then reinforcement learning where it practices tasks and gets rewarded when it gets them right. And Moonshot says they went into both of those stages and pulled out the cyber data, the exploit stuff, but they kept all the normal coding data in. Why? Their rule is they won't release open weights that push the absolute frontier of hacking capability. Their words, they want it to stay behind the closed models that have been out for at least 3 months. So, did it work? On their own internal hacking test, Kimmy K3 back in July scored 33%. The version of K3.1 Pro with the hacking data still in, which they kept in house, scored 90.5. And the version they
6:06actually released with the cut 81. So they took the hacking out and it still solves 81%. Up from 33. That's a weird flex. And on exploit bench, their table has Opus at 91, GPT6 Astra at 100, and Kimmy at 74.7. So, it's still behind the US labs, which is exactly the point they're making. Here's the catch, though. Every one of those numbers is moonshot grading its own homework. Back in July, the UK and US government testers checked K3, the older model, and it was way behind. But nobody independent has tested K3.1 Pros hacking yet. And honestly, I think it's the responsible move, but I'm not sure how much it really buys. Once the weights are out, people can train
6:51whatever they want back into it. So, I'd call it a speed bump, not a wall. So, what are people actually building with it? And the first 24 hours were wild. This demo website, one shot, about 15 minutes, almost 200,000 views. Welcome to the new era of webdev. This side by side got over 200,000 views. A low poly horse galloping across a savannah. Kimmy versus Opus versus Astra. And the caption, Kimmy cooked Astra. Remember this one. We're going to run it ourselves. And the cost comparisons are where people lost it. A rooftop glass pool. Same prompt. Opus took 26 minutes and about $12. Kimmy took 19 minutes 63.
7:36Another one. A Spongebob scene. Kimmy 731. Opus $7642. Absolutely insane. Simon Willis called it a big deal. And Nathan Lambert said Moonshot figured out their RL and that China is very much keeping pace. But then one day later, the vibe kind of flipped. Everyone loved the chart. Then people started using it. The same Nathan Lambert posted, "Not a lot of positive Kimmy 3.1 reactions on the timeline. People said Opus 5.5 is still more useful for real work. One guy had it build a backrooms horror game against Opus and Kimmy lost, and this one is the sharpest take. Sonnet 5.5 clearly beats K3.1 Pro despite both having similar API
8:22pricing. Is Kimmy just benchmaxed? And here's something to be wary about if you're thinking about the Kimmy subscription. People are burning through the 5hour limit in minutes. One person said they spent $400 on subs to send four prompts and through the API it would have been $29. Another person burned their whole 5 hour quota in 4 minutes and both prompts got cut off. That's the subscription, not the API. But still. So which is it? The hype or the doubt? I wanted to find out for myself. So we ran it through our own tests, three of them. Kimmy K 3.1 Pro versus Opus 5.5 versus Sonnet 5.5. Same exact prompt for every model. One shot, no help, no doovers on an answer. All
9:08through Open Router, all on high effort, and all with the same 120,000 token limit. We were going to have GPT6 Astra in there, too, but I ran out of Open Router credits halfway through. Astra got refused, and I decided it sits this one out. and one run each is a vibe check, not a verdict. Keep that in mind. Test one. Build a simple, beautiful Minecraft game. One HTML file, grass, dirt, stone, sand, water, trees, break blocks, place blocks, a hot bar, WD to walk around, and make it look beautiful. So, here's Kimmy. 4 minutes and 28 seconds, 15. You can see we've got a sandy beach world, water, trees, a hot bar at the bottom, and breaking and
9:54placing blocks works. Here's Summit. 7 minutes and 57 seconds, 67. A dense forest, stone hills in the back, and it spawned me inside a tree. Thanks, Sonnet. And here's Opus. 13 minutes and 19 seconds, $1.81. Like I said, slopus. But look at this. Flowers, tall grass, clouds, a forest you can walk through. And every game got the exact same inputs, the same moves, the same clicks at the same time. And look at how much each one wrote to get there. Kimmy about 15,000 tokens. Sonnet 67,000. Opus 90,000. Six times as much as Kimmy. Same story as artificial analysis. Which one looks better to you? Comment down below
10:39because I want to know. Test two. Remember that horse post with 200,000 views? The author never shared the prompt. So, we wrote our own. A low poly horse galloping across an African savannah at golden hour. Tall grass, acacia trees, mountains, dust, and a cinematic camera following the horse. Make it as beautiful as you can. Kimmy. 4 minutes and 39 seconds. 15. And you can see we've got the horse galloping, the acacia trees, the mountains, the golden light, the grass. Pretty awesome for 15 cents. And then claude opus ran for over 19 minutes. 116,000 tokens of thinking. Then it started writing the code and it hit our limit in
11:24the middle of the file. Cut off $2.40 for half a horse. And Sonnet Sonnet thought for 16.5 minutes, used all 120,000 tokens on thinking and wrote zero words of an answer. Zero. A$120 for nothing. So, what actually happened here? Here's how it works. When you give one of these models a limit, its thinking and its answer come out of the same bucket. So, if it spends the whole bucket thinking, there's nothing left to write the answer with. And that's not Claude being bad at horses. They just thought themselves out of budget. That's the same thing artificial analysis found. Kimmy writes less. So, I gave them one more shot. Second try with the limit raised to 128,000 tokens, which is
12:09the most open router allows for Claude. And same thing, Sonnet got further this time. It actually started writing. It even got to the birds in the sky. And then it got cut off in the middle of the file again. And Opus, Opus just thought even longer. 122,000 tokens of thinking. Cut off again. So that's about $75 of Claude and zero horses. Kimmy did it for 15 cents. Come on now. Test three. The bug hunt. A small to-do app with exactly five bugs hidden in it on purpose. Tasks don't save when you reload. Delete removes the wrong task. The done filter shows the wrong tasks. Typing code into a task can run it. And the counter never goes down.
12:55Find all five and fix them. And we score it against a hidden answer key. Two points for every bug found and fixed. Right. minus one for every bug it makes up. And full disclosure, Claude scored this one against that answer key. And yes, Claude is one of the contestants. So, every answer is right here on screen. Check it yourself. And the result, 10 out of 10. All three. Every model found all five bugs, fixed them right, and didn't make anything up. So, the only difference is speed and cost. Sonnet did it in 7 seconds. Opus in 11. Kimmy took 25 12. And all three cost under three cents. So this test is just too easy. Now we're going to need a harder bug hunt. So what does this actually mean as of today? Who should
13:42switch? If you're running big jobs through the API and cost matters, Kimmy K 3.1 Pro is the real deal. Same price per token as Sonnet, but it writes so much less that it's about two and a half times cheaper per task. And in our tests, Kimmy finished every single one for 32 cents total. Opus cost 423. Here's how I'd use it in my reselling business. The big boring batch jobs, tagging listings, sorting through stream data, the stuff where I need it done cheap and I don't need it perfect. But if you want the best model, I mean, it's still opus. three points ahead on artificial analysis and opus 5.5 is still my daily driver. I'm not switching. And be wary of the Kimmy subscription for now. If you want to try it, go through the API or open router.
14:29And the real story is November 4th. If those weights come out, artificial analysis says this becomes the best open weights model in the world. Something you could run yourself about three points behind Opus. I mean, this is a wild time to be alive. So, what do you think? Is Kimmy K 3.1 Pro the real deal or just benchmaxed? Let me know down below. And if you got something out of this, a like goes a long way. It helps the video reach more builders. Have a blessed day.
Ready to start building with AI?
Join the Shipping Skool community free. Paid members get live bootcamp calls every week and weekly-updated courses.
Join Shipping SkoolKeep reading
Gemini 4 Carbon vs Claude Opus 5.5: Benchmarks, Pricing, and Access
Gemini 4 Carbon is #1 on the Artificial Analysis Intelligence Index (59 vs Claude Opus 5.5 max at 58), but only cyber defenders and Fairwind partners can use it so far. On real coding agent work, Opus 5.5 in Claude Code still leads 62 to 59, though Carbon is faster and cheaper per task.
DeepSeek V4.1 Flash Explained: Pricing, Benchmarks, and Claude Code
DeepSeek V4.1 Flash costs $0.15 in and $0.60 out per million tokens off-peak. Artificial Analysis scores it 32, ahead of DeepSeek's own V4-Pro at 23 but well behind Claude Opus 5.5 at 58. I'd use it with Claude Code for cheap bulk chores and second-opinion bug hunting, not for hard reasoning or private data.
Kimi K3 vs Opus 4.8: Real Coding Tests, Cost, and Speed
In my tests, Moonshot's Kimi K3 split the results with Opus 4.8. Opus won the lava lamp sim and looked a bit better on the Blender donut. Kimi K3 built the only playable Minecraft clone and finished the donut in about 8 minutes compared with 31 for Opus, at roughly a third of the price.
