Robot Foundation Model API Explained: Dream Labs, Pi, and Gemini
Prefer to watch? Watch on YouTube · Subscribe for new builds · Read the transcript
In one week in October, robot brains turned into an API call. On October 13, Physical Intelligence opened a waitlist beta for its latest robot models, and on October 19, Dream Labs raised a $90M seed and opened a public API for its robot model at $3 an hour. In this video I explain what people mean by a "GPT-3 moment" for robots, how these models work, who is actually shipping, and what a software builder can do with them today. You don't need to own a robot to follow along.
What I cover in this video
- 0:46: What a robot API call looks like, and why it's being compared to GPT-3
- 1:33: Robot foundation models, and prompting with words or with video
- 3:06: How a robot brain works, in five steps
- 6:12: The data problem and the three tricks labs use to get around it
- 7:43: Who's actually shipping, ranked by what you can use today
- 9:14: What I'd build with Claude Code
- 11:30: The honest limits
- 13:01: Why the GPT-3 comparison still holds
What the "GPT-3 moment" for robots means
A robot API works like this: you send a server a few camera frames and one sentence, like "Put the cup in the bin," and it sends back 1.6 seconds of robot motion. Then you send the next request. That loop is all there is to it.
Before GPT-3, every language job needed its own model: one for spam, one for translation, one for sentiment. Then one big pre-trained model could handle all of them if you asked in plain English. Robots have been stuck in that earlier stage. You'd have one program for one task on one robot in one room, and if you moved the table or swapped the gripper, you basically started over. A robot foundation model is one model, pre-trained on a lot of robot data and video, that you steer with a prompt. The prompt can be words, or it can be a video. Skild's S1 watches a short video of a task, and Skild says it scored 66% on unseen long tasks with a video prompt versus 9% with a text prompt. That's their own test.
GPT-3 didn't matter just because it was smart. It mattered because you could call it. That's what changed in October: robot models are becoming something you call.
How a robot brain works in five steps
- Inputs. Camera images, the robot's own joint positions, and a sentence telling it what to do. Dream's model takes a camera on the robot's head and a camera on each wrist.
- The vision-language model. A model that already learned from the internet what a mug or a drawer is, and what "to the left of" means. Physical Intelligence's pi0 started from a 3B-parameter vision-language model.
- The action head. This step turns understanding into motion, and there are two main ways to do it. The first is action tokens: you predict chunks of motion the way a chatbot predicts words. Physical Intelligence's FAST compresses a chunk, a bit like a JPEG, into typically 30 to 60 tokens. The second is flow matching: the model starts from random noise and cleans it up step by step into a smooth arm path. pi0 and NVIDIA's GR00T both use flow matching. Vision, language, action is why these are called VLAs.
- Action chunks. For builders, this step matters most. The model sends the next second or so of moves at once instead of one move at a time. Physical Intelligence uses 50 actions per second-long chunk, and Dream returns 1.6 seconds per call. The model is slow: in one example it took 97 ms to think, while pi0 sends new motor commands up to 50 times a second. So the robot plays one chunk while you fetch the next. Physical Intelligence calls this real-time chunking, and in their tests speed stayed the same with up to 200 ms of added delay. That's why a robot brain can run in the cloud at all.
- One model, many bodies. If you train on many robots at once, the model learns "pick up the cup" separately from "this arm has seven joints." The open Open X-Embodiment dataset has over a million robot runs across 22 kinds of robots. Dream's model predicts where the gripper should end up, and each robot's own software works out the joints. Dream says it trained on one real robot and tested on 27 unseen robots in simulation. Fine-tuning still matters, though. Physical Intelligence built its API around it, and Dream says one-shot adaptation still beats zero-shot.
The data problem and who's shipping
Language models had the whole internet to read. Robots don't have an internet of robot moves. The best data comes from teleoperation, where a person drives the robot by remote control, and that's slow. Skild says 380 demos took 50 to 100 hours. Labs use three tricks to get around this:
- Human video. Figure says training on human video took Helix from 9% to 56% success in 30 homes it had never seen. NVIDIA's GR00T N1.7 trained on 20,000 hours of human video.
- Simulation and self-play. Skild ran a humanoid through about 140 years of simulated soccer before putting it on a real robot.
- World models. Meta FAIR's RoboJEPA preprint (not peer reviewed) is an 8B predictor whose error falls on a predictable curve as compute grows.
I think of it as a pyramid: lots of cheap human video at the bottom, simulation in the middle, and a thin layer of real robot driving on top.
Here's who you can actually use, ranked by access. Dream Labs is a public research beta: Dream-1 costs $3/hour, Dream-1-flash costs $2/hour, and there's a browser playground that controls a real robot in their San Francisco office. Google has Gemini Robotics ER 2 in preview in Google AI Studio, but that's the planner. The motion model, Gemini Robotics 2, is for trusted testers only. NVIDIA GR00T N1.7 is open weights at 3B parameters, so you can run it on your own hardware. Physical Intelligence has open-sourced pi0 and pi0.5, but the new API is waitlist only with no public price. Skild, Figure, 1X, and Generalist have demos but no API you can call.
What I'd build with Claude Code
- Start with no hardware. Type a task into Dream's playground and watch a real robot try it.
- Have Claude Code write the loop. Grab camera frames, call the API, play back the chunk, repeat. It's the same pattern as any other API. Test it on a simulated robot first, with something like NVIDIA's Isaac Sim.
- Split the brain and the body. Use Gemini Robotics ER 2 as the planner ("find the red block and put it in the bowl"). It returns points and function calls like move-to and grip into your own code. If you've built AI agents with tool calls, you already know this pattern.
- Add real hardware. Hugging Face's LeRobot project has a cheap open-source arm, the SO-101, with kits that run a few hundred dollars. You record demos and fine-tune an open model on them, and NVIDIA ships a GR00T example for the SO-100. Dream's model currently targets two-armed robots with parallel grippers, so it won't run on one small arm.
On cost: $3/hour at 1.6 seconds per call is about a tenth of a cent a call, so by my math a 4-hour afternoon of testing comes to about $12. For my resale business we fold and sort a lot of clothes every week. I wouldn't hand that job to a robot today, but I'd spend a few dollars to watch Dream's robot try to fold leggings.
The limits are real. Success rates are coin flips at best: Generalist 59% from one demo, Skild 66%, Figure 56%, RoboJEPA 27% on real pick-and-place. Every one of those numbers comes from the company's own tests. Dream says its model will make mistakes and hasn't been validated for safety-critical use, and it hasn't disclosed the model's size. Physical Intelligence hasn't shared a price or a robot list, and it mentioned partners running into network problems. Rodney Brooks has also argued that visual data alone isn't the right data, because human dexterity depends on touch. On the other hand, a company called Actor used the Pi API to run a wheel loader with no operator on a real commercial job, filling a dump truck.
Key takeaways
- Dream Labs' Dream-1 is a public API today: camera frames and a sentence in, 1.6 seconds of motion out, for $3/hour.
- Robot brains are VLAs: a vision-language model plus an action head, sending motion in chunks so a cloud model can keep up.
- For builders, it's a familiar loop or a tool-calling planner, the same patterns you already use with any API.
- Success rates on new tasks are around coin-flip level and self-reported, so prototype in a playground or simulator first.
FAQ
What is the Dream Labs robot API?
It's a public research beta for Dream Labs' Dream-1 robot foundation model. You send camera frames and a sentence, and it returns 1.6 seconds of robot motion. Dream-1 costs $3/hour and Dream-1-flash costs $2/hour.
Can I use the Physical Intelligence Pi API?
Not openly yet. The Pi API beta announced on October 13, 2026 is waitlist only. They onboard in batches, and the form is built for teams that already have robots and data. There's no public price or supported-robot list, though the older pi0 and pi0.5 models are open source.
What is a VLA model in robotics?
VLA stands for vision-language-action. A vision-language model understands the camera images and your instruction, and an action head turns that into robot motion, using either action tokens or flow matching.
Do I need a robot to try a robot foundation model?
No. Dream Labs has a browser playground that controls a real robot in their San Francisco office, and you can test your own code against a simulated robot in a tool like NVIDIA's Isaac Sim.
Chapters
Transcript
Read the full transcript
0:00In one week in October, robot brains started turning into an API call. On October 13th, physical intelligence opened a weightless beta for its latest robot models through an API. And then 6 days later, on October 19th, a company called Dream Labs, founded by researchers behind Nvidia's Groot robot models, raised $90 million and opened up a public API. No wait list, $3 an hour. You've probably seen people calling this the GPT3 moment for robotics. So, in this video, we're going to be breaking down what that actually means, how one model can drive a bunch of different robots, who's actually shipping, what you can build with it as a software builder, and where the hype is way ahead of the robots. Let's jump right into it.
0:46So, here's what a robot API actually looks like. You send a server a few camera frames and one sentence, like, "Put the cup in the bin." and it sends back 1.6 seconds of robot motion. Then you do it again. That's it. That's the whole loop. So that's basically what GPT3 did for text. So what is a GPT3 moment exactly? Before GPT3, if you wanted a computer to do something with language, you trained one model for one job. A model for spam, a model for translation, a model for sentiment. So then GPT3 came out and one big pre-trained model could do all of it if you just asked it in plain English. So robots have been stuck in the before picture. One program for one task on one
1:33robot in one room. You move the table 6 in or you swap the gripper and you basically start over, which is insane when you think about how much money goes into these things. So the new bet is a robot foundation model. one model pre-trained on a ton of robot data and video that you steer with a prompt. Sometimes the prompt is words like fold the towel and sometimes the prompt is a video. Skilled AI showed a model called S1 that watches a short video of the task and then tries it on long tasks it had never seen. Skilled says it scored 66% from a video prompt versus 9% from a text prompt. And that's their own test, so take that for what you will. But here's something most people don't think about. GPT3 didn't matter just because
2:20it was smart. It mattered because you could call it. You got an API key, you wrote 10 lines of code, and you built a product on top of it. So that's the part that changed in October. The robot model is turning into a thing you call. Real quick, every video I make is built in code with Claude Code. Inside Shipping School Premium, I show you how step by step to turn your product into videos that get views. Plus, new courses every day, 500 plus hours of replays, live calls every week, and a Q&A with me every Saturday. Join free link below. So, how does one of these robot brains actually work? Don't get lost in the sauce here. It's five steps. Step one is what goes in camera images, the robot's own joint positions, and a sentence
3:06telling it what to do. Dreams model, for example, takes a camera on the robot's head and a camera on each wrist. So step two, a vision language model reads all of that. This is a model that already learned from the internet what a mug is, what a drawer is, what to the left of means. Physical intelligence's first model, PI 0, started from a 3 billion parameter vision language model. So that's the V and the L, vision language. Step three is the A, action. The model has to turn what it understands into motion. And there are two main ways to do it. The first way is action tokens. You chop the motion up into tokens and predict them like words the same way a chatbot predicts the next word. Physical intelligence made a trick called fast
3:52that squeezes a chunk of motion down kind of like a JPEG into typically 30 to 60 tokens. So the second way is called flow matching. The model starts with random noise and cleans it up step by step into a smooth path for the arm. Pi 0 does that. Nvidia's Groot does that too. vision, language, action. That's why everybody calls these VLAS. Step four is the one that matters the most for us, action chunks. The model doesn't send one move at a time. It sends the next second or so of moves all at once. Physical intelligence works in chunks of 50 actions, which is 1 second. Dream sends back 1.6 seconds per call. And here's why. The model is slow. In one of physical intelligence's examples, the
4:38model itself took 97 milliseconds to think, but Pi 0 sends the motor's new commands up to 50 times a second. So, you give the robot a whole chunk to play, and while it's playing, you go get the next one. Physical intelligence calls that real time chunking. And in their tests, the robot kept working at the same speed, even with an extra 200 milliseconds of delay. That's the trick that makes a robot brain in the cloud possible at all. Your robot isn't waiting on the internet every single move. And then step five, one model, many bodies. You train it on lots of different robots at once. So it learns pick up the cup separately from this arm has seven joints. There's a big open data set called open X embodiment with more than a million robot runs across 22
5:25kinds of robots. So Dream does something clever here. Its model predicts where the gripper should end up and then each robot's own software figures out how to move the joints to get there. Dream says it trained on one real robot and then tested it on 27 robots it had never seen in simulation. But like I said, it's not magic. Fine-tuning on your own data hasn't gone away. Physical intelligence's API is literally built around it. And Dream says it straight up. Oneshot adaptation still outperforms zero shot. So a gap remains. So why did this take so long? Data. Language models had the whole internet to read. Robots don't have an internet of robot moves. The best data is a person driving the robot by remote control, which is called
6:12tea operation, and it's slow and expensive. Skilled says just 380 demos took 50 to 100 hours to collect. Come on now. So the labs have three new tricks. Trick one, human video. Instead of driving the robot, you just film people doing stuff and teach the robot from that. Figure says training on human video took its Helix model from 9% to 56% success in 30 homes it had never seen. Nvidia's Groot N 1.7 was trained on 20,000 hours of human video. And Dreams investors say its model learns from video of people doing everyday tasks. So trick two, simulation and selfplay. Skilled put a humanoid in a simulator with one goal, score a goal in
6:57soccer, and let it play for about 140 years of simulated time. Early on, it could barely walk. Then they moved it to a real robot. Trick three, world models. This is where the model imagines what's going to happen before it moves. Meta's research team put out a paper on October 7th called RoboJeppa. It's a pre-print, so it's not peer-reviewed yet. It's an 8 billion parameter model that predicts the future, not as pixels, but as a compressed idea of the scene. And the interesting part is that its error drops on a predictable curve as you give it more compute. That's a scaling law for robot imagination, which is pretty wild. And they release the checkpoints and the code. So, the way I think about it is a pyramid. Tons of cheap human video at
7:43the bottom, simulation in the middle, and a thin layer of real robot driving on top. So, who can you actually use? I'm going to rank these by how close you can get your hands on them, because that's what matters if you're a builder. Number one, Dream Labs. You can call it today. It's a public research beta. $3 an hour for Dream 1, $2 an hour for the Flash version, and they have a robot playground where you control a real robot in their San Francisco office from your browser. You don't even need to own a robot. DreamLabs calls it the first publicly available API for a robot foundation model. That's their words, not mine. So, number two, Google. Gemini Robotics ER2 is in Google AI Studio as a preview. But here's something to pay
8:29attention to. That's the planner. It looks at the scene, points at things, and calls your robot functions. The model that actually moves the motors, Gemini Robotics 2, is early access for trusted testers. Number three, Nvidia. Groot N 1.7 is open weights, 3 billion parameters. You download it and run it on your own hardware. So, number four, physical intelligence. Their older models, Pi 0 and PI 0.5, are open- source. The new API is a weight list. They're onboarding in batches and the form is built for teams that already have robots and data. No public price yet. And then everybody else is watch only for now. Skilled S1 has an early access signup. Figure 1X and Generalist
9:14all have impressive demos but no API you can call. So what does this actually mean for us as software builders using Cloud Code? Here's how I'd do it. Step one, no hardware. Go to Dream's playground, type a task, and watch a real robot try it. Simple. Don't overthink it. Step two, have Claude Code write the loop. Grab the camera frames, call the robot API, play back the chunk, repeat. It's not a lot of code, and it's the exact same pattern we already use with any API. And you test that loop on a simulated robot first with something like Nvidia's Isaac SIM. So, you're not breaking anything. So, step three, split the brain and the body. Use Gemini Robotics ER2 as the planner. You ask it, find the red block, and put it in the
10:00bowl. And it gives you back points and function calls into your own code, like move to and grip. If you've built an agent with tool calls, you already know how to do this. It's tool calling. That's it. And then step four. If you want real hardware, there's a cheap open- source arm from Hugging Face's lay robot project called the SO101. Kits run a few hundred. You record yourself showing it a task and then you fine-tune an open model on it. Nvidia even ships an example for the SO100 arm with Groot. But to be clear, Dream's model is built for two armed robots with parallel grippers for now. So don't expect it to run on one little arm. So now the cost. At $3 an hour and 1.6 seconds per call, that works out to about a tenth of a
10:45cent a call. So by my math, a full afternoon of testing, 4 hours, is about $12. That's less than my lunch. So, here's what I build first. For my resale business, we're folding and sorting a ton of clothes in our garage studio every single week. Would I let a robot do that today? No. But would I spend a few dollars in Dream's playground to watch their robot try to fold a pair of leggings? Absolutely. Now, I want to talk about where this falls apart because there's a lot of hype here. First, look at the success rates. Generalists model 59% on average from one demo. Skilled 66%. Figure 56% in new homes. Robojapa 27% on a real pick and
11:30place. Those are coin flips at best and some are worse. And every one of those numbers is the company's own test. So second, Dream says it right on their site. It will make mistakes and it has not been validated for safety critical use. They also haven't said how big the model is. Third, physical intelligence hasn't shared a price or a list of robots, and they even mentioned that some of their partners ran into network problems, which if your robot brain is in the cloud, yeah, that's a problem. And then the big one, Rodney Brooks, one of the best known names in robotics, wrote this last year. Collecting just visual data is not collecting the right data. Human dexterity relies on a rich sense of touch, and human video is
12:16exactly the bet most of these labs are making. So, that's a real debate. So, here's the other side, though. These aren't all just demos. A company called Actor used the PI API to run a wheel loader on a real commercial job, filling a dump truck with no operator inside the machine. And Joel Jen, Dream CEO, put it pretty well. Instead of edited demos, they're giving people a working model they can test themselves, mistakes included. I respect that. I mean, I'm so tired of edited demos. So, to sum it up, GPT3 wasn't the smartest model ever made. It was the one that got everybody building. And robots are right at that point. They're rough. They're coin flips at best on new tasks. They cost $3 an
13:01hour. And you can call them. So, if you had a robot brain for $3 an hour, what's the first job you'd hand it? Put it in the comments. I actually read them. And if you learned something new about robots here, tap that like button. That's how YouTube knows to show it to more people. Have a blessed day.
Ready to start building with AI?
Join the Shipping Skool community free. Paid members get live bootcamp calls every week and weekly-updated courses.
Join Shipping SkoolKeep reading
DeepSeek V4.1 Flash Explained: Pricing, Benchmarks, and Claude Code
DeepSeek V4.1 Flash costs $0.15 in and $0.60 out per million tokens off-peak. Artificial Analysis scores it 32, ahead of DeepSeek's own V4-Pro at 23 but well behind Claude Opus 5.5 at 58. I'd use it with Claude Code for cheap bulk chores and second-opinion bug hunting, not for hard reasoning or private data.
NVIDIA Nemotron 3 Nano Omni: The Open Model That Gives AI Agents Eyes And Ears
NVIDIA Nemotron 3 Nano Omni gives AI agents a single open perception layer for screens, audio, video, documents, charts, and images.
Jev by TypeSafe Explained: Pricing, Benchmarks, and Claude Code
Jev is TypeSafe's System One model: you send it a state plus typed questions and it returns a choice, a score, or a probability instead of text. I'd keep Claude Code and Opus for planning and building, and put Jev on the small yes or no calls inside the app.
