Robot Foundation Models (Vision-Language-Action)

Executive summary

A robot foundation model (vision-language-action, VLA) is a learned policy that maps visual observations and natural-language instructions to robot actions. The term became widely used after Google DeepMind released RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (S2; S8).

VLA models are technically real and demonstrate generalization in controlled laboratory/warehouse-style evaluations, but the public evidence does not yet support the strongest commercial claims often attached to “general-purpose robot foundation models.” Key findings:

• Technical foundation exists: RT-1, RT-2, PaLM-E, Open X-Embodiment, OpenVLA, Octo, and π0 are documented in arXiv preprints. Most technical/performance claims are still single-lineage Google/DeepMind or academic-preprint claims and should be treated as [SINGLE-SOURCE] where noted.
• A granted patent exists, but it is not VLA-specific: US10452978B2 is a granted patents.google.com record for the transformer attention architecture, not a VLA robot-control patent.
• No public company filing I could verify breaks out VLA/robot-foundation-model revenue. Alphabet, NVIDIA, Tesla, Amazon, Microsoft, and Qualcomm discuss AI/robotics in risk factors but do not report a VLA revenue line; therefore unit economics remain largely inferred.
• The moat question remains open: open-source VLA models (OpenVLA, Octo) and open datasets (Open X-Embodiment) reduce data/architecture exclusivity, but compute, robot fleet scale, and industrial deployment contracts may be the actual moats.

Source inventory

I did not use SEO content farms, anonymous benchmarks, or AI-generated summaries.

What a VLA model is

A vision-language-action model maps three input/output streams:

1. Visual observation — RGB images from one or more robot cameras.
2. Language instruction — e.g., “pick up the apple and place it on the plate.”
3. Action tokens — robot control commands or motion primitive tokens.

Google DeepMind’s RT-2 blog describes the approach as “co-fine-tuning” a large vision-language model so that it can output robot actions as text-like tokens, thereby transferring internet-scale visual and language knowledge into robot control (S8). The corresponding arXiv paper is S2, arXiv:2307.15818, submitted 2023-07-28, labeled preprint.

The transformer attention mechanism that underlies nearly all modern VLA backbones is recorded in the granted patent US10452978B2, “Attention-based sequence transduction neural networks,” assignee Google LLC, grant date 2019-10-22 (S9). The peer-reviewed NeurIPS 2017 paper is “Attention Is All You Need” (S10). Important boundary: the patent covers an architecture family, not a robot-control performance claim.

Technical lineage and load-bearing claims

4.1 RT-1: the robot-transformer predecessor

RT-1: Robotics Transformer for Real-World Control at Scale (S1, arXiv:2212.06817, submitted 2022-12-13, preprint) describes a 35M-parameter model trained on roughly 130,000 real robot demonstrations collected over 17 months from a fleet of everyday office-kitchen robots.

The paper reports that the policy used an EfficientNet-based image tokenizer, FiLM conditioning on language, a TokenLearner module, and a transformer to generate actions. This is a single-source technical claim from Google/Everyday Robots lineage; it should be tagged [SINGLE-SOURCE] for performance specifics because no independent replication with identical hardware was identified in this note.

4.2 RT-2: tokenizing actions

RT-2 (S2) discretizes robot actions into tokens that the model can emit in the same format as language tokens. The largest RT-2 variants were derived from PaLI-X at 55B parameters, with smaller variants from PaLM-E (S3, arXiv:2303.03378, submitted 2023-03-06, preprint).

The RT-2 paper reports improved generalization on novel objects, unseen backgrounds, and some emergent reasoning behaviors compared with prior robot policies. These are paper-reported performance claims, not independent Tier 1 evaluations. The company blog (S8) describes this as a “breakthrough,” but the blog is self-reported Tier 2.

Load-bearing technical claim:

A VLA can co-fine-tune a large web-pretrained vision-language model to output discretized robot actions.

Sources: RT-2 arXiv (S2) and Google DeepMind blog (S8).

Independence check: Same company lineage — DeepMind. Therefore this claim remains [SINGLE-SOURCE] unless independently replicated. The architectural attention mechanism is separately supported by the NeurIPS paper and the granted patent, but that does not independently support the robot-control performance claim.

4.3 PaLM-E: embodied multimodal LLM

PaLM-E (S3) introduces continuous robot sensor/state inputs into a large language model. The largest reported PaLM-E-562B configuration is a 562B-parameter model (S3). The paper reports that a single model can handle visual-language tasks, planning, and robot control across embodiments. This is a preprint and is Google lineage, so downstream VLA claims based on PaLM-E are also [SINGLE-SOURCE].

4.4 Open X-Embodiment: multi-embodiment data

Open X-Embodiment: Robotic Learning Datasets and RT-X Models (S4, arXiv:2310.08864, submitted 2023-10-13, preprint) consolidates more than one million real robot trajectories from multiple institutions and robot embodiments.

The paper reports that models trained on this multi-embodiment dataset often outperform single-embodiment models, but the improvement is not uniform across every robot/task. This creates a contradiction surface: the term “generalist” is supported on average, while per-embodiment specialization can still win. This nuance is often lost in marketing summaries.

4.5 OpenVLA and Octo: open-source generalist policies

OpenVLA: An Open-Source Vision-Language-Action Model (S5, arXiv:2406.09246, submitted 2024-06-13, preprint) reports a 7B-parameter open-source VLA trained on a large Open X-Embodiment subset. The paper reports competitive manipulation success on several benchmarks relative to closed models.

Octo: An Open-Source Generalist Robot Policy (S6, arXiv:2405.12213, submitted 2024-05-20, preprint) reports a 93M-parameter transformer-based policy designed for fine-tuning by different research groups.

These open releases weaken the “dataset/architecture secrecy” moat. However, model weights are not the same as deployment capability: fleet integration, safety certification, and maintenance remain separate barriers.

4.6 π0: flow-matching VLA

π0: A Vision-Language-Action Flow Model for General Robot Control (S7, arXiv:2410.24164, submitted 2024-10-31, preprint) from Physical Intelligence reports a flow-matching formulation for continuous robot actions rather than purely discrete token output. The company blog is self-reported Tier 2 (S20).

This is an important architectural distinction: early RT-2-style models discretized actions; π0 learns continuous actions through flow matching. The performance claim is preprint-only.

Worked quantitative calculation

5.1 Inference FLOPs estimate for a 55B VLA forward pass

Formula:

For a dense decoder-style transformer, a standard order-of-magnitude estimate is:

\[

C_{\text{forward}} \approx 2 \times P \times T

\]

where:

• \(P\) = number of model parameters
• \(T\) = number of tokens processed
• Factor 2 = multiply for forward pass matrix multiplications in both linear layers

This approximation is consistent with the scaling-law convention in Kaplan et al. (S22, arXiv:2001.08361, submitted 2020-01-23, preprint).

Inputs:

• \(P = 55 \times 10^9\) for RT-2-PaLI-X-55B (S2)
• \(T = 264\) tokens assumed: 256 vision tokens from the visual encoder plus 8 text/action tokens. The 256 image-token count is consistent with typical PaLI-X-style visual tokenization but should be treated as assumed unless verified in the specific checkpoint; exact token count needs primary source validation.

Calculation:

\[

C = 2 \times 55 \times 10^9 \times 264

\]

\[

C = 2 \times 55 \times 10^9 = 110 \times 10^9

\]

\[

110 \times 10^9 \times 264 = 2.904 \times 10^{13}

\]

So:

\[

C \approx 2.9 \times 10^{13} \text{ FLOPs} \approx 29.0 \text{ TFLOPs per action-step forward pass}

\]

Sensitivity:

• If \(P = 7 \times 10^9\) (OpenVLA-scale), then \(C \approx 2 \times 7 \times 10^9 \times 264 = 3.696 \times 10^{12}\) FLOPs ≈ 3.7 TFLOPs.
• If \(T\) doubles from 264 to 528 tokens, \(C\) doubles to 5.8 × 10^13 FLOPs.
• If the model is encoder-decoder rather than decoder-only, the true FLOPs can differ by a factor of roughly 2–4; this estimate is order-of-magnitude, not a benchmark measurement.

Why this matters: A 55B VLA requires substantially more inference compute per action than a 7B or 93M policy. The utility of the larger model depends on whether its measured success-rate gain justifies the additional compute, which the papers do not fully resolve with independent field data.

Parameter table

Contradictions and cross-file checks

7.1 “General-purpose” marketing vs paper-scope evaluations

Google DeepMind’s RT-2 blog says the model transfers web knowledge into robot control and can “learn from a small amount of robot data” (S8). The RT-2 paper reports strong gains on selected manipulation tasks (S2). That is not the same as deployment-grade reliability across industrial embodiments.

Discrepancy: The blog’s framing implies broad generality; the paper’s results are still mostly pick-and-place manipulation in Google-built or Google-partner environments. The performance claim is not independently reproduced in a Tier 1 source identified here. [SINGLE-SOURCE]

7.2 Company filings do not disclose VLA revenue

• Alphabet 10-K filings discuss AI competition, data, and regulatory risk, but I did not verify a separate robotics-foundation-model revenue line in the 10-K index (S11).
• Tesla 10-K filings discuss development risk around Optimus and AI but do not report a VLA product line (S13).
• NVIDIA 10-K filings do not separate robot-foundation-model software revenue; robotics appears inside broader platform/OEM categories. The major disclosed hardware economics are in GPU/data-center revenue and margin (S12).

Discrepancy conclusion: The marketing-level claim that robot foundation models are immediately transformative is not yet mirrored by line-item financial disclosure. This is a source-bound absence, not necessarily a falsification.

7.3 Patent vs performance

US10452978B2 is evidence of a recorded architecture claim, not proof that any VLA works in production. This is a critical epistemic boundary. A patent is legal record, not performance evidence. I did not verify a granted VLA-specific robot-control patent; if such a patent exists, its number and scope needs primary source validation.

7.4 Open-source tension

OpenVLA and Octo are open-source. Open X-Embodiment is an open dataset. Meanwhile, industrial humanoid firms largely keep their highest-performing VLA checkpoints closed. This creates a contradiction: the field publishes enough to commoditize some parts of the stack while retaining deployment-scale advantages out of public view.

Unit economics / business model

Who monetizes this technology

There are at least four distinct monetization positions:

1. Foundation-model developers — Google DeepMind/Physical Intelligence/Skild-style labs can monetize through APIs, licenses, or bespoke deployment contracts. Public pricing is not disclosed; exact unit economics needs primary source validation.
2. Robot OEMs — Tesla, Figure, Agility, Boston Dynamics, and others can sell or lease robots, potentially shifting from hardware sale to robot-as-a-service (RaaS). Margins are not publicly broken out for VLA features.
3. Compute/chip suppliers — NVIDIA is the clearest near-term value-capture beneficiary. NVIDIA reported 72.7% GAAP gross margin for fiscal 2024 in its 10-K (S12). This is a Tier 1 financial-statement number.
4. Cloud providers — Microsoft Azure, Google Cloud, and AWS monetize training/inference VM hours. No VLA-specific GPU-hour revenue split is disclosed.

Business model forms

• Chip sale: highest verified margin in the chain.
• Inference API / model license: likely subscription/consumption-based; unverified pricing.
• Robot-as-a-service: recurring operational fee + service; unverified margin.
• Data flywheel licensing: Open X-Embodiment-style datasets reduce data exclusivity, but industrial fleet data may become a proprietary asset.

Value capture

The strongest verified value capture today is at the compute layer. The VLA model layer may be partially commoditized by open-source releases. The robot OEM layer still bears deployment, safety, and reliability risk. The value may shift once buyers pay for outcomes, not models.

Disconfirming Evidence

The bullish thesis on robot foundation models / VLA would be weakened or falsified if:

1. Independent benchmark results fail to reproduce RT-2-style generalization gains outside Google/DeepMind installations. A non-Google peer-reviewed study showing no advantage over single-task policies would be strong disconfirming evidence.
2. No public company reports a VLA-related revenue line in subsequent 10-K or 10-Q filings. Continued absence is evidence that commercialization is not material.
3. Inference compute cost per successful action exceeds the labor cost it replaces. The FLOPs calculation above shows a 55B VLA requires ~29 TFLOPs per forward pass; if deployed action-success rates are low, economics invert.
4. Regulatory/standards barriers block deployment. If VLA-controlled manipulators cannot obtain safety certification under ISO/IEC functional-safety standards or equivalent regulatory assessments, industrial adoption stalls.
5. Open-source replication closes performance gaps without proprietary data. If OpenVLA/Octo-class open models reach parity with closed industrial VLA systems, the proprietary VLA layer loses pricing power.
6. The only verified granted patent remains a non-VLA transformer patent. If no VLA-specific enforceable IP emerges, the legal moat is weaker than the marketing suggests.