Element 9.25
Perception, learning and AI for machines
- Sheet
- 9.25
- Chapter
- 9 Manufacturing and repair
- State
- Drafted
- Release
- 16 November 2026
- Editor
- Contributed by Praecise as open research
- Reviewers
- To be named at review
- Safety
- S3, as the chapter
- Licence
- CC BY-SA 4.0
- Resources
- 13
The element
- Requirement
Where a machine uses a learned model to see, hear or act, the community knows what the model was trained on, how it was tested, what hardware it needs, and how it fails; and a learned controller never replaces the safety functions that stop a machine.
- How to tell
- The model and its maturity are named from its own release. One open release of robot foundation models calls itself an experiment that may or may not work on another robot (openpi).
- Hardware needs are known before purchase: that release states more than 8 GB of GPU memory for inference and more than 22.5 GB for low-rank fine-tuning (openpi); a small model is designed to train on one GPU and run on consumer GPUs or CPUs (SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics); microcontroller inference fits in tens to hundreds of kilobytes (TensorFlow Lite for Microcontrollers).
- The model is tested on the community's own task, with success counted over repeated trials, before it is used.
- Safety stops, limits and guarding work independently of any learned model (see the element on robot and drone safety).
- Options
- Perception models: detection, segmentation and pose models for cameras (Ultralytics YOLO models); sound models for machine fault detection (DCASE 2024 Task 2: First-shot unsupervised anomalous sound detection for machine condition monitoring); keyword spotting on microcontrollers (TensorFlow Lite for Microcontrollers, MLPerf Tiny Benchmark (paper)).
- Imitation learning on low-cost arms: record demonstrations with a leader arm and train a policy such as ACT or Diffusion Policy in a shared library and dataset format (LeRobot, Standard Open SO-100 and SO-101 Arms).
- Vision-language-action models that take a camera image and an instruction and output actions: OpenVLA (7 billion parameters, 970,000 demonstrations; its authors report higher success than a 55-billion-parameter model across 29 tasks) (OpenVLA: An Open-Source Vision-Language-Action Model); SmolVLA, trained on community datasets (SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics); the pi0 family (openpi).
- Pooled cross-robot data: 22 robot types, 527 skills, from 21 institutions (Open X-Embodiment: Robotic Learning Datasets and RT-X Models).
- Reinforcement learning in simulation with transfer to real robots (Isaac Lab).
- Touch: fingertip sensors and a touch model pre-trained on over 460,000 tactile images (Meta FAIR robotics release: Digit 360, Sparsh and Digit Plexus).
- Settled, open, contested
Settled: imitation learning on low-cost arms with open tools is reproducible by many groups (LeRobot, Standard Open SO-100 and SO-101 Arms). Settled: perception models run locally on small hardware (TensorFlow Lite for Microcontrollers, Ultralytics YOLO models). Open: how well vision-language-action models generalise to new robots and places; their own authors say results may not transfer (openpi). Contested: general-purpose robots. Developers present foundation models and humanoids as near; the International Federation of Robotics states mass adoption is uncertain and humanoids may not be common household helpers in the near or medium term (Humanoid Robots: Vision and Reality (IFR position paper)).
The four stages
- Learn: Open X-Embodiment: Robotic Learning Datasets and RT-X Models, OpenVLA: An Open-Source Vision-Language-Action Model, SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics.
- Design: choose model by task and hardware (openpi, SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics, TensorFlow Lite for Microcontrollers).
- Build: collect demonstrations and train with LeRobot; simulate with Isaac Lab.
- Deploy: test success rate on the real task; record model version, data and results with the machine's documentation.
Resources at each stage
1 Learn
Understand
- DCASE 2024 Task 2: First-shot unsupervised anomalous sound detection for machine condition monitoring Dataset, S0
- Humanoid Robots: Vision and Reality (IFR position paper) Framework, S0
- Isaac Lab Tool, S0
- LeRobot Library, S2
- Meta FAIR robotics release: Digit 360, Sparsh and Digit Plexus Reference design, S0
- Open X-Embodiment: Robotic Learning Datasets and RT-X Models Paper, S0
- OpenVLA: An Open-Source Vision-Language-Action Model Paper, S0
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics Paper, S0
- Standard Open SO-100 and SO-101 Arms Reference design, S2
2 Design
Size and choose
3 Build
Build or obtain
- LeRobot Library, S2
- openpi Repository, S0
- OpenVLA: An Open-Source Vision-Language-Action Model Paper, S0
- SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics Paper, S0
- Standard Open SO-100 and SO-101 Arms Reference design, S2
- TensorFlow Lite for Microcontrollers Library, S0
- Ultralytics YOLO models Library, S0
4 Deploy
Install, run and measure
- LeRobot Library, S2
- Standard Open SO-100 and SO-101 Arms Reference design, S2
What changes between 30 adults and 300 people
At 30 adults, learning robots are a shared study project: one low-cost arm, a consumer GPU and published datasets. At 300 people, learned perception can do routine work, such as sorting, counting and fault detection, if test results are kept and a person can override; foundation-model control of machines near people stays experimental.
Reference designs
-
Reference design
Meta FAIR robotics release: Digit 360, Sparsh and Digit Plexus
October 2024 release of an artificial fingertip (Digit 360) that Meta states has over 8 million taxels, more than 18 sensing features and detects forces of 1 millinewton; a self-supervised touch model (Sparsh) pre-trained on over 460,000 tactile images; and a platform to connect touch sensors across a robot hand. Code and designs released; manufacture by partner firms announced.
External resource. Not made or verified by Public ProtocolMaturity: PrototypeSafety S0Open, with paid products or servicesListed 16 November 2026 -
Reference design
Standard Open SO-100 and SO-101 Arms
Open, largely 3D-printed robot arms designed by RobotStudio with Hugging Face for the LeRobot learning library. Each arm uses six STS3215 serial bus servos; a leader arm moved by hand teaches a follower arm. The repository gives STEP and STL files, a bill of materials and print gauges.
External resource. Not made or verified by Public ProtocolMaturity: WorkingSafety S2Open, with paid products or servicesListed 16 November 2026
Parts list
Every resource that supports this element. Those marked External carry the label “External resource. Not made or verified by Public Protocol”.
| Resource | Type | Origin | Maturity | Safety | Licence, as stated at its source |
|---|---|---|---|---|---|
| DCASE 2024 Task 2: First-shot unsupervised anomalous sound detection for machine condition monitoring | Dataset | External | Working | S0 | Not stated on the page opened |
| Humanoid Robots: Vision and Reality (IFR position paper) | Framework | External | Not assessed | S0 | Not stated on the page opened |
| Isaac Lab | Tool | External | Working | S0 | BSD-3-Clause, with certain parts under Apache-2.0 |
| LeRobot | Library | External | Working | S2 | Apache-2.0 |
| Meta FAIR robotics release: Digit 360, Sparsh and Digit Plexus | Reference design | External | Prototype | S0 | Not stated on the page opened |
| MLPerf Tiny Benchmark (paper) | Paper | External | Established | S0 | Not stated on the page opened |
| Open X-Embodiment: Robotic Learning Datasets and RT-X Models | Paper | External | Working | S0 | CC BY 4.0 (paper) |
| openpi | Repository | External | Prototype | S0 | Apache-2.0 |
| OpenVLA: An Open-Source Vision-Language-Action Model | Paper | External | Prototype | S0 | CC BY 4.0 (paper); model licence not stated on the page opened |
| SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics | Paper | External | Prototype | S0 | Not stated on the page opened |
| Standard Open SO-100 and SO-101 Arms | Reference design | External | Working | S2 | "Apache-2.0 license", as the repository states |
| TensorFlow Lite for Microcontrollers | Library | External | Established | S0 | Apache-2.0 |
| Ultralytics YOLO models | Library | External | Established | S0 | Not stated on the page opened |
Depends on, and completes
Depends on 13.6.
It is one of 34 elements that complete Manufacturing and repair.
The standing questions
Who collects the demonstrations, and are they credited. Who checks a model before it runs. Who is accountable when a learned behaviour goes wrong.
Environments
No source found on how learned models perform across islands, cold, heat, humid forest, altitude or flood plains. Models trained in one setting are tested again in each new setting before use.
Safety level S3 where a learned model moves a machine near people. This text states the requirement and names sources; it is not a procedure.
Safety
State and evidence
Drafted
An editor has written the element's text from the resources listed under it, citing them. It is checked when it is reviewed.
Evidence from a real place or system is recorded here, with what did not work. What the states mean
History
- Drafted in the release of 16 November 2026.