# PI-0

## 1. What is PI0 and why KNX needs it

PI0 is a Large Behavior Model (LBM) for robotic manipulation. It takes a history of observations (camera images, joint states) and predicts low-level actions (e.g., torques or joint deltas). It’s trained on large, diverse demonstrations across multiple labs and robot platforms.

In the LeRobot ecosystem, PI0 is distributed as a ready-to-use policy:

* Base model on Hugging Face: `lerobot/pi0_base`

Why this matters for KNX:

* An open-source “brain” for manipulators you can run locally.
* A convenient baseline to compare against other VLA/LBM models.
* Useful for benchmarking KNX-related simulation scenes.

***

## 2. GPU server prep (A10 on vast.ai or similar)

Recommended minimum:

* GPU: NVIDIA A10 (24 GB) or better
* CPU: 8+ vCPU
* RAM: 32 GB
* OS: Ubuntu 20.04 / 22.04
* NVIDIA drivers + CUDA 12.x installed (common on vast.ai images)

Assumptions:

* You have SSH access, e.g.:

```bash
ssh USERNAME@your-server-ip
```

* Working under `/data`.

***

## 3. Miniconda and the `lerobot` environment

```bash
cd /data

# Install Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p /data/miniconda

# Activate conda in this shell
eval "$(/data/miniconda/bin/conda shell.bash hook)"
conda init
exec $SHELL

# Create env
conda create -y -n lerobot python=3.10
conda activate lerobot
```

Set up Hugging Face cache:

```bash
export HF_HOME=/data/hf_home
mkdir -p "$HF_HOME"
```

(Add these to `~/.bashrc` if you want them persistent.)

***

## 4. System deps (MuJoCo/robosim, ffmpeg, OpenGL)

```bash
sudo apt update
sudo apt install -y \
  git \
  ffmpeg \
  libgl1-mesa-glx \
  libglib2.0-0 \
  libglfw3 \
  libosmesa6
```

MuJoCo is pulled via Python packages, but GL libraries are still needed.

***

## 5. Install LeRobot from source

```bash
cd /data
git clone https://github.com/huggingface/lerobot.git
cd lerobot

conda activate lerobot
# LeRobot + policies (includes PI0)
pip install -e ".[policies]"
```

If `hf-egl-probe` build fails due to missing `cmake`:

```bash
pip install cmake
pip install hf-egl-probe
```

***

## 6. Smoke test: run PI0 on Aloha (example)

Create `eval_pi0_aloha_live.py` under `/data/lerobot`:

```python
# file: eval_pi0_aloha_live.py
import torch

from lerobot.envs.factory import make_env
from lerobot.policies.pi0.policy import PI0Policy


def main():
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # Example: Aloha insertion task (ensure gym_aloha is installed)
    env = make_env(
        env_type="gym_aloha",
        task_id="AlohaInsertion-v0",
        obs_type="image",
        render_mode="rgb_array",
    )

    policy = PI0Policy.from_pretrained("lerobot/pi0_base")
    policy.to(device)
    policy.eval()

    num_episodes = 5
    for ep in range(num_episodes):
        obs, info = env.reset()
        done = False
        step_idx = 0
        while not done:
            with torch.no_grad():
                action = policy.act(obs, device=device)
            obs, reward, terminated, truncated, info = env.step(action)
            done = bool(terminated or truncated)
            step_idx += 1
        print(f"Episode {ep} finished, steps={step_idx}, success={info.get('is_success', False)}")


if __name__ == "__main__":
    main()
```

Run:

```bash
cd /data/lerobot
conda activate lerobot
python eval_pi0_aloha_live.py
```

Expected:

* `lerobot/pi0_base` weights download from Hugging Face.
* Several episodes in simulation; success flag printed per episode.

***

## 7. KNX demo script: run and record a video

Save `pi0_run_and_record.py`:

```python
# file: pi0_run_and_record.py
import os
import cv2
import torch
import numpy as np

from lerobot.envs.factory import make_env
from lerobot.policies.pi0.policy import PI0Policy


def main():
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    env = make_env(
        env_type="gym_aloha",
        task_id="AlohaInsertion-v0",
        obs_type="image",
        render_mode="rgb_array",
        fps=30,
    )

    policy = PI0Policy.from_pretrained("lerobot/pi0_base")
    policy.to(device)
    policy.eval()

    out_path = "pi0_aloha_episode.mp4"
    fps = 30
    frames = []

    obs, info = env.reset()
    done = False
    step_idx = 0

    while not done:
        with torch.no_grad():
            action = policy.act(obs, device=device)
        obs, reward, terminated, truncated, info = env.step(action)
        done = bool(terminated or truncated)
        frame = env.render()
        if frame is not None:
            frames.append(frame)
        step_idx += 1

    print(f"Episode finished, steps={step_idx}, success={info.get('is_success', False)}")

    if len(frames) == 0:
        print("No frames collected, check env.render()")
        return

    h, w, _ = frames[0].shape
    writer = cv2.VideoWriter(
        out_path,
        cv2.VideoWriter_fourcc(*"mp4v"),
        fps,
        (w, h),
    )
    for f in frames:
        bgr = cv2.cvtColor(f, cv2.COLOR_RGB2BGR)
        writer.write(bgr)
    writer.release()
    print(f"Saved video to {out_path}")


if __name__ == "__main__":
    main()
```

Run:

```bash
cd /data/lerobot
conda activate lerobot
python pi0_run_and_record.py
```

Use the resulting `pi0_aloha_episode.mp4` in KNX docs or demos.

***

## 8. Pros and cons for KNX

Pros:

* Fully open model and weights (`lerobot/pi0_base`).
* Runs comfortably on a single A10.
* Integrated with LeRobot (multi-dataset, multiple policies).
* Strong baseline across different sim scenes (robosuite, LIBERO, gym\_aloha).

Cons:

* Not a text-conditioned VLA that follows arbitrary prompts; it’s a strong behavior policy for specific tasks.
* Without tailored integration to KNX scenes, looks more like a classic RL/IL baseline than a generalist KNX agent.
* Requires GPU and simulator setup, which is non-trivial for casual users.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.konnex.world/supported-ai-models/pi0.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.