> For the complete documentation index, see [llms.txt](https://docs.konnex.world/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.konnex.world/supported-ai-models/openvla-oft.md).

# OpenVLA · OFT

## 1. What is OpenVLA-OFT?

OpenVLA-OFT is a set of methods and code for parameter-efficient fine-tuning (OFT = Orthogonal/Offset Fine-Tuning) on top of the base OpenVLA 7B model. The goal is to cheaply adapt the same model to new domains/tasks by training small add-on parameterizations instead of touching all weights.

Official resources:

* Website: `https://openvla-oft.github.io`
* Code: `https://github.com/moojink/openvla-oft`

Conceptually:

* Start with the base 7B OpenVLA model.
* Add small OFT components (akin to PEFT/LoRA-family ideas).
* Train only those components on new data (e.g., LIBERO variants or new scenes).
* At inference, the base model + OFT layers together provide adapted behavior.

***

## 2. Architecture and differences vs. “vanilla” OpenVLA

Key differences:

* Depends on a custom Transformers fork: `transformers-openvla-oft` that knows how to load/use OFT layers.
* Checkpoints like `moojink/openvla-7b-oft-finetuned-libero-10` bundle the base model plus OFT parameters.
* The `openvla-oft` repository contains:
  * training scripts,
  * evaluation scripts on LIBERO,
  * configs for different modes (e.g., L1 regression vs. diffusion action decoders).

For KNX, this means you can:

* Efficiently adapt model behavior to your sim/robots without full retraining.
* Use the existing OFT checkpoints as strong baselines on LIBERO and extend from there.

***

## 3. Installation: a working pipeline

Below is a setup that ran `run_libero_eval.py` end-to-end without errors.

### 3.1. Conda environment

```bash
conda create -n openvla_oft python=3.10 -y
conda activate openvla_oft
```

### 3.2. PyTorch

```bash
pip install --index-url https://download.pytorch.org/whl/cu121 \
  "torch==2.2.0" "torchvision==0.17.0" "torchaudio==2.2.0"
```

Check:

```bash
python -c "import torch; print('torch:', torch.__version__)"
# torch: 2.2.0+cu121
```

### 3.3. Clone the repo and base install

```bash
cd /data
git clone https://github.com/moojink/openvla-oft.git
cd openvla-oft

pip install -e .
```

This will install part of the dependencies, but some will remain missing or have mismatched versions.

***

## 4. Installing additional dependencies

In practice, you’ll need the following (versions chosen for compatibility):

```bash
pip install accelerate==0.28.0 \
  diffusers==0.30.3 \
  einops \
  fastapi \
  huggingface_hub \
  imageio \
  json-numpy \
  jsonlines \
  matplotlib \
  "peft==0.11.1" \
  protobuf \
  rich \
  "sentencepiece==0.1.99" \
  "tensorflow==2.15.0" \
  "tensorflow_datasets==4.9.3" \
  "tensorflow_graphics==2021.12.3" \
  "timm==0.9.10" \
  "tokenizers==0.19.1" \
  uvicorn \
  wandb \
  "draccus==0.8.0"
```

Then install the custom Transformers fork and `dlimp`:

```bash
pip install "transformers @ git+https://github.com/moojink/transformers-openvla-oft.git"
pip install "dlimp @ git+https://github.com/moojink/dlimp_openvla.git"
```

Sanity checks:

```bash
python -c "import transformers, draccus, diffusers; print('OK')"
python -c "import tensorflow as tf; print('tf:', tf.__version__)"
# tf: 2.15.0
```

***

## 5. LIBERO + simulators (robosuite, MuJoCo)

Same as with base OpenVLA:

```bash
cd /data
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -e .
```

Then:

```bash
pip install mujoco==2.3.7
pip install "robosuite<2.0.0"
pip install gym==0.26.2 gym-notices
pip install pyopengl glfw opencv-python pynput easydict bddl
```

NumPy/TF conflict (same story):

```bash
pip install "numpy>=1.23.5,<2.0.0"
```

If `opencv-python` attempts to bump NumPy to 2.x, re-pin NumPy (e.g., `numpy==1.26.4`).

***

## 6. Environment variables

Use the same headless-friendly configuration; adjust `PYTHONPATH` for `openvla-oft`:

```bash
export HF_HOME=/data/hf_home
export HF_HUB_CACHE=/data/hf_home/hub
export TRANSFORMERS_CACHE=/data/hf_home/hub
export HF_MODULES_CACHE=/data/hf_home/modules

export PYTHONPATH=/data/LIBERO:/data/openvla-oft:$PYTHONPATH

export MUJOCO_GL=egl
export MUJOCO_EGL_DEVICE_ID=0
export TOKENIZERS_PARALLELISM=false
```

***

## 7. Test: `run_libero_eval.py` with an OFT checkpoint

Run the official evaluation script:

```bash
cd /data/openvla-oft
conda activate openvla_oft

python experiments/robot/libero/run_libero_eval.py \
  --model_family openvla \
  --pretrained_checkpoint moojink/openvla-7b-oft-finetuned-libero-10 \
  --task_suite_name libero_10 \
  --use_l1_regression True \
  --use_diffusion False \
  --use_film False \
  --num_images_in_input 2 \
  --use_proprio True \
  --center_crop True \
  --num_open_loop_steps 8 \
  --num_trials_per_task 1 \
  --env_img_res 256 \
  --local_log_dir ./experiments/logs_oft_libero10
```

Expected output:

* TensorFlow logs about CPU optimizations (fine).
* robosuite warnings about missing “private macro file” (fine).
* episodes progress and video saving under `./rollouts/...`.

If you see:

```
ValueError: Could not find a backend to open `...mp4` with iomode `w?`.
...
FFMPEG: pip install imageio[ffmpeg]
```

Install:

```bash
pip install "imageio[ffmpeg]"
```

***

## 8. Behavior: hi-res vs. low-res and seeds

We’ve observed small discrepancies between low-res control runs and hi-res replays:

* e.g., an object slightly tilted in one video but upright in another, even with the same `actions_log`.

Reasons:

* `OffScreenRenderEnv` and robosuite/MuJoCo can produce slightly different numeric trajectories with different resolutions/backends.
* Seeds:
  * we set `env.seed(0)` and global `set_seed_everywhere(cfg.seed)`,
  * but additional randomness in LIBERO/robosuite can exist if not fixed everywhere.
* Over long rollouts, tiny integration differences can accumulate.

Practically:

* For demos, this is acceptable—intent is preserved.
* For strict replication and comparisons:
  * fix seeds everywhere,
  * ensure identical init states and physics params,
  * consider keeping the same resolution for control and replay.

***

## 9. Pros and cons for KNX

Pros:

* Parameter-efficient fine-tuning:
  * no need to train all 7B parameters,
  * cheaper and faster adaptation to new tasks.
* Ready OFT checkpoints for LIBERO (e.g., `moojink/openvla-7b-oft-finetuned-libero-10`) with strong performance.
* Great fit for KNX:
  * base OpenVLA = “universal brain”,
  * OFT layers = “stickers” for specific robots/business use-cases.

Cons:

* Heavier dependency stack than base OpenVLA:
  * TensorFlow + TF-graphics + TFDS,
  * diffusers,
  * custom Transformers fork,
  * `dlimp`, etc.
* Requires careful env assembly, especially NumPy/TF/OpenCV versions.
* Research-oriented code and docs; for production KNX you’ll still need orchestration, safety layers, and integration with real robots and web infra.