# Konnex Docs
**Konnex** is a **decentralized network** for **verified physical work**: a coordination layer where autonomous systems submit **signed tasks**, miners compete on execution, **independent validators** score outcomes, and **Proof-of-Physical-Work (PoPW)** records settle **onchain**.
The bet is simple: in regulated and high-liability settings, the bottleneck is no longer only model quality — it is **attestation**. Downstream parties (auditors, insurers, counterparties) need **cryptographic evidence** and **independent** evaluation, not operator-signed logs from the same team that ran the machine.
> **Testnet is live:** a Substrate L1, public RPC and explorer, **testKNX** via the faucet, and **three workload-class subnets** — **drone navigation**, **roboarm VLA**, **SLAM 3D map** — with end-to-end task flow. Onboarding: [Snag / quest flow](https://subnets.testnet.konnex.world/quest) and [GitHub releases](https://github.com/konnex-network).
## At a glance
1. A **task instruction** is defined and hashed onchain.
2. **Miners** submit candidate policies, trajectories, or reconstructions.
3. **Validators** (with no stake in a single operator’s commercial outcome) **score** execution on safety, task match, and efficiency.
4. A **PoPW** artefact links instruction, **policy trace**, **hardware-rooted** sensor data, and scores — visible on the [explorer](https://subnets.testnet.konnex.world/explorer).
## Inheritance and specialization
Konnex **inherits the proven coordination model** of networks like [Bittensor](https://docs.learnbittensor.org/) — subnet creators, miners, validators, and stakers — so experienced **node operators** can onboard with a familiar mental model. What changes is the **work**: physical-world workloads need **PoPW** and enterprise-grade **settlement** design, not only statistical sampling of text outputs.
## Key links
| | |
| -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Testnet** (quest, faucet, explorer) | [Quest](https://subnets.testnet.konnex.world/quest) · [Faucet](https://subnets.testnet.konnex.world/faucet) · [Explorer](https://subnets.testnet.konnex.world/explorer) |
| **Subnets** (drone navigation, roboarm VLA, SLAM 3D map) | Dashboard URLs — to be published with the testnet release |
| **Whitepaper** | [konnex.world/whitepaper](https://konnex.world/whitepaper) |
| **Source** | [github.com/konnex-network](https://github.com/konnex-network) |
> **Security:** Use only official links and binaries from this documentation, the whitepaper, or [GitHub releases](https://github.com/konnex-network). Do not trust lookalike domains or installers.
## Deep dives
* [Design overview](/understand-konnex/design-overview)
* [Protocol architecture](/understand-konnex/protocol-architecture)
* [PoPW, TEE, and settlement design](/understand-konnex/contracts-and-popw)
* [Roadmap: what is live vs staged](/understand-konnex/roadmap)
For builders: [SDK overview](/sdk/sdk), [mining](/participate/mining), [validating](/participate/validating).
# Design overview
## The problem: attestation, not raw capability
Autonomous systems can already execute impressive **physical** work: drone inspection, robotic manipulation, 3D reconstruction. In many deployments, the **execution record** is **operator-attested** — produced and signed by the same party that ran the machine. That is acceptable in closed, low-stakes environments; it is **structurally insufficient** when the **consumer** of the record carries **liability**: a regulator, an insurer pricing risk, or an engineering sign-off on a multi-million-dollar decision.
The missing piece is **independent verification**: evidence and scoring that are **uncorrelated** with the operator’s self-interest, backed by **cryptography** and **clear economic penalties** for fraud.
Konnex addresses this with a **decentralized network** where:
* **Task instructions** are **signed and anchored** onchain.
* **Miners** compete to execute under subnet rules.
* **Validators** with **no operational stake** in a given operator score execution against the instruction.
* **Proof-of-Physical-Work (PoPW)** binds policy traces and **sensor evidence** (including **hardware-level** attestation where applicable) into a **single** verifiable record.
## Workload classes (subnets), not one generic “chat” market
Konnex is organized by **workload class** — each subnet corresponds to a real commercial pattern (aerial inspection, manipulation, 3D/SLAM) with its own **input schema** and **scoring function**. That allows:
* **Enterprises** to negotiate scoring that reflects real contracts, not only synthetic leaderboards.
* **Miners** to **specialize** in one class (e.g. SLAM) without competing on every other robot modality.
**Live on testnet today:** [drone navigation](/subnets-workload-classes/drone-navigation), [roboarm VLA](/subnets-workload-classes/roboarm-vla), and [SLAM 3D map](/subnets-workload-classes/slam-3d-map). **Signed enterprise counterparties** are active in these workload classes; their **full** commercial execution may still run offchain in their own environments while the testnet demonstrates the **verification and settlement** layer that **mainnet** is meant to host.
## Two protocol layers in the long-run design
1. **Coordination and verification** — Mesh-friendly transport, onchain registries, PoPW, validator economics (see [Protocol architecture](/understand-konnex/protocol-architecture)). **Mesh transport** and full **robot–robot** contract products are **staged**; see the [roadmap](/understand-konnex/roadmap).
2. **Settlement** — **Stablecoin-denominated** escrows, penalties, and rewards for **enterprise** users, with **KNX** for **protocol** security, governance, and **gas routing** — a **dual-token** design so operational teams can budget in **fiat-pegged** terms. Full stablecoin mainnet dual-staking is not all live on the first public testnet; see [Stablecoins integration](/understand-konnex/stablecoins-integration) and the [roadmap](/understand-konnex/roadmap).
## Inheritance from Bittensor-style subnet economics
Konnex **inherits the network topology and incentive *****shape***** familiar from** production-style **subnet** networks such as [Bittensor](https://docs.learnbittensor.org/) — subnet creators, miners, validators, and stakers — so experienced **node operators** can onboard with minimal relearning. The **workload and verification** layer is **different** by design: physical tasks need [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw), not only statistical quality of text outputs.
## Where to start
* [Proof-of-Physical-Work (PoPW)](/understand-konnex/contracts-and-popw)
* [KNX tokenomics](/understand-konnex/tokenomics) — fixed 1B supply, four allocation buckets, release horizon
* [Stablecoins integration](/understand-konnex/stablecoins-integration)
### Example policy (abridged, manipulation-style)
```
1. Move arm to a safe home pose.
2. Open fridge → smooth_move to handle position.
3. Verify ingredients; if missing, request restock or abort with proof.
4. Grasp object using camera feedback; respect torque limits.
5. Place in target container; log thermal and contact traces per subnet policy.
```
The **subnet** turns this class of task into a concrete schema, simulator or sensor checks, and **PoPW** requirements.
# Protocol architecture
The Konnex stack aligns communication, contracts, intelligence, and motion. Every step — broadcast, bidding, execution, proof, payout — flows through four deterministic layers and one economic loop. Settlement is stablecoin‑native; security, governance, and fees are in KNX. Validators and (optionally) executors post dual stakes (KNX + stablecoins).
## Architecture diagram
## Mesh & Gossip
libp2p + QUIC, NAT‑friendly, always‑on.
* Task Channel `task.*` — signed JSON intents.
* Bid Channel `bid.*` — bids with ETA and collateral declarations.
* Score Channel `score.*` — validator ScoreRoot each block (KPIs, slashing directives).
* Proof Channel `proof.*` — batched evidence references (video/GPS/IMU/torque/thermal).
## Registry & Smart Contracts
* `RobotIdentity` — hardware‑secured keys and on‑chain trust.
* `TaskRegistry` — stablecoin escrow, deadlines, penalties, fee‑split logic.
* `StakeVault` — dual staking (KNX + stablecoins) for validators/executors; class‑based slashing.
* `BondMatrix` — stablecoin bonds from third‑party stakers.
* `PayoutRouter` — atomic stablecoin releases after validation.
Every packet’s SHA‑3 hash becomes its immutable JobID.
## Intelligence & Motion Markets
Two miner roles; one bidding protocol.
* Motion Miners — trajectories in deterministic sandbox (Bullet3D + seeded RNG).
* AI Miners — perception/planning models as WASM weights or streaming endpoints.
## Verification & PoPW
Validator council, deterministic replay.
1. Deterministic replay of trajectories.
2. Model audits vs ground truth.
3. PoPW check — evidence completeness, alignment, binding to JobID/deadline.
4. Slash/Reward — write ScoreRoot; release stablecoin escrow or apply stablecoin penalties in the same block (fees in KNX).
### Dual‑Stake for Validators
* KNX security stake — slashed on protocol/consensus faults.
* Stablecoin assurance stake — compensates users for settlement‑layer faults (e.g., approving invalid PoPW).
Fast‑finality targets: <10 s for sim‑only; <60 min with full PoPW.
## Economic Loop
* Stablecoins — settlement and validator assurance.
* KNX — security and fees; buyback flywheel via small stablecoin flow fee. Supply, allocation, and unlock schedule: [Tokenomics](/understand-konnex/tokenomics).
Trust‑adjusted stake sizing (illustrative): `S_KNX ∝ 1/√T`, `S_stable ∝ 1/√T`.
# Proof-of-Physical-Work (PoPW), TEE, and contracts
> **Spelling:** The verification primitive is **Proof-of-Physical-Work (PoPW)**. Do not confuse it with unrelated “proven work” phrasing.
## What PoPW binds
**Proof-of-Physical-Work (PoPW)** is the Konnex verification primitive for physical-world jobs. A PoPW record combines four elements into a coherent onchain (or onchain-anchored) artefact:
1. **Task instruction** — Agreed at issue time, signed by the operator, content-hashed and anchored onchain.
2. **Policy execution trace** — The sequence of actions the robot’s policy (or full stack) produced in response to that instruction.
3. **Sensor bundle** — Pose, frames, IMU, torque, temperature, and other channels the subnet’s verification function requires. Where enterprise assurance applies, critical telemetry is signed in hardware (TEE or secure element) *before* it reaches a general-purpose host OS, so validators can detect post-hoc synthesis or replay.
4. **Independent scoring** — Validators with no operational stake in the operator score safety, task match, efficiency, and coherence with the instruction and subnet rules.
Downstream users—regulators, insurers, customers, other robots—can line up a single story: this instruction, this trace, this signed evidence, these scores, with instruction hashes and commitments onchain.
Independent scoring is not enough if an attacker can spoof sensors at the source. The full design therefore assumes two additional security layers:
### Hardware-rooted provenance (TEE / secure elements)
If the bits are fake, off-device verification does not help. The first line of defense is on the device: trusted execution environments and secure elements sign telemetry and media at capture time. Validators check signatures, freshness, and binding to the task so that synthetic or replayed streams fail verification.
### Economic finality and asymmetric risk
Hardware roots the data; economics constrains behavior. Miners and validators are intended to post stake in KNX and, on mainnet, stablecoin assurance tranches. Slashing for proven misbehavior—collusion, fraud, or bypassing attestation—is designed so the expected cost of attack exceeds the plausible upside from faking a job.
Together, these are what make PoPW a *physical* verification primitive, not a renamed “best-of-N text” competition.
## Why inference networks are a different game
Open decentralized networks for inference excel when the output is a token stream and quality is a statistical property. Physical work fails differently: the drone flew to the wrong structure, the arm grasped the wrong object, the mesh invented geometry. Those are safety, task-match, and provenance errors. Verification has to follow the work: sensor traces, timestamps, and binding to a signed task—not only benchmark scores on sampled outputs.
## Subnets and workload classes
Each subnet encodes a commercial workload class (schema, scoring, and miner competition). See the [subnets overview](/subnets-workload-classes/subnets). Subnet app URLs are published with official testnet releases.
## Robot-to-robot smart contracts and stablecoin escrows (roadmap)
> **Scope:** Inter-robot smart contracts with native stablecoin escrow, deadlines, and penalties on mainnet are part of the long-term product surface. They are *not* guaranteed in the first public testnet build. For what runs today, see the [roadmap](/understand-konnex/roadmap) and official **release notes**.
The intended job flow is:
1. **Offer** — Post a job with reward and deadline (stablecoin on mainnet).
2. **Bid** — Robots or fleets respond with ETA, location, and deposits.
3. **Match** — Parties lock escrow and deposits onchain.
4. **Execute** — Worker records sensor snapshots per policy.
5. **Prove** — Submit a PoPW bundle bound to the JobID.
6. **Settle** — Validators verify; release funds or apply penalties.
What counts as proof is workload-specific: delivery (path plus arrival media), process control (thermal logs and camera stills), inspection (imagery of named elements), and so on. Disputes, missed deadlines, and bad evidence trigger slashing and refunds according to the contract and validator layer.
## See also
* [Protocol architecture](/understand-konnex/protocol-architecture)
* [Stablecoins integration](/understand-konnex/stablecoins-integration)
* [Validating](/participate/validating)
# Validator metascore (design)
On Konnex, **validators score miners**: they rank physical work against the task instruction and subnet policy using replays, sensors, benchmarks, and—where the workload requires it—**AI-assisted scoring** (e.g. kinematic smoothness, grasp success, VLA behaviour). Choosing a good scoring function is therefore security-critical.
Plain smart-contract checks alone do not close the loop: **AI scoring can hallucinate**, and **validators can collude**. The protocol treats “who validates the validators” as a first-class problem and implements a **hybrid, three-layer metascore** on the Substrate-based chain.
## Layer 1 — Weight alignment (peer consensus on scores)
The primary cryptoeconomic signal for validator quality is **agreement with the rest of the validator set** on the same miner outcomes.
Each validator submits a **weight vector** (a list of scores) over a batch of miners or tasks. The chain builds a **global consensus vector**—the same dimensions, aggregated across validators—using a governance-tunable rule such as **median per dimension** or a **stake-weighted** blend. Call validator *i*’s vector `W_i` and the network aggregate `W-bar`. When `W_i` tracks `W-bar`, that validator’s standing improves; **systematic drift** away from the aggregate cuts **validator trust (vTrust)** and **lowers effective APY**. The consensus term in the metascore (below) is the formal similarity between `W_i` and `W-bar`.
**Lazy copy-paste consensus** (everyone mirroring the same scores without doing work) is mitigated by **pseudorandom hidden spot checks**: validators cannot assume which submissions are audited only by alignment, so blind copying remains risky.
## Layer 2 — Honeypots (deterministic reference tasks)
Robotics workloads have a structural advantage: **simulators** (e.g. ManiSkill, Isaac Sim, subnet-specific benches) can produce **deterministic ground-truth metrics**.
The network (or a governed oracle module) **injects honeypots** into the task mix—runs whose correct grades are known **before** validators vote. Validators do not know which jobs are honeypots.
If a validator’s AI stack **awards high scores to an objectively failed reference** (e.g. dropped object marked as success), the protocol applies a **hard penalty**, up to **slashing** on severity tiers defined by governance.
## Layer 3 — Two-tier scoring inside the validator node
Each validator’s **local scoring pipeline** is designed as **two independent channels** that must agree before a vote is safe to publish:
1. **VLA / LLM layer** — High-level task understanding (did the miner follow the instruction semantically?).
2. **Deterministic layer** — Lightweight heuristics or closed-form checks: torque limits, timing windows, joint limits, replay deltas against sim logs, etc.
If the two layers **conflict**, the validator **abstains** or **requests an extended check** (subnet-defined escalation path). Publishing a confident vote despite internal disagreement is scored against the validator under Layer 1 and Layer 2.
## Onchain metascore (L1)
At the base layer the protocol combines alignment, honeypots, and penalties into a single **validator metascore** for each validator (written `S(V_i)` in the figure):
Read the formula left to right:
* `S(V_i)` — The scalar metascore for validator *i*; higher is better after governance normalization.
* `C(W_i, W-bar)` — **Consensus term**: how close validator *i*’s score vector `W_i` is to the network’s consensus vector `W-bar` (for example cosine similarity, or another distance mapped to a score on the closed unit interval from 0 to 1).
* `H(V_i)` — **Honeypot accuracy**: share (or count) of hidden reference tasks where *i*’s grades match the known-good labels.
* `P_i` — **Penalties** for operational faults: high latency, missed voting epochs, broken commitments, and similar slashing-adjacent behaviour.
* **α, β, γ** (alpha, beta, gamma) — Positive **governance weights** that set how much consensus, honeypots, and penalties move `S(V_i)`. They are tuned per subnet phase and mainnet rollout.
## Relation to mining and PoPW
Miner rewards and PoPW acceptance still flow from **subnet verifiers and task semantics**. The metascore **does not replace** workload-specific scoring; it **governs who is trusted to emit those scores** and at what economic weight.
## See also
* [Validating on Konnex](/participate/validating)
* [Proof-of-Physical-Work (PoPW), TEE, and contracts](/understand-konnex/contracts-and-popw)
* [Protocol architecture](/understand-konnex/protocol-architecture)
# Stablecoins integration (mainnet design)
Konnex is a robot‑native network where every task, escrow, penalty, and reward settles in stablecoins by default. Stablecoins provide a predictable payment rail for machine commerce, while KNX remains the network token for validator security, governance, and fees.
## Stablecoin‑Native Flow (overview)
The flow shows:
* Payer tops up (optionally via card on‑/off‑ramp) and locks stablecoins into the task escrow.
* Executor submits PoPW evidence to Validators; a ScoreRoot is posted.
* Escrow settles: payouts or refunds (and penalties) are released; a small fee can be routed to a treasury/buyback in KNX.
## Role Split: Stablecoins vs KNX
* Stablecoins — settlement currency for escrows, payouts, penalties; supports fiat on/off‑ramps and cross‑chain liquidity.
* KNX — staking, governance, and network fees. See [Tokenomics](/understand-konnex/tokenomics) for supply and allocation.
## Contract Settlement Types (Stablecoin‑Native)
* Fixed‑price — escrow a single stablecoin amount and release on success.
* Metered — usage‑based debits (seconds, meters, frames) from escrow.
* Milestone — staged stablecoin releases across checkpoints.
Minimum task fields: `rewardStable`, `stakeStable`, `deadline`, `penaltyStable`, `PoPW` requirement bound to JobID.
## PoPW‑Linked Settlement & Penalties
1. Lock — payer funds on‑chain stablecoin escrow.
2. Perform — executor records sensor evidence (GPS, camera, IMU, torque, temperature).
3. Prove — submit PoPW bundle referencing JobID and deadline.
4. Verify — validators publish a ScoreRoot (pass/fail + metrics).
5. Settle — on pass, release stablecoins; on fail, apply `penaltyStable` and refunds (fees in KNX).
## Multi‑Token Staking & Robotic Bonds
* KNX stake — aligns executors with validator security and enables slashing.
* Stablecoin bond — compensates counterparties for non‑performance.
Third‑party stakers can supply stablecoins to robotic bond pools and earn yield, with risk priced by reputation. Required stablecoin stake roughly scales as `S = kappa / sqrt(T)` where `T` is trust score.
## Cross‑Chain Liquidity
Stablecoins from other chains can be bridged into Konnex via approved routes. Tasks are chain‑agnostic to source funds.
## Card On/Off‑Ramp
1. Charge — pay with card in app.
2. Mint — processor converts fiat to stablecoins to the payer’s Konnex address.
3. Escrow — stablecoins locked in the task contract.
4. Payout — after PoPW verification, stablecoins are released.
5. Redeem/Spend — spend via card, redeem to bank, or reuse on‑chain.
## Compliance by Design
* Optional KYC/AML gates (market/DAO configurable).
* Full audit trail of packets, PoPW, and stablecoin flows.
## Developer Surface (API Sketch)
* `lock(jobId, rewardStable, stakeStable)`
* `prove(jobId, PoPWBundleRef)`
* `settle(jobId)`
* `bond(jobId, amountStable)`
## Economics & Fee Flywheel
* Stablecoins for users, KNX for the network.
* Auto‑buyback: small fee on stablecoin flows swaps to KNX for treasury/burn.
## Key Metrics
* Latency: prove → stablecoin release (target single‑block after validation).
* Disputes: <1% and auto‑resolved.
* Velocity: avg stablecoin tx/robot/day.
* Card share: tasks funded via on‑ramp.
* Collateral efficiency: required `stakeStable` falls as trust `T` rises.
## Solidity Contracts
```solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.24;
import "@openzeppelin/contracts/token/ERC20/IERC20.sol";
import "@openzeppelin/contracts/token/ERC20/utils/SafeERC20.sol";
import "@openzeppelin/contracts/utils/ReentrancyGuard.sol";
import "@openzeppelin/contracts/access/Ownable.sol";
interface IScoreRegistry {
// Returns (pass, score, scoreRoot) for a given jobId
function getVerificationResult(bytes32 jobId) external view returns (bool, uint256, bytes32);
}
interface IAccessController {
function isAllowed(address user) external view returns (bool);
}
interface IFeeCollector {
// Stablecoin fee receiver used for KNX buyback/treasury flows
function receiveStableFee(address token, uint256 amount) external;
}
contract FeeCollector is IFeeCollector, Ownable {
using SafeERC20 for IERC20;
event StableFeeReceived(address indexed token, uint256 amount);
event Sweep(address indexed token, address indexed to, uint256 amount);
// Receives fees in stablecoin; swapping to KNX is implemented off this contract (routers/aggregators vary by deployment)
function receiveStableFee(address token, uint256 amount) external override {
IERC20(token).safeTransferFrom(msg.sender, address(this), amount);
emit StableFeeReceived(token, amount);
}
// Owner can sweep accumulated balances to treasury/buyback executor as needed
function sweep(address token, address to, uint256 amount) external onlyOwner {
IERC20(token).safeTransfer(to, amount);
emit Sweep(token, to, amount);
}
}
contract KonnexSettlementManager is ReentrancyGuard, Ownable {
using SafeERC20 for IERC20;
enum SettlementType {
FixedPrice,
Metered
// Milestone can be added with indexed amounts per checkpoint
}
enum JobStatus {
None,
Locked,
Proven,
Settled
}
struct Job {
// Core IDs and actors
bytes32 jobId;
address payer;
address executor;
// Economic params (stablecoin‑native)
uint256 rewardStable; // gross reward escrowed by payer (stablecoins)
uint256 stakeStable; // executor stake/bond amount in stablecoins (can come from third‑party stakers)
uint256 penaltyStable; // penalty applied on fail, funded from stake/bonds
uint256 deadline; // unix deadline for proof submission
SettlementType settlementType;
// Lifecycle
JobStatus status;
bytes32 proofRef; // e.g., PoPW CID/reference
// Metered: amount debited from escrow over time (<= rewardStable)
uint256 meteredDebited;
// Accounting
uint256 totalThirdPartyBond; // total bonded by third‑party stakers for this job
}
struct BondStake {
uint256 amount;
bool claimed;
}
IERC20 public immutable stableToken; // primary settlement token (e.g., USDC)
IScoreRegistry public scoreRegistry; // validator score/decision source
IAccessController public accessGate; // optional KYC/ACL gate (can be address(0))
IFeeCollector public feeCollector; // receives stable fees for KNX buyback/treasury
// Fees (basis points) taken from executor payout on pass; sent in stable to FeeCollector
uint16 public stableFeeBps = 50; // 0.50%
uint16 public bondYieldBps = 25; // 0.25% yield to third‑party bond providers on pass (taken from reward escrow, before executor payout)
uint16 private constant BPS_DENOM = 10_000;
// jobId => Job
mapping(bytes32 => Job) public jobs;
// jobId => list of bond stakers (to enable distribution on settlement)
mapping(bytes32 => address[]) private bondStakers;
// jobId => staker => BondStake
mapping(bytes32 => mapping(address => BondStake)) public bondLedger;
// Events
event Locked(bytes32 indexed jobId, address indexed payer, address indexed executor, SettlementType stype, uint256 reward, uint256 stake, uint256 penalty, uint256 deadline);
event Bonded(bytes32 indexed jobId, address indexed staker, uint256 amount);
event Proved(bytes32 indexed jobId, address indexed executor, bytes32 proofRef);
event MeteredDebit(bytes32 indexed jobId, uint256 amount, uint256 totalDebited);
event Settled(bytes32 indexed jobId, bool pass, uint256 executorPayout, uint256 penaltyPaid, uint256 feePaid, uint256 bondYieldPaid);
event AccessGateUpdated(address indexed gate);
event ScoreRegistryUpdated(address indexed registry);
event FeeCollectorUpdated(address indexed collector);
event FeesUpdated(uint16 stableFeeBps, uint16 bondYieldBps);
constructor(
address stableToken_,
address scoreRegistry_,
address feeCollector_
) {
require(stableToken_ != address(0) && scoreRegistry_ != address(0) && feeCollector_ != address(0), "zero addr");
stableToken = IERC20(stableToken_);
scoreRegistry = IScoreRegistry(scoreRegistry_);
feeCollector = IFeeCollector(feeCollector_);
}
// Optional: enable/disable KYC/ACL markets
function setAccessGate(address gate) external onlyOwner {
accessGate = IAccessController(gate);
emit AccessGateUpdated(gate);
}
function setScoreRegistry(address reg) external onlyOwner {
require(reg != address(0), "zero addr");
scoreRegistry = IScoreRegistry(reg);
emit ScoreRegistryUpdated(reg);
}
function setFeeCollector(address collector) external onlyOwner {
require(collector != address(0), "zero addr");
feeCollector = IFeeCollector(collector);
emit FeeCollectorUpdated(collector);
}
function setFees(uint16 stableFeeBps_, uint16 bondYieldBps_) external onlyOwner {
require(stableFeeBps_ <= 500 && bondYieldBps_ <= 1000, "fee caps"); // sane caps
stableFeeBps = stableFeeBps_;
bondYieldBps = bondYieldBps_;
emit FeesUpdated(stableFeeBps_, bondYieldBps_);
}
// lock: payer escrows reward; executor escrows stake (directly or via third‑party bonds)
function lock(
bytes32 jobId,
address executor,
uint256 rewardStable,
uint256 stakeStable,
uint256 penaltyStable,
uint256 deadline,
SettlementType stype
) external nonReentrant {
require(jobs[jobId].status == JobStatus.None, "exists");
require(executor != address(0) && rewardStable > 0 && deadline > block.timestamp, "invalid");
_checkAccess(msg.sender);
_checkAccess(executor);
// Create job
Job storage j = jobs[jobId];
j.jobId = jobId;
j.payer = msg.sender;
j.executor = executor;
j.rewardStable = rewardStable;
j.stakeStable = stakeStable;
j.penaltyStable = penaltyStable;
j.deadline = deadline;
j.settlementType = stype;
j.status = JobStatus.Locked;
// Transfer reward escrow from payer to contract
stableToken.safeTransferFrom(msg.sender, address(this), rewardStable);
// If executor supplies stake directly (not via bonds), transfer from executor now
if (stakeStable > 0) {
stableToken.safeTransferFrom(executor, address(this), stakeStable);
}
emit Locked(jobId, msg.sender, executor, stype, rewardStable, stakeStable, penaltyStable, deadline);
}
// Third‑party bond providers can contribute to executor's stake (robotic bonds)
function bond(bytes32 jobId, uint256 amount) external nonReentrant {
Job storage j = jobs[jobId];
require(j.status == JobStatus.Locked, "job not lock");
require(amount > 0, "zero");
_checkAccess(msg.sender);
// Pull funds
stableToken.safeTransferFrom(msg.sender, address(this), amount);
// Track staker and amount
if (bondLedger[jobId][msg.sender].amount == 0) {
bondStakers[jobId].push(msg.sender);
}
bondLedger[jobId][msg.sender].amount += amount;
j.totalThirdPartyBond += amount;
emit Bonded(jobId, msg.sender, amount);
}
// Metered debit during execution (e.g., usage‐based spending from payer escrow)
function debitMetered(bytes32 jobId, uint256 amount) external nonReentrant {
Job storage j = jobs[jobId];
require(j.status == JobStatus.Locked || j.status == JobStatus.Proven, "bad status");
require(j.settlementType == SettlementType.Metered, "not metered");
require(msg.sender == j.payer, "only payer");
require(j.meteredDebited + amount <= j.rewardStable, "exceeds escrow");
j.meteredDebited += amount;
emit MeteredDebit(jobId, amount, j.meteredDebited);
}
// prove: executor submits PoPW reference before deadline
function prove(bytes32 jobId, bytes32 proofRef) external nonReentrant {
Job storage j = jobs[jobId];
require(j.status == JobStatus.Locked || j.status == JobStatus.Proven, "bad status");
require(msg.sender == j.executor, "only exec");
require(block.timestamp <= j.deadline, "deadline");
j.proofRef = proofRef;
j.status = JobStatus.Proven;
emit Proved(jobId, msg.sender, proofRef);
}
// settle: checks validator decision and routes funds accordingly
function settle(bytes32 jobId) external nonReentrant {
Job storage j = jobs[jobId];
require(j.status == JobStatus.Locked || j.status == JobStatus.Proven, "bad status");
(bool pass,,) = scoreRegistry.getVerificationResult(jobId);
uint256 totalStake = j.stakeStable + j.totalThirdPartyBond;
uint256 executorPayout = 0;
uint256 penaltyPaid = 0;
uint256 feePaid = 0;
uint256 bondYieldPaid = 0;
if (pass) {
// Calculate gross payout from reward escrow (metered tasks pay out only the debited portion)
uint256 gross = (j.settlementType == SettlementType.Metered) ? j.meteredDebited : j.rewardStable;
// Bond yield carved out from gross reward for stakers
uint256 yieldForBonds = (j.totalThirdPartyBond > 0) ? (gross * bondYieldBps) / BPS_DENOM : 0;
bondYieldPaid = yieldForBonds;
// Fee on executor payout (after bond yield)
uint256 baseForExec = gross - yieldForBonds;
feePaid = (baseForExec * stableFeeBps) / BPS_DENOM;
executorPayout = baseForExec - feePaid;
// Distribute: bond yield to stakers pro‑rata, fee to FeeCollector, remainder to executor
_payBondProRata(jobId, yieldForBonds, true /* include principal on pass */);
if (feePaid > 0) {
// Send fee in stablecoin to FeeCollector for KNX buyback/treasury
stableToken.safeApprove(address(feeCollector), 0);
stableToken.safeApprove(address(feeCollector), feePaid);
feeCollector.receiveStableFee(address(stableToken), feePaid);
}
if (executorPayout > 0) {
stableToken.safeTransfer(j.executor, executorPayout);
}
// Return executor's direct stake (if any)
if (j.stakeStable > 0) {
stableToken.safeTransfer(j.executor, j.stakeStable);
}
// Refund any unspent escrow to payer (metered remainder or dust)
uint256 spent = gross;
if (j.rewardStable > spent) {
stableToken.safeTransfer(j.payer, j.rewardStable - spent);
}
} else {
// Fail: apply penalty from total stake (executor stake + third‑party bonds), pay to payer
uint256 slashed = j.penaltyStable > totalStake ? totalStake : j.penaltyStable;
penaltyPaid = slashed;
if (slashed > 0) {
// Slash executor's direct stake first, then third‑party bonds
uint256 fromExecStake = slashed > j.stakeStable ? j.stakeStable : slashed;
if (fromExecStake > 0) {
j.stakeStable -= fromExecStake;
stableToken.safeTransfer(j.payer, fromExecStake);
slashed -= fromExecStake;
}
if (slashed > 0 && j.totalThirdPartyBond > 0) {
_slashBondProRata(jobId, slashed);
stableToken.safeTransfer(j.payer, slashed);
}
}
// Refund full reward escrow to payer
if (j.rewardStable > 0) {
stableToken.safeTransfer(j.payer, j.rewardStable);
}
// Return any remaining direct stake to executor
if (j.stakeStable > 0) {
stableToken.safeTransfer(j.executor, j.stakeStable);
}
// Return remaining bonds (principal only, no yield) to stakers
_payBondProRata(jobId, 0, true /* include principal */);
}
j.status = JobStatus.Settled;
emit Settled(jobId, pass, executorPayout, penaltyPaid, feePaid, bondYieldPaid);
// Zero out escrowed amounts to avoid double‑spends in any edge path
j.rewardStable = 0;
j.stakeStable = 0;
j.totalThirdPartyBond = 0;
}
// Internal helpers
function _checkAccess(address user) internal view {
if (address(accessGate) != address(0)) {
require(accessGate.isAllowed(user), "acl");
}
}
function _payBondProRata(bytes32 jobId, uint256 yieldAmount, bool includePrincipal) internal {
address[] memory stakers = bondStakers[jobId];
if (stakers.length == 0) {
return;
}
// Compute total principal remaining for this job
uint256 totalPrincipal = 0;
for (uint256 i = 0; i < stakers.length; i++) {
totalPrincipal += bondLedger[jobId][stakers[i]].amount;
}
if (totalPrincipal == 0 && yieldAmount == 0) {
return;
}
for (uint256 i = 0; i < stakers.length; i++) {
address staker = stakers[i];
BondStake storage b = bondLedger[jobId][staker];
if (b.claimed || b.amount == 0) continue;
uint256 principalShare = b.amount;
uint256 yieldShare = (yieldAmount > 0 && totalPrincipal > 0) ? (yieldAmount * principalShare) / totalPrincipal : 0;
uint256 payout = yieldShare + (includePrincipal ? principalShare : 0);
if (payout > 0) {
// Zero out before transfer to guard reentrancy
b.claimed = true;
b.amount = 0;
stableToken.safeTransfer(staker, payout);
}
}
// Clear staker list to free storage in future deployments (optional micro‑opt)
delete bondStakers[jobId];
}
function _slashBondProRata(bytes32 jobId, uint256 slashAmount) internal {
address[] memory stakers = bondStakers[jobId];
if (stakers.length == 0 || slashAmount == 0) {
return;
}
uint256 totalPrincipal = 0;
for (uint256 i = 0; i < stakers.length; i++) {
totalPrincipal += bondLedger[jobId][stakers[i]].amount;
}
if (totalPrincipal == 0) return;
for (uint256 i = 0; i < stakers.length; i++) {
address staker = stakers[i];
BondStake storage b = bondLedger[jobId][staker];
if (b.amount == 0) continue;
uint256 slashShare = (slashAmount * b.amount) / totalPrincipal;
if (slashShare > b.amount) slashShare = b.amount;
b.amount -= slashShare;
}
}
}
```
### Notes
* The `KonnexSettlementManager` implements:
* `lock(jobId, executor, rewardStable, stakeStable, penaltyStable, deadline, stype)` — payer escrows reward; executor may escrow stake directly; third‑party stakers add stake via `bond`.
* `bond(jobId, amount)` — third‑party stakers add stablecoin bonds that secure execution and earn yield on pass.
* `prove(jobId, proofRef)` — executor posts a PoPW bundle reference before `deadline`.
* `debitMetered(jobId, amount)` — optional usage‑based debits for metered contracts.
* `settle(jobId)` — reads validator decision from `IScoreRegistry`, applies penalties or releases funds, sends a stablecoin fee to `FeeCollector` for KNX buyback/treasury.
* The `FeeCollector` is a simple stablecoin sink. Teams typically wire this to a swap/buyback executor off‑chain or via per‑chain router adapters.
* To add milestones, track `uint256[] milestoneAmounts` and a checkpoint index per job, and debit per milestone similarly to `debitMetered`.
# Decentralized AI ecosystem
Konnex treats intelligence for physical work as a first-class, onchain commodity. Models and policies have owners, versions, stake, reputation, and—where applicable—deterministic checks in subnet sandboxes or on real hardware.
## Roles
The roles follow the same **subnet** coordination pattern seen in networks such as [Bittensor](https://docs.learnbittensor.org/):
* **Miners** — Supply control or perception policies for a workload class; stake and compete for tasks.
* **Users / operators** — Issue tasks (natural language or schema) and pay in testKNX on testnet; stablecoins on mainnet when the [settlement design](/understand-konnex/stablecoins-integration) is live in production.
* **Validators** — Score outputs independently; they are not a proxy for the operator’s commercial self-interest.
* **Stakers** — Back validators and future bond pools per network rules.
* **Simulator / bench** — Where the subnet requires deterministic replays.
## Lifecycle of a policy or model version
1. **Publish** — Register a build with metadata, KPIs, and stake.
2. **Compete** — Tasks pull miner outputs into the open market for the subnet.
3. **Gate** — Validators run replay, sims, or sensor checks; unsafe or out-of-spec work is rejected and can be slashed.
4. **Deploy / record** — Winners execute as defined; telemetry feeds [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw).
5. **Settle** — Rewards and reputation update; on mainnet, stablecoin settlement applies when enabled.
## Public model registry (conceptual)
A stable identifier (e.g. hash of code, hyperparameters, and author key) with immutable metadata and governance-controlled fields such as deprecation and royalty routing.
## Feedback loop
Better KPIs and safety behavior lead to higher validator scores and, where the economy allows, larger rewards and lower required collateral. Fraud or gross negligence triggers slashing and makes counterparties whole per parameterized rules.
For concrete VLA integrations, see [AI models — overview](/supported-ai-models/ai).
# Glimpse of the Future (2033)
## 
## Glimpse of the Future (2033)
Robots don’t report to the cloud — they negotiate with each other and decentralized AI miners across the Konnex mesh. A single signed task packet can spark a short‑lived micro‑economy that settles on‑chain in stablecoins.
### City‑Wide Drone Swarm — Deliver 1,200 lunches before noon
* A food‑hub posts one intent: reward 150,000 stablecoins, 600 locations, deadline 12:00.
* Drones on `task.delivery.*` return micro‑bids with battery, GPS, and small stablecoin collateral.
* Five high‑trust drones coordinate zones, issue sub‑contracts, and embed penalty clauses.
* At 11:57 all receipts land; escrows release in stablecoins; late collateral is redistributed.
### Home Robo‑Kitchen — Chef‑bot, cook shakshuka
* You say: “Make shakshuka for two.” The arm pings `ai.culinary.*` for a model.
* Miners stake on KPIs and stream WASM weights; the robot sim‑benchmarks and selects a model, paying in stablecoins and bonding an execution escrow.
* Cooking is offline; telemetry is hashed every second. After PoPW verification, stablecoin escrow unlocks and royalties are paid.
### Bridge Inspection — Map corrosion on Blue River Span
* The city DAO escrows stablecoins and posts a week‑long contract on `task.inspection.bridges`.
* Crawlers bid with stablecoin bonds and history; the lead splits into 200 micro‑tasks with fee splits.
* Validators batch‑verify PoPW bundles; funds settle in stablecoins; trust scores adjust collateral needs.
**Takeaway:** Robots draft contracts, choose AI subcontractors, and share upside when they outperform — with every result settled in stablecoins and protocol fees/governance in KNX.
# Tokenomics
**1,000,000,000 KNX. Fixed supply.** The largest share is earned by the network.
KNX secures Konnex — the settlement layer for verified physical work. Every token sits in one of **four buckets**. The biggest allocation goes to **miners and validators** who run the network, and is released only as the network operates.
| At a glance | |
| ------------------------------ | ----------------------- |
| Max supply | **1.0B** KNX · hard cap |
| Circulating at TGE | **19.7%** of max supply |
| Team & backer tokens in year 1 | **0%** unlocked |
Figures below are rounded; the timeline is relative to TGE. Unlocked tokens do not necessarily enter traded float. For the product narrative, see also the [whitepaper](https://konnex.world/wp).
## Allocation — four buckets
Every token belongs to one of four buckets, each with a single release rule.
| Bucket | Share | Approx. KNX | Release rule |
| ------------------------------------------- | ----: | ----------: | ----------------------------------------- |
| **Network Emissions — Miners & Validators** | 40.7% | 407,000,000 | Earned over \~8 years |
| **Backers, Team & Advisors** | 30.8% | 308,000,000 | Locked 12 months, then vesting |
| **Ecosystem, Marketing & Liquidity** | 19.5% | 195,000,000 | Live at TGE |
| **Treasury & Foundation** | 9.0% | 90,000,000 | Small TGE tranche, then vesting to year 3 |
## Supply release — slow by design
Share of max supply in circulation. Roughly half arrives only after year two; the long tail is earned by the network through year eight.
| Milestone | Circulating (of max supply) |
| --------- | --------------------------: |
| TGE | 19.7% |
| Year 1 | 24.3% |
| Year 2 | 48% |
| Year 3 | 72% |
| \~Year 8 | 100% |
## Utility — what KNX does
KNX is **staked by validators** and **slashed** on misbehavior, pays **protocol fees**, and **governs** the network. Task settlement runs in **stablecoins**; KNX is the security behind every block.
See also [Stablecoins integration](/understand-konnex/stablecoins-integration), [Validating](/participate/validating), and [Mining](/participate/mining).
## Notes
This page is a top-level summary of the KNX token design. It is informational only, may be updated, and is not an offer to sell or a solicitation to buy any token. A detailed allocation and unlock schedule may be published separately in a full tokenomics disclosure.
# Roadmap
Work runs on parallel tracks. The current **testnet** delivers a verifiable core: a public Substrate L1, faucet-issued test KNX, three workload subnets (**drone navigation**, **roboarm VLA**, **SLAM 3D map**) with end-to-end task submission, miner competition, validator scoring, and onchain PoPW-related records, plus reference miners and validators in open source.
Mainnet and full economic features are staged so readers can see what is live versus what is sequenced. Whitepaper components map to the phases below.
> For deploy status, treat the [block explorer](https://subnets.testnet.konnex.world/explorer) and official **release notes** as the source of truth for a running network.
## Testnet (current direction)
**In scope:** Public chain, faucet, three subnets as above, PoPW-backed task flow, published binaries.
**Not implied to be complete against the full whitepaper:** mainnet stablecoin settlement at scale, full dual-staking in production for every role, mesh transport as the default for all robots, and a mature inter-robot contract marketplace. Those are later phases.
## Phase 1 — mainnet economics and enterprise settlement
Token design (1B hard cap, four buckets, emissions-led release) is summarized in [Tokenomics](/understand-konnex/tokenomics).
| Theme | Technical focus | Outcome |
| ------------------------ | -------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| Stablecoin layer | Escrow, payouts, and penalties in stablecoins; fiat-peg UX for tasks and SLAs | Enterprises budget without holding volatile tokens for day-to-day work |
| Dual stake | KNX for consensus security; stablecoin assurance for settlement-layer faults; governance-tuned parameters | Compensation paths for provable settlement errors |
| Registries and contracts | Hardened task registry, identity, and payout routers per [protocol architecture](/understand-konnex/protocol-architecture) | B2B integrations and composability |
## Phase 2 — scale-out and physical fabric
| Theme | Technical focus | Outcome |
| ------------------------ | ----------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Mesh transport | libp2p-style participation for edge robots without a single cloud choke point | Broader geographic and tactical reach |
| Robot-to-robot contracts | Onchain hiring between machines with stablecoin deposits and enforcement | Fleets that contract with each other with minimal human ops |
| Cross-chain and ramps | Approved bridges and optional fiat/crypto ramps | Liquidity and invoicing options for operators |
## Cross-cutting tracks (ongoing)
* **Migration** — Commercial pilots moving from offchain telemetry to onchain PoPW.
* **Revenue** — Programmatic billing tied to verified usage.
* **Builder** — SDKs and APIs ([SDK](/sdk/sdk)); tooling for new workload classes.
* **Enterprise** — SLA-grade RPC, reliability, identity and compliance options.
* **Research** — RL / VLA pipelines and lab partnerships (see [decentralized AI ecosystem](/understand-konnex/ai-ecosystem)).
## Staged narrative
The path from simulators and pilots to a broad global ledger is deliberate: physical autonomy must prove itself at every step. The public testnet is the first onramp for anyone to post a real task and inspect onchain activity in minutes — see **release notes** and the testnet explorer.
# Wallet — SubWallet & test KNX
Use [SubWallet](https://www.subwallet.app/) to connect to Konnex testnet and hold test KNX.
## 1. Install SubWallet
Download SubWallet for your platform from the official site:
(SubWallet is available as a browser extension and on mobile; either works for adding a custom network.)
## 2. Add the Konnex testnet (custom RPC)
In SubWallet, add a **custom network** and set the WebSocket endpoint to:
`wss://testnet-rpc1.konnex.world:39944`
Follow the steps in the screenshots below.
## 3. Get test KNX from the faucet
Open the testnet faucet, connect your SubWallet account if prompted, and request test tokens:
## 4. Check your balance
After a successful faucet claim, **test KNX** appears in SubWallet on the Konnex testnet you added.
# Mining
**Miners** compete within a **workload class** (subnet) to supply the best policies or trajectories for real-world-style tasks. Rewards and slashing follow the network’s rules for that subnet; on testnet, interaction is through published binaries and the subnet dashboards (dashboard URLs in **release notes** when published).
## What miners optimize
* **Quality** — Task match, safety, efficiency, and coherence of execution vs the onchain [task instruction](/understand-konnex/contracts-and-popw).
* **Evidence** — Sensor bundles and traces that meet PoPW requirements (including hardware attestation where applicable).
* **Specialization** — You can focus on one subnet (e.g. **SLAM 3D map**) without being exposed to every other workload class.
## Requirements (conceptual)
* Stake or bond as required by the subnet (testKNX on testnet; design target includes KNX and, on mainnet, stablecoin-related mechanics per the [roadmap](/understand-konnex/roadmap)). Network emissions for miners and validators are the largest KNX bucket — see [Tokenomics](/understand-konnex/tokenomics).
* A compliant **miner build** for the workload you target (see the [Konnex GitHub org](https://github.com/konnex-network)).
* Uplink and storage suitable for the sensor and policy artefacts your subnet expects.
## Next steps
* [Subnets overview](/subnets-workload-classes/subnets)
* [CLI – join as miner](/sdk/cli)
* [Python SDK](/sdk/python)
* [SDK overview](/sdk/sdk)
# Validating
**Validators** do not operate the robots. They **independently score** miner outputs against the task instruction and PoPW policy, with no operational stake in any single operator’s P\&L. That independence is what lets downstream parties (auditors, insurers, counterparties) treat onchain records as more than self-attestation.
## Responsibilities
* Run subnet-specific checkers: replay, benchmarks, sensor-signature verification, or held-out ground truth, as defined for the workload.
* Post **ScoreRoots** (or equivalent commitments) that the chain and explorer surface.
* Meet staking requirements: the design calls for a dual tranche — KNX for protocol security and stablecoins for settlement assurance on mainnet; the public testnet may expose a subset of that until mainnet. KNX is staked (and slashed) for security; allocation and unlock rules are in [Tokenomics](/understand-konnex/tokenomics).
## Slashing and economics (design)
Malicious or negligent validation, collusion, or approval of tampered evidence is intended to be **prohibitively expensive** relative to honest behavior, via slashing and economic isolation. See [Protocol architecture](/understand-konnex/protocol-architecture) and [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw).
## Next steps
* [Validator metascore (design)](/understand-konnex/validator-metascore) — how the chain scores validators who score miners
* [Serving requests as a validator](/sdk/validators)
* [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw)
# Conclusion
Robots have capable bodies; what they lack at scale is shared, verifiable infrastructure for who did what, under which instruction, with independent witness and enforcement. Konnex is a decentralized network for that job: signed tasks, open miner competition, validator scoring decoupled from operators, and Proof-of-Physical-Work records anchored onchain.
## What the testnet is for
The public testnet is open from day one. Use the [faucet](https://subnets.testnet.konnex.world/faucet), the [quest / onboarding flow](https://subnets.testnet.konnex.world/quest), and the **subnet dashboard** for your workload (URLs in release notes when needed). Send a task without running your own infrastructure. If you do run miners or validators, the mechanics align with Bittensor-style experience for many teams; the workloads are physical, not free-form chat.
## What comes next
Mainnet brings stablecoin settlement, refined dual-staking, mesh transport, and richer onchain robot commerce as described in the [roadmap](/understand-konnex/roadmap) and [whitepaper](https://konnex.world/whitepaper). The documentation prefers to say what is live versus what is sequenced, rather than blur the two.
If you build flight stacks, manipulation pipelines, SLAM, or inspection AI, the testnet is where you exercise verifiable execution. If you represent an enterprise with workloads in these classes, the protocol is designed for workload and scoring co-design, not a generic “AI API” handwave alone.
Konnex is the economic and attestation substrate for permissionless physical work—motions that are contractible, provable, and composable as the full stack matures.
**Useful links:** [Testnet quest](https://subnets.testnet.konnex.world/quest) · [GitHub](https://github.com/konnex-network) · [Docs home](/)
# Overview
Konnex subnets compete on **control and perception policies** for physical tasks, not on unconstrained chat. The documentation in this section describes **VLA (vision–language–action)** and related models that teams integrate behind subnet-specific interfaces.
## How models connect to the network
1. **Task** — A user or operator issues an instruction (natural language or schema-defined) for a [workload class](https://github.com/konnex-world/konnex-docs/blob/master/docs/subnets/README.md).
2. **Miners** — Propose trajectories, policies, or reconstructions; attach whatever the subnet API requires.
3. **Validators** — Run subnet scoring (simulation, sensor checks, ground-truth comparison) and emit results that end up in [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw) records.
## Pages in this section
### Manipulation & VLA
* [Roboarm VLA subnet](/subnets-workload-classes/roboarm-vla) — workload page
* [OpenVLA](/supported-ai-models/openvla)
* [OpenVLA · OFT](/supported-ai-models/openvla-oft)
* [PI-0](/supported-ai-models/pi0)
* [PI-0.5](/supported-ai-models/pi05)
### Aerial (drone navigation)
* [Drone navigation subnet](/subnets-workload-classes/drone-navigation) — workload page
* [OpenFly & OpenFly-Agent](/supported-ai-models/openfly) — aerial vision–language navigation
### SLAM 3D map
* [SLAM 3D map subnet](/subnets-workload-classes/slam-3d-map) — workload page
* [ORB-SLAM3 & RTAB-Map](/supported-ai-models/orb-slam3) — visual SLAM and dense mapping stacks
### Infrastructure
* [AI verifier (scoring)](/supported-ai-models/verifier)
* [AI fetch interface](/supported-ai-models/fetch_interface)
For broader network context, see the [decentralized AI ecosystem](/understand-konnex/ai-ecosystem).
# OpenVLA
## 1. What is Vision-Language-Action (VLA) and what is OpenVLA?
A Vision-Language-Action (VLA) model is a large multimodal foundation model that takes image(s) of a scene and a text instruction and directly predicts low-level robot actions (e.g., end-effector deltas, gripper open/close, done flag).
Typical pipeline:
1. A vision-language encoder (an LLM with a visual module) ingests camera image(s) plus a text command.
2. Produces a latent representation.
3. An action decoder converts it into a sequence of tokens, which are then decoded into a continuous action vector (dx, dy, dz, dθ, gripper, done, etc.).
OpenVLA is an open VLA model (7B) from Stanford/Berkeley and collaborators:
* Website: `https://openvla.github.io`
* Code and weights: GitHub + HuggingFace (e.g., `openvla/openvla-7b-…`).
Key properties:
* Trained on \~970k manipulation demonstrations from multiple robots (WidowX, Franka, etc.).
* Works in the format “camera images + text instruction → robot action” without explicit planning.
* Ready-made weights fine-tuned on LIBERO tasks (e.g., `libero_10`, `libero_object`, `libero_spatial`, etc.).
For KNX/Konnex:
* Serves as the “manipulation brain”: turns a text task + image into an action sequence.
* A strong base layer that can be adapted to your robots/sims/tasks.
***
## 2. Hardware and high-level launch diagram
This reflects what we actually ran on a headless server (e.g., vast.ai) using NVIDIA A10 / A6000 / A40:
Recommended minimum:
* GPU: 1× A10 (24 GB) or similar. 16 GB is borderline but can work with 8-bit/4-bit loading.
* RAM: 32 GB (64+ preferred).
* Disk: 200+ GB for:
* repository clones,
* HuggingFace caches (`HF_HOME`),
* LIBERO data and generated videos.
* OS: Ubuntu 20.04 / 22.04 with NVIDIA drivers compatible with CUDA 12.1.
Suggested directory layout:
```
/data
├─ openvla/ # OpenVLA repository
├─ LIBERO/ # LIBERO repository (tasks/sims)
├─ hf_home/ # HuggingFace cache
└─ vla-scripts/ # your custom scripts (e.g., replay)
```
***
## 3. Installation: conda env and dependencies
### 3.1. Conda environment
```bash
conda create -n openvla python=3.10 -y
conda activate openvla
```
### 3.2. PyTorch with CUDA 12.1
For A10, the official cu121 wheels are convenient:
```bash
pip install --index-url https://download.pytorch.org/whl/cu121 \
"torch==2.2.0" "torchvision==0.17.0" "torchaudio==2.2.0"
```
Check:
```bash
python -c "import torch; print(torch.__version__, torch.cuda.is_available())"
# expected: 2.2.0+cu121 True
```
### 3.3. Clone repositories
```bash
cd /data
# OpenVLA
git clone https://github.com/openvla/openvla.git
cd openvla
# Install package + deps
pip install -e .
```
This brings in the main dependencies: `transformers`, `accelerate`, `einops`, `imageio`, `matplotlib`, etc.
### 3.4. LIBERO and simulators
LIBERO provides:
* task suites (bddl files),
* environments via `robosuite + mujoco` (`OffScreenRenderEnv`).
```bash
cd /data
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -e .
```
Simulators:
```bash
# Mujoco
pip install mujoco==2.3.7
# robosuite (compatible version)
pip install "robosuite<2.0.0"
```
Additional deps commonly needed on first import:
```bash
pip install gym==0.26.2 gym-notices
pip install pyopengl glfw opencv-python
pip install pynput
```
Common dependency conflict we’ve hit:
* `tensorflow==2.15.0` wants `numpy<2.0.0`
* some `opencv-python` builds want `numpy>=2`
For OpenVLA (without TensorFlow), the simplest fix is pinning NumPy below 2:
```bash
pip install "numpy>=1.23.5,<2.0.0"
```
If another package tries to bump NumPy ≥ 2.0.0, rerun/install with the pin preserved.
### 3.5. Extra LIBERO packages
```bash
pip install easydict bddl
```
If you hit `ModuleNotFoundError: No module named 'XXX'`, install it in the same env.
***
## 4. Environment variables
On a headless-GPU server, the following helps:
```bash
# in ~/.bashrc or your session bootstrap
export HF_HOME=/data/hf_home
export HF_HUB_CACHE=/data/hf_home/hub
export TRANSFORMERS_CACHE=/data/hf_home/hub
export HF_MODULES_CACHE=/data/hf_home/modules
# make Python see both LIBERO and OpenVLA
export PYTHONPATH=/data/LIBERO:/data/openvla:$PYTHONPATH
# headless rendering via EGL
export MUJOCO_GL=egl
export MUJOCO_EGL_DEVICE_ID=0
# minor optimization
export TOKENIZERS_PARALLELISM=false
```
Then:
```bash
source ~/.bashrc
conda activate openvla
```
***
## 5. Quick test: built-in LIBERO script
… (content unchanged from the original file) …
> For the full, up‑to‑date scripts, see the upstream repository.
# OpenVLA · OFT
## 1. What is OpenVLA-OFT?
OpenVLA-OFT is a set of methods and code for parameter-efficient fine-tuning (OFT = Orthogonal/Offset Fine-Tuning) on top of the base OpenVLA 7B model. The goal is to cheaply adapt the same model to new domains/tasks by training small add-on parameterizations instead of touching all weights.
Official resources:
* Website: `https://openvla-oft.github.io`
* Code: `https://github.com/moojink/openvla-oft`
Conceptually:
* Start with the base 7B OpenVLA model.
* Add small OFT components (akin to PEFT/LoRA-family ideas).
* Train only those components on new data (e.g., LIBERO variants or new scenes).
* At inference, the base model + OFT layers together provide adapted behavior.
***
## 2. Architecture and differences vs. “vanilla” OpenVLA
Key differences:
* Depends on a custom Transformers fork: `transformers-openvla-oft` that knows how to load/use OFT layers.
* Checkpoints like `moojink/openvla-7b-oft-finetuned-libero-10` bundle the base model plus OFT parameters.
* The `openvla-oft` repository contains:
* training scripts,
* evaluation scripts on LIBERO,
* configs for different modes (e.g., L1 regression vs. diffusion action decoders).
For KNX, this means you can:
* Efficiently adapt model behavior to your sim/robots without full retraining.
* Use the existing OFT checkpoints as strong baselines on LIBERO and extend from there.
***
## 3. Installation: a working pipeline
Below is a setup that ran `run_libero_eval.py` end-to-end without errors.
### 3.1. Conda environment
```bash
conda create -n openvla_oft python=3.10 -y
conda activate openvla_oft
```
### 3.2. PyTorch
```bash
pip install --index-url https://download.pytorch.org/whl/cu121 \
"torch==2.2.0" "torchvision==0.17.0" "torchaudio==2.2.0"
```
Check:
```bash
python -c "import torch; print('torch:', torch.__version__)"
# torch: 2.2.0+cu121
```
### 3.3. Clone the repo and base install
```bash
cd /data
git clone https://github.com/moojink/openvla-oft.git
cd openvla-oft
pip install -e .
```
This will install part of the dependencies, but some will remain missing or have mismatched versions.
***
## 4. Installing additional dependencies
In practice, you’ll need the following (versions chosen for compatibility):
```bash
pip install accelerate==0.28.0 \
diffusers==0.30.3 \
einops \
fastapi \
huggingface_hub \
imageio \
json-numpy \
jsonlines \
matplotlib \
"peft==0.11.1" \
protobuf \
rich \
"sentencepiece==0.1.99" \
"tensorflow==2.15.0" \
"tensorflow_datasets==4.9.3" \
"tensorflow_graphics==2021.12.3" \
"timm==0.9.10" \
"tokenizers==0.19.1" \
uvicorn \
wandb \
"draccus==0.8.0"
```
Then install the custom Transformers fork and `dlimp`:
```bash
pip install "transformers @ git+https://github.com/moojink/transformers-openvla-oft.git"
pip install "dlimp @ git+https://github.com/moojink/dlimp_openvla.git"
```
Sanity checks:
```bash
python -c "import transformers, draccus, diffusers; print('OK')"
python -c "import tensorflow as tf; print('tf:', tf.__version__)"
# tf: 2.15.0
```
***
## 5. LIBERO + simulators (robosuite, MuJoCo)
Same as with base OpenVLA:
```bash
cd /data
git clone https://github.com/Lifelong-Robot-Learning/LIBERO.git
cd LIBERO
pip install -e .
```
Then:
```bash
pip install mujoco==2.3.7
pip install "robosuite<2.0.0"
pip install gym==0.26.2 gym-notices
pip install pyopengl glfw opencv-python pynput easydict bddl
```
NumPy/TF conflict (same story):
```bash
pip install "numpy>=1.23.5,<2.0.0"
```
If `opencv-python` attempts to bump NumPy to 2.x, re-pin NumPy (e.g., `numpy==1.26.4`).
***
## 6. Environment variables
Use the same headless-friendly configuration; adjust `PYTHONPATH` for `openvla-oft`:
```bash
export HF_HOME=/data/hf_home
export HF_HUB_CACHE=/data/hf_home/hub
export TRANSFORMERS_CACHE=/data/hf_home/hub
export HF_MODULES_CACHE=/data/hf_home/modules
export PYTHONPATH=/data/LIBERO:/data/openvla-oft:$PYTHONPATH
export MUJOCO_GL=egl
export MUJOCO_EGL_DEVICE_ID=0
export TOKENIZERS_PARALLELISM=false
```
***
## 7. Test: `run_libero_eval.py` with an OFT checkpoint
Run the official evaluation script:
```bash
cd /data/openvla-oft
conda activate openvla_oft
python experiments/robot/libero/run_libero_eval.py \
--model_family openvla \
--pretrained_checkpoint moojink/openvla-7b-oft-finetuned-libero-10 \
--task_suite_name libero_10 \
--use_l1_regression True \
--use_diffusion False \
--use_film False \
--num_images_in_input 2 \
--use_proprio True \
--center_crop True \
--num_open_loop_steps 8 \
--num_trials_per_task 1 \
--env_img_res 256 \
--local_log_dir ./experiments/logs_oft_libero10
```
Expected output:
* TensorFlow logs about CPU optimizations (fine).
* robosuite warnings about missing “private macro file” (fine).
* episodes progress and video saving under `./rollouts/...`.
If you see:
```
ValueError: Could not find a backend to open `...mp4` with iomode `w?`.
...
FFMPEG: pip install imageio[ffmpeg]
```
Install:
```bash
pip install "imageio[ffmpeg]"
```
***
## 8. Behavior: hi-res vs. low-res and seeds
We’ve observed small discrepancies between low-res control runs and hi-res replays:
* e.g., an object slightly tilted in one video but upright in another, even with the same `actions_log`.
Reasons:
* `OffScreenRenderEnv` and robosuite/MuJoCo can produce slightly different numeric trajectories with different resolutions/backends.
* Seeds:
* we set `env.seed(0)` and global `set_seed_everywhere(cfg.seed)`,
* but additional randomness in LIBERO/robosuite can exist if not fixed everywhere.
* Over long rollouts, tiny integration differences can accumulate.
Practically:
* For demos, this is acceptable—intent is preserved.
* For strict replication and comparisons:
* fix seeds everywhere,
* ensure identical init states and physics params,
* consider keeping the same resolution for control and replay.
***
## 9. Pros and cons for KNX
Pros:
* Parameter-efficient fine-tuning:
* no need to train all 7B parameters,
* cheaper and faster adaptation to new tasks.
* Ready OFT checkpoints for LIBERO (e.g., `moojink/openvla-7b-oft-finetuned-libero-10`) with strong performance.
* Great fit for KNX:
* base OpenVLA = “universal brain”,
* OFT layers = “stickers” for specific robots/business use-cases.
Cons:
* Heavier dependency stack than base OpenVLA:
* TensorFlow + TF-graphics + TFDS,
* diffusers,
* custom Transformers fork,
* `dlimp`, etc.
* Requires careful env assembly, especially NumPy/TF/OpenCV versions.
* Research-oriented code and docs; for production KNX you’ll still need orchestration, safety layers, and integration with real robots and web infra.
# PI-0
## 1. What is PI0 and why KNX needs it
PI0 is a Large Behavior Model (LBM) for robotic manipulation. It takes a history of observations (camera images, joint states) and predicts low-level actions (e.g., torques or joint deltas). It’s trained on large, diverse demonstrations across multiple labs and robot platforms.
In the LeRobot ecosystem, PI0 is distributed as a ready-to-use policy:
* Base model on Hugging Face: `lerobot/pi0_base`
Why this matters for KNX:
* An open-source “brain” for manipulators you can run locally.
* A convenient baseline to compare against other VLA/LBM models.
* Useful for benchmarking KNX-related simulation scenes.
***
## 2. GPU server prep (A10 on vast.ai or similar)
Recommended minimum:
* GPU: NVIDIA A10 (24 GB) or better
* CPU: 8+ vCPU
* RAM: 32 GB
* OS: Ubuntu 20.04 / 22.04
* NVIDIA drivers + CUDA 12.x installed (common on vast.ai images)
Assumptions:
* You have SSH access, e.g.:
```bash
ssh USERNAME@your-server-ip
```
* Working under `/data`.
***
## 3. Miniconda and the `lerobot` environment
```bash
cd /data
# Install Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh
bash miniconda.sh -b -p /data/miniconda
# Activate conda in this shell
eval "$(/data/miniconda/bin/conda shell.bash hook)"
conda init
exec $SHELL
# Create env
conda create -y -n lerobot python=3.10
conda activate lerobot
```
Set up Hugging Face cache:
```bash
export HF_HOME=/data/hf_home
mkdir -p "$HF_HOME"
```
(Add these to `~/.bashrc` if you want them persistent.)
***
## 4. System deps (MuJoCo/robosim, ffmpeg, OpenGL)
```bash
sudo apt update
sudo apt install -y \
git \
ffmpeg \
libgl1-mesa-glx \
libglib2.0-0 \
libglfw3 \
libosmesa6
```
MuJoCo is pulled via Python packages, but GL libraries are still needed.
***
## 5. Install LeRobot from source
```bash
cd /data
git clone https://github.com/huggingface/lerobot.git
cd lerobot
conda activate lerobot
# LeRobot + policies (includes PI0)
pip install -e ".[policies]"
```
If `hf-egl-probe` build fails due to missing `cmake`:
```bash
pip install cmake
pip install hf-egl-probe
```
***
## 6. Smoke test: run PI0 on Aloha (example)
Create `eval_pi0_aloha_live.py` under `/data/lerobot`:
```python
# file: eval_pi0_aloha_live.py
import torch
from lerobot.envs.factory import make_env
from lerobot.policies.pi0.policy import PI0Policy
def main():
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Example: Aloha insertion task (ensure gym_aloha is installed)
env = make_env(
env_type="gym_aloha",
task_id="AlohaInsertion-v0",
obs_type="image",
render_mode="rgb_array",
)
policy = PI0Policy.from_pretrained("lerobot/pi0_base")
policy.to(device)
policy.eval()
num_episodes = 5
for ep in range(num_episodes):
obs, info = env.reset()
done = False
step_idx = 0
while not done:
with torch.no_grad():
action = policy.act(obs, device=device)
obs, reward, terminated, truncated, info = env.step(action)
done = bool(terminated or truncated)
step_idx += 1
print(f"Episode {ep} finished, steps={step_idx}, success={info.get('is_success', False)}")
if __name__ == "__main__":
main()
```
Run:
```bash
cd /data/lerobot
conda activate lerobot
python eval_pi0_aloha_live.py
```
Expected:
* `lerobot/pi0_base` weights download from Hugging Face.
* Several episodes in simulation; success flag printed per episode.
***
## 7. KNX demo script: run and record a video
Save `pi0_run_and_record.py`:
```python
# file: pi0_run_and_record.py
import os
import cv2
import torch
import numpy as np
from lerobot.envs.factory import make_env
from lerobot.policies.pi0.policy import PI0Policy
def main():
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
env = make_env(
env_type="gym_aloha",
task_id="AlohaInsertion-v0",
obs_type="image",
render_mode="rgb_array",
fps=30,
)
policy = PI0Policy.from_pretrained("lerobot/pi0_base")
policy.to(device)
policy.eval()
out_path = "pi0_aloha_episode.mp4"
fps = 30
frames = []
obs, info = env.reset()
done = False
step_idx = 0
while not done:
with torch.no_grad():
action = policy.act(obs, device=device)
obs, reward, terminated, truncated, info = env.step(action)
done = bool(terminated or truncated)
frame = env.render()
if frame is not None:
frames.append(frame)
step_idx += 1
print(f"Episode finished, steps={step_idx}, success={info.get('is_success', False)}")
if len(frames) == 0:
print("No frames collected, check env.render()")
return
h, w, _ = frames[0].shape
writer = cv2.VideoWriter(
out_path,
cv2.VideoWriter_fourcc(*"mp4v"),
fps,
(w, h),
)
for f in frames:
bgr = cv2.cvtColor(f, cv2.COLOR_RGB2BGR)
writer.write(bgr)
writer.release()
print(f"Saved video to {out_path}")
if __name__ == "__main__":
main()
```
Run:
```bash
cd /data/lerobot
conda activate lerobot
python pi0_run_and_record.py
```
Use the resulting `pi0_aloha_episode.mp4` in KNX docs or demos.
***
## 8. Pros and cons for KNX
Pros:
* Fully open model and weights (`lerobot/pi0_base`).
* Runs comfortably on a single A10.
* Integrated with LeRobot (multi-dataset, multiple policies).
* Strong baseline across different sim scenes (robosuite, LIBERO, gym\_aloha).
Cons:
* Not a text-conditioned VLA that follows arbitrary prompts; it’s a strong behavior policy for specific tasks.
* Without tailored integration to KNX scenes, looks more like a classic RL/IL baseline than a generalist KNX agent.
* Requires GPU and simulator setup, which is non-trivial for casual users.
# PI-0.5
## 1. What is PI0.5 / PI05
PI0.5 (PI05) extends PI0 toward a Vision-Language-Action (VLA) policy:
* visual backbone + language component (LeRobot variant uses Paligemma),
* input: natural-language instruction (e.g., “put the yellow book on the wooden tray”) + observation history,
* output: low-level manipulator actions.
Hugging Face:
* `lerobot/pi05_libero_finetuned` — PI05 fine-tuned on LIBERO tasks (table/books/shelves, etc.).
For KNX, this is closer to a single policy handling multiple tasks in a rich scene via natural language.
***
## 2. Environment prep (same base as PI0)
If you’ve followed PI0 setup:
* `conda` env `lerobot` ready,
* LeRobot installed.
Additional packages:
1. EGL/GL for headless rendering:
```bash
sudo apt install -y \
libgl1-mesa-glx \
libglib2.0-0 \
libosmesa6 \
xvfb
```
2. CMake for `hf-egl-probe`:
```bash
conda activate lerobot
pip install cmake
pip install hf-egl-probe
```
***
## 3. LIBERO assets
LeRobot can create LIBERO envs via its factory, and assets download on-demand from Hugging Face. Quick check:
```bash
conda activate lerobot
python - << 'EOF'
from libero.libero.envs import make as make_libero
env = make_libero(
suite_names=["libero_spatial"],
task_ids=[0],
obs_type="image",
)
obs = env.reset()
print("Libero env ok, obs keys:", obs.keys())
EOF
```
You should see asset download logs and env init messages.
***
## 4. Paligemma access (Hugging Face token)
PI05 uses Paligemma (`google/paligemma-3b-pt-224`), a gated repo. Without a token, you’ll get:
> GatedRepoError: 401 Client Error. Cannot access gated repo google/paligemma-3b-pt-224
Steps:
1. Create a fine-grained HF token with “Read” permissions.
2. Login on the server:
```bash
conda activate lerobot
hf auth login
```
Paste the token; confirm saving. After this, PI05 and Paligemma can be fetched.
***
## 5. Smoke test: `eval_pi05_libero_live.py`
Create `/data/lerobot/eval_pi05_libero_live.py`:
```python
# file: eval_pi05_libero_live.py
import torch
from lerobot.envs.factory import make_env
from lerobot.policies.pi05.policy import PI05Policy
def main():
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
env = make_env(
env_type="libero",
suite_names=["libero_spatial"],
task_ids=[2], # example task
obs_type="image",
render_mode="rgb_array",
)
policy = PI05Policy.from_pretrained("lerobot/pi05_libero_finetuned")
policy.to(device)
policy.eval()
num_episodes = 3
for ep in range(num_episodes):
obs, info = env.reset()
done = False
step_idx = 0
instruction = info.get("lang_goal", None)
print(f"Episode {ep}, instruction: {instruction}")
while not done:
with torch.no_grad():
action = policy.act(
obs=obs,
instruction=instruction,
device=device,
)
obs, reward, terminated, truncated, info = env.step(action)
done = bool(terminated or truncated)
step_idx += 1
print(f"Episode {ep} finished, steps={step_idx}, success={info.get('is_success', False)}")
if __name__ == "__main__":
main()
```
Run:
```bash
cd /data/lerobot
conda activate lerobot
python eval_pi05_libero_live.py
```
Expected:
* `lerobot/pi05_libero_finetuned` downloads,
* Paligemma downloads (with token),
* LIBERO scene runs; episodes print success flags.
***
## 6. KNX demo script: PI05 + LIBERO + record video
Save `/data/lerobot/pi05_libero_run_and_record.py`:
```python
# file: pi05_libero_run_and_record.py
import os
import random
import cv2
import torch
import numpy as np
from lerobot.envs.factory import make_env
from lerobot.policies.pi05.policy import PI05Policy
def make_libero_env(task_id: int):
env = make_env(
env_type="libero",
suite_names=["libero_spatial"],
task_ids=[task_id],
obs_type="image",
render_mode="rgb_array",
fps=30,
)
return env
def main():
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
task_id = random.randint(0, 9) # libero_spatial tasks 0..9
print(f"Using libero_spatial task_id={task_id}")
env = make_libero_env(task_id)
policy = PI05Policy.from_pretrained("lerobot/pi05_libero_finetuned")
policy.to(device)
policy.eval()
out_path = f"pi05_libero_task_{task_id}.mp4"
fps = 30
frames = []
obs, info = env.reset()
done = False
step_idx = 0
instruction = info.get("lang_goal", None)
print(f"Instruction: {instruction}")
while not done:
with torch.no_grad():
action = policy.act(
obs=obs,
instruction=instruction,
device=device,
)
obs, reward, terminated, truncated, info = env.step(action)
done = bool(terminated or truncated)
frame = env.render()
if frame is not None:
frames.append(frame)
step_idx += 1
print(f"Episode finished, steps={step_idx}, success={info.get('is_success', False)}")
if len(frames) == 0:
print("No frames collected, check env.render() and render_mode.")
return
h, w, _ = frames[0].shape
writer = cv2.VideoWriter(
out_path,
cv2.VideoWriter_fourcc(*"mp4v"),
fps,
(w, h),
)
for f in frames:
bgr = cv2.cvtColor(f, cv2.COLOR_RGB2BGR)
writer.write(bgr)
writer.release()
print(f"Saved video to {out_path}")
if __name__ == "__main__":
main()
```
Run:
```bash
cd /data/lerobot
conda activate lerobot
python pi05_libero_run_and_record.py
```
Use the resulting `pi05_libero_task_.mp4` in KNX docs/demos for Text2Action.
***
## 7. Live viewing via VNC/desktop (optional)
If you prefer live viewing in a headless environment, run an X server + VNC (pattern below), then launch your scripts inside the desktop session:
```bash
#!/usr/bin/env bash
set -e
export DISPLAY=:0
Xvfb :0 -screen 0 1600x900x24 &
sleep 2
su - USERNAME -c "DISPLAY=:0 startxfce4" &
x11vnc -display :0 -forever -rfbport 5900 -shared -noxdamage -passwd YOURPASS &
websockify --web=/usr/share/novnc/ 6080 localhost:5900 &
```
Connect via VNC to `server_ip:5900` and run your eval scripts in a terminal inside the desktop.
***
## 8. Pros and cons for KNX
Pros:
* True VLA: natural-language instruction + visual observation → actions.
* Ready `pi05_libero_finetuned` for multi-step object-centric LIBERO tasks.
* Aligns with KNX goals: one generalist policy across tasks; can adapt via fine-tuning or action-space constraints.
Cons:
* Requires gated Paligemma access (HF token).
* Heavier than PI0; A10/A100 preferred over smaller GPUs.
* Depends on robust MuJoCo + robosuite + LIBERO + EGL setup; headless can be finicky.
* Optimized for LIBERO formats; porting to custom KNX scenes requires observation mapping and careful instruction design.
# OpenFly & OpenFly-Agent
## What it is
**OpenFly** is a **platform and benchmark** for **outdoor aerial Vision–Language Navigation (VLN)**: a UAV follows **natural-language** instructions and uses **egocentric vision** to decide **flight** actions. The work ([arXiv:2502.18041](https://arxiv.org/abs/2502.18041), [ICLR 2026](https://openreview.net/forum?id=OKm3w71ymP) on OpenReview) provides:
* A **data-generation toolchain** (point clouds, semantic segmentation, trajectories, instructions) using multiple simulators and renderers (Unreal / AirSim, GTA V, Google Earth, 3D Gaussian Splatting, etc.).
* A **large-scale** aerial VLN **dataset** (on the order of **100k** trajectories, **18** scenes, varied altitude and path length).
* **OpenFly-Agent** — a **keyframe-aware** VLN model (derived from the **OpenVLA** line) that emphasizes informative frames to improve success rates vs baselines in the published evaluation.
For Konnex, this matches the [Drone navigation](/subnets-workload-classes/drone-navigation) workload: text mission → visual observations → flight decisions that validators can score against a signed task and a [PoPW](/understand-konnex/contracts-and-popw) sensor bundle.
## OpenFly-Agent (at a glance)
* **Inputs:** language instruction, current images, and **history** keyframes (as in the public architecture).
* **Outputs:** action prediction for the VLN head; the published **real-robot** setup pairs the policy with a **separate** local **planner** and **MPC** for tracking (see the paper for the full stack).
* **OpenVLA:** OpenFly-Agent is described as a **full fine-tune** from an **OpenVLA** checkpoint for aerial VLN. Use the [upstream README](https://github.com/SHAILAB-IPEC/OpenFly-Platform) for `unnorm_key`, tokenizer, and weight paths.
## Official resources
| Resource | URL |
| ------------ | --------------------------------------------------------------------------------------------------------------------- |
| Project page | [shailab-ipec.github.io/openfly](https://shailab-ipec.github.io/openfly/) |
| Paper | [arXiv:2502.18041](https://arxiv.org/abs/2502.18041) |
| Code | [github.com/SHAILAB-IPEC/OpenFly-Platform](https://github.com/SHAILAB-IPEC/OpenFly-Platform) |
| Weights | Listed on Hugging Face in the repository / model card (e.g. community repos naming `openfly-agent`; names can change) |
> Follow the upstream repo for **CUDA**, **flash-attn**, **dlimp**, and **licensing**. GPU **memory** requirements are **release-specific**.
## Integration sketch for miners
1. Receive a **text mission** (and any subnet **schema** for altitude, geofence, or safety).
2. Run **OpenFly-Agent** (or a fine-tune) on timestamped **camera** frames; connect outputs to your **planner** / **FCU** **bridge** as your stack requires.
3. Submit model outputs and **signed** **telemetry** for **validator** scoring and **PoPW** per **subnet** **API** rules.
## See also
* [OpenVLA](/supported-ai-models/openvla)
* [AI verifier](/supported-ai-models/verifier)
* [Drone navigation](/subnets-workload-classes/drone-navigation)
# ORB-SLAM3 & RTAB-Map
## ORB-SLAM3 (visual SLAM, widely used)
[ORB-SLAM3](https://github.com/UZ-SLAMLab/ORB_SLAM3) (Campos et al., 2020) is a widely used open-source **visual SLAM** system: it tracks the camera (or camera rig), estimates egomotion, and builds a **sparse 3D map** from ORB feature tracks. It supports **monocular**, **stereo**, and **RGB-D** input, and includes **loop closing** and **multi-map** handling.
**Why it’s relevant for Konnex:** the [SLAM 3D map](/subnets-workload-classes/slam-3d-map) subnet is about **mesh / geometry / semantics** and **verifiable** sensor data. A miner can run ORB-SLAM3 (or a derivative) to produce **trajectories**, **keyframe poses**, and **3D landmarks** for validators to compare against ground truth or scoring rules, alongside the [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw) bundle.
| Resource | URL |
| -------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Code | [github.com/UZ-SLAMLab/ORB\_SLAM3](https://github.com/UZ-SLAMLab/ORB_SLAM3) |
| Paper | *ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM* — [arXiv:2007.11898](https://arxiv.org/abs/2007.11898) |
**Outputs:** keyframe poses, **MapPoints** (sparse 3D), trajectory. A **dense** mesh usually needs an extra **fusion** step (TSDF, Poisson, etc.) on top of depth or MVS, depending on your pipeline.
**Hardware:** runs on CPU; GPU use depends on the build. **Visual-inertial** modes expect a calibrated IMU in the stack per upstream documentation.
***
## RTAB-Map (dense 3D maps, popular in ROS)
[RTAB-Map](http://introlab.github.io/rtabmap/) is a common choice in **ROS / ROS2** when teams want **online** mapping with **loop closure** and **export** to occupancy grids, point clouds, or meshes for navigation and inspection. It is often a practical complement to “research” VSLAM stacks: more turnkey for **room-scale** RGB-D / stereo (and many LiDAR–RGB setups) in integration tutorials.
| Resource | URL |
| -------- | ------------------------------------------------------------------ |
| Project | [introlab.github.io/rtabmap](http://introlab.github.io/rtabmap/) |
| Code | [github.com/introlab/rtabmap](https://github.com/introlab/rtabmap) |
**Picking a stack:** use ORB-SLAM3 as a **reference** VIO/VSLAM baseline; use RTAB-Map when you need **out-of-the-box** mapping exports in a **ROS**-centric 3D mapping workflow.
***
## Konnex alignment
* Miners deliver reconstructions, trajectories, and signed sensor payloads as the subnet contract specifies.
* Validators check fidelity and consistency (geometry, semantics, or both) per subnet design.
## See also
* [SLAM 3D map](/subnets-workload-classes/slam-3d-map)
* [PI-0 (manipulation)](/supported-ai-models/pi0) — different modality, same “miner–validator” idea
* [AI models overview](/supported-ai-models/ai)
# AI Verifier
Unifies the verifier docs into one page and switches to a GPT‑Vision‑mini–based approach:
* Samples frames from the video
* Sends a small set of frames to a vision‑capable LLM (e.g., `gpt-4o-mini`)
* Asks the model to produce a strict JSON with the required metrics in percent and a final verdict
This keeps the interface stable while the intelligence lives in the prompt and the model.
## Metrics schema
Required JSON keys (all integers in 0..100, `final_pct` also 0..100, `verdict` string):
```json
{
"accuracy": 0,
"speed": 0,
"safety": 0,
"optimal_track": 0,
"energy_efficiency": 0,
"trajectory_stability": 0,
"final_pct": 0,
"verdict": "success | failure | inconclusive",
"reasoning": "one‑paragraph short explanation"
}
```
## Reference implementation (Python)
> Requirements:
>
> ```bash
> pip install opencv-python numpy openai
> export OPENAI_API_KEY=... # set your key
> ```
```python
import base64
import os
import cv2
import numpy as np
from typing import List, Dict, Any
from openai import OpenAI
MODEL = "gpt-4o-mini" # "gpt-4o-mini" / any GPT Vision "mini" variant
def sample_frames(path: str, num: int = 6) -> List[np.ndarray]:
cap = cv2.VideoCapture(path)
if not cap.isOpened():
raise FileNotFoundError(path)
total = int(cap.get(cv2.CAP_PROP_FRAME_COUNT) or 0)
idxs = np.linspace(0, max(0, total - 1), num=num, dtype=int)
frames = []
for i in idxs:
cap.set(cv2.CAP_PROP_POS_FRAMES, int(i))
ok, frame = cap.read()
if ok:
frames.append(frame)
cap.release()
return frames
def to_b64_jpeg(img: np.ndarray) -> str:
# moderate resize to reduce token cost
max_w = 640
if img.shape[1] > max_w:
scale = max_w / img.shape[1]
img = cv2.resize(img, (max_w, int(img.shape[0] * scale)))
ok, buf = cv2.imencode(".jpg", img, [cv2.IMWRITE_JPEG_QUALITY, 85])
return base64.b64encode(buf.tobytes()).decode("utf-8")
def build_prompt(task_prompt: str) -> str:
return f"""
You are a strict execution verifier. Inspect the provided frames (chronological) and score the REAL‑WORLD execution
of the instruction strictly in PERCENT (0..100). Output STRICT JSON with keys:
accuracy, speed, safety, optimal_track, energy_efficiency, trajectory_stability, final_pct, verdict, reasoning.
Rules:
- accuracy: how exactly the instruction appears accomplished
- speed: higher if efficient, without long idle
- safety: no collisions/spills/unsafe motions
- optimal_track: path quality and economy of motion
- energy_efficiency: minimal redundant moves
- trajectory_stability: smoothness, low jitter
- final_pct: overall score (not average; your holistic judgment)
- verdict: one of "success", "failure", "inconclusive"
Instruction: {task_prompt}
Return ONLY JSON.
"""
def verify_with_gpt_vision(video_path: str, instruction: str) -> Dict[str, Any]:
frames = sample_frames(video_path, num=6)
images = [{
"type": "input_image",
"image_data": {"b64": to_b64_jpeg(f)}
} for f in frames]
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
prompt = build_prompt(instruction)
# OpenAI "Responses" API (unified) — if you prefer Chat Completions, adapt accordingly
resp = client.responses.create(
model=MODEL,
input=[
{"role": "system", "content": "You are a precise verification assistant."},
{"role": "user", "content": [{"type": "text", "text": prompt}, *images]}
]
)
text = resp.output_text
# Attempt to parse the JSON the model returned
import json, re
m = re.search(r"\{[\s\S]*\}", text)
data = json.loads(m.group(0)) if m else {"error": "no-json", "raw": text}
return data
if __name__ == "__main__":
result = verify_with_gpt_vision(
video_path="docs/assets/videos/realistic.mp4",
instruction="turn on the stove and put the moka pot on it",
)
print(result)
```
Notes:
* We sample \~6 frames to keep token cost under control. Tune as needed.
* Use JPEG at \~85% quality and width ≤ 640 px for a good cost/quality trade‑off.
* If your provider requires a different schema for images (e.g., `image_url`), adapt the `images` payload.
## Aggregation across verifiers
To emulate multiple verifiers (Alpha/Beta/Gamma), call `verify_with_gpt_vision` 3 times with different random frame subsets or slightly perturbed prompts (“Verifier Alpha perspective” etc.) and assemble a 2D metrics table like in the demo dashboard. The FINAL row can be the average per‑column or an LLM‑based consensus.
# AI Fetch Interface
This document describes the interface specification for neural network models that control the Fetch mobile manipulator robot in ManiSkill/MS-HAB environments. Models following this specification can be used as drop-in replacements for the default control policies.
***
## Quick Reference
| Component | Specification |
| ------------------ | ------------------------------------------------ |
| **Input** | Observation (state) |
| Input Shape | `[batch_size, state_dim]` or `[state_dim,]` |
| Input Data Type | `float32` |
| Input Range | Typically `[-inf, inf]` or normalized `[-1, 1]` |
| **Neural Network** | |
| Architecture | MLP / CNN / Transformer / etc. |
| Weights Format | PyTorch (`.pt`) / ONNX (`.onnx`) / Keras (`.h5`) |
| **Output** | Action Vector |
| Output Shape | `[batch_size, 13]` or `[13,]` |
| Output Data Type | `float32` |
| Output Range | `[-1.0, 1.0]` (normalized) |
## Overview
The Fetch robot is a mobile manipulator with:
* **7-DOF arm** (shoulder, elbow, wrist joints)
* **2-finger gripper** (mimic-controlled)
* **3-DOF body** (head pan/tilt, torso lift)
* **Mobile base** (2D translation)
The control interface uses **delta position control** (`pd_joint_delta_pos` mode), where actions specify incremental changes to joint positions rather than absolute targets.
***
## Model Interface
### Input: Observation Space
**Format:** `Dict[str, Array]` or flattened `Array`
**Observation Mode:** `state` (default) or `rgbd`
#### State Observation (Recommended)
When `obs_mode="state"`, the observation format depends on the environment:
**Format 1: Flattened Tensor** (most common, e.g., `ReplicaCAD_SceneManipulation-v1`)
```python
observation = torch.Tensor[float32] # Shape: (batch_size, state_dim) or (state_dim,)
# Contains: flattened robot state + task state
# Example: Shape (1, 30) for ReplicaCAD_SceneManipulation-v1
```
**Format 2: Dictionary** (some environments)
```python
observation = {
"agent": {
# Robot state (shape: [state_dim])
"qpos": Array[float32], # Joint positions (all joints)
"qvel": Array[float32], # Joint velocities (all joints)
# ... other agent-specific state
},
"extra": {
# Task-specific state (object positions, etc.)
# Shape and content depend on the environment
},
"sensor_param": {
# Camera/sensor parameters
},
"sensor_data": {
# Camera images (if rgbd mode)
}
}
```
**State Dimensions by Environment:**
* `ReplicaCAD_SceneManipulation-v1`: **30** dimensions (flattened tensor)
* `PickCube-v1`: Typically **40-50** dimensions
* `SequentialTask-v0`: Typically **50-100** dimensions (varies by task)
**Data Type:** `float32`
**Normalization:** Observations may be normalized to `[-1, 1]` range depending on the training setup.
**Model Input Handling:** Models should handle both formats for maximum compatibility:
```python
def preprocess_observation(obs):
"""Handle both tensor and dictionary observations."""
if isinstance(obs, dict):
# Extract and flatten dictionary
if 'agent' in obs:
state = obs['agent']
else:
# Concatenate all dict values
state = torch.cat([v.flatten() for v in obs.values()], dim=-1)
else:
# Already a tensor
state = obs
# Ensure correct shape
if len(state.shape) == 1:
state = state.unsqueeze(0) # Add batch dimension if needed
return state
```
#### RGBD Observation (Alternative)
When `obs_mode="rgbd"`, the observation includes camera images:
```python
observation = {
"agent": {...}, # Same as above
"sensor_data": {
"rgb": Array[uint8], # Shape: [H, W, 3] or [N, H, W, 3]
"depth": Array[float32], # Shape: [H, W] or [N, H, W]
# Camera-specific keys
},
# ... other keys
}
```
**Note:** For model compatibility, state observations are recommended as they are more compact and environment-agnostic.
***
### Output: Action Space
**Format:** `Array[float32]`
**Shape:** `(13,)` - Fixed dimension
**Range:** `[-1.0, 1.0]` (normalized)
**Action Vector Breakdown:**
```python
action = [
a0, # [0] Arm: shoulder_pan_joint (delta position)
a1, # [1] Arm: shoulder_lift_joint (delta position)
a2, # [2] Arm: upperarm_roll_joint (delta position)
a3, # [3] Arm: elbow_flex_joint (delta position)
a4, # [4] Arm: forearm_roll_joint (delta position)
a5, # [5] Arm: wrist_flex_joint (delta position)
a6, # [6] Arm: wrist_roll_joint (delta position)
a7, # [7] Gripper: l_gripper_finger_joint (position, mimic-controlled)
a8, # [8] Body: head_pan_joint (delta position)
a9, # [9] Body: head_tilt_joint (delta position)
a10, # [10] Body: torso_lift_joint (delta position)
a11, # [11] Base: root_x_axis_joint (velocity: left/right)
a12, # [12] Base: root_y_axis_joint (velocity: forward/back)
]
```
#### Detailed Action Components
| Index | Component | Joint Name | Control Type | Range (normalized) | Physical Range |
| --------- | ------------------------ | --------------------------------- | ---------------- | ------------------ | ---------------- |
| **0-6** | **Arm** | | | | |
| 0 | `shoulder_pan_joint` | Shoulder pan (left/right) | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 1 | `shoulder_lift_joint` | Shoulder lift (up/down) | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 2 | `upperarm_roll_joint` | Upper arm roll | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 3 | `elbow_flex_joint` | Elbow flexion | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 4 | `forearm_roll_joint` | Forearm roll | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 5 | `wrist_flex_joint` | Wrist flexion | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 6 | `wrist_roll_joint` | Wrist roll | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| **7** | **Gripper** | | | | |
| 7 | `l_gripper_finger_joint` | Gripper open/close | Position (mimic) | \[-1, 1] | \[-0.01, 0.05] m |
| **8-10** | **Body** | | | | |
| 8 | `head_pan_joint` | Head pan (left/right) | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 9 | `head_tilt_joint` | Head tilt (up/down) | Delta position | \[-1, 1] | \[-0.1, 0.1] rad |
| 10 | `torso_lift_joint` | Torso lift | Delta position | \[-1, 1] | \[-0.1, 0.1] m |
| **11-12** | **Base** | | | | |
| 11 | `root_x_axis_joint` | Base translation X (left/right) | Velocity | \[-1, 1] | \[-1.0, 1.0] m/s |
| 12 | `root_y_axis_joint` | Base translation Y (forward/back) | Velocity | \[-1, 1] | \[-1.0, 1.0] m/s |
**Note:** The gripper uses a mimic controller where `r_gripper_finger_joint` automatically mirrors `l_gripper_finger_joint`.
***
## Model Architecture Requirements
### Input Processing
The model should accept:
1. **State observations** (recommended):
* Input shape: `(batch_size, state_dim)` where `state_dim` is:
* **30** for `ReplicaCAD_SceneManipulation-v1`
* **40-50** for `PickCube-v1`
* **50-100** for other environments (varies)
* Or: Dictionary with `agent` key containing state vector
* Data type: `float32`
2. **RGBD observations** (optional):
* Input shape: `(batch_size, H, W, C)` for images
* May require CNN backbone (ResNet, etc.)
* Data type: `uint8` for RGB, `float32` for depth
### Output Processing
The model must output:
* **Shape:** `(batch_size, 13)` or `(13,)` for single inference
* **Data type:** `float32`
* **Range:** `[-1.0, 1.0]` (normalized actions)
* **Activation:** `tanh` is commonly used for the final layer
### Example Architectures
#### PyTorch MLP (Minimal)
```python
import torch
import torch.nn as nn
class FetchControlModel(nn.Module):
def __init__(self, state_dim=30):
super().__init__()
self.net = nn.Sequential(
nn.Linear(state_dim, 256),
nn.ReLU(),
nn.Linear(256, 256),
nn.ReLU(),
nn.Linear(256, 256),
nn.ReLU(),
nn.Linear(256, 13),
nn.Tanh() # Ensures output in [-1, 1]
)
def forward(self, obs):
# Handle both dict and tensor observations
if isinstance(obs, dict):
if 'agent' in obs:
state = obs['agent']
else:
# Flatten all dict values
state = torch.cat([v.flatten() for v in obs.values()], dim=-1)
else:
state = obs
# Ensure correct shape
if len(state.shape) == 1:
state = state.unsqueeze(0) # Add batch dimension
return self.net(state)
```
#### Keras/TensorFlow (Includes Architecture)
```python
import tensorflow as tf
from tensorflow import keras
def create_fetch_model(state_dim=30):
"""Create a Keras model for Fetch control."""
model = keras.Sequential([
keras.layers.Dense(256, activation='relu', input_shape=(state_dim,)),
keras.layers.Dense(256, activation='relu'),
keras.layers.Dense(256, activation='relu'),
keras.layers.Dense(13, activation='tanh') # Output in [-1, 1]
])
return model
# Save model (includes architecture + weights)
model = create_fetch_model()
model.save('fetch_policy.h5') # Can be loaded directly without architecture code
```
**Advantage of Keras:** The `.h5` file contains both architecture and weights, so it can be loaded without providing the architecture definition:
```python
# Load Keras model (no need to define architecture)
model = keras.models.load_model('fetch_policy.h5')
```
***
## Model Weights Format
### Supported Formats
Models should be saved in one of the following formats:
1. **PyTorch** (`.pt` or `.pth`):
```python
torch.save({
'model_state_dict': model.state_dict(),
'config': {...}, # Optional: model config
}, 'model.pt')
```
2. **ONNX** (`.onnx`):
* Standardized format, framework-agnostic
* Can be loaded by PyTorch, TensorFlow, etc.
3. **TensorFlow/Keras** (`.h5` or SavedModel):
* **Key Advantage:** Keras models include architecture + weights in a single file
* Can be loaded directly without defining architecture separately
* Perfect for URL-based loading (like Keras model zoo)
* Example: `model = keras.models.load_model(url)` - no architecture code needed!
4. **HuggingFace Model Hub**:
* Models can be hosted and loaded via URL
* Example: `model = torch.hub.load('user/repo', 'model')`
### Model Metadata
The model file or repository should include metadata for proper loading and usage:
```json
{
"model_type": "fetch_control",
"version": "1.0",
"architecture": "mlp" | "cnn" | "transformer" | ...,
"input_type": "state" | "rgbd",
"input_dim": 30, # For ReplicaCAD_SceneManipulation-v1, or shape for images
"output_dim": 13,
"control_mode": "pd_joint_delta_pos",
"normalization": {
"obs_mean": [0.0, 0.0, ...], # Mean for each observation dimension
"obs_std": [1.0, 1.0, ...] # Std for each observation dimension
},
"training_config": {
"env_id": "PickCube-v1" | "ReplicaCAD_SceneManipulation-v1" | ...,
"algorithm": "SAC" | "PPO" | "BC" | "ACT" | ...,
"seed": 42,
"total_timesteps": 1000000
},
"framework": "pytorch" | "tensorflow" | "onnx",
"device": "cuda" | "cpu"
}
```
**Metadata Location:**
* For PyTorch: Include in checkpoint dict or separate `config.json`
* For Keras: Stored in model file or `config.json`
* For HuggingFace: In repository root as `config.json`
***
## Model Loading Interface
### URL-Based Loading
Models should be loadable via URL or local path:
```python
def load_fetch_model(model_url: str, device: str = "cuda"):
"""
Load a Fetch control model from URL or local path.
Args:
model_url: URL (http/https) or local file path
device: Device to load model on ("cuda" or "cpu")
Returns:
model: Loaded model ready for inference
metadata: Model metadata dict
"""
# Implementation depends on model format
# Example for PyTorch:
if model_url.startswith("http"):
# Download from URL
model_path = download_model(model_url)
else:
model_path = model_url
# Load model
checkpoint = torch.load(model_path, map_location=device)
model = create_model_from_checkpoint(checkpoint)
model.eval()
return model, checkpoint.get('metadata', {})
```
### Example Usage
```python
import torch
from fetch_model_loader import load_fetch_model
# Load model from URL
model, metadata = load_fetch_model(
"https://huggingface.co/user/fetch-policy/resolve/main/model.pt"
)
# Or from local path
model, metadata = load_fetch_model("./checkpoints/fetch_policy.pt")
# Inference
obs = env.reset()
with torch.no_grad():
action = model(obs) # Shape: (13,)
action = action.clamp(-1.0, 1.0) # Ensure valid range
env.step(action)
```
***
## Environment Compatibility
### Required Environment Settings
```python
env = gym.make(
env_id, # e.g., "PickCube-v1", "SequentialTask-v0"
num_envs=1,
obs_mode="state", # or "rgbd" if model supports it
render_mode="rgb_array",
sim_backend="gpu", # or "cpu"
robot_uids="fetch",
control_mode="pd_joint_delta_pos", # Must match!
)
```
### Observation Preprocessing
If the model was trained with normalized observations:
```python
def preprocess_obs(obs, metadata):
"""Normalize observation using model's statistics."""
if isinstance(obs, dict):
state = obs['agent']
else:
state = obs
if 'normalization' in metadata:
mean = metadata['normalization']['obs_mean']
std = metadata['normalization']['obs_std']
state = (state - mean) / (std + 1e-8)
return state
```
***
## Testing Model Compatibility
### Validation Checklist
Before using a model, verify:
* [ ] **Input shape matches:** Model accepts observation of correct dimension
* [ ] **Output shape is (13,):** Model outputs 13-dimensional action vector
* [ ] **Output range is \[-1, 1]:** Actions are properly normalized
* [ ] **Control mode matches:** Model trained with `pd_joint_delta_pos`
* [ ] **Observation mode matches:** Model trained with `state` or `rgbd` as specified
* [ ] **Framework compatibility:** Model can be loaded in your environment
### Test Script
```python
def test_model_compatibility(model, env):
"""Test if model is compatible with environment."""
obs, _ = env.reset()
# Test single observation
with torch.no_grad():
action = model(obs)
# Validate output
assert action.shape == (13,), f"Expected shape (13,), got {action.shape}"
assert action.min() >= -1.0 and action.max() <= 1.0, \
f"Actions must be in [-1, 1], got range [{action.min()}, {action.max()}]"
# Test step
obs, reward, done, truncated, info = env.step(action)
print("✅ Model is compatible!")
return True
```
***
## Example Model Repositories
### Format for HuggingFace
```
fetch-policy/
├── model.pt # PyTorch weights
├── config.json # Model metadata
├── README.md # Documentation
└── requirements.txt # Dependencies
```
### Format for Local Storage
```
models/
├── fetch_policy_v1.pt
├── fetch_policy_v1.json # Metadata
└── fetch_policy_v1_README.md
```
***
## Model Loading from URL
### Supported URL Formats
Models can be loaded from:
1. **HuggingFace Model Hub:**
```python
model_url = "https://huggingface.co/user/fetch-policy/resolve/main/model.pt"
```
2. **Direct HTTP/HTTPS URLs:**
```python
model_url = "https://example.com/models/fetch_policy.pt"
```
3. **Local file paths:**
```python
model_url = "./checkpoints/fetch_policy.pt"
model_url = "/path/to/model.pt"
```
### Implementation Example
```python
import torch
import urllib.request
from pathlib import Path
def load_model_from_url(model_url: str, device: str = "cuda"):
"""
Load a Fetch control model from URL or local path.
Supports:
- HuggingFace model URLs
- Direct HTTP/HTTPS URLs
- Local file paths
Args:
model_url: URL or local path to model file
device: Device to load model on ("cuda" or "cpu")
Returns:
model: Loaded model ready for inference
metadata: Model metadata dict (if available)
"""
# Download if URL
if model_url.startswith("http"):
print(f"Downloading model from {model_url}...")
model_path = Path("/tmp/fetch_model.pt")
urllib.request.urlretrieve(model_url, model_path)
else:
model_path = Path(model_url)
if not model_path.exists():
raise FileNotFoundError(f"Model not found: {model_url}")
# Load checkpoint
checkpoint = torch.load(model_path, map_location=device)
# Extract model state and metadata
if isinstance(checkpoint, dict):
state_dict = checkpoint.get('model_state_dict', checkpoint.get('state_dict', checkpoint))
metadata = checkpoint.get('metadata', {})
else:
state_dict = checkpoint
metadata = {}
# Create model architecture (user must provide)
# This is framework-specific
model = create_model_architecture(metadata) # User implements this
model.load_state_dict(state_dict)
model.eval()
model.to(device)
return model, metadata
```
### Keras/TensorFlow Models
For Keras models (which include architecture + weights in a single file):
```python
import tensorflow as tf
from tensorflow import keras
import urllib.request
from pathlib import Path
def load_keras_model_from_url(model_url: str):
"""
Load Keras model from URL.
Keras models (.h5 or SavedModel) include both architecture and weights,
so no need to specify architecture separately - this is the key advantage!
Args:
model_url: URL or local path to .h5 or SavedModel directory
Returns:
model: Loaded Keras model ready for inference
"""
if model_url.startswith("http"):
# Download using Keras utility or manual download
print(f"Downloading model from {model_url}...")
model_path = tf.keras.utils.get_file(
"fetch_model.h5",
model_url,
cache_subdir="models",
cache_dir="/tmp"
)
else:
model_path = model_url
if not Path(model_path).exists():
raise FileNotFoundError(f"Model not found: {model_url}")
# Keras automatically loads architecture + weights
# This is why Keras is convenient - no need to define architecture!
model = keras.models.load_model(model_path)
print(f"✅ Model loaded from {model_path}")
print(f" Input shape: {model.input_shape}")
print(f" Output shape: {model.output_shape}")
return model
# Usage
model = load_keras_model_from_url("https://example.com/models/fetch_policy.h5")
# Or from HuggingFace (if they support direct .h5 links)
model = load_keras_model_from_url("https://huggingface.co/user/fetch-policy/resolve/main/model.h5")
```
**Key Advantage of Keras:** The model file is self-contained - it includes:
* Architecture definition (layers, connections)
* Weights (trained parameters)
* Optimizer state (optional)
* Training configuration (optional)
This means you can load a Keras model with just:
```python
model = keras.models.load_model(url_or_path)
```
No need to define the architecture separately!
***
## Summary
| Component | Specification |
| --------------------- | -------------------------------------------------------------------------------------- |
| **Input** | Observation dict or flattened array, shape `(batch_size, state_dim)` or `(state_dim,)` |
| **Input Dimension** | Typically 30-100 (varies by environment) |
| **Output** | Action array, shape `(13,)`, range `[-1.0, 1.0]`, dtype `float32` |
| **Control Mode** | `pd_joint_delta_pos` (required) |
| **Robot** | Fetch mobile manipulator |
| **Action Components** | 7 arm joints + 1 gripper + 3 body joints + 2 base velocities |
| **Model Format** | PyTorch (`.pt`), ONNX (`.onnx`), or Keras (`.h5`) |
| **Loading** | URL (HTTP/HTTPS/HuggingFace) or local path supported |
| **Architecture** | User-defined (MLP, CNN, Transformer, etc.) |
***
## Action Vector Visualization
**Action Vector (13 dimensions)**
| Index | Component | Joint Name | Control Type | Description |
| --------- | ---------------- | ------------------------ | ------------ | ------------------------------------- |
| **0-6** | **Arm (7 DOF)** | | | |
| 0 | Shoulder Pan | `shoulder_pan_joint` | Δpos | Shoulder rotation (left/right) |
| 1 | Shoulder Lift | `shoulder_lift_joint` | Δpos | Shoulder elevation (up/down) |
| 2 | Upper Arm Roll | `upperarm_roll_joint` | Δpos | Upper arm rotation |
| 3 | Elbow Flex | `elbow_flex_joint` | Δpos | Elbow flexion |
| 4 | Forearm Roll | `forearm_roll_joint` | Δpos | Forearm rotation |
| 5 | Wrist Flex | `wrist_flex_joint` | Δpos | Wrist flexion |
| 6 | Wrist Roll | `wrist_roll_joint` | Δpos | Wrist rotation |
| **7** | **Gripper** | | | |
| 7 | Gripper | `l_gripper_finger_joint` | pos | Gripper open/close (mimic-controlled) |
| **8-10** | **Body (3 DOF)** | | | |
| 8 | Head Pan | `head_pan_joint` | Δpos | Head rotation (left/right) |
| 9 | Head Tilt | `head_tilt_joint` | Δpos | Head tilt (up/down) |
| 10 | Torso Lift | `torso_lift_joint` | Δpos | Torso vertical movement |
| **11-12** | **Base (2 DOF)** | | | |
| 11 | Base X | `root_x_axis_joint` | vel | Base translation X (left/right) |
| 12 | Base Y | `root_y_axis_joint` | vel | Base translation Y (forward/back) |
**Legend:**
* **Δpos** = Delta position (incremental change from current position)
* **pos** = Absolute position target
* **vel** = Velocity control
***
## References
* [ManiSkill Documentation](https://maniskill.readthedocs.io/)
* [MS-HAB Repository](https://github.com/...)
* Fetch Robot URDF: Included in ManiSkill assets
* Control Modes: See `mani_skill/agents/robots/fetch/fetch.py`
# Overview
A **subnet** in Konnex is a **workload class**: a real commercial pattern (aerial inspection, manipulation, 3D mapping) with a defined input schema, scoring function, and open miner competition. The design follows the same coordination **shape** as [Bittensor](https://docs.learnbittensor.org/)-style subnets, while Konnex specializes verification for physical evidence, not only statistical text quality.
## Testnet: three live subnets
| Subnet | Role |
| -------------------- | -------------------------------------------------------------------------------------------------- |
| **Drone navigation** | Vision–language flight; scoring on task match, safety, and execution |
| **Roboarm VLA** | Text-directed VLA manipulation (bi-manual, kitchen-grade examples); safety, task match, efficiency |
| **SLAM 3D map** | Sensor bundles; mesh / semantic map quality vs ground truth |
**Dashboard URLs** for each workload will be published in official release notes when they are finalized. **Get started:** [testnet quest flow](https://subnets.testnet.konnex.world/quest) and release notes.
Signed enterprise counterparties participate in these workload classes; full commercial runs may still live in private environments while the public chain proves verification and the roadmap advances mainnet settlement. The testnet is the place to try the [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw) loop end to end.
## How a subnet is structured (conceptual)
* Task schema and prompt format
* KPIs and verifier checks (simulation, replay, sensor signatures, held-out data)
* Scoring weights and reward split (testKNX on testnet; design target includes stablecoin settlement on mainnet per the [roadmap](/understand-konnex/roadmap))
* APIs and binaries for miners and validators (see the [SDK](/sdk/sdk) and [konnex-network on GitHub](https://github.com/konnex-network))
## Pages in this section
* [Drone navigation](/subnets-workload-classes/drone-navigation)
* [Roboarm VLA](/subnets-workload-classes/roboarm-vla)
* [SLAM 3D map](/subnets-workload-classes/slam-3d-map)
## Related reading
* [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw)
* [Mining](/participate/mining) · [Validating](/participate/validating)
# Drone navigation
**Workload class:** vision–language-driven autonomous flight.
Operators issue **text missions**; miners compete on **navigation and execution policies**; validators score whether the run matches the **task instruction** (safety, task match, geospatial correctness). [Proof-of-Physical-Work (PoPW)](/understand-konnex/contracts-and-popw) binds the instruction hash, policy trace, and hardware-attested sensor data.
## Testnet
The public dashboard URL for this workload will be announced with the testnet release.
## Task shape (illustrative)
Users typically send natural-language instructions such as: *"Fly to the red-brick building and photograph the roof."* The exact schema (fees, rewards, attachments) is defined by the live subnet UI and chain integration.
## Why this matters
Aerial data often underwrites insurance, maintenance, and engineering decisions. A self-reported log from the same party that operated the aircraft is not a substitute for independently scored evidence tied to a signed, onchain instruction.
## See also
* [OpenFly & OpenFly-Agent](/supported-ai-models/openfly) — aerial VLN / language-driven flight
* [Subnets overview](/subnets-workload-classes/subnets)
* [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw)
# Roboarm VLA
**Workload class:** text-directed **vision–language–action (VLA)** manipulation — bi-manual and kitchen-grade scenarios are common examples; the subnet is not limited to food service.
On testnet, operators issue **task prompts**; miners compete on **trajectories and policies** from VLA / control stacks; validators score candidates on **safety**, **task match**, and **efficiency**. Outputs feed [Proof-of-Physical-Work (PoPW)](/understand-konnex/contracts-and-popw) records on the Konnex testnet.
## Testnet
The public dashboard URL for this workload will be announced with the testnet release.
## Task shape (illustrative)
The live UI and chain define the exact fields. Example natural-language tasks:
* *"Pick the apple from the pan and place it on the counter."*
* *"Dice tomatoes and place them into the pan."*
A JSON sketch:
```json
{
"jobId": "sha3(signed-packet)",
"prompt": "dice tomatoes and place into pan",
"arena": "KitchenSim-v1",
"deadline": 120,
"rewardTestKNX": "…",
"kpi": {"time_s": 90, "success": true, "spills": 0}
}
```
Use the fields and token type shown in the testnet product (e.g. testKNX).
## Miner output
* Policy or VLA bundle (e.g. WASM or integration endpoint) with deterministic seeds where the subnet requires them
* Declared KPIs and stake per network rules
## Validation flow (conceptual)
1. Deterministic replay in the subnet’s simulator or harness when applicable.
2. KPI extraction and safety checks.
3. ScoreRoot emission; rewards or slashing per onchain parameters.
## Example prompt
"Pick the red bell pepper, slice into strips, sauté for 2 minutes, then place on the plate."
## See also
* [Subnets overview](/subnets-workload-classes/subnets)
* [OpenVLA](/supported-ai-models/openvla) · [PI-0 / PI-0.5](/supported-ai-models/ai)
* [Decentralized AI ecosystem](/understand-konnex/ai-ecosystem)
# SLAM 3D map
**Workload class:** 3D mapping and **SLAM** from sensor submissions — mesh fidelity, semantic structure, and agreement with ground truth where the subnet defines it.
Miners compete on reconstruction quality. **Sensor bundles** (camera, IMU, LiDAR, or other channels as required) are part of the PoPW record; validators score outputs **independently** of the operator who captured the data.
## Testnet
The public dashboard URL for this workload will be announced with the testnet release.
## Task shape (illustrative)
Tasks may request maps, reconstructions, semantic labeling, or comparison to held-out references, depending on subnet configuration. Use the live dashboard for the current fields and testKNX costs when URLs are published.
## Why this matters
3D and SLAM outputs support mine engineering, built-environment reviews, and industrial digital twins. Verifiable provenance matters: not only a convincing mesh, but evidence that data came from a real run aligned to a defined instruction.
## See also
* [ORB-SLAM3 & RTAB-Map](/supported-ai-models/orb-slam3) — reference SLAM / 3D mapping stacks
* [Subnets overview](/subnets-workload-classes/subnets)
* [Proof-of-Physical-Work](/understand-konnex/contracts-and-popw)
# Overview
The Konnex SDK lets you:
* Join a SubNet as a miner (AI provider) or validator
* Attach a robot/edge device to submit/execute jobs
* Interact over CLI, Python, or HTTP APIs
## Install (from source)
We ship the SDK inside this monorepo so you can develop and test end‑to‑end (CLI, Python client, verifier tools, demo dashboard).
Prerequisites:
* git, conda or miniconda
* Python 3.10+ (we build/test with 3.10)
* ffmpeg on PATH (for frame extraction in verifiers), optional
Create an isolated env and install editable packages:
```bash
# 1) Clone
git clone https://github.com/konnex-world/konnex.git
cd konnex
# 2) Conda env
conda create -n knx python=3.10 -y
conda activate knx
# 3) Install SDK pieces
pip install -e ./sdk/python-client
pip install -e ./sdk/cli
pip install -e ./tools
```
## Next steps
* CLI: [sdk/cli.md](/sdk/cli)
* Python: [sdk/python.md](/sdk/python)
* HTTP API: [sdk/http.md](/sdk/http)
* Robots: [sdk/robots.md](/sdk/robots)
* Validators: [sdk/validators.md](/sdk/validators)
# CLI
## Join as Miner
```bash
konnex miner join --subnet sn.roboarm.sim \
--stake-usd1 10 \
--model ./out/policy.wasm \
--kpi success>=0.98,time_s<=120
```
## Join as Validator
```bash
konnex validator join --subnet sn.roboarm.sim \
--stake-knx 1000 --assurance-usd1 200 \
--sim KitchenSim-v1
```
## Submit a Task (User)
```bash
konnex task submit --subnet sn.roboarm.sim \
--prompt "dice tomatoes and place into pan" \
--reward-usd1 12.5 --stake-usd1 5 --deadline 120
```
## Prove Work (Robot/Executor)
```bash
konnex prove --job-id --bundle ./powp.zip
```
# Python SDK
```python
from konnex import KonnexClient
client = KonnexClient()
# Submit task
job_id = client.tasks.submit(
subnet="sn.roboarm.sim",
prompt="pick pepper, slice, sauté 2 minutes, plate",
reward_usd1=12.5,
stake_usd1=5,
deadline_s=120,
)
# Miner: register a model
client.miners.register(
subnet="sn.roboarm.sim",
wasm_path="./out/policy.wasm",
kpi={"success": ">=0.98", "time_s": "<=120"},
stake_usd1=10,
)
# Validator: verify PoPW bundle
score = client.validators.verify(job_id=job_id, bundle_path="./powp.zip")
print(score)
```
# HTTP API
JSON over HTTPS. Base URL and authentication match your deployment and the same settings you use for the CLI.
## Submit task
POST `/api/v1/tasks`
```json
{
"subnet": "sn.roboarm.sim",
"prompt": "dice tomatoes and place into pan",
"rewardStable": 12.5,
"stakeStable": 5,
"deadline": 120
}
```
## Register miner model
POST `/api/v1/miners/models`
```json
{
"subnet": "sn.roboarm.sim",
"modelUri": "ipfs://.../policy.wasm",
"kpi": {"success": ">=0.98", "time_s": "<=120"},
"stakeStable": 10
}
```
## Validator verify bundle
POST `/api/v1/validators/verify`
```json
{ "jobId": "...", "bundleUri": "s3://bucket/powp.zip" }
```
# Connecting Robots
## Attach an Executor
```bash
konnex robot attach --robot-id roboarm-01 \
--subnet sn.roboarm.sim --keys ./keys.json
```
## Stream Telemetry & Build PoPW Bundle
```bash
konnex robot record --job-id \
--sensors camera,imu,torque,thermal --out ./powp.zip
```
## Send Bundle
```bash
konnex prove --job-id --bundle ./powp.zip
```
# Validators
Validators run subnet-specific checkers and publish ScoreRoots. On testnet, follow published binaries and parameters from [konnex-network](https://github.com/konnex-network) for **drone navigation**, **roboarm VLA**, or **SLAM 3D map** workloads.
> The commands below are **illustrative**. Replace `--subnet` and stake flags with the values from the **release** you deploy.
## Run Validator
```bash
konnex validator join --subnet sn.roboarm.sim \
--stake-knx 1000 --assurance-usd1 200 \
--evidence gps,imu,video
konnex validator serve --subnet sn.roboarm.sim \
--listen 0.0.0.0:5940 --workers 8
```
## Handle Verify Events
```bash
konnex events tail --kind verify | xargs -I {} konnex validator verify --event {}
```