PanoVLD: Fast Multimodal Decision Model

Automation and physical AI often come down to two things: understanding a scene, and making a concrete decision about it, usually by picking from a short, fixed list. Is the gate open? Is anyone in the loading bay? Is the warning light on that machine red, yellow or green? Vision-language models can answer such questions, but only by generating text token by token. That text then has to be parsed, and it comes with no honest estimate of how confident the model is.

TypeSafe AI's Jev showed a better route. Jev is a decision model: it takes a state and a set of typed questions (pick one of these options, give a score, is this statement true?) and answers all of them in a single parallel pass, each with a calibrated probability. No text is generated and nothing needs parsing, so code can act when the model is confident and escalate when it is not. TypeSafe calls these System One models, built for fast, structured decisions that software can use directly. But Jev's state is text or JSON, and a camera frame is neither.

PanoVLD-500M brings decision models to vision. It takes an image, a question and its options, and returns a calibrated probability for each option in one forward pass, fast enough to run on an edge device next to the camera.

Model architecture

PanoVLD-500M is our PanoVLM vision-language model with one change: its text-generation head is replaced by a small decision head. A FastViT-HD vision encoder turns the image into tokens, a linear projector maps them into the language model, and PanoLM, our hybrid language model in which only one layer in six uses full attention, reads them together with the question and its options. The decision head then turns one hidden state per option into a score, and a softmax turns the scores into probabilities.

... USER: <image x144> choice question: What color is the overhead signal light on the machine bay in front of the robot? ASSISTANT: Options:
 - Green (Clear)⏎
 - Yellow (Caution)⏎
 - Red (Stop / Fault)⏎
 - Off⏎

PanoVLD handles three kinds of question, and every answer comes back as a probability for each possible outcome:

In every case the probabilities are calibrated: answers given with 80% confidence are right about 80% of the time, so a system can act when the model is sure and ask for help when it is not. You can see this at work in the demos below.

Results

We compared PanoVLD-500M with six open decision models of up to 853M active parameters, each run through its authors' released code: Laya Vision (its current 201M and earlier 237M checkpoints), Liquid AI's d1-omni-600M, vLLM-SR's d3-edge, JPT-0.8B and OneJev-0.8B. All seven models answer the same questions from eleven benchmarks in three groups:

• Laya Vision's three sets: A-OKVQA, ScienceQA and VQAv2 yes/no, with everyday photos and science diagrams.
• Visual-Jev's four sets: choice versions of GQA, SNLI-VE, TextVQA and TallyQA.
• Laya Vision's four rating sets: score questions on VLFeedback (answers about an image, rated for helpfulness and faithfulness), RichHF-18K (generated images), AVA (photo aesthetics) and CrisisMMD (damage in disaster photos).

Training data

A benchmark score only means something if the model has never seen the test images. We compared every training image with every test image of the eleven benchmarks, counting resized and re-encoded copies as matches, and removed each training question with a match: 1.5% of our training questions.

The same check found a leak in The Cauldron, a public training collection that many open vision-language models use: it contains 551 of the 2,097 ScienceQA questions we evaluate on, with the same image, question and answer. To keep the comparison fair, we score every model on the other 1,546.

Accuracy Analysis

Each benchmark is scored by plain accuracy: the share of its questions on which the answer the model gives the highest probability is the right one. On the rating sets, that means picking the right level of the scale. All seven models answer the same 21,053 test questions, and a model's mean accuracy weighs the eleven benchmarks equally. The radar chart below plots the accuracy on every benchmark for PanoVLD-500M and three of the other models; the buttons above it add the rest, and Show table lists every number.

PanoVLD-500M is the most balanced; the only model within 5 points of the best on every question type (choice, yes / no and score). The four models never trained on rating questions collapse on the rating sets, scoring 30 to 34, worse than always guessing each set's most common answer (54.8). Laya Vision, the one other model trained on ratings, falls about 20 points behind us on Visual-Jev's sets. PanoVLD-500M leads on Visual-Jev's sets, matches Laya Vision on the rating sets and stays within 8 points of the best on Laya Vision's own sets. That balance earns it the highest mean accuracy of the seven, 70.5 against 62.7 for the runner-up, in a model just over half the size of the two largest.

Latency Analysis

PanoVLD is built to run next to the camera. We timed it on an NVIDIA Jetson AGX Orin, an embedded computer made for robots, and on an NVIDIA A100 data-center GPU. Each call answers 16 questions about one photo and encodes the image only once. On the Orin, a call takes 202 ms: 79 questions per second, or sixteen questions about a frame five times a second. The A100 makes the same call in 48 ms.

To compare PanoVLD-500M with the other six models, we timed all seven on one NVIDIA A100 with Laya Vision's own latency protocol: its 512-pixel test card and 15 typed questions about it, timed from a decoded image to Python values, preprocessing included. Each model ran through its released code at its fastest, with CUDA graphs wherever they helped. On this test, PanoVLD-500M answers one question in 18 ms and all fifteen in 43 ms.

Demos

Here is where PanoVLD fits in physical AI. A mobile robot on a factory floor has to keep checking its surroundings: whether the people near it are safe, what state the machines it passes are in, and whether its path is clear. Each check is a question with a fixed set of answers, and each answer is a probability the robot could act on directly: slow down, reroute or raise an alert. The three clips below show what the robot's camera sees, with PanoVLD's answers updating beside each one. All the questions about a frame are answered together, in one call.

Try it

PanoVLD-500M is small, fast and calibrated: a component that sits behind a camera or inside an agent and turns a question with known answers into a probability your code can act on. The model is available on Hugging Face.

Hugging Face PanocularAI/PanoVLD-500M Image + text → decision482.5M paramsChoice · Yes / no · Score View model →

← All posts