Meta has released SAM 3.1, a significant update to its Segment Anything Model 3, introducing Object Multiplex — a shared-memory architecture that enables joint multi-object tracking at substantially higher speeds than SAM 3, with no loss in accuracy. The update, quietly published on GitHub and Hugging Face in late March, represents the most important improvement to Meta’s open-source vision foundation model since SAM 3’s initial release in July 2025.
SAM 3.1 builds on the foundation of SAM 3 (Segment Anything with Concepts), Meta’s unified model for promptable segmentation in images and videos. Unlike SAM 2, which could segment any object but required separate inference runs for each concept, SAM 3 introduced the ability to exhaustively segment all instances of an open-vocabulary concept using a single text phrase, visual exemplar, or combination of prompts. SAM 3.1 takes this further by making multi-object video tracking dramatically more efficient.
What Is SAM 3?
SAM 3 is a foundational computer vision model developed by Meta Superintelligence Labs (MSL), led by a large team of researchers including Nicolas Carion, Laura Gustafson, and project leads Piotr Dollar, Kate Saenko, and Christoph Feichtenhofer. The model is described in the paper “SAM 3: Segment Anything with Concepts” and is available under a research license on GitHub, where it has already gathered nearly 10,000 stars and over 1,400 forks.
The architecture is a 848M-parameter model combining a detector and a tracker that share a vision encoder:
| Component | Architecture | Role |
|---|---|---|
| Detector | DETR-based, conditioned on text, geometry, and image exemplars | Identifies objects matching a concept prompt |
| Tracker | SAM 2 transformer encoder-decoder architecture | Tracks segmented objects across video frames |
| Encoder | Shared vision backbone | Extracts features for both detection and tracking |
What Makes SAM 3 Different from SAM 2
The headline innovation in SAM 3 is concept-conditioned segmentation. Where SAM 2 could segment “everything” based on point, box, or mask prompts, SAM 3 can segment “everything matching this concept” using natural language. This is a fundamentally different paradigm:
- Text prompting: “Find all red cars” → segments every red car in the image or video
- Visual exemplars: Show one instance → segment all similar instances
- Combined prompts: “Find this type of bird” + an image example → precise concept segmentation
This capability is enabled by the model’s DETR-based detector, which is conditioned on text embeddings, geometric features, and visual exemplars simultaneously. The detector outputs bounding boxes and masks for every instance of the target concept, while the tracker maintains identity across frames.
SAM 3.1: What Changed
SAM 3.1, released March 27, 2026, introduces Object Multiplex — a new shared-memory approach for joint multi-object tracking.
The Multiplex Problem
In SAM 3, tracking multiple distinct objects required running the tracker independently for each object or concept. If you wanted to track “pedestrians,” “vehicles,” and “traffic signs” simultaneously in a video, you’d need three separate inference passes — each consuming GPU memory proportional to the number of detected instances.
This approach scales linearly with the number of tracked concepts, quickly exhausting GPU memory for complex scenes.
Object Multiplex: The Solution
SAM 3.1’s Object Multiplex introduces a shared memory bank that all tracked objects read from and write to simultaneously. Instead of maintaining separate memory states per object or per concept, the multiplexer uses a single, unified memory representation that stores features for all tracked instances:
| Feature | SAM 3 (before) | SAM 3.1 (with Multiplex) |
|---|---|---|
| Memory architecture | Per-object memory states | Shared multiplexed memory |
| Multi-concept tracking | Sequential, O(n) passes | Joint, O(1) pass |
| GPU memory | Scales with object count | Near-constant regardless of object count |
| Frame processing | Independent per-object | Batched with shared features |
The result is significantly faster inference — Meta reports speed improvements without sacrificing accuracy across the SA-V and YT-Temporal-1B benchmarks. The multiplexer handles occlusion, object re-entry, and identity assignment automatically, using global reasoning across the entire memory bank.
Global Reasoning
SAM 3.1 also adds global reasoning capabilities — the model can now consider the full video context when making tracking decisions. Rather than making frame-by-frame predictions based on local context, global reasoning allows the model to:
- Re-identify objects that temporarily leave the frame
- Resolve ambiguous occlusion scenarios by reasoning across timestamps
- Maintain consistent object identities even during rapid motion
This is particularly impactful for surveillance, sports analysis, wildlife monitoring, and autonomous driving applications where objects regularly exit and re-enter the frame.
Benchmarks and Performance
Meta released a suite of improved model checkpoints on Hugging Face (facebook/sam3.1) alongside SAM 3.1. The updated checkpoints show:
- SA-V benchmark: Competitive or improved performance on video segmentation accuracy
- YT-Temporal-1B: Strong results on large-scale video understanding
- Real-time throughput: Dramatically improved frames-per-second for multi-object tracking scenarios
The SA-Co dataset, released with SAM 3, continues to serve as the primary evaluation benchmark. It includes:
- SA-Co/Gold: Human-verified image benchmarks with annotated noun phrases
- SA-Co/Silver: Larger-scale automatically annotated image benchmarks
- SA-Co/VEval: Video evaluation dataset with instance masks and unique object IDs
How to Use SAM 3.1
SAM 3.1 requires Python 3.12+, PyTorch 2.7+, and a CUDA-compatible GPU with CUDA 12.6+. Installation is straightforward:
git clone https://github.com/facebookresearch/sam3.git
cd sam3
pip install -e .
Then use the model for text-prompted image segmentation:
from sam3.model_builder import build_sam3_image_model
from sam3.model.sam3_image_processor import Sam3Processor
model = build_sam3_image_model()
processor = Sam3Processor(model)
inference_state = processor.set_image("photo.jpg")
output = processor.set_text_prompt(
state=inference_state,
prompt="all red cars"
)
Or for video tracking with the new multiplexer:
from sam3.model_builder import build_sam3_video_predictor
predictor = build_sam3_video_predictor()
# SAM 3.1 automatically uses Object Multiplex
# for multi-concept tracking
The Broader Context
SAM 3.1 arrives at a time when Meta is doubling down on both open-source computer vision and its Superintelligence Labs (MSL) — the same division behind Meta’s closed-source Muse Spark model. The SAM family has been one of Meta’s most successful open-source contributions, with the original SAM accumulating over 54,000 GitHub stars and SAM 2 nearly 20,000.
The release also signals Meta’s continuing commitment to open-weight vision models even as its flagship language model (Llama 4) remains unreleased and its Muse Spark model is hosted-only. SAM 3.1 is fully open-source, with model weights available on Hugging Face and code on GitHub — a strategy that has made Meta’s vision models the de facto standard for research and commercial applications alike.
For the broader AI ecosystem, SAM 3.1’s multiplexing architecture is an important technical contribution. The shared-memory tracking pattern could influence other domains where multi-instance reasoning is required — from multi-agent tracking in robotics to simultaneous object detection in autonomous systems. This mirrors the broader shift toward multi-agent architectures documented in our Ultimate Guide to Open Source AI Agent Frameworks.
Summary
| Aspect | Detail |
|---|---|
| Model | SAM 3.1 (Segment Anything Model 3.1) |
| Key innovation | Object Multiplex — shared-memory multi-object tracking |
| Release date | March 27, 2026 (updated model pushed May 16) |
| Parameters | 848M |
| License | Research license (open weights) |
| GitHub | facebookresearch/sam3 |
| Hugging Face | facebook/sam3.1 |
| Paper | SAM 3: Segment Anything with Concepts |
SAM 3.1 may not have the flash of a new LLM release, but for anyone building computer vision systems that need real-time, open-vocabulary object detection and tracking, it’s one of the most practical releases of the year — and a reminder that Meta’s open-source AI contributions extend far beyond the Llama family.