Understanding XMem Through Synthetic Benchmarks

XMem is a video object segmentation model built around an explicit memory system.

Unlike an object detector, it does not discover the target from a category prompt. It receives an initial segmentation mask that identifies the object of interest, then propagates that object identity through the rest of the video.

The underlying idea is broader than segmentation: preserve a compact, searchable representation of earlier observations so that information can be recovered over long sequences.

I wanted to understand what this memory mechanism costs, what the initial mask actually does, and how the model behaves when the target is occluded. I built a small synthetic sequence, ran XMem on my RTX 5090, counted FLOPs across its inference phases, and recorded the IoU before and after occlusion.

The main results were:

• XMem propagated a first-frame target mask without fine-tuning on the synthetic video.
• It preserved target identity in the presence of a similar-looking distractor.
• Its predicted mask disappeared under full occlusion and recovered as the target became visible again. It didn’t confound the occluded object with the similar looking one still visible.
• At 480 short-side resolution, it processed the 180-frame sequence at ~64 FPS using ~940 MiB of peak CUDA allocation with AMP on my 5090.
• At 720 short-side resolution, throughput fell to ~42 FPS and peak allocation increased to ~2500 MiB.
• At 480p, mean visible-target IoU was 0.864; the prediction was correctly empty for all six fully occluded frames.
• The segmentation decoder, not the memory read alone, dominated propagation FLOPs.

The task is mask propagation, not object detection

XMem solves semi-supervised video object segmentation. For one object, its initial inputs are conceptually:

RGB frame:    [3, H, W]
object mask:  [1, H, W]

The mask answers a question that RGB alone cannot answer: which object should the model follow? This matters when a frame contains multiple plausible targets. The model does not need a semantic label such as “car” or “person,” but it does need a seed that establishes object identity.

The frame is encoded into spatial features. XMem creates compact keys for matching and richer, mask-conditioned values for object information. At a high level:

initial frame + mask -> object-specific key/value memory
current frame        -> query keys
memory match         -> retrieved object features
retrieved features   -> current-frame mask

The synthetic frames are generated at 864x480. Both dimensions are divisible by 16, the stride-16 feature grid is 54x30, or 1,620 positions. With the released checkpoint, a simplified single-object memory entry has shapes similar to:

key:    [1, 64, 30, 54]
value:  [1, 1, 512, 30, 54]

The key is shared matching information. The value includes an object dimension because it carries information conditioned by the input mask. This distinction is the core of the model: use a relatively small representation to find relevant memories, then retrieve a larger representation containing target-specific detail.

A synthetic occlusion test

The synthetic sequence includes three elements:

• a masked target moving through the frame;
• a similar red distractor following a different trajectory;
• an opaque foreground occluder that fully covers the target during part of the sequence.

Only the first-frame target mask is placed in XMem’s Annotations input. XMem must propagate that initial identity through all later frames without correction. A separate EvaluationMasks directory contains the visible target mask for every frame. These evaluation masks subtract the opaque occluder and are never passed to the model. A second evaluation-only mask tracks the visible distractor.

The following contact sheet samples the sequence before, during, and after occlusion. Each sample contains two panels. The left panel is the RGB frame with XMem’s prediction overlaid in cyan. The right panel isolates the predicted foreground mask. The reported area is the number of predicted foreground pixels, while the bounding box gives the prediction’s extent in the original frame coordinates.

This makes the failure and recovery sequence visible rather than reducing it to one aggregate metric:

• Before occlusion, XMem followed the seeded target.
• As the occluder covered the target, the predicted foreground area shrank.
• Under full occlusion, the prediction became empty.
• As the target reappeared, XMem first recovered partial masks and then returned to the target.
• It did not visibly switch identity to the similar distractor in the sampled frames.

This is not evidence that XMem predicts an object’s hidden extent. It did not produce an amodal mask behind the occluder. The result is better understood as visible-region segmentation with identity recovery after reappearance.

The per-frame masks turn this into a quantitative test. IoU is computed against the visible target rather than its hidden full extent. Empty-vs-empty IoU is undefined, so the six fully occluded frames are excluded from mean IoU and reported separately. XMem produced an empty prediction on all six. On the remaining frames, the 480p AMP run achieved:

The target first reappeared at frame 69 and crossed IoU 0.5 at frame 71, a recovery delay of two frames. On average, 0.85% of predicted pixels overlapped the distractor; the maximum on any frame was 12.90%. The partial-occlusion and after-reappearance slices are intentionally not disjoint: the target is still partially covered for some frames after it first reappears.

The timeline exposes behavior that the aggregate means hide. The shaded interval and dashed boundaries mark the start and end of occlusion, while the orange line marks the target’s reappearance. Missing points correspond to the six empty-ground-truth frames, where IoU is undefined; those frames are covered by the separate correct-empty result rather than being assigned an IoU of 1.

Inference cost depends on the frame type

XMem does not perform exactly the same work on every frame. I counted operations for three phases at 864x480 with one object:

The first annotated frame initializes object memory and is the least expensive of these phases. A normal propagation frame reads memory and decodes a segmentation. A memory-update frame encodes a new value to write back into memory, adding about 86.25 GFLOPs.

The ValueEncoder accounted for about 88.86 GFLOPs in the module table, which closely matches the measured increase. This makes the tradeoff behind XMem’s memory-update interval concrete: writing more often gives the model fresher object information, but value encoding is not free.

The decoder dominates propagation compute

I initially expected content-addressed memory matching to dominate inference. At this resolution, it did not. The propagation breakdown was:

Most decoder cost came from higher-resolution fusion and upsampling:

This changes how I would reuse XMem’s ideas in a smaller system. The key/value memory pattern is attractive, but copying the full segmentation decoder and mask-conditioned value encoder would bring most of the compute with it.

Resolution is the practical constraint

The wall-clock benchmark used the same 180-frame synthetic sequence:

Moving from a 480 to 720 short side increases pixel count by 2.25x. Throughput decreased by about 36%, while measured peak allocation increased by 2.65x.

The memory increase is not only caused by larger CNN feature maps. XMem compares current-frame query positions against stored memory positions. In this implementation, the dense similarity tensor is constructed before top-k selection. Increasing resolution therefore expands both sides of the matching operation: there are more locations in each stored frame and more query locations in the current frame.

An experiment above 720 short side made the workstation unresponsive, so I removed that case from the benchmark. This is not a universal resolution limit for XMem. It is a practical limit for this sequence, configuration, and workstation. Memory cadence, working-memory size, video length, object count, and software versions can all change the outcome.

One other result was counterintuitive: FP32 was faster than AMP at 480p, although it used more memory. On this run, FP32 delivered 74.59 FPS versus 63.87 FPS with AMP. Autocast overhead, kernel selection, TF32 behavior, and run-to-run variance could all contribute. I would not generalize from one run; repeated warmed-up measurements are needed before concluding that FP32 is consistently faster.

Closing thoughts

XMem is a useful example of explicit memory applied to video. Its first-frame mask creates object-specific memory, its compact keys support content-based lookup, and its values preserve richer information for mask decoding. On my synthetic test, this was enough to maintain identity through an occlusion and reappearance event without fine-tuning on the sequence.

The implementation is not lightweight. At 480p, a normal propagation frame required about 295 GFLOPs, and the decoder accounted for most of that cost. Memory-update frames were more expensive because value encoding added roughly 86 GFLOPs. Increasing resolution also produced a sharp memory increase because spatial memory matching grows with both stored and current-frame positions.

XMem’s most reusable idea is not necessarily its complete architecture. It is the separation between a small key used to find relevant history and a larger, object-conditioned value used to reconstruct detail. That pattern is applicable beyond segmentation, provided the memory lookup and decoder are designed around the resolution and hardware constraints of the final system.

fork: https://github.com/hugovergnes/XMem/tree/experiment/synthetic-occlusion-benchmark