SAM2Long

A plugin for interactive 3D (volumetric or time-lapse) segmentation using Meta's Segment Anything Model 2 (SAM2).

Designed for bioimaging researchers working with 3D volumetric or time-lapse images, this plugin supports TIFF files in ZYX or TYX format. Users can provide input and make corrections through point clicks or manually drawn masks.

The tool leverages the SAM2Long model, an optimized version of Meta's SAM 2 with enhancements to the memory module for improved performance on long videos. It was built to support long videos, but it remains effective for shorter videos as well.

This napari plugin was generated with copier using the napari-plugin-template.

Select object with point prompts

Refine object selection with napari tools

Installation

Please see the official SAM 2 repo and the INSTALL.md for notes and FAQs on potential installation issues.

1. Create a new conda environment with python>=3.10 and install napari:

   conda create -n napari-sam2long python==3.10 pip
   conda activate napari-sam2long
   python -m pip install "napari[all]"
   
   

2. Install PyTorch and TorchVision. Select preferences here to find correct installation command.

   Example command can look like this:

   pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
   
   

3. Install SAM2Long:

   git clone git@github.com:maihanhoang/napari-sam2long.git
   
   cd napari-sam2long/SAM2Long && python -m pip install -e .
   
   

4. Install napari-SAM2Long plugin:

   cd .. && python -m pip install e .
   
   

Usage

Segmenting & tracking first object

1. Open a 3D tiff in napari, make sure it's in TYX or ZYX format.

2. Select the image in the Input dropdown.

3. Add a new labels layer after the input image, then select it in the Labels dropdown.

   The labels layer must be added after the image to ensure dimension alignment.

4. Select the Model.

5. Click Initialize to load the image and initialize the inference state.

6. Define the initial object mask on any frame:

   • Use the mouse middle-click to prompt the model:
     • Middle-click = add region
     • Ctrl + middle-click = remove region
   • Or use napari's built-in tools (paintbrush, eraser, etc.) to draw the mask manually.

   If the model doesn’t segment the object accurately with point prompts, manual correction using napari tools can be useful.

7. Once satisfied with the initial mask, click Propagate from current frame to obtain segmentations for all subsequent frames. The result will be added to the labels layer.

   Propagation only affects future frames. It does not recompute previous ones or consider prompts from other frames. Only the current mask is used to propagate forward.

Making corrections

8. To refine segmentation, add/remove regions use:

   • (Ctrl+) middle-click prompts
   • napari’s label tools

9. Propagate from current frame to re-run the model's predictions with the new mask.

   The plugin treats this as a new initial mask and discards earlier prompts on that frame.

Segmenting another object in the same image/video

10. Save the current labels layer (to preserve previous segmentation).
11. Click Reset, or add a new labels layer and select it in the Labels dropdown. Then repeat steps from Step 6 for the next object.

Segment new image/video

12. Reset inference state.
13. Load new image and follow instructions starting from Step 1. Initialize is necessary to load the new image.

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Bibliography

[1]: Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, & Christoph Feichtenhofer. (2024). SAM 2: Segment Anything in Images and Videos.

[2]: Ding, S., Qian, R., Dong, X., Zhang, P., Zang, Y., Cao, Y., Guo, Y., Lin, D., & Wang, J. (2024). SAM2Long: Enhancing SAM 2 for Long Video Segmentation with a Training-Free Memory Tree. arXiv preprint arXiv:2410.16268.

License & Attribution

This project integrates code from:

• SAM2 by Meta (Apache 2.0 License)
• SAM2Long by Shuangrui Ding et al. (CC-BY-NC 4.0 License)
• napari-samv2 by Krishnan Venkataraman (BSD-3 License)

The following changes were made to SAM2Long:

• integrated SAM2Long into a napari plugin
• modified the video predictor to support the progress bar in the plugin

Since this project includes SAM2Long, it inherits the CC-BY-NC 4.0 license, meaning commercial use is not allowed.