EfficientTAM for real-time tracking ?

[patrick91p]

Hi, and thanks for your work !!
Are you planning to release a predictor for real time tracking ?
Similarly to this for example : https://github.com/zdata-inc/sam2_realtime/blob/main/sam2/sam2_object_tracker.py

In there, the prompt is only needed for the first frame. Then, we call a predict() function, that only takes as argument the current frame of the video/camera to propagate. This would help reduce the memory usage of loading all the frames of a video first before propagating.

Thanks !

[villimikko]

+1 Would be really intrested in this as well.

[SebasGarcia08]

+1, I would be really interested in this too.

[rodneyML]

The codebase of EfficientTAM is very similar to that of SAM2, therefore you can refer to this repo to implement a "real-time" tracker. This issue was also mentioned in this discussion regarding online tracking - which refers to exactly what @patrick91p describes. Refer to the sam2_camera_predictor.py in the segment-anything-2-real-time repo, specifically this function, to build a real-time tracker.

[yformer]

Hi, and thanks for your work !! Are you planning to release a predictor for real time tracking ? Similarly to this for example : https://github.com/zdata-inc/sam2_realtime/blob/main/sam2/sam2_object_tracker.py

In there, the prompt is only needed for the first frame. Then, we call a predict() function, that only takes as argument the current frame of the video/camera to propagate. This would help reduce the memory usage of loading all the frames of a video first before propagating.

Thanks !

Hi @patrick91p, thanks for your interest in our work! I will find a time to add this feature.

[yformer]

@villimikko @SebasGarcia08, I will let you know when this is supported.

[yformer]

The codebase of EfficientTAM is very similar to that of SAM2, therefore you can refer to this repo to implement a "real-time" tracker. This issue was also mentioned in this discussion regarding online tracking - which refers to exactly what @patrick91p describes. Refer to the sam2_camera_predictor.py in the segment-anything-2-real-time repo, specifically this function, to build a real-time tracker.

@patrick91p, did you have a chance to follow @reynhardtstef shared?

[DonggeonKim1012]

Could you guys try out if this works: https://github.com/robrosinc/REALTIME_SAM2

[patrick91p]

Wow thanks a lot @DonggeonKim1012 ! I am going to try this as soon as possible :)

[chenweisomebody126]

@patrick91p @SebasGarcia08 @rodneyML @DonggeonKim1012 @villimikko
For real-time frame-by-frame tracking on iOS — I built a full on-device implementation using EdgeTAM (same SAM 2 architecture). The API is exactly what @patrick91p described: provide a point prompt on the first frame, then call track(frame:) for each subsequent frame with no prompt needed.

Under the hood it uses a ring-buffer memory bank (7 frames) so memory usage stays constant regardless of video length — no need to load the full video into memory.

• GitHub: https://github.com/chenweisomebody126/SegmentKit
• Docs: https://segmentkit.dev

It's iOS-specific (CoreML + Metal), not a general Python solution, but if mobile deployment is your goal this might be useful.