Instagram‘s new standalone video creation app, “Edits,” features Cutouts, an AI-powered tool that enables precise object segmentation in videos. Available globally on iOS and Android devices, the app integrates Meta’s Segment Anything Model (SAM) 2.1 technology to provide professional-level video editing capabilities without requiring expensive software or advanced expertise.

Source: Instagram

How Cutouts Works

The Cutouts feature in Edits uses an object detection pipeline that automatically suggests objects in video frames for segmentation. Users can also employ manual mode for more precise control.

“There are three main steps: first, a user needs to be able to select the object interactively and correctly,” Nikhila Ravi, Research Engineering Manager at Meta, explains in a blog post. “Then they need to be able to track the object through the video correctly, even when the object goes out of frame. And finally, we need to be able to run the SAM 2.1 model fast enough to give the user a real-time experience.”

Once an object is selected, SAM 2.1 predicts a high-quality mask defining the boundary of the object. Users can then hit “track,” allowing the AI to maintain consistent object recognition across every frame, even when the object is temporarily hidden or out of frame.

Technical Enhancements

Meta’s FAIR team implemented several improvements in SAM 2.1 over its predecessor, including additional data augmentation techniques to handle visually similar and small objects. The update also improved occlusion handling by training the model on longer sequences of frames and refining positional encoding.

“Initially, we thought we would need to pursue more aggressive methods to increase model efficiency like quantization, but we were pleasantly surprised to see how effective Torch Inductor was at optimizing model throughput with a minimal amount of code modification,” says Joseph Greer, Research Scientist at Meta.

Performance optimizations increased model throughput by 1.8x and reduced first frame preview latency by 3x on NVIDIA H100 GPUs.

User Adoption and Functionality

In the first 24 hours after Edits launched, Cutouts was used hundreds of thousands of times. The feature allows creators to edit across several layers of video, apply filters to specific parts of videos, and place elements like text and stickers behind objects.

“In 2024, we built a demo as part of our research and as a way to showcase SAM 2 externally to a research audience,” states Ravi. “Less than a year later, the research developed as Segment Anything Model 2.1 is now an important part of Edits.”

Market Positioning

The release comes as ByteDance-owned apps TikTok and CapCut face regulatory challenges in the United States. Instagram head Adam Mosseri noted in January: “We think it’s our job to create the most compelling creative tools for those of you who make videos for not just Instagram but for platforms out there.”

Meta continues development on SAM 3, automatically detecting, segmenting, and tracking objects in images and videos using open vocabulary text or click prompts.