Drawing onto the Camera

Rousavy describes real-time overlays, color effects, face blurring, and foreground segmentation as camera drawing use cases. A Margelo client project dims the background and adds an animated sticker around a person's hand, while the captured video keeps only the foreground. Building such features natively requires separate iOS and Android implementations.

VisionCamera provides camera features such as photo and video capture, QR scanning, and controls for buffer compression and pixel format. Its frame processors run on a separate camera thread for each frame. Frames stay as native C++ objects exposed to JavaScript through JSI, avoiding the cost of copying or serializing large image buffers.

Rousavy describes a hybrid approach: JavaScript handles camera setup, plugin configuration, and decisions about results, while native plugins perform high-performance processing. Plugins can wrap libraries such as ML Kit, Apple Vision, or TensorFlow Lite. This lets apps use native processing while sharing their JavaScript logic across iOS and Android.

Updating UI state or using Reanimated shared values to draw over the preview can lag because the camera and UI run on separate threads, and those overlays do not become part of the video stream. VisionCamera's Skia frame processor lets developers draw back onto the frame. In Rousavy's face-blurring example, a native detector finds faces and Skia applies a blur inside each face region.

Rousavy says the face-blur app takes about 40 lines of code and runs in 10 milliseconds on the demonstrated iPhone 11, including detection, JSI, and drawing. He says it also runs on Android without writing Java or Kotlin. He points listeners to VisionCamera, Reanimated, a face-blur demo, and his Shadow Lens app.