On-Device AI With Transformers.js, WebGPU, and Hugging Face

Martin works on Transformers.js and Tokenizers.js to bring machine learning to JavaScript. Transformers.js supports tasks including text embeddings, audio transcription, image processing, and vision-language work. Its pipeline API accepts task-appropriate inputs and handles preprocessing, inference, and postprocessing.

Models take tensors as input, while pipelines let developers provide text, images, or audio and receive task-specific results. Transformers.js uses ONNX models with ONNX Runtime Web in browsers, and can also use other runtimes across JavaScript environments. Martin recommends WebGPU in general, while noting developers can check for support and select CPU as a fallback.

Transformers.js downloads the files a pipeline needs from the Hugging Face Hub and stores them in the browser cache. Large language models can take several gigabytes, while some task-specific models are around 100 to 200 MB and speech detection models can be only a few megabytes. Download size and inference speed vary by model and device, so Martin recommends considering the user experience before loading a model.

Martin says browser-based AI can handle tasks such as background removal and speech recognition, sometimes avoiding the time needed to upload data and receive a result from a server. He has built offline browser agent demos where a language model selects JavaScript tools to run. He describes these agents as demos for now, while suggesting that individual tasks such as transcription and tool execution can already run in the browser alongside a cloud model.

Martin identifies cross-origin storage as a potential way to avoid downloading the same model separately for each site, though the API is still early. Transformers.js is also exploring structured output for text generation and an in-house WebGPU inference engine with custom kernels. He says the early project aims to optimize operations for specific devices, with potential speedups of 5x or 10x.