google/magika

What it solves

Magika is an AI-powered tool designed to accurately identify the content type of a file, regardless of its extension or name. It addresses the problem of unreliable file identification, providing a high-precision alternative to traditional methods, especially for textual content types.

How it works

Magika uses a custom, highly optimized deep learning model (a few MBs in size) that analyzes a limited subset of a file's content to determine its type. It was trained on a dataset of approximately 100 million samples across more than 200 content types, covering both binary and textual formats. The tool employs a per-content-type threshold system and adjustable prediction modes (high-confidence, medium-confidence, and best-guess) to control the tolerance for errors.

Who it’s for

It is intended for security researchers, developers, and system administrators who need to fast and precise file identification at scale. It is used by Google to route files in Gmail, Drive, and Safe Browsing to appropriate security scanners.

Highlights

• High Accuracy: Achieves ~99% average precision and recall on its test set.
• Fast Inference: Inference time is approximately 5ms per file on a single CPU after the model is loaded.
• Multi-language Support: Available as a Rust-based CLI, Python API, and bindings for JavaScript/TypeScript and Go.
• Scalability: Capable of processing thousands of files simultaneously or recursively scanning directories.
• Efficient: Near-constant inference time regardless of file size because it only processes a limited subset of the content.