tensorflow/text

What it solves

TensorFlow Text provides a set of text processing tools that integrate directly into the TensorFlow graph. This ensures that preprocessing steps—like tokenization and normalization—are identical during both training and inference, eliminating the need to manage separate preprocessing scripts or risk inconsistencies between the two stages.

How it works

The library implements text operations as TensorFlow ops. It handles UTF-8 strings and provides tools for:

• Normalization: Applying case folding and Unicode normalization (e.g., NFKC) to ensure consistent character representation.
• Tokenization: Breaking strings into tokens using various methods, such as WhitespaceTokenizer (splitting on ICU whitespace) and UnicodeScriptTokenizer (splitting based on Unicode script boundaries).
• Offset Tracking: Using TokenizerWithOffsets to track the exact byte positions of tokens relative to the original string.
• Feature Extraction: Using Wordshape to identify patterns like capitalization, numeric values, or punctuation via regular expressions.
• Sequence Generation: Creating N-grams using a sliding window approach.

Who it’s for

Developers and machine learning engineers building text-based models in TensorFlow who need robust, graph-integrated preprocessing pipelines.

Highlights

• Graph Integration: Preprocessing is part of the TensorFlow graph, ensuring parity between training and inference.
• Unicode Support: Comprehensive support for UTF-8 and Unicode normalization.
• Ragged Tensors: Native support for Ragged Tensors, allowing for variable-length sequences without manual padding.
• Keras Integration: Includes a ToDense layer for easy integration with Keras models using ragged data.