juliasilge/tidytext

What it solves

It simplifies text mining by applying tidy data principles to text analysis. It allows users to convert unstructured text into a structured, "one-token-per-row" format, making it easier to perform analysis using standard data manipulation tools rather than specialized text mining software.

How it works

The package provides functions to transform text data frames into tidy formats. A key function, unnest_tokens(), breaks text down into individual tokens (such as words, n-grams, or sentences). Once tidied, the data can be processed using common R tools like dplyr for filtering and counting, tidyr for reshaping, and ggplot2 for visualization. It also includes utilities to remove stop words and integrate sentiment lexicons for sentiment analysis.

Who it’s for

It is designed for data analysts and researchers who use R and want to perform text mining using a consistent, tidy workflow.

Highlights

• Tidy Transformation: Converts text to a one-token-per-row format for seamless integration with the tidyverse.
• Flexible Tokenization: Supports tokenizing by words, characters, n-grams, sentences, lines, and paragraphs.
• Sentiment Analysis: Provides built-in access to sentiment lexicons (e.g., Bing) to calculate sentiment scores.
• Interoperability: Includes functions to convert DocumentTermMatrix objects from other text mining packages into tidy data frames.