word_counts <- cleaned_austen %>% count(word, sort = TRUE) word_counts %>% head(10)
This write-up outlines a reproducible workflow for text mining using R, emphasizing tidy data principles. | Package | Purpose | | :--- | :--- | | tidytext | Converts text to tidy data frames (one token per row). Integrates with dplyr , ggplot2 . | | dplyr | Data manipulation (filter, group, mutate). | | ggplot2 | Visualization of text metrics (word frequencies, sentiment scores). | | janeaustenr | Sample texts for practice. | | tidyverse | Meta-package for data science. | | wordcloud | Generates word clouds. | | quanteda | Advanced text analysis (DFM, keywords-in-context). | | tm | Classic text mining (corpus, term-document matrix). | Installation: install.packages(c("tidytext", "tidyverse", "wordcloud", "quanteda")) 3. The Text Mining Workflow A standard text mining pipeline in R consists of these steps: Text Mining With R
tf_idf <- cleaned_austen %>% count(book, word) %>% bind_tf_idf(word, book, n) %>% arrange(desc(tf_idf)) tf_idf %>% group_by(book) %>% slice_max(tf_idf, n = 3) 4.1. N-grams (Pairs of Words) austen_bigrams <- austen_books() %>% unnest_tokens(bigram, text, token = "ngrams", n = 2) Count common bigrams bigram_counts <- austen_bigrams %>% separate(bigram, into = c("word1", "word2"), sep = " ") %>% filter(!word1 %in% stop_words$word) %>% filter(!word2 %in% stop_words$word) %>% count(word1, word2, sort = TRUE) 4.2. Topic Modeling (Latent Dirichlet Allocation) Using tidytext + topicmodels to discover hidden themes. | | dplyr | Data manipulation (filter, group, mutate)
with a bar chart: