Therefore sometimes it is necessary to use the tf–idf(term frequency–inverse document frequency) instead of the frequencies of the term as entries, tf-idf measures the relative importance of a word to a document. One may argue that in the wordcloud, words such as one, film, movie do not carry too much meaning in the setting, since we know that the entire corpus is about movies. Wordcloud(rownames(freq), freq, max.words=50, colors=brewer.pal(1, 'Dark2')) # 'well' 'will' 'work' freq = ame(sort(colSums(as.matrix(review_dtm)), decreasing=TRUE)) # 'scene' 'scenes' 'see' 'seems' 'something' # 'best' 'can' 'character' 'characters' 'director'

We can draw a simple word cloud findFreqTerms(review_dtm, 1000) # 'action' 'also' 'another' 'back' 'bad' # Docs acceptable accepts accident accidentally # Docs absolute absolutely absurd abuse abyss academy accent accept # Docs abandon abandoned abilities ability able aboard absence absent The first review now looks like inspect(review_dtm) # > To reduce the dimension of the DTM, we can emove the less frequent terms such that the sparsity is less than 0.95 review_dtm = removeSparseTerms(review_dtm, 0.99) # Docs adolescents adolf adolph adopt adopted adopter # Weighting : term frequency (tf) inspect(review_dtm) # > Because the number of unique words in the corpus the dimension can be large. To analyze the textual data, we use a Document-Term Matrix (DTM) representation: documents as the rows, terms/words as the columns, frequency of the term in the document as the entries. # Metadata: corpus specific: 0, document level (indexed): 0 Review_corpus = tm_map(review_corpus, stripWhitespace)Īfter the above transformations the first review looks like inspect(review_corpus) # > Review_corpus = tm_map(review_corpus, removeWords, c('the', 'and', stopwords('english'))) Review_corpus = tm_map(review_corpus, removePunctuation) Review_corpus = tm_map(review_corpus, removeNumbers) Remove extra whitespaces review_corpus = tm_map(review_corpus, content_transformer(tolower)) Remove punctuation marks and stopwords 4. Next we normalize the texts in the reviews using a series of pre-processing steps: 1. To use the tm package we first transfrom the dataset to a corpus: review_corpus = Corpus(VectorSource(reviews$content)) The review dataset has two variables: content of the review, and polarity of the review (0 or 1).

Next we load the move review dataset reviews = read.csv('movie_reviews.csv', stringsAsFactors = F, row.names = 1) Library(wordcloud) # Loading required package: RColorBrewer

install.packages(c('tm', 'SnowballC', 'wordcloud', 'topicmodels')) library(tm) # Loading required package: NLP library(SnowballC) First install and load packages needed for text mining.

R studio review