Text data is used in natural language processing (NLP), which interacts between humans and machines using natural language. Text data helps analyze movie reviews, products using Amazon reviews, etc. But the question that arises here is how to deal with text data when building a machine learning model?
Text data is converted to a real-valued vector by various techniques. One such approach is Bag of Words (BoW), which will be discussed in this article. But why do we have to convert the text into a vector? Why can’t we use text data to build a machine learning model?
Need for text vectorization
Let’s say we have reviews of a product. Text reviews provided by the customers are of different lengths. By converting from text to numbers, we can represent a review by a finite length of the vector. In this way, the length of the vector will be equal for each review, irrespective of the text length.
Bag of words is the most trivial representation of text into vectors. Each column of a vector represents a word. The values in each cell of a row show the number of occurrences of a word in a sentence.
Example
The initial step is to find a vocabulary of unique words (ignoring the punctuation and cases). Vocabulary in the above example:
[This, movie, is, the, good, of, times, not, I, love, watch, you, will, it, too]
In our vocabulary, we have 15 unique words. Therefore, each movie review is represented by a vector of 15 dimensions (each word representing a dimension). For the first review:
The values corresponding to each word shows the number of occurrences of a word in a review. Similarly, 15-dimension vectors represent the remaining reviews.
In real-world problems, text data must be preprocessed before vectorization. Preprocessing includes removing punctuations, converting all words into lowercase, and removing unnecessary words that are not adding any value to the text.
In Natural Language Processing (NLP), unnecessary words are called stopwords. nltk library already contains the list of stopwords. There are 179 stopwords in English.
We’ll see our example after removing the stopwords.
Vocabulary — [movie, good, times, love, watch]
Note that the vector for review 1 and 2 are the same because ‘not’ is present in stopwords. The model considers both the review the same since the vectors for both are equal. One solution to this problem is using n-grams.
n-grams
n-grams are a neighboring sequence of n-words. n can be any positive integer.
Example – "Bag of words" is a three-gram, "text vectorization" is a two-gram.
We have used Uni-gram (1-gram) in our example. That means each word is considered as a feature. Removing stopwords will remove words such as ‘not’ which can be useful. Uni-gram based bag of words (BoW) does not take sequence information into account. To consider sequence information, we use bi-grams, tri-grams, etc.
In our example, if we use bi-grams vocabulary can be changed to:
[This movie, movie is, is good, the movie, is not, not good, I love, love this, movie watch, watch you, you will, will love, love it, it too]
As a result, n-grams will give better results than the uni-gram based model.
Drawbacks of BoW
- The length of the vector can be high, and most of the values are zero. Computationally, it is not efficient if the vocabulary is very high.
- Uni-gram based BoW is unable to capture the context of the text. Bi-grams and tri-grams can do the job, but they are computationally expensive.
Conclusion
Bag of words is a text vectorization technique that converts the text into finite length vectors. The boW model is easy to implement and understand. Bag of words has few drawbacks, which can be overcome by using advanced techniques.
Thanks for reading!
