In this blog, we will discuss text to features methods. It will include:
One Hot Encoding
Count Vectorizer
N-grams
Hash vectorizer
Term Frequency-Inverse Document Frequency(TF-IDF)
In [1]:
import pandas as pd
In [2]:
text = "I love my family"
In [3]:
tokenize_words = text.split()
In [4]:
pd.get_dummies(tokenize_words)
Out[4]:
Output has 4 features as the number of unique words present in the input was 4.
In [5]:
from sklearn.feature_extraction.text import CountVectorizer
class sklearn.feature_extraction.text.CountVectorizer(*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None, lowercase=True, preprocessor=None, tokenizer=None, stop_words=None, token_pattern='(?u)\b\w\w+\b', ngram_range=(1, 1), analyzer='word', max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, dtype=<class 'numpy.int64'>):
Convert a collection of text documents to a matrix of token counts.
CountVectorizer implements both tokenization and occurrence counting in a single class. The vectorization the general process of turning a collection of text documents into numerical feature vectors.
In [6]:
text = ["I love my family and in my family there are ten memebers."]
In [7]:
vectorizer = CountVectorizer()
In [8]:
vector = vectorizer.fit_transform(text)
vector
Out[8]:
In [9]:
vectorizer.get_feature_names()
Out[9]:
In [10]:
vectorizer.vocabulary_
Out[10]:
In [11]:
vector.toarray()
Out[11]:
We can see word 'my' and 'family' are showing twice.
In [12]:
text1 = [
"I love my family",
"In my family there are ten memebers",
"All of them are stay together"
]
In [13]:
vectorizer1 = CountVectorizer()
In [14]:
vector1 = vectorizer1.fit_transform(text1)
vector1
Out[14]:
In [15]:
vectorizer1.get_feature_names()
Out[15]:
In [16]:
vectorizer1.vocabulary_
Out[16]:
In [17]:
vector1.toarray()
Out[17]:
In [18]:
text = ["I love my family and in my family there are ten memebers."]
ngram_range: tuple (min_n, max_n), default=(1, 1)
The lower and upper boundary of the range of n-values for different word n-grams or char n-grams to be extracted. All values of n such such that min_n <= n <= max_n will be used. For example an ngram_range of (1, 1) means only unigrams, (1, 2) means unigrams and bigrams, and (2, 2) means only bigrams. Only applies if analyzer is not callable.In [19]:
vectorizer2 = CountVectorizer(ngram_range=(2, 2))
In [20]:
vector2 = vectorizer2.fit_transform(text)
vector2
Out[20]:
In [21]:
vectorizer2.vocabulary_
Out[21]:
In [22]:
vector2.toarray()
Out[22]:
We can see "my family" bigram came twice.
In [23]:
text1 = [
"I love my family",
"In my family there are ten memebers",
"All of them are stay together"
]
In [24]:
vectorizer3 = CountVectorizer(ngram_range=(2, 2))
vector3 = vectorizer3.fit_transform(text1)
vector3
Out[24]:
In [25]:
vectorizer3.vocabulary_
Out[25]:
In [26]:
vector3.toarray()
Out[26]:
In [27]:
vectorizer4 = CountVectorizer(ngram_range=(3, 3))
In [28]:
vector4 = vectorizer4.fit_transform(text)
vector4
Out[28]:
In [29]:
vectorizer4.vocabulary_
Out[29]:
In [30]:
vector4.toarray()
Out[30]:
In [31]:
from sklearn.feature_extraction.text import HashingVectorizer
class sklearn.feature_extraction.text.HashingVectorizer(*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None, lowercase=True, preprocessor=None, tokenizer=None, stop_words=None, token_pattern='(?u)\b\w\w+\b', ngram_range=(1, 1), analyzer='word', n_features=1048576, binary=False, norm='l2', alternate_sign=True, dtype=<class 'numpy.float64'>)
Convert a collection of text documents to a matrix of token occurrences.This text vectorizer implementation uses the hashing trick to find the token string name to feature integer index mapping.
This strategy has several advantages:
I. it is very low memory scalable to large datasets as there is no need to store a vocabulary dictionary in memory.
II. it is fast to pickle and un-pickle as it holds no state besides the constructor parameters.
III. it can be used in a streaming (partial fit) or parallel pipeline as there is no state computed during fit.There are also a couple of cons (vs using a CountVectorizer with an in-memory vocabulary):
I. there is no way to compute the inverse transform (from feature indices to string feature names) which can be a problem when trying to introspect which features are most important to a model.
II. there can be collisions: distinct tokens can be mapped to the same feature index. However in practice this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems).
III. no IDF weighting as this would render the transformer stateful.In [32]:
text = ["I love my family and in my family there are ten memebers."]
In [33]:
vectorizer5 = HashingVectorizer(n_features = 10)
In [34]:
vector5 = vectorizer5.fit_transform(text)
print(vector5.shape)
In [35]:
vector5.toarray()
Out[35]:
It created vector of size 10.
In [36]:
text1 = [
"I love my family",
"In my family there are ten memebers",
"All of them are stay together"
]
In [37]:
vectorizer6 = HashingVectorizer(n_features = 10)
vector6 = vectorizer6.fit_transform(text1)
print(vector6.shape)
In [38]:
vector6.toarray()
Out[38]:
In [39]:
from sklearn.feature_extraction.text import TfidfVectorizer
class sklearn.feature_extraction.text.TfidfVectorizer(*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None, lowercase=True, preprocessor=None, tokenizer=None, analyzer='word', stop_words=None, token_pattern='(?u)\b\w\w+\b', ngram_range=(1, 1), max_df=1.0, min_df=1, max_features=None, vocabulary=None, binary=False, dtype=<class 'numpy.float64'>, norm='l2', use_idf=True, smooth_idf=True, sublinear_tf=False)
Convert a collection of raw documents to a matrix of TF-IDF features.
In [40]:
text = ["I love my family", "In my family there are ten memebers."]
In [41]:
vectorizer7 = TfidfVectorizer()
vectorizer7.fit_transform(text)
Out[41]:
In [42]:
vectorizer7.vocabulary_
Out[42]:
In [48]:
vectorizer7.idf_
Out[48]:
If we observe, "my" and "family" words are appearing two times in both the documents. That's why the vector value are 1, which is less than all the other vector representations of the tokens.
In [44]:
text1 = [
"I love my family",
"In my family there are ten memebers",
"All of them are stay together"
]
In [45]:
vectorizer8 = TfidfVectorizer()
vectorizer8.fit_transform(text1)
Out[45]:
In [46]:
vectorizer8.vocabulary_
Out[46]:
In [47]:
vectorizer8.idf_
Out[47]: