Language Representation

Language representation is the process of converting text into numerical formats that machine learning algorithms can process. These representations aim to capture semantic, syntactic, and contextual information from language in a way that computers can understand and manipulate.

Traditional Representation Methods #

Before the advent of neural network-based embeddings, several statistical approaches were used to represent text data.

Bag of Words (BoW) #

The Bag of Words model represents text as an unordered collection of words, disregarding grammar and word order but keeping track of word frequency.

# Bag of Words example
from sklearn.feature_extraction.text import CountVectorizer
corpus = [
    "This is the first document.",
    "This document is the second document.",
    "And this is the third document.",
    "Is this the first document?"
]
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
print("Feature names (vocabulary):", vectorizer.get_feature_names_out())
print("Document-term matrix shape:", X.shape)
print("Document-term matrix:")
print(X.toarray())

Key characteristics of BoW:

• Simplicity: Easy to implement and understand.
• Sparsity: Results in high-dimensional, sparse vectors.
• Loss of Order: Ignores word order and context.
• Frequency Capture: Retains information about word frequency.

Applications include document classification, spam filtering, and simple information retrieval.

TF-IDF (Term Frequency-Inverse Document Frequency) #

TF-IDF extends the BoW model by weighting terms based on their frequency in a document and their rarity across the corpus.

# TF-IDF example
from sklearn.feature_extraction.text import TfidfVectorizer
corpus = [
    "This is the first document.",
    "This document is the second document.",
    "And this is the third document.",
    "Is this the first document?"
]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(corpus)
print("Feature names:", vectorizer.get_feature_names_out())
print("TF-IDF matrix shape:", X.shape)
print("TF-IDF matrix:")
print(X.toarray())

The TF-IDF value increases with:

• Higher term frequency in the document (TF component)
• Lower document frequency of the term across the corpus (IDF component)

This approach helps to:

• Highlight Important Words: Words that are frequent in a document but rare across the corpus.
• Reduce Stop Word Impact: Common words receive lower weights.
• Improve Retrieval Quality: Better ranking of documents in information retrieval.