NLP (Natural Language Processing)

NLP is a branch of AI that deals with the interaction between computers and human language. It covers text analysis, translation, sentiment analysis, and chatbots.

Tokenization

from sklearn.feature_extraction.text import CountVectorizer

texts = ["Machine learning is fascinating", "Python for data science", "Deep learning is powerful"]

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

print(f"Vocabulary: {vectorizer.get_feature_names_out()}")
print(f"Matrix:\n{X.toarray()}")

TF-IDF

from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

texts = [
    "the cat sat on the mat",
    "the dog sat on the log",
    "cats and dogs are friends"
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(texts)

feature_names = vectorizer.get_feature_names_out()
for i, text in enumerate(texts):
    scores = tfidf_matrix[i].toarray().flatten()
    top_indices = np.argsort(scores)[::-1][:3]
    print(f"Text {i+1}: {[feature_names[j] for j in top_indices]}")

Word Embeddings

Word2Vec (pre-trained)

from gensim.models import Word2Vec

sentences = [
    ["machine", "learning", "is", "fun"],
    ["deep", "learning", "is", "powerful"],
    ["python", "is", "great", "for", "ml"]
]

model = Word2Vec(sentences, vector_size=50, window=5, min_count=1)
vector = model.wv["learning"]
print(f"Embedding shape: {vector.shape}")

Sentiment Analysis

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import train_test_split

reviews = [
    "This movie is great and entertaining",
    "The acting is amazing, the story is engaging",
    "Very disappointing, a waste of time",
    "Not recommended, the story is boring",
    "This product has excellent quality",
    "The item was damaged and not as described",
    "Fast delivery and neat packaging",
    "Poor product, do not buy it"
]
labels = [1, 1, 0, 0, 1, 0, 1, 0]

X_train, X_test, y_train, y_test = train_test_split(
    reviews, labels, test_size=0.25, random_state=42
)

pipeline = make_pipeline(
    TfidfVectorizer(),
    LogisticRegression()
)
pipeline.fit(X_train, y_train)

test = ["The movie was awesome!", "The item was badly damaged"]
pred = pipeline.predict(test)
for t, p in zip(test, pred):
    print(f"'{t}' → {'Positive' if p == 1 else 'Negative'}")

Text Preprocessing

import re

def preprocess(text):
    text = text.lower()
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    text = re.sub(r'\s+', ' ', text).strip()
    return text

texts = ["Machine Learning! It's Amazing...", "Python 3.9+ version"]
for t in texts:
    print(f"'{t}' → '{preprocess(t)}'")

Transformers Overview

# pip install transformers
# from transformers import pipeline

# sentiment_pipeline = pipeline("sentiment-analysis")
# result = sentiment_pipeline("I love this product!")
# print(result)

N-grams

from sklearn.feature_extraction.text import CountVectorizer

texts = ["machine learning is great", "deep learning is powerful"]

# Unigrams
vec_1 = CountVectorizer(ngram_range=(1, 1))
print("Unigrams:", vec_1.fit_transform(texts).toarray().shape)

# Bigrams
vec_2 = CountVectorizer(ngram_range=(2, 2))
print("Bigrams:", vec_2.fit_transform(texts).toarray().shape)

# Uni + Bigrams
vec_3 = CountVectorizer(ngram_range=(1, 2))
print("Uni+Bigrams:", vec_3.fit_transform(texts).toarray().shape)

Bag of Words vs TF-IDF vs Embeddings

NLP Tips

• Always lowercase and clean the text first
• Use TF-IDF as a good baseline
• Use word embeddings for semantic understanding
• Use transformers for state-of-the-art performance
• Use stopwords removal when needed
• Use stemming/lemmatization to reduce words

Final chapter: a Full ML Project, combining all the concepts in one end-to-end project.

Next: Project Machine Learning →