Natural Language Processing

Natural Language Processing (NLP) enables machines to understand, interpret, and generate human language.

NLTK Basics

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('wordnet')

text = "Python is amazing for AI and machine learning!"

# Tokenization
words = word_tokenize(text)
sentences = sent_tokenize(text)
print("Words:", words)
print("Sentences:", sentences)

# Remove stopwords
stop_words = set(stopwords.words('english'))
filtered = [w for w in words if w.lower() not in stop_words]
print("Filtered:", filtered)

# Lemmatization
lemmatizer = WordNetLemmatizer()
lemmatized = [lemmatizer.lemmatize(w) for w in filtered]
print("Lemmatized:", lemmatized)

spaCy Basics

import spacy

nlp = spacy.load("en_core_web_sm")
doc = nlp("Apple is looking at buying U.K. startup for $1 billion")

# Named Entity Recognition
for ent in doc.ents:
    print(ent.text, ent.label_)

# Part-of-speech tagging
for token in doc:
    print(token.text, token.pos_, token.dep_)

Sentiment Analysis

from transformers import pipeline

classifier = pipeline("sentiment-analysis")
results = classifier([
    "I love this product!",
    "This is terrible, worst experience ever.",
    "It's okay, nothing special."
])

for result in results:
    print(f"{result['label']}: {result['score']:.4f}")

Text Generation with LLMs

generator = pipeline("text-generation", model="gpt2")

result = generator(
    "The future of artificial intelligence",
    max_length=100,
    num_return_sequences=1,
    temperature=0.7
)

print(result[0]['generated_text'])

Summary

You learned the core NLP stack with NLTK for text processing, spaCy for linguistic pipelines like tokenization and named entities, and transformers for pretrained sequence models. The next chapter applies these same models to generative tasks.