LLM Applications with Python

Large language models (LLMs) turn natural language into a programmable interface: they draft, summarize, classify, extract, translate, and answer questions about your own documents. Building with them is mostly an engineering discipline — managing context, structuring prompts, validating output, and controlling cost and latency. This chapter covers the OpenAI-style chat API, retrieval-augmented generation, structured extraction, and streaming.

The Chat Completions Pattern

Modern LLM APIs accept a message list with roles: system fixes behavior and constraints, user carries the request, assistant holds prior replies. Keeping a persona and rules in the system message is more stable than restating them per turn.

from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from the environment

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a concise technical assistant."},
        {"role": "user", "content": "Explain idempotent HTTP methods in 3 bullets."},
    ],
    temperature=0.2,   # low = factual, high = creative
    max_tokens=200,
)
print(resp.choices[0].message.content)
print(resp.usage.total_tokens)  # cost control starts here

Retrieval-Augmented Generation

LLMs do not know your private documents, and they cannot cite sources out of nowhere. Retrieval-augmented generation (RAG) fixes this: embed your text into a vector index, retrieve the passages relevant to the question, and paste them into the prompt as grounding evidence. The transformers library also lets you run a local sentence-embedding model for the indexing step.

from sentence_transformers import SentenceTransformer
import numpy as np

embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunks = load_document_chunks()  # list of strings, ~300 tokens each
M = embedder.encode(chunks, normalize_embeddings=True)

def retrieve(question, k=4):
    q = embedder.encode([question], normalize_embeddings=True)
    sims = M @ q[0]
    return [chunks[i] for i in np.argsort(-sims)[:k]]

context = "\n---\n".join(retrieve("What SLA covers the reporting service?"))
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "Answer only from the provided context."},
        {"role": "user", "content": f"Context:\n{context}\n\nQuestion: ..."},
    ],
)
print(resp.choices[0].message.content)

Structured Output and Extraction

Downstream code should never parse free text. Ask for JSON with an explicit schema, or use the SDK's structured-output mode, and validate with Pydantic. Temperature near zero and a one-shot example in the prompt make extraction reliable.

from pydantic import BaseModel, ValidationError

class Ticket(BaseModel):
    category: str      # enum values in your system prompt
    priority: int      # 1-5
    summary: str

raw = client.chat.completions.create(
    model="gpt-4o-mini",
    response_format={"type": "json_object"},
    messages=[{"role": "user", "content": email_text}],
).choices[0].message.content

try:
    ticket = Ticket.model_validate_json(raw)
except ValidationError as e:
    ticket = retry_with_bigger_model(email_text)

Streaming Responses

Chat UIs feel snappy only when tokens appear as they are generated. Stream by iterating the completion, printing each delta, and only then accumulating the full text for logging:

stream = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    stream=True,
)
full = ""
for chunk in stream:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)
    full += delta

Operating LLM Applications

Cache identical prompts, batch requests through one client with retries and backoff for rate limits, and log prompts, outputs, token usage, and latency per request. Local open-source models served with Ollama or vLLM drop into the same OpenAI-compatible endpoint, which keeps your code portable across hosted and self-hosted backends.

Summary

LLM applications are built on the chat completions pattern with role-structured messages, grounded in private data through retrieval-augmented generation with embeddings, and made machine-readable through JSON-validated structured output. Streaming keeps interfaces responsive, and production readiness is an operations problem of caching, retries, logging, and cost tracking. Treat prompts as code — version them, test them, and measure them.