Tokens and Keywords in C++

Tokens are the smallest units of a C++ program that are recognized by the compiler. Every element in C++ code — from variable names and operators to separator symbols — is a token. Understanding tokens is a fundamental step in understanding how the compiler reads and processes C++ code.

What Is a Token?

A token is a building block of a program. When the compiler reads source code, it breaks the code down into a sequence of tokens, then validates and processes each token according to the language rules.

C++ has 6 token categories:

1. Identifiers
2. Keywords
3. Literals
4. Operators
5. Separators/Punctuators
6. Comments (ignored by the compiler)

Code Example

#include <iostream>

int main() {
    // Identifiers: variable and function names
    int age = 25;             // age = identifier, int = keyword
    double average = 89.5;    // average = identifier, 89.5 = literal

    // Keywords and identifiers
    if (age >= 18) {          // if = keyword, age = identifier
        std::cout << "Adult" << std::endl;
    }

    // Operator
    int result = age * 2 + 5; // * + = operator

    // Literal
    std::string name = "John"; // "John" = string literal
    char grade = 'A';          // 'A' = character literal
    bool passed = true;        // true = boolean literal

    return 0;
}

Code Explanation

1. Identifiers

Identifiers are names given by a programmer to identify variables, functions, classes, or other entities in a program.

Rules for Writing Identifiers:

• Must start with a letter (a-z, A-Z) or an underscore (_)
• Subsequent characters can be letters, digits (0-9), or underscores
• Case-sensitive — name, Name, and NAME are different identifiers
• C++ keywords cannot be used as identifier names
• Cannot contain spaces or special characters other than underscores

// ✅ Valid identifiers
int age;
double _score;
std::string fullName;
int data2024;
float _pi_ = 3.14f;
int MAX_SIZE = 100;

// ❌ Invalid identifiers
int 123abc;      // Starts with a digit
int my-name;     // Uses a hyphen
int my name;     // Uses a space
int class;       // Uses a keyword
int namespace;   // Uses a keyword

Naming Conventions in C++:

2. Keywords

Keywords are reserved words that have special meaning in the C++ language. Keywords cannot be used as identifiers.

Here is the complete list of C++ keywords (based on the C++23 standard):

Flow Control:
- if, else — Conditional statement
- switch, case, default — Switch statement
- for, while, do — Loops
- break, continue — Loop flow control
- goto — Jump to a specific label (not recommended)
- return — Returns a value from a function

Data Types:
- bool — Boolean type (true/false)
- char — Single character
- int — Integer
- float — Single-precision floating-point number
- double — Double-precision floating-point number
- void — Empty data type
- wchar_t — Wide character
- signed, unsigned — Sign modifiers
- short, long — Size modifiers

OOP:
- class — Defines a class
- struct — Defines a struct
- union — Defines a union
- enum — Defines an enumeration
- virtual — Virtual function (polymorphism)
- override — Marks a virtual function override
- final — Prevents inheritance/overriding
- this — Pointer to the current object
- friend — Grants private access to another function/class
- public, protected, private — Access modifiers

Memory Management:
- new — Dynamic memory allocation
- delete — Frees memory
- nullptr — Null pointer (C++11)

Modifiers and Qualifiers:
- const — Value that cannot be changed
- constexpr — Expression evaluated at compile time (C++11)
- static — Static storage
- extern — Declares a variable from another compilation unit
- mutable — Allows changes to a const member
- volatile — Prevents compiler optimization on a variable
- register — Suggests storage in a register (mostly ignored)
- inline — Suggests inlining the function

Templates and Generics:
- template — Defines a template
- typename — Declares a type name in a template
- typedef — Creates an alias for a data type
- using — Declares an alias (modern replacement for typedef)

Error Handling:
- try — Exception handling block
- catch — Catches an exception
- throw — Throws an exception
- noexcept — Marks a function that does not throw exceptions

Others:
- auto — Automatic type inference (C++11)
- sizeof — Calculates the size of a data type in bytes
- operator — Defines a custom operator
- asm — Inline assembly
- alignas, alignof — Memory alignment attributes (C++11)
- decltype — Deduces the type from an expression (C++11)
- static_assert — Assert evaluated at compile time (C++11)
- thread_local — Thread-local variable (C++11)
- co_await, co_yield, co_return — Coroutines (C++20)
- concept, requires — Concepts (C++20)
- consteval — Function that must be evaluated at compile time (C++20)
- constinit — Constant initialization (C++20)
- char8_t, char16_t, char32_t — Additional character types

3. Literals

Literals are constant values written directly in the program code.

// Integer literals
int decimal = 42;
int octal = 052;       // Base 8 (starts with 0)
int hexadecimal = 0x2A; // Base 16 (starts with 0x)
int binary = 0b101010;  // Base 2 (starts with 0b, C++14)

// Floating-point literals
double pi = 3.14159;
double scientific = 1.5e10; // 1.5 × 10^10

// Character literals
char letter = 'A';
wchar_t wideCharacter = L'Ω';
char8_t utf8 = u8'A';  // C++20

// String literals
const char* text = "Hello, World!";
std::string name = "John";
const wchar_t* wideText = L"Welcome";

// Boolean literals
bool valid = true;
bool invalid = false;

// Null pointer literal
int* ptr = nullptr;

4. Operators

Operators are symbols that perform operations on one or more operands.

// Arithmetic
int result = 10 + 5;   // + addition operator
int difference = 10 - 5; // - subtraction operator
int product = 10 * 5;    // * multiplication operator
int quotient = 10 / 5;    // / division operator
int remainder = 10 % 3;    // % modulus operator

// Comparison
bool isEqual = (5 == 5);  // == equal to
bool isNotEqual = (5 != 3); // != not equal to
bool isGreater = (5 > 3); // > greater than

// Logic
bool andResult = true && false;  // && AND
bool orResult = true || false;  // || OR
bool notResult = !true;         // ! NOT

// Assignment
int x = 10;    // = assignment
x += 5;        // += compound assignment
x++;           // ++ increment

5. Separators (Punctuators)

Separators or punctuators are symbols that have a structural function in the C++ language.

Everyday Use Cases

Breaking Code Down Into Tokens

Imagine the compiler reading the following code:

int total = price + tax;

The compiler breaks it down into the following tokens:

Each token is processed according to its category — int specifies the data type, total becomes the variable name, = is the assignment operator, and so on.

Recognizing Keywords in Context

// A simple calculation program
#include <iostream>

int main() {
    // Keywords: int (data type), const (qualifier)
    const int MAX_LIMIT = 100;

    // Keyword: auto (type inference)
    auto name = std::string("John");

    // Keywords: if, else (conditional)
    if (name.empty()) {
        std::cout << "Name is empty" << std::endl;
    } else {
        std::cout << "Name: " << name << std::endl;
    }

    // Keyword: for (loop)
    for (int i = 0; i < MAX_LIMIT; i++) {
        if (i % 10 == 0) {
            // Keyword: continue (jump to the next iteration)
            continue;
        }
        if (i > 20) {
            // Keyword: break (stop the loop)
            break;
        }
    }

    // Keyword: return (return a value)
    return 0;
}

Identifiers vs Keywords: Summary

Allowed vs Disallowed Identifiers

// ✅ Valid identifiers
int x;
int _count;
int MAX_VALUE;
int nombreUtilisateur;
int data2024;
int _internal_var;

// ❌ Invalid identifiers
int 2x;          // Starts with a digit
int my-var;      // Contains a hyphen
int my var;      // Contains a space
int class;       // Keyword
int return;      // Keyword
int int;         // Keyword
int "string";    // Contains illegal characters

// ⚠️ Valid but risky (reserved for the compiler/standard library)
int __count;     // Leading double underscore = reserved
int _Count;      // Underscore followed by an uppercase letter = reserved in global scope

Conclusion

Tokens are the smallest units of a C++ program that the compiler recognizes. The six main token categories are identifiers, keywords, literals, operators, separators, and comments. Identifiers must follow specific naming rules, while keywords are reserved words established by the language. Understanding tokens helps you write valid code and understand how the compiler processes every element in a C++ program.