What the path develops
Observability engineering is the design and operation of useful evidence. It includes what applications emit, how context crosses boundaries, what a Collector accepts and changes, which signals are retained, how user reliability is calculated, and how the telemetry platform reveals its own loss and cost. The goal is not maximum data. It is enough well-governed data to answer important questions within reliability and resource constraints.
Telemetry and reliability foundations
Week 1Learn to define the questions and user outcomes before selecting components. Distinguish data generation from storage, visualization, and paging.
- Explain observability as the ability to investigate system behavior, including novel problems
- Distinguish traces, metrics, logs, baggage, resources, and emerging signal areas
- Separate the OpenTelemetry API used by instrumentation from the application-owned SDK
- Model a trace as causally related spans with unique trace and span identity
- Recognize metrics as measurements transformed into aggregations and time series
- Use logs as event records and plan trace and span correlation fields
- Define user-centered SLIs and SLOs rather than substituting CPU or uptime blindly
- Write an observability question inventory for success, latency, errors, saturation, and dependency behavior
- Classify telemetry data by sensitivity, retention, access, and purpose
- Baseline signal rates and storage before adding new instrumentation
Instrumentation, resources, and context propagation
Weeks 2-3Instrument a small distributed system in two languages. Build coherent causality and shared meaning without duplicating spans or leaking data.
- Evaluate current automatic instrumentation support and stability for each selected language
- Add manual spans only for meaningful application operations and coverage gaps
- Choose span names, kinds, status, events, attributes, and links according to their semantics
- Use counters, gauges, and histograms for the quantities each instrument represents
- Set stable service and deployment resource identity across traces, metrics, and logs
- Follow current semantic conventions for HTTP, databases, messaging, resources, and exceptions
- Inject and extract W3C trace context across HTTP boundaries
- Use carried context or span links deliberately for queues, batches, and asynchronous work
- Keep secrets and personal data out of baggage, tracestate, attributes, and routine log bodies
- Test duplicate instrumentation, broken context, malformed headers, shutdown flush, and signal correlation
Collector pipelines, resilience, and troubleshooting
Weeks 4-5Make the Collector a measurable policy and transport layer rather than an opaque YAML relay.
- Choose an appropriate Collector distribution and verify every component and stability level
- Configure and enable signal-specific receivers, processors, exporters, connectors, and extensions
- Order processors so validation and redaction happen before batching and export
- Use memory limiting, batching, sending queues, retries, and timeouts with tested bounds
- Protect receivers and diagnostics with constrained binding, TLS, authentication, and network policy
- Validate configuration and inspect available components before deployment
- Monitor accepted, refused, dropped, queued, retried, failed, and sent telemetry by signal
- Inject backend outages and overload to expose degraded and data-loss behavior
- Follow a controlled signal hop by hop when telemetry is missing
- Complete the polyglot service and Collector gateway portfolio project
Sampling, cardinality, privacy, and cost
Weeks 6-7Control telemetry by measured value, not arbitrary percentages. Preserve the data needed for reliability while bounding operational and privacy risk.
- Compare consistent probability head sampling with stateful tail sampling
- Design tail policies for errors, latency, selected services or attributes, and a baseline sample
- Route spans for one trace coherently and capacity-plan decision wait and trace state
- Measure retained and dropped traces by policy and validate late-span behavior
- Audit metric series and reject user, email, raw URL, request, trace, and span IDs as labels
- Use normalized routes, bounded status or outcome sets, and backend-appropriate base units
- Redact or drop prohibited telemetry as early as practical and test canary fields
- Set span limits, event limits, log severity and body policies, and retention by signal
- Attribute bytes, events, series, CPU, memory, storage, query, and egress to owners
- Review sampling's compute, engineering, and missed-information costs before adoption
Kubernetes SLO platform and operational evidence
Weeks 8-10Operate a disposable shared platform with least privilege, tail sampling, SLO math, metamonitoring, capacity tests, and verified teardown.
- Deploy node agents and gateway services with explicit resource, probe, disruption, and network settings
- Enrich workload telemetry using only the Kubernetes metadata and RBAC required
- Review Kubernetes system metric lifecycle before depending on alpha, beta, or deprecated data
- Calculate availability and threshold-latency SLIs from valid user-oriented requests
- Build SLO and error-budget views alongside dependency and capacity diagnostics
- Page on sustained user symptoms and actionable capacity risks rather than every cause
- Add metamonitoring, an external synthetic probe, telemetry freshness, and alert-delivery validation
- Load test gateways and stateful samplers under trace, metric, log, and destination pressure
- Publish sanitized architecture, tests, policy decisions, SLO math, limits, and honest caveats
- Destroy namespaces, volumes, credentials, certificates, endpoints, retained data, and the disposable cluster
PrepKloud OpenTelemetry learning surfaces
Twenty-five practical scenarios across instrumentation, signals, Collector operations, sampling, SLOs, privacy, cost, and troubleshooting. OpenTelemetry flashcards
Review context, spans, resources, semantic conventions, Collector components, cardinality, sampling, and reliability. Two portfolio projects
Build a correlated polyglot gateway and an advanced Kubernetes SLO and tail-sampling platform. Complete practical guide
Read the architecture, study sequence, signal controls, project strategy, and operational principles. SRE and observability career stack
Place OpenTelemetry within the broader reliability engineering toolset and workflow. Kubernetes cloud-native career stack
Connect telemetry platform skills to Kubernetes operations and shared platform engineering. Explore related roles
Research SRE, observability, platform, cloud, and DevOps role requirements without treating outcomes as guaranteed. Editorial policy
Review PrepKloud sourcing, originality, update, independence, and correction standards.
Official resources
Use the observability primer, signals, resources, context, trace, metrics, logs, API/SDK architecture, stability, and protocol documentation.
Open OpenTelemetry docsOpen specification
Check current naming, attributes, units, types, requirements, and stability for resources and instrumented technologies.
Open semantic conventionsStudy deployment, configuration, components, resilience, scaling, internal telemetry, security, transformation, and troubleshooting.
Open Collector docsGround traceparent and tracestate parsing, propagation, mutation, privacy, security, and interoperability in the Recommendation.
Open W3C RecommendationUse official metric naming, labels, histograms, recording rules, alerting, and operational guidance for the metric and SLO portions.
Open Prometheus docsReview logging, system metrics, traces, resource behavior, metric lifecycle, and cluster access requirements.
Open Kubernetes observability docsConfirm project governance, maturity, community, and cloud-native ecosystem context through CNCF.
Open CNCF project pageFrequently asked questions
Is this OpenTelemetry path a certification or exam course?
No. It is explicitly a practical skill path. There is no claimed exam, passing score, credential, vendor blueprint, or question dump. Working telemetry, controlled failures, reliability math, security evidence, and cleanup demonstrate progress.
Do I need a commercial observability backend?
No. The projects can use small local OpenTelemetry-compatible and Prometheus-compatible components plus local trace and log stores. Backend choice is replaceable; instrumentation and pipeline contracts remain the main learning goal.
Should I learn automatic or manual instrumentation first?
Use supported automatic instrumentation to establish common framework coverage, inspect the result, and then add manual instrumentation for meaningful business operations and missing context. Audit for duplicate spans and unstable packages.
Is tail sampling always better than head sampling?
No. Head sampling is simpler, early, and efficient. Tail sampling can retain traces based on completed error, latency, and attribute evidence, but it needs trace-aware routing, state, memory, decision deadlines, scaling, and ongoing policy maintenance.
What portfolio projects are included?
Project one instruments a synthetic polyglot request path and a Collector gateway with traces, metrics, logs, redaction, resilience, and failure injection. Project two deploys a Kubernetes platform with tail sampling, cardinality and privacy controls, SLOs, alerts, capacity and cost measurements, and teardown.
Build evidence, not just dashboards
Practice the decisions, correlate all three signals, operate the pipeline under failure, and finish two projects with measurable privacy, reliability, capacity, and cleanup outcomes.