Observability Examples¶

All examples include built-in OpenTelemetry support for distributed tracing and log forwarding.

Overview¶

Learn how to:

Configure OpenTelemetry tracing
Enable log forwarding to APM
Correlate logs with traces
Monitor API operations

Quick Start¶

from kibana import Kibana
from examples.utils import configure_example_telemetry
import logging

# Configure telemetry
traces_enabled, logs_enabled = configure_example_telemetry(
    enabled=True,
    logs_enabled=True
)

# Create logger
logger = logging.getLogger("my-app")

# Initialize client
client = Kibana("http://localhost:5601")

# Operations are automatically traced
connector = client.actions.create(
    name="My Connector",
    connector_type_id=".index",
    config={"index": "my-index"}
)

# Logs are automatically forwarded
logger.info(
    "Connector created",
    extra={
        "connector_id": connector.body["id"],
        "connector_type": ".index"
    }
)

Configuration¶

Environment Variables¶

# Trace configuration
export KIBANA_OTEL_ENABLED=true
export OTEL_SERVICE_NAME=my-app
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:8200

# Log forwarding
export KIBANA_OTEL_LOGS_ENABLED=true
export KIBANA_OTEL_LOGS_LEVEL=WARNING
export KIBANA_OTEL_LOGS_LOGGERS=kibana,my-app

Programmatic Configuration¶

from kibana import configure_opentelemetry

configure_opentelemetry(
    enabled=True,
    service_name="my-app",
    exporter="otlp",
    endpoint="http://localhost:8200"
)

Structured Logging¶

import logging

logger = logging.getLogger("my-app")

# Log with structured data
logger.info(
    "API request completed",
    extra={
        "http_method": "POST",
        "http_path": "/api/actions/connector",
        "http_status_code": 200,
        "response_time_ms": 150,
        "connector_id": "conn-123"
    }
)

Log-Trace Correlation¶

When both traces and logs are enabled, logs automatically include trace correlation:

# This creates a trace
response = client.actions.list()

# This log is correlated with the trace
logger.info(
    "Retrieved connectors",
    extra={
        "connector_count": len(response.body),
        "operation": "list_connectors"
    }
)

In APM, you’ll see:

The trace for client.actions.list()
The log message with the same trace_id and span_id
Ability to navigate between traces and logs

Viewing Telemetry Data¶

In Kibana APM¶

Open Kibana: http://localhost:5601/app/apm
Select your service name
View traces and logs
Navigate between correlated data

Example Trace¶

Service: my-app
Operation: POST /api/actions/connector
Duration: 150ms
Status: 200

Spans:
  ├─ HTTP POST /api/actions/connector (150ms)
  │  └─ Elasticsearch index operation (45ms)
  └─ Space validation (5ms)

Logs:
  [INFO] Connector created (connector_id=conn-123)

Best Practices¶

Use meaningful logger names for filtering
Include structured data in log extra fields
Set appropriate log levels (WARNING for production)
Monitor trace performance for slow operations
Use log forwarding for error tracking

Automatic Configuration¶

With elastic-start-local:

./local-stack.sh -o start

# Telemetry is automatically configured
python examples/simple_index_connector.py

Performance Considerations¶

Log Level Filtering¶

# Production: Only forward important logs
export KIBANA_OTEL_LOGS_LEVEL=ERROR

# Development: Forward all logs
export KIBANA_OTEL_LOGS_LEVEL=DEBUG

Logger Selection¶

# Forward specific loggers only
export KIBANA_OTEL_LOGS_LOGGERS=kibana,my-app,critical-module

Zero Overhead When Disabled¶

# No performance impact when disabled
export KIBANA_OTEL_ENABLED=false
export KIBANA_OTEL_LOGS_ENABLED=false

Troubleshooting¶

Check Telemetry Status¶

All examples display telemetry status:

📊 OpenTelemetry Status:
  ✅ Traces: Enabled (endpoint: http://localhost:8200)
  ✅ Logs: Enabled (level: WARNING, loggers: kibana,my-app)
  📡 Protocol: grpc
  🔑 Authentication: Bearer token

Common Issues¶

Traces not appearing: Check KIBANA_OTEL_ENABLED=true
Logs not appearing: Check KIBANA_OTEL_LOGS_ENABLED=true
Authentication errors: Verify APM token