OpenAI Monitoring with OpenTelemetry

This guide walks you through setting up OpenAI monitoring using OpenTelemetry and SigNoz. You will instrument your OpenAI applications to capture traces, logs, and metrics and visualize them in real time in SigNoz.

OpenAI monitoring with SigNoz gives you visibility into:

Token usage — track input and output tokens per request to understand cost drivers
Model latency — measure response times across models and request types
Error rates — detect API failures, rate limit errors, and timeouts before they impact users
Request traces — follow individual LLM calls through your full application stack

Requirements

Python 3.8 or newer
OpenAI Python library (openai >= 1.0.0)
Valid OpenAI API key
SigNoz setup (choose one):
- SigNoz Cloud account with an active ingestion key
- Self-hosted SigNoz instance

Setup

Step 1. Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

Step 2. Install the OpenTelemetry dependencies

pip install opentelemetry-distro~=0.51b0
pip install opentelemetry-exporter-otlp~=1.30.0
pip install opentelemetry-instrumentation-openai-v2

Step 3. Add automatic instrumentation

opentelemetry-bootstrap --action=install

Step 4. Run your application

OTEL_SERVICE_NAME=<service-name> \
OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.us.signoz.cloud:443" \
OTEL_EXPORTER_OTLP_HEADERS="signoz-ingestion-key=<your-ingestion-key>" \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true \
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true \
OPENAI_API_KEY=<your-openai-key> \
opentelemetry-instrument <your_run_command>

Environment Variables Explained:

OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
- Enables automatic logging instrumentation
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT
- Captures prompts and completions as logs
us: Your SigNoz Cloud region
<your-ingestion-key>: Your SigNoz ingestion key
Replace <service-name> with the name of your service
Replace <your-openai-key> with your OpenAI API key

Requirements for Java

Java 8 or newer
OpenAI Java SDK (openai-java >= 1.1.0)
Valid OpenAI API key
SigNoz setup (choose one):
- SigNoz Cloud account with an active ingestion key
- Self-hosted SigNoz instance

Setup for Java

Step 1. Add dependencies

For Gradle (build.gradle.kts):

dependencies {
    implementation("com.openai:openai-java:3.6.1")
    implementation("io.github.cdimascio:dotenv-java:3.0.0")
    implementation("io.opentelemetry:opentelemetry-api:1.54.1")
    implementation("io.opentelemetry:opentelemetry-sdk:1.54.1")
    implementation("io.opentelemetry:opentelemetry-sdk-trace:1.54.1")
    implementation("io.opentelemetry:opentelemetry-sdk-metrics:1.54.1")
    implementation("io.opentelemetry:opentelemetry-sdk-logs:1.54.1")
    implementation("io.opentelemetry:opentelemetry-exporter-otlp:1.54.1")
    implementation("io.opentelemetry:opentelemetry-semconv:1.27.0-alpha")
    implementation(platform("io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:2.20.1-alpha"))
    implementation("io.opentelemetry.instrumentation:opentelemetry-openai-java-1.1")
}

For Maven (pom.xml):

<dependencies>
    <dependency>
        <groupId>com.openai</groupId>
        <artifactId>openai-java</artifactId>
        <version>3.6.1</version>
    </dependency>
    <dependency>
        <groupId>io.github.cdimascio</groupId>
        <artifactId>dotenv-java</artifactId>
        <version>3.0.0</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-api</artifactId>
        <version>1.54.1</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk</artifactId>
        <version>1.54.1</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk-trace</artifactId>
        <version>1.54.1</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk-metrics</artifactId>
        <version>1.54.1</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-sdk-logs</artifactId>
        <version>1.54.1</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-exporter-otlp</artifactId>
        <version>1.54.1</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry</groupId>
        <artifactId>opentelemetry-semconv</artifactId>
        <version>1.27.0-alpha</version>
    </dependency>
    <dependency>
        <groupId>io.opentelemetry.instrumentation</groupId>
        <artifactId>opentelemetry-openai-java-1.1</artifactId>
        <version>2.20.1-alpha</version>
    </dependency>
</dependencies>

Step 2. Import the necessary modules in your Java application

OpenTelemetry:

import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.instrumentation.openai.v1_1.OpenAITelemetry;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.semconv.ServiceAttributes;

Traces:

import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.BatchSpanProcessor;
import io.opentelemetry.exporter.otlp.http.trace.OtlpHttpSpanExporter;

Logs:

import io.opentelemetry.sdk.logs.SdkLoggerProvider;
import io.opentelemetry.sdk.logs.export.BatchLogRecordProcessor;
import io.opentelemetry.exporter.otlp.http.logs.OtlpHttpLogRecordExporter;

Metrics:

import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;
import io.opentelemetry.exporter.otlp.http.metrics.OtlpHttpMetricExporter;

Step 3. Setup Traces

import io.github.cdimascio.dotenv.Dotenv;

Dotenv dotenv = Dotenv.load();

String serviceName = "<service_name>";

Resource resource = Resource.getDefault()
            .toBuilder()
            .put(ServiceAttributes.SERVICE_NAME, serviceName)
            .build();

OtlpHttpSpanExporter otlpSpanExporter = OtlpHttpSpanExporter.builder()
            .setEndpoint(System.getenv("OTEL_EXPORTER_TRACES_ENDPOINT"))
            .addHeader("signoz-ingestion-key", System.getenv("SIGNOZ_INGESTION_KEY"))
            .build();

SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
            .setResource(resource)
            .addSpanProcessor(BatchSpanProcessor.builder(otlpSpanExporter).build())
            .build();

Step 4. Setup Logs

OtlpHttpLogRecordExporter otlpLogExporter = OtlpHttpLogRecordExporter.builder()
            .setEndpoint(System.getenv("OTEL_EXPORTER_LOGS_ENDPOINT"))
            .addHeader("signoz-ingestion-key", System.getenv("SIGNOZ_INGESTION_KEY"))
            .build();

SdkLoggerProvider loggerProvider = SdkLoggerProvider.builder()
            .setResource(resource)
            .addLogRecordProcessor(BatchLogRecordProcessor.builder(otlpLogExporter).build())
            .build();

Step 5. Setup Metrics

OtlpHttpMetricExporter otlpMetricExporter = OtlpHttpMetricExporter.builder()
            .setEndpoint(System.getenv("OTEL_EXPORTER_METRICS_ENDPOINT"))
            .addHeader("signoz-ingestion-key", System.getenv("SIGNOZ_INGESTION_KEY"))
            .build();

SdkMeterProvider meterProvider = SdkMeterProvider.builder()
            .setResource(resource)
            .registerMetricReader(PeriodicMetricReader.builder(otlpMetricExporter).build())
            .build();

Step 6. Setup OpenTelemetry SDK with your created providers

OpenTelemetrySdk openTelemetry = OpenTelemetrySdk.builder()
            .setTracerProvider(tracerProvider)
            .setMeterProvider(meterProvider)
            .setLoggerProvider(loggerProvider)
            .build();

Step 7. Run an Example using OpenAI Instrumentor

OpenAIClient client = OpenAIOkHttpClient.builder()
            .apiKey(System.getenv("OPENAI_API_KEY"))
            .build();

OpenAIClient otelClient = OpenAITelemetry.builder(openTelemetry)
            .build()
            .wrap(client);

ChatCompletionCreateParams params = ChatCompletionCreateParams.builder()
            .addUserMessage("What is SigNoz?")
            .model("gpt-4.1")
            .build();

ChatCompletion chatCompletion = otelClient.chat().completions().create(params);
System.out.println(chatCompletion);

// Force flush with timeout to ensure all telemetry is exported before shutdown
sdk.getSdkTracerProvider().forceFlush().join(10, java.util.concurrent.TimeUnit.SECONDS);
sdk.getSdkMeterProvider().forceFlush().join(10, java.util.concurrent.TimeUnit.SECONDS);
sdk.getSdkLoggerProvider().forceFlush().join(10, java.util.concurrent.TimeUnit.SECONDS);

sdk.getSdkTracerProvider().shutdown();
sdk.getSdkMeterProvider().shutdown();
sdk.getSdkLoggerProvider().shutdown();

What Does OpenAI Monitoring Capture?

This instrumentation captures comprehensive telemetry data for your OpenAI applications:

Traces

Spans for each OpenAI API call with timing information
Operation details like model name, max tokens etc.
Token usage including input and output
Request and response metadata such as finish reason and model used

Logs

Structured logs for each API call when OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED=true is set
Message content logs (prompts and completions) when OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true is enabled
Error logs for failed API calls with detailed error information and stack traces
Performance logs showing request duration and timing information

Metrics

Duration metrics showing how long OpenAI calls take
Token usage metrics tracking consumption over time
Request rate metrics showing API call frequency
Error rate metrics for monitoring API failures

Validating Your OpenAI Monitoring Setup

Validate your traces, logs, and metrics in SigNoz:

Trigger OpenAI API calls in your app. Make several API calls to generate some data. Then, wait for some time.
In SigNoz, open the Services tab. Hit the Refresh button on the top right corner, and your application should appear in the list of Applications.
Go to the Traces tab, and apply relevant filters to see your application's traces.
Check the Logs tab to see captured logs from your OpenAI calls.
Visit the Metrics tab to view token usage and performance metrics.