Application Performance Monitoring (APM)

Access the APM dashboard at https://app.watchlog.io/apm to view your applications and performance metrics.

Overview

Watchlog APM uses OpenTelemetry for instrumentation. You don't need to install any Watchlog-specific packages. Simply:

1. Install OpenTelemetry SDK for your platform (Node.js, Python, Go, Java, .NET, etc.)
2. Configure OTLP exporter to send traces to your Watchlog agent
3. Set your application name - this will appear in the Watchlog dashboard

Agent URL Format

The trace endpoint follows this format:

http://<agent-host>:3774/apm/<your-app-name>/v1/traces

Examples:

• Local: http://localhost:3774/apm/my-service/v1/traces
• Docker: http://watchlog-agent:3774/apm/my-service/v1/traces

Prerequisites

Before configuring APM, make sure you have:

1. Watchlog Agent installed and running (see Host Map for installation)
2. Agent accessible at http://localhost:3774 (local) or http://watchlog-agent:3774 (Docker)
3. Your application name ready (this will be used to identify your service in Watchlog)

Node.js

Installation

Install OpenTelemetry packages:

npm install @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node @opentelemetry/exporter-trace-otlp-http

Basic Usage

Create an otel-config.js file (must be loaded first, before your application code):

// otel-config.js — must be first
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_SERVICE_VERSION } from '@opentelemetry/semantic-conventions';

// Your application name
const APP_NAME = process.env.APP_NAME || 'my-service';
// Watchlog agent URL (local: http://localhost:3774 or Docker: http://watchlog-agent:3774)
const WATCHLOG_URL = process.env.WATCHLOG_URL || 'http://localhost:3774';

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: APP_NAME,
    [SEMRESATTRS_SERVICE_VERSION]: '1.0.0',
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${WATCHLOG_URL}/apm/${APP_NAME}/v1/traces`,
    headers: {},
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // Disable unnecessary instrumentations to reduce overhead
      '@opentelemetry/instrumentation-fs': {
        enabled: false,
      },
    }),
  ],
});

sdk.start();

console.log(`OpenTelemetry initialized for app: ${APP_NAME}`);
console.log(`Sending traces to: ${WATCHLOG_URL}/apm/${APP_NAME}/v1/traces`);

// Graceful shutdown
process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('OpenTelemetry terminated'))
    .catch((error) => console.log('Error terminating OpenTelemetry', error))
    .finally(() => process.exit(0));
});

Then load it before your application:

// index.js
// Load OpenTelemetry first
import './otel-config.js';

// Continue loading your application
import express from 'express';
const app = express();

app.get('/', (req, res) => res.send('Hello World!'));
app.listen(3000, () => console.log('Listening on 3000'));

Docker Setup

When running your Node.js app in Docker, set the WATCHLOG_URL environment variable to point to your Watchlog Agent container:

// otel-config.js
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME } from '@opentelemetry/semantic-conventions';

const APP_NAME = process.env.APP_NAME || 'my-service';
// For Docker: use container name 'watchlog-agent'
// For local: use 'localhost'
const WATCHLOG_URL = process.env.WATCHLOG_URL || 'http://watchlog-agent:3774';

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: APP_NAME,
  }),
  traceExporter: new OTLPTraceExporter({
    url: `${WATCHLOG_URL}/apm/${APP_NAME}/v1/traces`,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

Docker Compose Example:

version: '3.8'

services:
  watchlog-agent:
    image: watchlog/agent:latest
    container_name: watchlog-agent
    ports:
      - "3774:3774"
    environment:
      - WATCHLOG_APIKEY=your-api-key
      - WATCHLOG_SERVER=https://log.watchlog.ir
    networks:
      - app-network

  node-app:
    build: .
    container_name: node-app
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - APP_NAME=my-service
      - WATCHLOG_URL=http://watchlog-agent:3774  # Use container name
    depends_on:
      - watchlog-agent
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

Docker Run Example:

# 1. Create network
docker network create app-network

# 2. Run Watchlog Agent
docker run -d \
  --name watchlog-agent \
  --network app-network \
  -p 3774:3774 \
  -e WATCHLOG_APIKEY="your-api-key" \
  -e WATCHLOG_SERVER="https://log.watchlog.ir" \
  watchlog/agent:latest

# 3. Run Node.js app with environment variables
docker run -d \
  --name node-app \
  --network app-network \
  -p 3000:3000 \
  -e APP_NAME=my-service \
  -e WATCHLOG_URL=http://watchlog-agent:3774 \
  my-node-app

Important Notes:

• When using Docker, use the container name as the hostname (e.g., watchlog-agent)
• Both containers must be on the same Docker network
• The agent must be running before your app starts
• Set the WATCHLOG_URL environment variable to http://watchlog-agent:3774 for Docker
• Set the APP_NAME environment variable to your application name
• The trace endpoint format is: ${WATCHLOG_URL}/apm/${APP_NAME}/v1/traces

Python

Watchlog APM supports Python applications using OpenTelemetry. Install the OpenTelemetry SDK and configure it to send traces to your Watchlog agent.

Installation

Install OpenTelemetry packages:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation

For specific frameworks, install the corresponding instrumentation packages:

• Flask: opentelemetry-instrumentation-flask
• Django: opentelemetry-instrumentation-django
• FastAPI: opentelemetry-instrumentation-fastapi
• PostgreSQL: opentelemetry-instrumentation-psycopg2
• Requests: opentelemetry-instrumentation-requests

See the OpenTelemetry Python Instrumentation documentation for a complete list of available instrumentations.

Basic Usage

# otel_config.py
import os
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# Your application name
APP_NAME = os.getenv('APP_NAME', 'my-python-service')
# Watchlog agent URL (local: http://localhost:3774 or Docker: http://watchlog-agent:3774)
WATCHLOG_URL = os.getenv('WATCHLOG_URL', 'http://localhost:3774')

# Set up OpenTelemetry
resource = Resource.create({
    "service.name": APP_NAME,
    "service.version": "1.0.0",
})

trace.set_tracer_provider(TracerProvider(resource=resource))

# Configure OTLP exporter
otlp_exporter = OTLPSpanExporter(
    endpoint=f"{WATCHLOG_URL}/apm/{APP_NAME}/v1/traces",
)

span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

Then import this configuration before your application code:

# main.py
# Import OpenTelemetry configuration first
import otel_config

# Now import and use your framework
from flask import Flask  # or Django, FastAPI, etc.
from opentelemetry.instrumentation.flask import FlaskInstrumentor

app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)

@app.route("/")
def hello():
    return "Hello, Watchlog APM!"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=6000)

Framework-Specific Instrumentation

For Flask:

from opentelemetry.instrumentation.flask import FlaskInstrumentor
FlaskInstrumentor().instrument_app(app)

For Django (in wsgi.py or asgi.py, before Django setup):

from opentelemetry.instrumentation.django import DjangoInstrumentor
DjangoInstrumentor().instrument()

For FastAPI:

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
FastAPIInstrumentor.instrument_app(app)

Docker Setup

Docker Compose Example:

version: '3.8'

services:
  watchlog-agent:
    image: watchlog/agent:latest
    container_name: watchlog-agent
    ports:
      - "3774:3774"
    environment:
      - WATCHLOG_APIKEY=your-api-key
      - WATCHLOG_SERVER=https://log.watchlog.ir
    networks:
      - app-network

  python-app:
    build: .
    container_name: python-app
    ports:
      - "8000:8000"
    environment:
      - APP_NAME=my-python-service
      - WATCHLOG_URL=http://watchlog-agent:3774  # Use container name
    depends_on:
      - watchlog-agent
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

Docker Run Example:

# 1. Create network
docker network create app-network

# 2. Run Watchlog Agent
docker run -d \
  --name watchlog-agent \
  --network app-network \
  -p 3774:3774 \
  -e WATCHLOG_APIKEY="your-api-key" \
  -e WATCHLOG_SERVER="https://log.watchlog.ir" \
  watchlog/agent:latest

# 3. Run Python app with environment variables
docker run -d \
  --name python-app \
  --network app-network \
  -p 8000:8000 \
  -e APP_NAME=my-python-service \
  -e WATCHLOG_URL=http://watchlog-agent:3774 \
  my-python-app

Important Notes:

• When using Docker, use the container name as the hostname (e.g., watchlog-agent)
• Both containers must be on the same Docker network
• The agent must be running before your app starts
• Set the WATCHLOG_URL environment variable to http://watchlog-agent:3774 for Docker
• Set the APP_NAME environment variable to your application name
• The trace endpoint format is: ${WATCHLOG_URL}/apm/${APP_NAME}/v1/traces
• For Django, OpenTelemetry instrumentation must be initialized before Django setup
• Install framework-specific instrumentation packages as needed

Go

Installation

Install OpenTelemetry packages:

go get go.opentelemetry.io/otel \
  go.opentelemetry.io/otel/trace \
  go.opentelemetry.io/otel/sdk \
  go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp \
  go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp

Basic Usage

// main.go
package main

import (
    "context"
    "fmt"
    "log"
    "net/http"
    "os"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

var (
    appName    = getEnv("APP_NAME", "my-go-service")
    watchlogURL = getEnv("WATCHLOG_URL", "http://localhost:3774")
)

func getEnv(key, defaultValue string) string {
    if value := os.Getenv(key); value != "" {
        return value
    }
    return defaultValue
}

func initTracer() func() {
    ctx := context.Background()

    // Create resource
    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceNameKey.String(appName),
            semconv.ServiceVersionKey.String("1.0.0"),
        ),
    )
    if err != nil {
        log.Fatalf("failed to create resource: %v", err)
    }

    // Create OTLP exporter
    endpoint := fmt.Sprintf("%s/apm/%s/v1/traces", watchlogURL, appName)
    exporter, err := otlptracehttp.New(ctx,
        otlptracehttp.WithEndpoint(endpoint),
        otlptracehttp.WithInsecure(), // Use WithInsecure() for HTTP
    )
    if err != nil {
        log.Fatalf("failed to create exporter: %v", err)
    }

    // Create trace provider
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
    )
    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    return func() {
        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        if err := tp.Shutdown(ctx); err != nil {
            log.Fatal(err)
        }
    }
}

func main() {
    shutdown := initTracer()
    defer shutdown()

    // Create HTTP handler with OpenTelemetry instrumentation
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, Watchlog APM + Go!"))
    })
    http.Handle("/", otelhttp.NewHandler(handler, "root"))

    log.Printf("Starting server on :8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Docker Setup

Docker Compose Example:

version: '3.8'

services:
  watchlog-agent:
    image: watchlog/agent:latest
    container_name: watchlog-agent
    ports:
      - "3774:3774"
    environment:
      - WATCHLOG_APIKEY=your-api-key
      - WATCHLOG_SERVER=https://log.watchlog.ir
    networks:
      - app-network

  go-app:
    build: .
    container_name: go-app
    ports:
      - "8080:8080"
    environment:
      - APP_NAME=my-go-service
      - WATCHLOG_URL=http://watchlog-agent:3774
    depends_on:
      - watchlog-agent
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

Additional Resources

For more information about OpenTelemetry instrumentation for other languages and frameworks, visit:

• OpenTelemetry Documentation
• Node.js Instrumentation
• Python Instrumentation
• Go Instrumentation
• Java Instrumentation
• .NET Instrumentation

Common Configuration

Regardless of the language or framework, the key configuration is:

1. Install OpenTelemetry SDK for your platform
2. Configure OTLP exporter with endpoint: http://<agent-host>:3774/apm/<your-app-name>/v1/traces
3. Set service name to identify your application in Watchlog
4. Enable auto-instrumentation for your framework (see OpenTelemetry documentation for framework-specific instrumentations)

Troubleshooting

• No traces appearing? Check that:

  • Watchlog agent is running and accessible
  • Agent URL is correct (use http://watchlog-agent:3774 for Docker)
  • Application name matches in both code and agent configuration
  • OpenTelemetry SDK is initialized before your application code

• Connection errors? Verify:

  • Both agent and app are on the same Docker network (if using Docker)
  • Agent port 3774 is accessible
  • Firewall rules allow communication between containers