Datadog Agent(s) to OTel

The Omlet Datadog-to-OTel service can receive metrics, logs and traces from Datadog Agents

Omlet Datadog-to-OTel Service

Container

Pull psharma1989/datadog-to-otel:latest

Set environment variables:

DD_API_KEY="${DATADOG_API_KEY}"
OMLET_INTAKE="${OMLET_INTAKE}"
OMLET_INTAKE_API_KEY="${OMLET_INTAKE_API_KEY}"
DD_SITE="datadoghq.com"

Datadog Agent Config

Variables:

DD_OMLET_INTAKE="${OMLET_INTAKE/intake/ddintake}"
DD_OMLET_INTAKE_HOST="${DD_OMLET_INTAKE#https://}"

Dual Shipping

Set "additional endpoints" for metrics logs and traces:

# metrics
DD_ADDITIONAL_ENDPOINTS="{\"${DD_OMLET_INTAKE}\": [\"${DATADOG_API_KEY}\"]}"
# traces
DD_APM_ADDITIONAL_ENDPOINTS="{\"${DD_OMLET_INTAKE}\": [\"${DATADOG_API_KEY}\"]}"
# logs
DD_LOGS_CONFIG_FORCE_USE_HTTP=true
DD_LOGS_CONFIG_ADDITIONAL_ENDPOINTS="[{\"api_key\": \"${DATADOG_API_KEY}\", \"Host\": \"${DD_OMLET_INTAKE_HOST}\", \"Port\": 443, \"is_reliable\": true}]"

Note: ${DD_OMLET_INTAKE_HOST} is ${DD_OMLET_INTAKE} without protocol (https://)

Lambda Extension vs Datadog Agent

Due to "additional endpoints" configurations not being supported in the new Datadog Lambda Extension (https://github.com/DataDog/datadog-lambda-extension/issues/493), the legacy extension must be used. In your lambda environment, set: DD_EXTENSION_VERSION=compatibility

Proxy

# metrics
DD_DD_URL="${DD_OMLET_INTAKE}"
# traces
DD_APM_DD_URL="${DD_OMLET_INTAKE}"
# logs
DD_LOGS_CONFIG_LOGS_DD_URL="${DD_OMLET_INTAKE}"

Host Tags

To retain host tags, configure the Datadog Agent YAML:

Bare-Metal

Set expected_tags_duration . Example with very long duration.

expected_tags_duration: "80000h"
logs_config:
  expected_tags_duration: "80000h"

Kubernetes (Helm)

Set expected_tags_duration . Example with very long duration.

datadog:
  tags: 
    - "foo:baz"
agents:
  useConfigMap: true
  customAgentConfig:
    expected_tags_duration: "80000h"
    logs_config:
      expected_tags_duration: "80000h"

Kubernetes Example:

The example below highlights deploying this service as a K8s deployment:

kubectl create secret generic datadog-secrets \
  --from-literal=DATADOG_API_KEY="${DATADOG_API_KEY}" \
  --from-literal=OMLET_INTAKE="${OMLET_INTAKE}" \
  --from-literal=OMLET_INTAKE_API_KEY="${OMLET_INTAKE_API_KEY}"

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: datadog-to-otel
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: datadog-to-otel
  template:
    metadata:
      labels:
        app: datadog-to-otel
    spec:
      containers:
        - name: datadog-to-otel
          image: psharma1989/datadog-to-otel:latest
          ports:
            - containerPort: 8080
          resources: 
            requests:
              cpu: 50m   
              memory: 128Mi 
            limits:
              cpu: 100m
              memory: 256Mi
          env:
            - name: DD_SITE
              value: 'datadoghq.com'
            - name: DD_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-secrets
                  key: DATADOG_API_KEY
            - name: OMLET_INTAKE
              valueFrom:
                secretKeyRef:
                  name: datadog-secrets
                  key: OMLET_INTAKE
            - name: OMLET_INTAKE_API_KEY
              valueFrom:
                secretKeyRef:
                  name: datadog-secrets
                  key: OMLET_INTAKE_API_KEY
---
apiVersion: v1
kind: Service
metadata:
  name: datadog-to-otel-service
  namespace: monitoring
spec:
  selector:
    app: datadog-to-otel
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

You can set this up behind a load balancer for security and scale.