Bare-Metal

Gathering and sending telemetry from Bare Metal Hosts is easy. An example config is shown below.

Prerequisite

Download the relevant binary from: https://opentelemetry.io/docs/collector/installation

An example:

sudo yum update
sudo yum -y install wget systemctl
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.116.1/otelcol-contrib_0.116.1_linux_amd64.rpm
sudo rpm -ivh otelcol-contrib_0.116.1_linux_amd64.rpm

Remove default config

sudo rm /etc/otelcol-contrib/config.yaml

Copy "Example Config" (below). Replace {INTAKE} and {AUTH_TOKEN} :

sudo nano /etc/otelcol-contrib/config.yaml

Restart Collector Service:

sudo systemctl restart otelcol-contrib

Check Status:

sudo systemctl status otelcol-contrib

Example Config

This is a full-fledged example you can use to gather system metrics (excluding per process) and traces locally:

receivers:
  hostmetrics:
    collection_interval: 15s
    scrapers:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network:
      processes:
  otlp:
    protocols:
      grpc:
        endpoint: "localhost:4317"
      http:
        endpoint: "localhost:4318"
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1000
  batch:
  resourcedetection:
    system:
      resource_attributes:
        os.description:
          enabled: true
        host.arch:
          enabled: true
        host.cpu.vendor.id:
          enabled: true
        host.cpu.family:
          enabled: true
        host.cpu.model.id:
          enabled: true
        host.cpu.model.name:
          enabled: true
        host.cpu.stepping:
          enabled: true
        host.cpu.cache.l2.size:
          enabled: true
    detectors: [env, ecs, ec2, gcp, azure, system]
    timeout: 2s
    override: false
exporters:
  otlphttp/omlet:
    endpoint: "{INTAKE}"
    headers:
      Authorization: "Bearer {AUTH_TOKEN}"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resourcedetection]
      exporters: [otlphttp/omlet]
    metrics:
      receivers: [hostmetrics, otlp]
      processors: [memory_limiter, batch, resourcedetection]
      exporters: [otlphttp/omlet]

Metrics (System)

The hostmetrics section in the Example Config sets typical system scrapers. You can always customize this: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/hostmetricsreceiver#getting-started

For example, setting different collection interval and adding per-process collection:

hostmetrics:
    collection_interval: 30s
    scrapers:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network:
      processes:
      process:

Logs

To add logs, you can add logs to the pipeline.

Example with OTLP

logs:
  receivers: [otlp]
  processors: [memory_limiter, batch, resourcedetection]
  exporters: [otlphttp/omlet]

Example with filelog receiver (https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). Note how fault-tolerant log collection is enabled with storage and sending queue (this can be disabled):

receivers:
  filelog:
    include: [/var/log/app.log]
    storage: file_storage/filelogreceiver
  hostmetrics:
    collection_interval: 15s
    scrapers:
      paging:
        metrics:
          system.paging.utilization:
            enabled: true
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
      disk:
      filesystem:
        metrics:
          system.filesystem.utilization:
            enabled: true
      load:
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      network:
      processes:
  otlp:
    protocols:
      grpc:
        endpoint: "localhost:4317"
      http:
        endpoint: "localhost:4318"
processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 1000
  batch:
  resourcedetection:
    system:
      resource_attributes:
        os.description:
          enabled: true
        host.arch:
          enabled: true
        host.cpu.vendor.id:
          enabled: true
        host.cpu.family:
          enabled: true
        host.cpu.model.id:
          enabled: true
        host.cpu.model.name:
          enabled: true
        host.cpu.stepping:
          enabled: true
        host.cpu.cache.l2.size:
          enabled: true
    detectors: [env, ecs, ec2, gcp, azure, system]
    timeout: 2s
    override: false
exporters:
  otlphttp/omlet:
    endpoint: "{INTAKE}"
    sending_queue:
      storage: file_storage/otlpoutput
    headers:
      Authorization: "Bearer {AUTH_TOKEN}"
extensions:
  file_storage/filelogreceiver:
    directory: /var/lib/otelcol/file_storage/receiver
    create_directory: true
  file_storage/otlpoutput:
    directory: /var/lib/otelcol/file_storage/output
    create_directory: true
service:
  extensions: [file_storage/filelogreceiver, file_storage/otlpoutput]
  pipelines:
    logs:
      receivers: [filelog, otlp]
      processors: [memory_limiter, batch, resourcedetection]
      exporters: [otlphttp/omlet]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch, resourcedetection]
      exporters: [otlphttp/omlet]
    metrics:
      receivers: [hostmetrics, otlp]
      processors: [memory_limiter, batch, resourcedetection]
      exporters: [otlphttp/omlet]