---
title: profile_qdisc_latency
sidebar_position: 0
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# profile_qdisc_latency

The `profile_qdisc_latency` gadget gathers information about the usage of the
network interfaces, generating a histogram distribution of latency caused by
the network scheduler when consuming packets off qdiscs, when the gadget is stopped.

The histogram shows the number of packets enqueued to qdiscs (`count` column) that lie in the
latency range `interval-start` -> `interval-end` (`µs` column). By default the latency is
measured in microseconds. If the `--ms` flag is passed, it will be shown in milliseconds.

This guide will use the [netem qdisc](https://man7.org/linux/man-pages/man8/tc-netem.8.html) to
emulate delay in sending packets. To configure it, the [tc](https://man7.org/linux/man-pages/man8/tc.8.html)
program of the `iproute2` is used.

## Requirements

- Minimum Kernel Version : 5.15

## Getting started

Running the gadget:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        $ kubectl gadget run ghcr.io/inspektor-gadget/gadget/profile_qdisc_latency:%IG_TAG% [flags]
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        $ sudo ig run ghcr.io/inspektor-gadget/gadget/profile_qdisc_latency:%IG_TAG% [flags]
        ```
    </TabItem>
</Tabs>

## Guide

Run the gadget in a terminal:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        $ kubectl gadget run profile_qdisc_latency:%IG_TAG% --node minikube-docker
        ```

        It will start to display the qdisc latency distribution as follows:
        ```bash
        latency
        µs               : count    distribution
               0 -> 1          : 2        |****************************************|
               1 -> 2          : 0        |                                        |
               2 -> 4          : 0        |                                        |
               4 -> 8          : 0        |                                        |
               8 -> 16         : 0        |                                        |
              16 -> 32         : 0        |                                        |
              32 -> 64         : 0        |                                        |
              64 -> 128        : 0        |                                        |
             128 -> 256        : 0        |                                        |
             256 -> 512        : 0        |                                        |
             512 -> 1024       : 0        |                                        |
            1024 -> 2048       : 0        |                                        |
            2048 -> 4096       : 0        |                                        |
            4096 -> 8192       : 0        |                                        |
            8192 -> 16384      : 0        |                                        |
           16384 -> 32768      : 0        |                                        |
           32768 -> 65536      : 0        |                                        |
           65536 -> 131072     : 0        |                                        |
          131072 -> 262144     : 0        |                                        |
          262144 -> 524288     : 0        |                                        |
          524288 -> 1048576    : 0        |                                        |
         1048576 -> 2097152    : 0        |                                        |
         2097152 -> 4194304    : 0        |                                        |
         4194304 -> 8388608    : 0        |                                        |
         8388608 -> 16777216   : 0        |                                        |
        16777216 -> 33554432   : 0        |                                        |
        33554432 -> 67108864   : 0        |                                        |
        ```

        Now to introduce some more latency, let's add a `netem` qdisc with some latency and jitter.

        ```bash
        # Start by creating our testing namespace
        $ kubectl create ns qdisc-latency-test

        # Run a pod that we will emulate network latency in
        $ kubectl run -n qdisc-latency-test --rm -it netem-test \
            --image=alpine --restart=Never \
            --privileged

        # Inside the container run the following commands
        $ apk update
        $ apk add iproute2
        # This will introduce a latency of 100ms with 100ms jitter
        $ tc qdisc add dev eth0 root netem delay 100ms 100ms

        # Now let's ping some remote host
        $ ping google.com
        ```

        Using the profile qdisc-latency gadget, we can generate another histogram to analyse the
        latency of scheduled network packets:

        ```bash
        # Run the gadget again
        $ kubectl gadget run profile_qdisc_latency:%IG_TAG% --node minikube-docker
        latency
              µs               : count    distribution
               0 -> 1          : 2        |****                                    |
               1 -> 2          : 0        |                                        |
               2 -> 4          : 0        |                                        |
               4 -> 8          : 1        |**                                      |
               8 -> 16         : 0        |                                        |
              16 -> 32         : 5        |***********                             |
              32 -> 64         : 14       |********************************        |
              64 -> 128        : 17       |****************************************|
             128 -> 256        : 0        |                                        |
             256 -> 512        : 0        |                                        |
             512 -> 1024       : 0        |                                        |
            1024 -> 2048       : 0        |                                        |
            2048 -> 4096       : 0        |                                        |
            4096 -> 8192       : 0        |                                        |
            8192 -> 16384      : 0        |                                        |
           16384 -> 32768      : 0        |                                        |
           32768 -> 65536      : 0        |                                        |
           65536 -> 131072     : 0        |                                        |
          131072 -> 262144     : 0        |                                        |
          262144 -> 524288     : 0        |                                        |
          524288 -> 1048576    : 0        |                                        |
         1048576 -> 2097152    : 0        |                                        |
         2097152 -> 4194304    : 0        |                                        |
         4194304 -> 8388608    : 0        |                                        |
         8388608 -> 16777216   : 0        |                                        |
        16777216 -> 33554432   : 0        |                                        |
        33554432 -> 67108864   : 0        |                                        |
        ```

        The new histogram shows how the latency numbers increased.
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        $ sudo ig run profile_qdisc_latency:%IG_TAG%
        ```

        It will start to display the qdisc latency distribution as follows:

        ```bash
        latency
        µs               : count    distribution
               0 -> 1          : 2        |****************************************|
               1 -> 2          : 0        |                                        |
               2 -> 4          : 0        |                                        |
               4 -> 8          : 0        |                                        |
               8 -> 16         : 0        |                                        |
              16 -> 32         : 0        |                                        |
              32 -> 64         : 0        |                                        |
              64 -> 128        : 0        |                                        |
             128 -> 256        : 0        |                                        |
             256 -> 512        : 0        |                                        |
             512 -> 1024       : 0        |                                        |
            1024 -> 2048       : 0        |                                        |
            2048 -> 4096       : 0        |                                        |
            4096 -> 8192       : 0        |                                        |
            8192 -> 16384      : 0        |                                        |
           16384 -> 32768      : 0        |                                        |
           32768 -> 65536      : 0        |                                        |
           65536 -> 131072     : 0        |                                        |
          131072 -> 262144     : 0        |                                        |
          262144 -> 524288     : 0        |                                        |
          524288 -> 1048576    : 0        |                                        |
         1048576 -> 2097152    : 0        |                                        |
         2097152 -> 4194304    : 0        |                                        |
         4194304 -> 8388608    : 0        |                                        |
         8388608 -> 16777216   : 0        |                                        |
        16777216 -> 33554432   : 0        |                                        |
        33554432 -> 67108864   : 0        |                                        |
        ```

        Now to introduce some more latency, let's add a `netem` qdisc with some latency and jitter.

        ```bash
        # We spawn a container where we will add network latency to
        $ docker run --rm --privileged --name qdisc-latency-test -it alpine

        # Inside the container run the following commands
        apk update
        apk add iproute2
        # This will introduce a latency of 100ms with 100ms jitter
        tc qdisc add dev eth0 root netem delay 100ms 100ms

        # Now let's ping some remote host
        ping google.com
        ```

        Using the profile qdisc-latency gadget, we can generate another histogram to analyse the
        latency of scheduled network packets:

        ```bash
        $ sudo ig run profile_qdisc_latency:%IG_TAG%
        latency
              µs               : count    distribution
               0 -> 1          : 2        |****                                    |
               1 -> 2          : 0        |                                        |
               2 -> 4          : 0        |                                        |
               4 -> 8          : 1        |**                                      |
               8 -> 16         : 0        |                                        |
              16 -> 32         : 5        |***********                             |
              32 -> 64         : 14       |********************************        |
              64 -> 128        : 17       |****************************************|
             128 -> 256        : 0        |                                        |
             256 -> 512        : 0        |                                        |
             512 -> 1024       : 0        |                                        |
            1024 -> 2048       : 0        |                                        |
            2048 -> 4096       : 0        |                                        |
            4096 -> 8192       : 0        |                                        |
            8192 -> 16384      : 0        |                                        |
           16384 -> 32768      : 0        |                                        |
           32768 -> 65536      : 0        |                                        |
           65536 -> 131072     : 0        |                                        |
          131072 -> 262144     : 0        |                                        |
          262144 -> 524288     : 0        |                                        |
          524288 -> 1048576    : 0        |                                        |
         1048576 -> 2097152    : 0        |                                        |
         2097152 -> 4194304    : 0        |                                        |
         4194304 -> 8388608    : 0        |                                        |
         8388608 -> 16777216   : 0        |                                        |
        16777216 -> 33554432   : 0        |                                        |
        33554432 -> 67108864   : 0        |                                        |
        ```
    </TabItem>
</Tabs>

You can clean up the resources created during this guide by running the following commands:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        $ kubectl delete ns qdisc-latency-test
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        $ docker rm -f qdisc-latency-test
        ```
    </TabItem>
</Tabs>

## Exporting metrics

The `profile_qdisc_latency` gadget can expose the histograms it generates to a
Prometheus endpoint. To do so, you need to activate both the metrics listener as
well as the gadget collector. To enable the metrics listener, check the
[Exporting Metrics](../reference/export-metrics.mdx) documentation. To enable
the collector for the `profile_qdisc_latency` gadget with the metrics name
`qdisc-latency-metrics`, run the following command:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        WIP: Headless mode for kubectl gadget is under development
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        gadgetctl run ghcr.io/inspektor-gadget/gadget/profile_qdisc_latency:%IG_TAG% \
                    --annotate=qdisc_latency:metrics.collect=true \
                    --otel-metrics-name=qdisc:qdisc-latency-metrics \
                    --detach
        ```
    </TabItem>
</Tabs>

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        WIP: Headless mode for kubectl gadget is under development
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        Unless you configured the metrics listener to do differently, the
        metrics will be available at `http://localhost:2224/metrics` on the
        server side. For the `profile_qdisc_latency` gadget we ran above, you
        can find the metrics under the `qdisc-latency-metrics` scope:

        ```bash
        $ curl http://localhost:2224/metrics -s | grep qdisc-latency
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="1"} 0
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="2"} 0
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="4"} 0
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="8"} 0
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="16"} 0
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="32"} 0
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="64"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="128"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="256"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="512"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="1024"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="2048"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="4096"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="8192"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="16384"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="32768"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="65536"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="131072"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="262144"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="524288"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="1.048576e+06"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="2.097152e+06"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="4.194304e+06"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="8.388608e+06"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="1.6777216e+07"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="3.3554432e+07"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="6.7108864e+07"} 3
        latency_bucket{otel_scope_name="qdisc-latency-metrics",otel_scope_version="",le="+Inf"} 3
        latency_sum{otel_scope_name="qdisc-latency-metrics",otel_scope_version=""} 192
        latency_count{otel_scope_name="qdisc-latency-metrics",otel_scope_version=""} 3
        otel_scope_info{otel_scope_name="qdisc-latency-metrics",otel_scope_version=""} 1
        ```
    </TabItem>
</Tabs>

Finally, stop metrics collection:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        WIP: Headless mode for kubectl gadget is under development
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        $ gadgetctl list
        ID           NAME                                                      TAGS                                                      GADGET
        3e68634c4c28 amazing_payne                                                                                                       ghcr.io/inspektor-gadget/gadget/profile_qdisc_latency:latest
        ```

        ```bash
        $ gadgetctl delete 3e68634c4c28
        3e68634c4c28a981a60fe96a29d24a99
        ```
    </TabItem>
</Tabs>
