---
title: bpfstats
sidebar_position: 0
---

import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';

# bpfstats

The bpfstats Gadget provides CPU and memory usage for Gadgets and eBPF programs.

## Getting started

Running the gadget:

<Tabs groupId="env">
  <TabItem value="kubectl-gadget" label="kubectl gadget">
```bash
$ kubectl gadget run ghcr.io/inspektor-gadget/gadget/bpfstats:%IG_TAG% [flags]
```
  </TabItem>

  <TabItem value="ig-daemon" label="ig-daemon">
```bash
$ sudo gadgetctl run ghcr.io/inspektor-gadget/gadget/bpfstats:%IG_TAG% [flags]
```
  </TabItem>
</Tabs>

## Flags

### `--gadgets-only`

Show statistics for only gadget-related eBPF programs

Default value: "false"

## Guide

Start the server part of IG:

<Tabs groupId="env">
  <TabItem value="kubectl-gadget" label="kubectl gadget">
```bash
$ kubectl gadget deploy
```
  </TabItem>

  <TabItem value="ig-daemon" label="ig-daemon">
```bash
$ sudo ig daemon --tls-insecure
```
  </TabItem>
</Tabs>

Then, let's run the gadget:

<Tabs groupId="env">
  <TabItem value="kubectl-gadget" label="kubectl gadget">

```bash
$ kubectl gadget run bpfstats:%IG_TAG%
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPC…
minikube                                                                2 hid_tail_call                0            0  0.00   0.00      0:00.000         8512     1
minikube                                                               38 sched_process_e       89770034        15277  0.05   0.01      0:00.089      4391808     4
minikube                                                               39 task_newtask         172413186        91790  0.10   0.02      0:00.172      4391808     4
minikube                                                               40 sched_process_e      144593647        91635  0.08   0.02      0:00.144      4391808     4
minikube                                                               41 __x64_sys_renam        7185786         2173  0.01   0.00      0:00.007      1083840     4
```
  </TabItem>

  <TabItem value="ig-daemon" label="ig-daemon">
```bash
$ sudo gadgetctl run bpfstats:%IG_TAG%
GADGETID    GADGETNAME           GADGETIMAGE          PROGID PROGNAME                     RUNTIME        RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPCO…
                                                           2 hid_tail_call                      0               0  0.00   0.00      0:00.000         8512      1
                                                          38 sched_process_e              1212539             180  0.00   0.00      0:00.001      4391808      4
                                                         198                                    0               0  0.00   0.00      0:00.000        48672      1
                                                         736 ig_execve_e                  1103998             317  0.00   0.00      0:00.001       849792      2
                                                         737 ig_execve_x                   160239             317  0.00   0.00      0:00.000       837504      1
```
  </TabItem>
</Tabs>

You can run the gadget with `--help` to get the meaning of the different
columns:

- comms
  List of processes using the eBPF program
- gadgetID
  Unique ID assigned to each Gadget instance
- gadgetImage
  Name of the Gadget image (like trace_open, trace_exec, etc.)
- gadgetName
  Name of the Gadget instance
- mapCount
  Number of maps used by the eBPF program or Gadget
- mapMemory
  Memory used by maps in bytes
- pids
  List of PIDs using the eBPF program
- progID
  eBPF program ID assigned by the Linux kernel
- progName
  Name of the eBPF program
- progType
  Type of the eBPF program
- cpuTimeStr
  Total CPU time formatted as duration
- cpuUsage
  CPU usage percentage of the eBPF program
- cpuUsageRelative
  CPU usage relative to the number of available CPUs
- runcount
  Number of times the eBPF program or Gadget has run
- runtime
  Time that the eBPF program or Gadget has run in nanoseconds

By default, it will print statistics for all eBPF programs running on the system.
You can restrict the view to only Inspektor Gadget programs by using the
`--gadgets-only` flag. Let's create a couple of gadgets:

<Tabs groupId="env">
  <TabItem value="kubectl-gadget" label="kubectl gadget">
```bash
$ kubectl gadget run --name mytraceopen trace_open:%IG_TAG% --detach
$ kubectl gadget run --name mytraceexec trace_exec:%IG_TAG% --detach
```
  </TabItem>

  <TabItem value="ig-daemon" label="ig-daemon">
```bash
$ sudo gadgetctl run --name mytraceopen trace_open:%IG_TAG% --detach
$ sudo gadgetctl run --name mytraceexec trace_exec:%IG_TAG% --detach
```
  </TabItem>
</Tabs>

The bpfstats will print the statistics for these two gadgets:

<Tabs groupId="env">
  <TabItem value="kubectl-gadget" label="kubectl gadget">
```bash
$ kubectl gadget run bpfstats:%IG_TAG% --gadgets-only
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPC…
minikube           6bcfcca7a mytraceopen        trace_open:%IG_TAG%     0                        2537846        10798  0.12   0.03      0:02.537     1710240   6
minikube           040e3fc1d mytraceexec        trace_exec:%IG_TAG%     0                           8265           12  0.00   0.00      0:00.008     8742176   7
```
  </TabItem>

  <TabItem value="ig-daemon" label="ig-daemon">
```bash
$ sudo gadgetctl run bpfstats:%IG_TAG% --gadgets-only
GADGETID    GADGETNAME           GADGETIMAGE          PROGID PROGNAME                     RUNTIME        RUNCOUNT      %CPU  %RELCPU   TIME+       MAPMEMORY MAPCO…
a624bdbf27f mytraceexec          trace_exec:%IG_TAG%       0                                    0               0      0.00   0.00     0:00.000    8742176      7
bb95fa2c8ab mytraceopen          trace_open:%IG_TAG%       0                                95233             242      0.12   0.03     0:02.537    1710240      6
```
  </TabItem>
</Tabs>

In this mode, the gadget shows consolidated information for the running Gadgets.
Here, PROGID is 0 and PROGNAME is empty, as a Gadget can have multiple programs.
Let's view the per-program statistics by removing the `--gadgets-only` flag.

<Tabs groupId="env">
  <TabItem value="kubectl-gadget" label="kubectl gadget">
```bash
$ kubectl gadget run bpfstats:%IG_TAG%
NODENAME           GADGETID  GADGETNAME         GADGETIMAGE        PROGID PROGNAME               RUNTIME     RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPC…
minikube                                                                2 hid_tail_call                0            0  0.00   0.00      0:00.000         8512     1
minikube                                                               38 sched_process_e       89770034        15277  0.05   0.01      0:00.089      4391808     4
minikube                                                               39 task_newtask         172413186        91790  0.10   0.02      0:00.172      4391808     4
minikube                                                               40 sched_process_e      144593647        91635  0.08   0.02      0:00.144      4391808     4
minikube                                                               41 __x64_sys_renam        7185786         2173  0.01   0.00      0:00.007      1083840     4
minikube                                                               42 __x64_sys_renam       23368056         2184  0.02   0.00      0:00.023      6467272     7
minikube                                                               43 __x64_sys_renam        4649957         2069  0.01   0.00      0:00.004      1083840     4
minikube                                                               44 __x64_sys_renam       18957942         2072  0.02   0.00      0:00.018      6467272     7
minikube                                                               45 __x64_sys_renam              0            0  0.00   0.00      0:00.000      1083840     4
minikube                                                               46 __x64_sys_renam              0            0  0.00   0.00      0:00.000      6467272     7
minikube                                                               47 __x64_sys_unlin       37799602        14147  0.03   0.01      0:00.037      1083840     4
minikube                                                               48 __x64_sys_unlin      120719371        14370  0.07   0.02      0:00.120      6467272     7
minikube                                                               49 __x64_sys_unlin       17037065         9561  0.02   0.00      0:00.017      1083840     4
minikube                                                               50 __x64_sys_unlin       56861068         9561  0.04   0.01      0:00.056      6467272     7
minikube                                                             1244 ig_execve_e           15430709         3239  0.02   0.00      0:00.015       849792     2
minikube                                                             1245 ig_execve_x            1792662         3655  0.01   0.00      0:00.001       837504     1
minikube                                                             1246 ig_fa_pick_e           2899018         4581  0.01   0.00      0:00.002        21632     2
minikube                                                             1247 ig_fa_pick_x           4478788         4581  0.01   0.00      0:00.004      1194336     4
minikube                                                             1248 ig_sched_exec          2759885         2728  0.01   0.00      0:00.002       837504     1
minikube           6bcfcca7a mytraceopen        trace_open:%IG_TAG%  1259 ig_open_x                    0            0  0.00   0.00      0:00.000      1615904     4
minikube           6bcfcca7a mytraceopen        trace_open:%IG_TAG%  1260 ig_openat_x           14944051        83160  0.02   0.00      0:00.014      1615904     4
minikube           6bcfcca7a mytraceopen        trace_open:%IG_TAG%  1261 ig_open_e                    0            0  0.00   0.00      0:00.000      1098240     3
minikube           6bcfcca7a mytraceopen        trace_open:%IG_TAG%  1262 ig_openat_e           33097520        83005  0.03   0.01      0:00.033      1098240     3
minikube           040e3fc1d mytraceexec        trace_exec:%IG_TAG%  1263 ig_execve_x             200186          570  0.00   0.00      0:00.000      8647840     5
minikube           040e3fc1d mytraceexec        trace_exec:%IG_TAG%  1264 ig_execveat_x                0            0  0.00   0.00      0:00.000      8647840     5
minikube           040e3fc1d mytraceexec        trace_exec:%IG_TAG%  1265 security_bprm_c         225868          362  0.00   0.00      0:00.000      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:%IG_TAG%  1266 ig_execve_e             340822          518  0.00   0.00      0:00.000      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:%IG_TAG%  1267 ig_execveat_e            37108           52  0.00   0.00      0:00.000      8372040     4
minikube           040e3fc1d mytraceexec        trace_exec:%IG_TAG%  1268 ig_sched_exec           226692          362  0.00   0.00      0:00.000      8647840     1
```
  </TabItem>

  <TabItem value="ig-daemon" label="ig-daemon">
```bash
$ sudo gadgetctl run bpfstats
GADGETID    GADGETNAME           GADGETIMAGE          PROGID PROGNAME                     RUNTIME        RUNCOUNT  %CPU   %RELCPU   TIME+        MAPMEMORY MAPCO…
                                                           2 hid_tail_call                      0               0  0.00   0.00      0:00.000         8512      1
                                                          38 sched_process_e              1212539             180  0.00   0.00      0:00.001      4391808      4
                                                         198                                    0               0  0.00   0.00      0:00.000        48672      1
                                                         736 ig_execve_e                  1103998             317  0.00   0.00      0:00.001       849792      2
                                                         737 ig_execve_x                   160239             317  0.00   0.00      0:00.000       837504      1
                                                         738 ig_fa_pick_e                  373093             509  0.00   0.00      0:00.000        21632      2
                                                         739 ig_fa_pick_x                  378858             509  0.00   0.00      0:00.000      1194336      4
                                                         740 ig_sched_exec                 112317              67  0.00   0.00      0:00.000       837504      1
a624bdbf27f mytraceexec          trace_exec:%IG_TAG%     743 security_bprm_c                49435              67  0.00   0.00      0:00.000      8372040      4
a624bdbf27f mytraceexec          trace_exec:%IG_TAG%     744 ig_execve_e                   140296             317  0.00   0.00      0:00.000      8372040      4
a624bdbf27f mytraceexec          trace_exec:%IG_TAG%     745 ig_execveat_e                      0               0  0.00   0.00      0:00.000      8372040      4
a624bdbf27f mytraceexec          trace_exec:%IG_TAG%     746 ig_execve_x                    89667             317  0.00   0.00      0:00.000      8647840      5
a624bdbf27f mytraceexec          trace_exec:%IG_TAG%     747 ig_sched_exec                  76171              67  0.00   0.00      0:00.000      8647840      5
a624bdbf27f mytraceexec          trace_exec:%IG_TAG%     748 ig_execveat_x                      0               0  0.00   0.00      0:00.000      8647840      5
bb95fa2c8ab mytraceopen          trace_open:%IG_TAG%     749 ig_open_e                          0               0  0.00   0.00      0:00.000      1098240      3
bb95fa2c8ab mytraceopen          trace_open:%IG_TAG%     750 ig_openat_e                 21538317           48300  0.02   0.00      0:00.021      1098240      3
bb95fa2c8ab mytraceopen          trace_open:%IG_TAG%     751 ig_open_x                          0               0  0.00   0.00      0:00.000      1615904      4
bb95fa2c8ab mytraceopen          trace_open:%IG_TAG%     752 ig_openat_x                  9076187           48503  0.01   0.00      0:00.009      1615904      4
```
  </TabItem>
</Tabs>

Finally, clean the system:

<Tabs groupId="env">
  <TabItem value="kubectl-gadget" label="kubectl gadget">
```bash
$ kubectl gadget remove mytraceexec mytraceopen
```
  </TabItem>

  <TabItem value="ig-daemon" label="ig-daemon">
```bash
$ sudo gadgetctl remove mytraceexec mytraceopen
```
  </TabItem>
</Tabs>

### Exporting metrics

The `bpfstats` Gadget provides the following metrics for `mapCount`,
`mapMemory`, `runcount` and `runtime`. To enable the metrics listener, check the
[Exporting Metrics](../reference/export-metrics.mdx) documentation. To enable
the collector for this gadget, run the following command:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        WIP: Headless mode for kubectl gadget is under development
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        $ sudo gadgetctl run bpfstats --name mystats --otel-metrics-name=bpfstats:bpfstats --annotate=bpfstats:metrics.collect=true --detach
        INFO[0000] installed as "8423cd8e53339c8d4501ec7cdff436bc"
        ```
    </TabItem>
</Tabs>

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        WIP: Headless mode for kubectl gadget is under development
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        Unless you configured the metrics listener to do differently, the
        metrics will be available at `http://localhost:2224/metrics` on the
        server side. For the `bpfstats` gadget we ran above, you
        can find the metrics under the `bpfstats` scope:

        ```bash
        $ curl http://localhost:2224/metrics -s  | grep bpfstats
        mapCount{gadgetImage="trace_exec:latest",gadgetName="mytraceexec",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 7
        mapCount{gadgetImage="trace_open:latest",gadgetName="mytraceopen",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 6
        mapMemory{gadgetImage="trace_exec:latest",gadgetName="mytraceexec",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 1.83170336e+08
        mapMemory{gadgetImage="trace_open:latest",gadgetName="mytraceopen",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 1.71024e+06
        otel_scope_info{otel_scope_name="bpfstats",otel_scope_version=""} 1
        runcount_total{gadgetImage="trace_exec:latest",gadgetName="mytraceexec",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 636
        runcount_total{gadgetImage="trace_open:latest",gadgetName="mytraceopen",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 219513
        runtime_total{gadgetImage="trace_exec:latest",gadgetName="mytraceexec",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 445114
        runtime_total{gadgetImage="trace_open:latest",gadgetName="mytraceopen",otel_scope_name="bpfstats",otel_scope_version="",progName=""} 7.3383934e+07
        ```
    </TabItem>
</Tabs>

Finally, stop metrics collection:

<Tabs groupId="env">
    <TabItem value="kubectl-gadget" label="kubectl gadget">
        ```bash
        WIP: Headless mode for kubectl gadget is under development
        ```
    </TabItem>

    <TabItem value="ig" label="ig">
        ```bash
        $ sudo gadgetctl delete mystats
        8423cd8e53339c8d4501ec7cdff436bc
        ```
    </TabItem>
</Tabs>

### Limitations

#### Memory usage of maps

The shown value for MapMemory is read from `/proc/<pid>/fdinfo/<map_id>`. This
is the maximum size the map can have, but it doesn't necessarily reflect its
current memory allocation. Additionally, maps can be used by more than one
program and would account towards the MapMemory of all those programs.

Also note:
* `BPF_MAP_TYPE_PERF_EVENT_ARRAY`: value_size is not counting the ring buffers,
  but only their file descriptors (i.e. sizeof(int) = 4 bytes)
* `BPF_MAP_TYPE_{HASH,ARRAY}_OF_MAPS`: value_size is not counting the inner maps,
  but only their file descriptors (i.e. sizeof(int) = 4 bytes)
