This document describes the detailed usage of damo. This doesn‘t cover all details of damo but only major features. This document may not complete and up to date sometimes. Please don’t hesitate at asking questions and improvements of this document via GitHub issues or mails.
Prerequisites
Kernel
You should first ensure your system is running on a kernel built with at least CONFIG_DAMON, CONFIG_DAMON_VADDR, CONFIG_DAMON_PADDR, and CONFIG_DAMON_SYSFS. Depending on the kernel version, you may need to enable CONFIG_DAMON_DBGFS instead of CONFIG_DAMON_SYSFS.
Sysfs or Debugfs
Because damo is using the sysfs or debugfs interface of DAMON, you should ensure at least one of those is mounted. Note that DAMON debugfs interface is deprecated. Please use sysfs. If you depend on DAMON debugfs interface and cannot use sysfs interface, report your usecase to the community.
Perf
damo uses perf[1] for recording DAMON‘s access monitoring results. Please ensure your system is having it if you will need to do record full monitoring results of DAMON (damo supports recording partial snapshots of DAMON monitoring results). If you will not do the full-recording, you don’t need to install perf on your system, though.
[1] https://perf.wiki.kernel.org/index.php/Main_Page
Basic Concepts of DAMON
damo is a user space tool for DAMON. Hence, for advanced and optimized use of damo rather than simple “Getting Started” tutorial, you should first understand the concepts of DAMON. There are a number of links to resources including DAMON introduction talks and publications at the project site. The official design document is recommended among those, since we will try to keep it up to date always, and appropriate for DAMON users.
Specifically, if you want to use damo for profiling purpose, please ensure you understand what nr_accesses and age mean. ‘Region Based Sampling’ and ‘Age Tracking’ sections of the kernel documentation should be helpful for that.
You should also understand DAMOS if you want to use damo for simple access-aware system operations. ‘Operation Schemes’ section of the kernel documentation should be helpful for that.
Install
Using Packaging Systems
There are a number of packaging systems that support damo. For example, below can be used if you want to install damo via dnf or PyPi.
$ sudo dnf install damo # if you prefer dnf $ sudo pip3 install damo # if you prefer PyPi
Note that dnf and PyPi are just two of the options. Refer to below repology data to show the packaging status of damo on your favorite packaging system.
Using Source Code
Simply downloading the source code and using damo file at the root of the source tree is also a completely supported method for damo installation. It would be particularly preferred if you want to participate in damo development. In the case, you could add the path to the source tree to your $PATH for convenience.
Overview
damo provides a subcommands-based interface. You can show the list of the available commands and brief description of those via damo --help. The major commands can be categorized as below:
- For controlling DAMON (monitoring and monitoring-based system optimization)
start,tune, andstopare included
- For recording, snapshot, and visualization of useful data including DAMON‘s monitoring results, DAMON’s working status, and additional system information
recordandreportare included
- For more convenient use of
damoversionandargsare included
Every subcommand also provides --help option, which shows the basic usage of it. Below sections introduce more details about the major subcommands.
Note that some of the subcommands that not described in this document would be in experimental stage, or not assumed to be used in major use cases. Those could be deprecated and removed without any notice and grace periods. Refer to FEATURES_DEPRECATION_PROCESS.md for more details.
DAMON Control (Access Monitoring and Monitoring-based System Optimization)
The main purposes of damo is operating DAMON, as the name says (DAMO: Data Access Monitor Operator). In other words, damo is for helping control of DAMON and retrieval/interpretation of the results.
damo start
damo start starts DAMON as users request. Specifically, users can specify how and to what address spaces DAMON should do monitor accesses, and what access monitoring-based system optimizations to do. The request can be made via several command line options of the command.
Refer to files under scripts/ directory for example usages of this command for memory tiering and LRU lists sorting. For more details about the command line options, use damo help damon_param_options -h.
The command exits immediately after starting DAMON as requested. It exits with exit value 0 if it successfully started DAMON. Otherwise, the exit value will be non-zero.
Simple Target Argument
The command receives one positional argument called deducible target. It could be used for specifying monitoring target, or full DAMON parameters. The command will try to deduce the type of the argument value and use it.
With the argument, users can specify the monitoring target with 1) the command for execution of the monitoring target process, 2) pid of running target process, or 3) the special keyword, paddr, if you want to monitor the system's physical memory address space.
Below example shows a command target usage:
# damo start "sleep 5"
The command will execute sleep 5 by itself and start monitoring the data access patterns of the process.
Note that the command requires the root permission, and hence executes the monitoring target command as a root. This means that the user could execute arbitrary commands with root permission. Hence, sysadmins should allow only trusted users to use damo.
Below example shows a pid target usage:
# sleep 5 & # damo start $(pidof sleep)
Finally, below example shows the use of the special keyword, paddr:
# damo start paddr
In this case, the monitoring target regions defaults to the largest ‘System RAM’ region specified in /proc/iomem file. Note that the initial monitoring target region is maintained rather than dynamically updated like the virtual memory address spaces monitoring case.
Partial DAMON Parameters Update
damo sets DAMON parameters such as monitoring intervals with its default values. Users can continue using the default values but set specific parameters as they want, via command line options. For the list and brief explanation of the options for those options, use damo help damon_param_options monitoring.
To understand what each of the command line options really mean, you may need to read DAMON core concepts document.
For keyword parameters such as operations set name (--ops), those on DAMON design document is samely applied. For example the keywords for --ops can be found from the kernel doc session for operations set.
Users can create multiple monitoring targets, multiple DAMON contexts and multiple kdamonds. For that, users can specify related options multiple times to set the parameter values with non-default ones. Users should also set --nr_targets and --nr_ctxs when multiple contexts are used, for assigning monitoring targets to each context, and contexts to each kdamond.
To see what full DAMON parameters are created with given command line, users can use damo args damon --format report. For example:
$ sudo ./damo args damon --format report \
--ops paddr --regions 100-200 --damos_action migrate_cold 1 \
--ops paddr --regions 400-700 --damos_action migrate_hot 0 \
--nr_targets 1 1 --nr_schemes 1 1 --nr_ctxs 1 1
kdamond 0
context 0
ops: paddr
target 0
region [100, 200) (100 B)
intervals: sample 5 ms, aggr 100 ms, update 1 s
nr_regions: [10, 1,000]
scheme 0
action: migrate_cold to node 1 per aggr interval
target access pattern
sz: [0 B, max]
nr_accesses: [0 %, 18,446,744,073,709,551,615 %]
age: [0 ns, max]
quotas
0 ns / 0 B / 0 B per max
priority: sz 0 %, nr_accesses 0 %, age 0 %
watermarks
metric none, interval 0 ns
0 %, 0 %, 0 %
kdamond 1
context 0
ops: paddr
target 0
region [400, 700) (300 B)
intervals: sample 5 ms, aggr 100 ms, update 1 s
nr_regions: [10, 1,000]
scheme 0
action: migrate_hot to node 0 per aggr interval
target access pattern
sz: [0 B, max]
nr_accesses: [0 %, 18,446,744,073,709,551,615 %]
age: [0 ns, max]
quotas
0 ns / 0 B / 0 B per max
priority: sz 0 %, nr_accesses 0 %, age 0 %
watermarks
metric none, interval 0 ns
0 %, 0 %, 0 %
Partial DAMOS Parameters Update
Command line options having prefix of --damos_ are for DAMON-based operation schemes. Those options are allowed to be specified multiple times for requesting multiple schemes. When user creates schemes with multiple DAMON contexts, --nr_schemes should also be provided for assigning the schemes to each context. For example, below shows how you can start DAMON with two DAMOS schemes, one for proactive LRU-prioritization of hot pages and the other one for proactive LRU-deprioritization of cold pages.
# damo start \
--damos_action lru_prio --damos_access_rate 50% max --damos_age 5s max \
--damos_action lru_deprio --damos_access_rate 0% 0% --damos_age 5s max
This command will ask DAMON to find memory regions that showing >=50% access rate for >=5 seconds and prioritize the pages of the regions on the Linux kernel's LRU lists, while finding memory regions that not accessed for >=5 seconds and deprioritizes the pages of the regions from the LRU lists.
For the list and brief explanations of the command line options, damo help damon_param_options damos can help. For options that need more explanations of usage, please read below additional section.
For keyword parameters such as DAMOS action or DAMOS filter types, those on DAMOS design document are samely applied. For example, those for --damos_action can be found from kernel doc session for DAMOS action.
Access rate
Access rate is the ratio of nr_accesses of the region to maximum nr_accesses that possible on given DAMON parameters. For example, if the sampling interval is 5 milliseconds and the aggregation interval is 100 milliseconds, the “maximum nr_accesses” is 20 (100 milliseconds divided by 5 milliseconds). And if a region has nr_accesses value 4, it's access rate is 20% (4 / 20 * 100).
Access hz
Access hz is the ratio of nr_accesses of the region to the aggregation interval, simply speaking, Hertz. That is, if a region's “access hz” is 100, it means the region is accessed 100 times per second.
--damos_filter Option Format
--damos_filter option's format is as below:
<allow|reject> [none] <type> [<additional type options>...] [<damos filter>....]
The first argument (allow or reject) specifies if the filter should allow or reject the memory. If it is not given, it applies reject by default. Note that kernel support of allow behavior is not yet mainlined as of 2025-01-19. It is expected to be supported from Linux v6.14.
<type> is the type of the memory that the filter should work for. Depending on the <type>, <additional type options> need to be given. For example, if <type> is memcg, the memory cgroup's mount point should be passed as <additional type options>. Supported types and required additional options are as below.
- anon: no additional options are required.
- memcg: the path to the memory cgroup should be provided.
- young: no additional options are required.
- hugeapge_size: minimum and maximum size of hugepages should be provided.
- addr: start and end addresses of the address range should be provided.
- target: the DAMON target index should be provided.
If the filter is for memory exclude the given type, none keyword can be given before the <type> part. For example,
reject young: Reject applying the DAMOS action to young pages. In other words, apply the action to non-young pages only.reject none young: Reject applying the DAMOS action to none-young pages. In other words, apply the action to young pages only.reject none addr 1234 56678: Reject applying the DAMOS action to address ranges except 1234-5678. In other words, apply the action to only 1234-5678 address range.
To use multiple filters, users can put the options in single --[snapshot_]damos_filter option, or do that with another --[snapshot_]damos_filter flag. For example,
--damos_filter allow anon reject memcg foo
and
--damos_filter allow anon --damos_filter reject memcg foo
Will install two DAMOS filters in same way.
Read DAMON design documentation for more details including how filters work.
Old --damos_filter Format
Before damo v2.6.4, only below old version of the format was supported. The format is still supported, but might be deprecated in future.
<type> <matching|nomatching> [additional type options>...] [allow|reject]
Meaning of <type>, <additional type options> are same to that of the above format.
Second argument says if the filter is for memory that matches (matching) or not matches (nomatching) the <type>. Same to not giving none keyword or not to the above format.
Finally, users can specify if the filter should allow of reject the memory. Same to what allow or reject on the above format means. If it is not given, it applies reject by default.
Unlike the new format, this format doesn't support specifying multiple filter options with single --[snapshot]_damos_filter flag. For multiple filters, multiple --[snapshot]_damos_filter flags should be provided.
Full DAMON Parameters Update
As mentioned above, the partial DAMON parameters update command line options support only single kdamond and single DAMON context. That should be enough for many use cases, but for system-wide dynamic DAMON usages, that could be restrictive. Also, specifying each parameter that different from their default values could be not convenient. Users may want to specify full parameters at once in such cases. For such users, the command supports --kdamonds option. It receives a specification of kdamonds that would contains all DAMON parameters in json or yaml format. Either a string of the format, or a path to a file containing the string can be passed to the option. Then, damo starts DAMON with the specification.
For the full DAMON parameters input format, please refer to damo args damon documentation below, or simply try the command. The --kdamonds option keyword can also simply omitted because the full DAMON parameters input can be used as is for the deducible target (refer to “Simple Target Argument” section above).
Full DAMOS Parameters Update
The Partial DAMOS parameters update options support multiple schemes as abovely mentioned. However, it could be still too manual in some cases and users may want to provide all inputs at once. For such cases, --schemes option receives a json-format specification of DAMOS schemes. The format is same to schemes part of the --kdamonds input.
You could get some example json format input for --schemes option from any .json files in damon-tests repo.
damo tune
damo tune applies new DAMON parameters while DAMON is running. It provides the set of command line options that same to that of damo start. Note that users should provide the full request specification to this command. If only a partial parameters are specified via the command line options of this command, unspecified parameters of running DAMON will be updated to their default values.
The command exits immediately after updating DAMON parameters as requested. It exits with exit value 0 if the update succeeded. Otherwise, the exit value will be non-zero.
damo stop
damo stop stops the running DAMON.
The command exits immediately after stopping DAMON. It exits with exit value 0 if it successfully terminated DAMON. Otherwise, the exit value will be non-zero.
For Recording, Snapshot, and Visualization of Data
There are two major DAMON usages. Automated access-aware system operation, and profiling.
Automated access-aware system operation specifies what kind of access-aware system operation the users want to do, and ask DAMON to do that on its own. That is, using DAMON-based Operation Schemes, aka DAMOS. damo users can do this via damo start or damo tune with DAMOS parameter options.
The second major usage of DAMON is profiling. In this usage, users retrieve DAMON‘s access monitoring results, visualize it, and better understand the behavior of the system and workloads. With the understanding, users could further make profiling-guided optimizations. Utilizing not only DAMON’s monitoring results but also some additional information can make it more powerful. Status of DAMON, memory footages, CPU usage, and hot code paths of system and processes could be such example. damo provides two commands for retrieving such data as snapshots and save as record files, and visualizing those.
damo record
damo record records data including DAMON‘s monitoring results, DAMON’s status, and additional system/workloads information as snapshots, and save those in files. Users can set the prefix of the resulting files' paths using --out option. It is ./damon.data by default. The command requires root permission. The output files will also be owned by root and have 600 permission by default, so only root can read those. Users can change the permission via --output_permission option.
If the command runs long time, resulting output files could be too huge to process in a practically short time. To avoid the problems from such too huge output files, damo record flushes the outputs per user-specified time into intermediate output files. The time can be specified in seconds, using --output_flush_sec option. The default value is 3600 seconds (one hour). The intermediate output files are complete ones, so can be processed like normal output files. The autoamtically split outputs are stored in directories named as <given_out_file>.<timestamp>. <given_out_file> is same to the --out option input, which is damon.data by default. <timestamp> is the time of the intermediate record creation, in %Y-%m-%d-%H-%M-%S format.
For the DAMON‘s monitoring results, it retrieves and saves every DAMON-generated monitoring result snapshots. Because DAMON’s monitoring result snapshot contains age information, the full record is not always required. Users can retrieve and save only specific number of snapshots with a specific time delay between snapshots, using --snapshot option.
damo record records monitoring results and status of running DAMON by default. If no DAMON is running, users can start DAMON first using damo start, and then run damo record. For example, below will start DAMON for physical address space monitoring, record the monitoring results, and save the records in damon.data file.
# damo start # damo record
To make the commands shorter, users can also ask damo record to start DAMON by themselves, together with the monitoring target command if needed, and then start the recording of the newly-started DAMON. For this use case, damo record receives command line options that same to those for damo start. For example, below command can be used.
# damo record "sleep 5"
or, for already running process, like below:
# damo record $(pidof my_workload)
Recording Profile Information
Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future.
damo record commands records record profiling information of the system together with the access pattern. Internally, it runs perf record while damo record is running, and store the perf output as a file of name same to the access pattern record file (specified by --out option of damo record) except having .profile suffix. Hence, damon.data.profile is the default name of the profile information.
Because the profile information record file is simply perf record output, users can further analyze the profile information using perf or any perf record output compatible tools.
Recording Memory Footprints
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
damo record command records memory usage information of the record target processes and the system together with the access pattern. Internally, it parses /proc/meminfo and /proc/<pid>/statm files for the monitoring target processes, and save the results as a json file of the name same to the access pattern record file (specified by --out option of damo record) except having .mem_footprint suffix. Hence, damon.data.mem_footprint is the default name of the profile information.
Users could use the files for vaious purpose. For an example, users could find when how much memory is allocated by the process and really accessed, by comparing the recorded residential set size and DAMON-based working set size. damo report footprints and damo report wss could be used for the purpose.
Recording Memory Mappings
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
damo record command records virtual memory mapping information of the record target processes. Internally, it parses /proc/<pid>/maps files of the monitoring target processes, and save the results as a json file of the name same to the access pattern record file (specified by --out option of damo record) except having .vmas suffix. Hence, damon.data.vmas is the default name of the memory mapping information.
Recording CPU Usages
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
damo record command records CPU usages of monitoring target processes and kdamonds by default. The record data is saved as a json file of the name same to the access pattern record file (specified by --out option of damo record) except having .proc_stats suffix. Hence, damon.data.proc_stats is the default name of the CPU usage information.
damo report
damo report visualizes the damo record-retrieved data in specific report format. Users can specify damo record-generated record files as the source of the data to visualize. For some report formats, users can also ask damo report to retrieve the data on its own, and directly do visualization of the self-retrieved data. Input data, command line options, and usages are different for specific report formats. Following sections describe some of the report formats.
damo report damon
damo report damon shows the current or recorded status of DAMON. It gets snapshot of the current status of DAMON and shows it by default. Users can use --input_file option to show the damo record-recorded DAMON status file or damo args damon-generated DAMON parameters as the source.
It shows every kdamond with the parameters that applied to it, running status (on or off), and DAMOS schemes status including their statistics and detailed applied regions information. It supports more command line options for retrieving status of specific parts.
damo report access
damo report access visualizes DAMON's access monitoring result snapshots in customizable formats. From where the data to visualize is retrieved is decided based on the given command line options and system status.
- If
--input_fileoption is given withdamo record-generated monitoring results record, the record is used. - If no data source specification command line option is given, the source of data is decided based on the system status.
- If any
damo start-started DAMON instance is running, its current monitoring results snapshot is used. - Else if
./damon.datafile exists, use./damon.dataas--input_file. - Else if
DAMON_STATis running, itsmemory_idle_ms_percentilessnapshot is used. Note thatmemory_idle_ms_percentilesprovides only a subset of the access pattern, and hence re-constructed access pattern information lacks many details and accuracy.
- If any
Users can set it to use damo record-generated monitoring results record as the source using --input_file option. If --input_file is not provided and DAMON is running, it captures snapshot from the running DAMON and uses it as the source. If --input_file is not provided, DAMON is not running, but ./damon.data file does exist, use ./damon.data as --input_file.
For example:
# damo start # damo report access heatmap: 77788899998666666654444443333333322222221111111100000000000000000000000000000000 # min/max temperatures: -2,400,000,000, 1,500, column size: 766.312 MiB intervals: sample 5 ms aggr 100 ms (max access hz 200) 0 addr 4.000 GiB size 2.326 GiB access 0 hz age 2.800 s 1 addr 6.326 GiB size 8.000 KiB access 200 hz age 2.800 s 2 addr 6.326 GiB size 12.000 KiB access 10 hz age 800 ms [...] 71 addr 51.462 GiB size 5.534 GiB access 0 hz age 23.700 s 72 addr 56.996 GiB size 5.908 GiB access 0 hz age 23.900 s 73 addr 62.903 GiB size 988.039 MiB access 0 hz age 24 s memory bw estimate: 121.143 GiB per second total size: 59.868 GiB record DAMON intervals: sample 5 ms, aggr 100 ms
The first line of the output shows the hotness (temperature) of regions on the address range as a heatmap visualization. The location and value of each column on the line represents the relative location and the access temperature of each memory region on the monitoring target address space. The second line shows scales of the temperature number and size of the heatmap. The third line shows the sampling and aggregation intervals that used for this snapshot. DAMON checks if each page is accessed every “sampling interval”, count the times it shown access for each page, and resets the counter every “aggregation interval”. Hence the observable maximum access frequency in hz depends on the intervals, and the maximum hz is shown on the same line.
Lines showing more detailed properties of each region follow. The detailed properties include the start address, (addr), size (size), access frequency in hz (access), and the age of the region. “Access frequency” represents how many times each page in the region is accessed per second. “Age” means how long the access frequency to the region was kept. For example the sixth line (starts with “2”) says it found 12 KiB memory region that starts from ~6.326 GiB address on the address space. The region was accessed about 10 times per second, and the frequency was kept for last 800 milliseconds.
Final three lines show the memory bandwidth usage that is estimated from the snapshot, and the total size of the regions that are listed on the output, respectively.
Access temperature
Access temperature is an abstract representing holistic access-hotness of a given region. It is calculated as a weighted sum of the access pattern values (size in bytes, access rate in percent, and age in microseconds) of each region. If access_rate is zero, the hotness becomes the weighted sum multiplies -1. By default, the weights for the three values are 0, 100, and 100, respectively. Users can set custom weights using --temperature_weights option.
Access report styles
damo report access provides --style option that allows users set the report style for commonly useful cases. The default style is ‘detailed’, which shown on example of the previous section.
recency-percentiles style shows the distribution of per-byte idle time in percentiles. For example,
$ sudo damo report access --style recency-percentiles # total recency percentiles <percentile> <idle time> 0 0 ns | | 1 0 ns | | 25 5 m 3.500 s |******** | 50 8 m 56.400 s |************** | 75 11 m 9.500 s |****************** | 99 11 m 56.300 s |********************| 100 11 m 56.300 s |********************| memory bw estimate: 5.886 GiB per second total size: 59.868 GiB
temperature-sz-hist style shows the distribution of per-byte access temperature in percentiles. For example,
$ sudo damo report access --style temperature-percentiles # total temperature percentiles <percentile> <temperature (weights: [0, 100, 100])> 0 -337,330,000,000 | | 1 -337,330,000,000 | | 25 -312,180,000,000 |* | 50 -209,510,000,000 |******* | 75 -85,890,000,000 |************** | 99 120,000,500 |********************| 100 120,000,500 |********************| memory bw estimate: 10.402 GiB per second total size: 59.868 GiB
recency-sz-hist style shows the distribution of per-byte idle time in histogram. For example,
$ sudo damo report access --style recency-sz-hist <idle time (us)> <total size> [0 ns, 1 m 11.500 s) 12.101 GiB |*************** | [1 m 11.500 s, 2 m 23.000 s) 0 B | | [2 m 23.000 s, 3 m 34.500 s) 4.435 GiB |****** | [3 m 34.500 s, 4 m 46.000 s) 0 B | | [4 m 46.000 s, 5 m 57.500 s) 5.441 GiB |******* | [5 m 57.500 s, 7 m 9.000 s) 5.816 GiB |******* | [7 m 9.000 s, 8 m 20.500 s) 0 B | | [8 m 20.500 s, 9 m 32.000 s) 5.666 GiB |******* | [9 m 32.000 s, 10 m 43.500 s) 9.545 GiB |************ | [10 m 43.500 s, 11 m 55.000 s) 16.864 GiB |********************| memory bw estimate: 9.180 MiB per second total size: 59.868 GiB
temperature-sz-hist style shows the distribution of per-byte access temperature in histogram. For example,
$ sudo damo report access --style temperature-sz-hist <temperature> <total size> [-71,450,000,000, -63,707,998,999) 18.284 GiB |********************| [-63,707,998,999, -55,965,997,998) 5.729 GiB |******* | [-55,965,997,998, -48,223,996,997) 5.650 GiB |******* | [-48,223,996,997, -40,481,995,996) 5.953 GiB |******* | [-40,481,995,996, -32,739,994,995) 5.732 GiB |******* | [-32,739,994,995, -24,997,993,994) 0 B | | [-24,997,993,994, -17,255,992,993) 5.954 GiB |******* | [-17,255,992,993, -9,513,991,992) 0 B | | [-9,513,991,992, -1,771,990,991) 12.567 GiB |************** | [-1,771,990,991, 5,970,010,010) 92.000 KiB |* | memory bw estimate: 10.625 MiB per second total size: 59.868 GiB
simple-boxes style is similar to detailed style, but provides a box visualization for access frequency and age of each region. It is useful for understanding overall access pattern of the system in a glance. For example,
$ sudo damo report access --style simple-boxes
|000000000000000000000000000000000000000| size 54.996 MiB access rate 0 % age 7.300 s
|0000000000000000000000000000000000000000| size 292.797 MiB access rate 0 % age 10.400 s
|000000000000000000000000000000000| size 54.062 MiB access rate 0 % age 700 ms
|00000000000000000000000000000000| size 31.820 MiB access rate 0 % age 500 ms
|99999999999999999999999999999999| size 9.402 MiB access rate 100 % age 500 ms
|00000000000000000000000000000000000| size 6.277 MiB access rate 0 % age 1.400 s
|000000000000000000000000000000000000000| size 116.000 KiB access rate 0 % age 9.900 s
|4| size 8.000 KiB access rate 55 % age 0 ns
|000000000000000000000000000000000000000| size 8.000 KiB access rate 0 % age 9.800 s
total size: 559.688 MiB
damo report access further provides flexible customization features. Actually --style option is also built on top of the customization features. Below sections provide more details about the background and usages of the features.
damo report access: Page Level Properties Based Access Monitoring
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
damo supports DAMON's page level properties based access monitoring feature, which has been merged into the mainline Linux kernel starting from v6.14.
damo report access shows the per-region size of the pages of specific properties by default if following two conditions met.
First, the access pattern to visulize should contain the information. Such collection can be made by taking access snapshot with page level properties-based DAMOS filters. For example, damo record or damo report for live snapshot use case with --snapshot_damos_filter options can generate such access pattern collection. The input format for --snapshot_damos_filter is same to that of --damos_filter.
Second, the visualization format supports the page level properties based information. detailed and histogram report styles will be automatically extended to show the information when it is included in the access pattern. To make custom format of the reports with the information, users can use format keywords such as <filters passed bytes> and <filters passed type> snapshot format keywords. Use damo help access_format_options {region,snapshot,record}_format_keywords for list and meaning of the keywords.
For example, users can show how much of anonymous and PG_young pages reside in regiosn of different access patterns, like below:
$ sudo damo start $ sudo damo report access --snapshot_damos_filter allow anon allow young 89888888885555555522222222111111100000000000000000000000000000000000000000000000 # min/max temperatures: -9,460,000,000, -368,113,080, column size: 766.312 MiB intervals: sample 5 ms aggr 100 ms (max access hz 200) # damos filters (df): allow anon, allow young df-pass: 777777777987777777777777777777777777777777777777777775444444444444446[...]400000000000 # min/max temperatures: -9,460,000,000, 870,000,500, column size: 143.883 MiB 0 addr 4.000 GiB size 25.285 MiB access 0 hz age 8.900 s df-passed 16.000 KiB 1 addr 4.025 GiB size 227.574 MiB access 0 hz age 8.900 s df-passed 7.320 MiB 2 addr 4.247 GiB size 1011.438 MiB access 0 hz age 8.900 s df-passed 172.566 MiB 3 addr 5.235 GiB size 8.000 KiB access 200 hz age 3.900 s df-passed 8.000 KiB 4 addr 5.235 GiB size 7.387 MiB access 0 hz age 8.700 s df-passed 1.336 MiB 5 addr 5.242 GiB size 17.242 MiB access 0 hz age 8.700 s df-passed 3.645 MiB [...] 26 addr 29.101 GiB size 5.620 GiB access 0 hz age 1 m 31.400 s df-passed 0 B 27 addr 34.721 GiB size 5.677 GiB access 0 hz age 1 m 33.600 s df-passed 0 B 28 addr 40.398 GiB size 5.934 GiB access 0 hz age 1 m 34.300 s df-passed 0 B 29 addr 46.332 GiB size 15.140 GiB access 0 hz age 1 m 34.600 s df-passed 0 B 30 addr 61.472 GiB size 1.615 GiB access 0 hz age 1 m 34.600 s df-passed 476.000 KiB 31 addr 63.087 GiB size 799.883 MiB access 0 hz age 1 m 34.600 s df-passed 0 B memory bw estimate: 2.171 GiB per second df-passed: 377.578 MiB per second total size: 59.868 GiB df-passed 2.190 GiB record DAMON intervals: sample 5 ms, aggr 100 ms
The line starting with df-pass: shows the access snapshot heatmap for regions that having the filters passed. Regions that not having the filters passed are represented as a gap ([...]). Note that the heatmap is in experimental support now.
damo report access: Programming Visualization
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
damo report access provides a highly customizable visualization formats. It cannot fits all, though. Users can implement and use their own visualization of the given access monitoring results in Python code using --exec option of damo report access be below two ways.
Users can implement the visualization in a Python script and pass the path to the script with optional arguments via --exec. The --exec parameter value can also have optional inputs for the script. The given Python script should have a function name main(), which receives two parameters. damo report access will call the main() function. The invocation will pass monitoring results record information to the first parameter. The tokens of --exec parameter value will be passed as the second parameter. For example:
$ cat foo.py
def main(records, cmd_fields):
print('hello')
print(records)
print(cmd_fields)
for r in records:
for s in r.snapshots:
for r in s.regions:
print(r.start, r.end)
$ sudo ./damo report access --exec "./foo.py a b c"
hello
[<_damo_records.DamonRecord object at 0x7f24e3678190>]
['./foo.py', 'a', 'b', 'c']
4294967296 10660085760
10660085760 17044361216
17044361216 68577918976
Users can also start Python interpreter with the access information records by passing interpreter as --exec value. For example:
$ sudo ./damo report access --exec interpreter DAMON records are available as 'records' >>> print(records) [<_damo_records.DamonRecord object at 0x7fe468d13d90>] >>> help(records[0]) Help on DamonRecord in module _damo_records object: class DamonRecord(builtins.object) | DamonRecord(kd_idx, ctx_idx, intervals, scheme_idx, target_id, scheme_filters) | | Contains data access monitoring results for single target | | Methods defined here: | | __init__(self, kd_idx, ctx_idx, intervals, scheme_idx, target_id, scheme_filters) | Initialize self. See help(type(self)) for accurate signature. [...]
records is a list of DamonRecord class that defined on src/_damo_records.py of damo repo. The methods and fields could be changed in future, so no strict script backward compatibility is guaranteed. Instead, we provide a compatibility strategy that is similar to that of Linux kernel's in-tree modules only API compatibility guarantee. We will keep some --exec scripts on report_access_exec_scripts directory of damo repo, and do our best to keep those not broken. Hence, if you need backward compatibility of your --exec script, please consider upstreaming it into damo repo.
DAMON Monitoring Results Structure
The monitoring results are constructed with three data structures called ‘record’, ‘snapshot’, and ‘region’.
‘Record’ contains monitoring results for each kdamond/context/target combination.
Each ‘record’ contains multiple ‘snapshots’ of the monitoring results. If it was made with --snapshot option, the snapshots are retrieved for each --snapshot-specified snapshot dealy time. If it was made as a result of full recording (no --snapshot given), the snapshots are retrieved for each aggregation interval. For example, if damo report access was used for live snapshot visualization, each record will contain only one single snapshot.
Each ‘snapshot’ contains multiple ‘regions’ information. Each region information contains the monitoring results for the region including the start and end addresses of the memory region, nr_accesses, and age. The number of regions per snapshot depends on the min_nr_regions and max_nr_regions DAMON parameters, and actual data access pattern of the monitoring target address space.
damo's way of showing DAMON Monitoring Results
damo report access shows the information in an enclosed hierarchical way like below:
<record 0 head>
<snapshot 0 head>
<region 0 information>
[...]
<snapshot 0 tail>
[...]
<record 0 tail>
[...]
That is, information of record and snapshot can be shown twice, once at the beginning (before showing it‘s internal data), and once at the end. Meanwhile, the information of regions can be shown only once since it is the lowest level that not encloses anything. By default, record and snapshot head/tail are skipped if there is only one record that has only one snapshot. That’s why above damo report access example output shows only regions information.
Customization of The Output
Users can customize what information to be shown in which way for the each position using --format option. The option receives fine-grained format command for each position of the hierarchical output structure (head and tail of record and snapshot, and region). The fine-grained per-output position format command is constructed in below format.
<action> <position> <format string>
<position> specifies the output position on the hierarchical structure. It can be one of record_head, snapshot_head, region, snapshot_tail, and record_tail.
<action> can be set or append. set is for setting the format of the <position> output. append is for append a line of specific format to the current output of the <position>.
Finally, <format string> is the template string for the output. The template can have any words and special format keywords for each position. For example, <start address>, <end address>, <access rate>, or <age> keywords are available for region position. The template can also have arbitrary strings. The newline character (\n) is also supported. Each of the keywords for each position and their brief description can be shown via damo help access_format_options {record,snapshot,region}_format_keywords command.
For example:
# damo start
# damo report access \
--format set region "region that starts from <start address> and ends at <end address> was having <access rate> access rate for <age>." \
--format set snapshot_tail "total size: <total bytes>"
region that starts from 4.000 GiB and ends at 16.251 GiB was having 0 % access rate for 40.700 s.
region that starts from 16.251 GiB and ends at 126.938 GiB was having 0 % access rate for 47.300 s.
total size: 122.938 GiB
Region Visualization via Boxes
For region information customization, a special keyword called <box> is provided. It represents each region‘s access pattern with its shape and color. By default it represents each region’s relative age, access rate, and size with its length, color, and height, respectively. For example, damo report access --format set region "<box>" shows visualization of the access pattern, by showing location of each region in Y-axis, the hotness with color of each box, and how long the hotness has continued in X-axis. Showing only the first column of the output would be somewhat similar to an access heatmap of the target address space.
For convenient use of it with a default format, damo report access provides --region_box option. Output of the command with the option would help users better to understand.
Users can further customize the box using damo report access options that having --region_box_ prefix. For example, users can set what access information to be represented by the length, color, and height, and whether the values should be represented in logscale or linearscale.
Sorting and Filtering Regions Based on Access Pattern
By default, damo report access shows all regions that sorted by their start address. Different users would have different interest to regions having specific access pattern. Someone would be interested in hot and small regions, while some others are interested in cold and big regions.
For such cases, users can make it to sort regions with specific access pattern values as keys including access_rate, age, and size via --sort_regions_by option. --sort_regions_dsc option can be used to do desscending order sorting.
Further, users can make damo report access to show only regions of specific access pattern and address ranges using options including --sz_region, --access_rate, --age, and --address. Note that the filtering could reduce DAMON‘s overhead, and therefore recommended to be used if you don’t need full results and your system is sensitive to any resource waste.
Sorting Regions Based on Hotness
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
Users can sort the regions based on hotness of the regions by providing access temperature as the sort key (--sort_regions_by).
For example:
snapshot time: [0 ns, 100 ms] (100 ms)
|0000000000000000000000000000000000000000| size 3.022 GiB access 0 hz age 11 m 31.300 s
|000000000000000000000000000000000000000| size 5.899 GiB access 0 hz age 11 m 28.300 s
|000000000000000000000000000000000000000| size 5.319 GiB access 0 hz age 11 m 19.900 s
|000000000000000000000000000000000000000| size 5.557 GiB access 0 hz age 11 m 4.900 s
|000000000000000000000000000000000000000| size 4.509 GiB access 0 hz age 10 m 35.400 s
|000000000000000000000000000000000000000| size 5.563 GiB access 0 hz age 9 m 33.200 s
|00000000000000000000000000000000000000| size 5.512 GiB access 0 hz age 7 m 52.100 s
|000000000000000000000000000000000000| size 5.699 GiB access 0 hz age 5 m 57.500 s
|000000000000000000000000000000000| size 5.854 GiB access 0 hz age 2 m 53.900 s
|00000000000000000000000000000| size 4.306 GiB access 0 hz age 1 m 13.100 s
|00000000000000000| size 5.949 GiB access 0 hz age 6.200 s
|00000000000| size 2.074 MiB access 0 hz age 1.800 s
|00000000| size 1.242 GiB access 0 hz age 900 ms
|3| size 8.000 KiB access 80 hz age 200 ms
|00| size 4.000 KiB access 20 hz age 300 ms
|99| size 4.000 KiB access 200 hz age 300 ms
|9999999999| size 16.000 KiB access 200 hz age 1.600 s
|00000000000000000000000000000| size 1.435 GiB access 10 hz age 1 m 13.100 s
memory bw estimate: 14.358 GiB per second
total size: 59.868 GiB
record DAMON intervals: sample 5 ms, aggr 100 ms
damo report access --raw_form
--raw_form option of damo report access directly transforms the recorded access monitoring results into a human-readable text. For example:
$ damo report access --raw_form --input damon.data base_time_absolute: 8 m 59.809 s monitoring_start: 0 ns monitoring_end: 104.599 ms monitoring_duration: 104.599 ms target_id: 18446623438842320000 nr_regions: 3 563ebaa00000-563ebc99e000( 31.617 MiB): 1 7f938d7e1000-7f938ddfc000( 6.105 MiB): 0 7fff66b0a000-7fff66bb2000( 672.000 KiB): 0 monitoring_start: 104.599 ms monitoring_end: 208.590 ms monitoring_duration: 103.991 ms target_id: 18446623438842320000 nr_regions: 4 563ebaa00000-563ebc99e000( 31.617 MiB): 1 7f938d7e1000-7f938d9b5000( 1.828 MiB): 0 7f938d9b5000-7f938ddfc000( 4.277 MiB): 0 7fff66b0a000-7fff66bb2000( 672.000 KiB): 5
The first line shows the recording started timestamp (base_time_absolute). Records of data access patterns follow. Each record is separated by a blank line. Each record first specifies when the record started (monitoring_start) and ended (monitoring_end) relative to the start time, the duration for the recording (monitoring_duration). Recorded data access patterns of each target follow. Each data access pattern for each task shows the target's id (target_id) and a number of monitored address regions in this access pattern (nr_regions) first. After that, each line shows the start/end address, size, and the number of observed accesses of each region.
damo report heatmap
damo report heatmap plots damo record-generated monitoring results in 3-dimensional form, which represents the time in x-axis, address of regions in y-axis, and the access frequency in z-axis. Users can optionally set the resolution of the map (--resol) and start/end point of each axis (--time_range and --address_range). For example:
$ sudo ./damo report heatmap --resol 15 80 11111111111111111111111111111111111111111111111111111111111111111111112211110000 11111111111111111111111111111111111111111111111111111111111111111111111111100000 00000000000000000000000000000000000000000000000000000000000000000001455555530000 00000000000000000000000000000000000000000000000000000000000013333333344444430000 00000000000000000000000000000000000000000000000000000111111127777777000000000000 00000000000000000000000000000000000000000000000000000688888830000000000000000000 00000000000000000000000000000000000000000000037777775000000000000000000000000000 00000000000000000000000000000000000000555555522222222000000000000000000000000000 00000000000000000000000000000003333332555555510000000000000000000000000000000000 00000000000000000000000001111147777774000000000000000000000000000000000000000000 00000000000000000000001888888800000000000000000000000000000000000000000000000000 00000000000000067777773000000000000000000000000000000000000000000000000000000000 00000002555555423333331000000000000000000000000000000000000000000000000000000000 33333332555555400000000000000000000000000000000000000000000000000000000000000000 77777771000000000000000000000000000000000000000000000000000000000000000000000000 # access_frequency: 0123456789 # x-axis: space (139905344733184-139905455165424: 105.316 MiB) # y-axis: time (3347892748000-3407716412995: 59.824 s) # resolution: 80x15 (1.316 MiB and 3.988 s for each character)
As the above example shows, it plots the heatmap on the terminal by default. Users can ask the heatmap to be made as image files using --output option. Currently pdf, png, jpeg, and svg file formats are supported. For example, below command will create ‘heatmap.png’ file that contains the visualized heatmap.
$ sudo ./damo report heatmap --output heatmap.png
Heatmap Scoping
If the recorded time and space are huge, users would want to do proper zoom in/out. Particularly if the target address space is a virtual memory address space and the user plots the entire address space, the huge unmapped regions will make the picture looks only black. For this, command line options for setting the resolution (--resol) and x/y axis boundaries (--time_range and --address_range) are provided.
To make this effort minimal, --guide_human option can be used as below:
# ./damo report heatmap --guide_human target_id: 18446623438842320000 time: [8 m 59.809 s, 9 m 56.607 s) (56.797 s) region 0: [86.245 TiB, 86.245 TiB) (31.617 MiB) region 1: [127.576 TiB, 127.576 TiB) (196.848 MiB) region 2: [127.998 TiB, 127.998 TiB) (672.000 KiB)
The output shows unions of monitored regions (start and end addresses in byte) and the union of monitored time duration (start and end time in nanoseconds) of each target task. Therefore, it would be wise to plot the data points in each union. If no axis boundary option is given, it will automatically find the biggest union in --guide_human output and set the boundary in it.
Users can also interactively find the right scope using --interactive_edit option.
For a case that the user still unsure which range to draw heatmap for, --draw_range option can be used. The option receives either all or hottest. If all is passed, damo report heatmap draws heatmaps for all the three regions. If file output is requested, the output for first region will have the user-specified file name. For those of second and third regions, .1 and .2 will be added to the file name, before the file format extension part (e.g., heatmap.1.png). If hottest is passed, damo report heatmap will draw the heatmap for hottest region among the three regions.
Raw Heatmap Data for Custom Plots
In some cases, users may want to have only the raw data points of the heatmap so that they can do their own heatmap visualization. For such use case, a special keyword, raw can be given to --output option, like below.
# damo report heatmap --output raw --resol 3 3 0 0 0.0 0 7609002 0.0 0 15218004 0.0 66112620851 0 0.0 66112620851 7609002 0.0 66112620851 15218004 0.0 132225241702 0 0.0 132225241702 7609002 0.0 132225241702 15218004 0.0
This command shows a recorded access pattern in a heatmap of 3x3 resolution. Therefore it shows 9 data points in total. Each line shows each of the data points. The three numbers in each line represent time in nanoseconds, address in bytes and the observed access frequency.
Users can convert this text output into a heatmap image (represents z-axis values with colors) or other 3D representations using various tools such as gnuplot.
damo report wss
The wss type extracts the distribution and chronological working set size changes from the record.
By default, the working set is defined as memory regions shown any access within each snapshot. Hence, for example, if a record is having N snapshots, the record is having N working set size values, and wss report type shows the distribution of the N values in size order, or chronological order.
For example:
$ ./damo report wss # <percentile> <wss> # target_id 18446623438842320000 # avr: 107.767 MiB 0 0 B | | 25 95.387 MiB |**************************** | 50 95.391 MiB |**************************** | 75 95.414 MiB |**************************** | 100 196.871 MiB |***********************************************************|
Without any option, it shows the distribution of the working set sizes as above. It shows 0th, 25th, 50th, 75th, and 100th percentile and the average of the measured working set sizes in the access pattern records. In this case, the working set size was 95.387 MiB for 25th to 75th percentile but 196.871 MiB in max and 107.767 MiB on average.
By setting the sort key of the percentile using --sortby, you can show how the working set size has chronologically changed. For example:
$ ./damo report wss --sortby time # <percentile> <wss> # target_id 18446623438842320000 # avr: 107.767 MiB 0 0 B | | 25 95.418 MiB |***************************** | 50 190.766 MiB |***********************************************************| 75 95.391 MiB |***************************** | 100 95.395 MiB |***************************** |
The average is still 107.767 MiB, of course. And, because the access was spiked in very short duration and this command plots only 4 data points, we cannot show when the access spikes made. Users can specify the resolution of the distribution (--range). By giving more fine resolution, the short duration spikes could be more easily found.
Similar to that of heats --heatmap, it also supports gnuplot based simple visualization of the distribution via --plot option.
damo report footprints
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
The footprints type extracts the distribution of the recorded in chronological or size order. By comparing the output against that of wss report type, users can know how much memory is being allocated for the workload and what amount of the allocated memory is really being accessed. The output format is similar to that of wss.
Because there are various memory footprint metrics, the command asks users to specify what memory footprint metric they want to visualize. Currently, below metrics are supported.
vsz: The amount of virtual memory that allocated to the workloads; a.k.a “virtual set size”.rss: The amount of physical memory that allocated to the workloads; a.k.a “residential set size”.sys_used: The amount of system's memory that allocated for any usage (MemTotal - MemFree).
For example:
$ ./damo report footprints vsz # <percentile> <footprint> # avr: 199.883 MiB 0 199.883 MiB |***********************************************************| 25 199.883 MiB |***********************************************************| 50 199.883 MiB |***********************************************************| 75 199.883 MiB |***********************************************************| 100 199.883 MiB |***********************************************************| $ $ ./damo report footprints rss # <percentile> <footprint> # avr: 196.168 MiB 0 196.168 MiB |***********************************************************| 25 196.168 MiB |***********************************************************| 50 196.168 MiB |***********************************************************| 75 196.168 MiB |***********************************************************| 100 196.168 MiB |***********************************************************|
damo report profile
Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future.
The profile type shows profiling report for specific access pattern.
It requires two files, namely an access pattern record file and a profiling information record file that recorded together with the access pattern record file. Those can be generated by damo record (Refer to ‘Recording Profile Information’ section for details). Users can further describe access pattern of their interest that they want to know what happens when the access pattern occurs. Then, damo report profile command read the access pattern, find times when the specific access pattern happened, collect profiling information for the time ranges, and generate the report with the filtered information.
For example, below shows what was consuming CPU while 50% or more access rate was made towards 50 MiB size address range starting from 139,798,348,038,144, and the total size of the memory regions that got the access in the address range was 40 or more MiB.
$ sudo ./damo report profile --access_rate 50% 100% \
--address 139798348038144 $((139798348038144 + 50 * 1024 * 1024)) \
--sz_snapshot 40MiB max
Samples: 69K of event 'cpu-clock:pppH', Event count (approx.): 17449500000
Overhead Command Shared Object Symbol
70.32% swapper [kernel.vmlinux] [k] pv_native_safe_halt
28.83% masim masim [.] do_seq_wo
0.03% masim [kernel.vmlinux] [k] _raw_spin_unlock_irqrestore
0.03% ps [kernel.vmlinux] [k] do_syscall_64
0.03% swapper [kernel.vmlinux] [k] __do_softirq
damo report times
Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future.
The times type shows time intervals in an access pattern record that showing specific access pattern. This can be useful when user runs damo together with other tools such as profilers.
For example, below shows when there was no access to 50 MiB size address range starting from 139,798,348,038,144.
$ sudo ./damo report times --access_rate 0% 0% \
--address 139798348038144 $((139798348038144 + 50 * 1024 * 1024))
93904.291408-93904.393156
93905.919058-93910.903176
93915.994039-93920.876248
93926.049032-93930.918094
93935.988105-93940.956402
93946.027539-93950.997432
93956.067597-93961.036500
93966.101779-93966.910657
damo report holistic
Note: This is an experimental feature at the moment. Many changes would be made, or the support can be dropped in future.
The holistic type shows holistic view of the recorded access pattern, memory footprints, and CPU consuming functions. As of v2.4.1, it shows access heatmap and distributions of working set size and memory footprints. It will further updated to provide more detailed information in a concise manner, and add the hot functions information.
For example:
$ sudo damo record "./masim ./configs/stairs.cfg"
[...]
$ sudo damo report holistic
# Heatmap
# target 0, address range 94737092173824-94737384947712
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000000000000000000000000000
# x-axis: space (94737092173824-94737384947664: 279.211 MiB)
# y-axis: time (12681562920000-12741660534000: 1 m 0.098 s)
# resolution: 80x5 (3.490 MiB and 12.020 s for each character)
# target 0, address range 140223579160576-140223690682368
33333222222222222222222222222222222222232222222222222222222222222223666666500000
00000000000000000000000000000000000000000000000000002666677567777773333333300000
00000000000000000000000000000022222223777778667777772000000000000000000000000000
00000000000000045555554777777545555551000000000000000000000000000000000000000000
77777885777777512222220000000000000000000000000000000000000000000000000000000000
# x-axis: space (140223579160576-140223690682336: 106.355 MiB)
# y-axis: time (12681562920000-12741660534000: 1 m 0.098 s)
# resolution: 80x5 (1.329 MiB and 12.020 s for each character)
# target 0, address range 140725996015616-140725996150784
00000000000000000000000000000000000000000000000000000000000000000000000000068888
00000000000000000000000000000000000000000000000000000000000000000000000000068888
00000000000000000000000000000000000000000000000000000000000000000000000000067777
00000000000000000000000000000000000000000000000000000000000000000000000000067777
00000000000000000000000000000000000000000000000000000000000000000000000000067777
# x-axis: space (140725996015616-140725996150736: 131.953 KiB)
# y-axis: time (12681562920000-12741660534000: 1 m 0.098 s)
# resolution: 80x5 (1.649 KiB and 12.020 s for each character)
# Memory Footprints Distribution
percentile 0 25 50 75 100
wss 0 B 9.547 MiB 9.566 MiB 9.828 MiB 106.363 MiB
rss 100.879 MiB 100.879 MiB 100.879 MiB 100.879 MiB 100.879 MiB
vsz 104.535 MiB 104.535 MiB 104.535 MiB 104.535 MiB 104.535 MiB
sys_used 2.159 GiB 2.234 GiB 2.241 GiB 2.245 GiB 2.253 GiB
# Hotspot functions
# Samples: 680K of event 'cpu-clock:pppH'
# Event count (approx.): 170020000000
#
# Overhead Command Shared Object Symbol
# ........ ............... ..................................... ...........................................
#
63.01% swapper [kernel.vmlinux] [k] pv_native_safe_halt
34.96% masim masim [.] do_seq_wo
0.08% python3 python3.11 [.] _PyEval_EvalFrameDefault
0.07% ps [kernel.vmlinux] [k] do_syscall_64
0.05% ps [kernel.vmlinux] [k] memset_orig
0.04% ps libc.so.6 [.] open64
0.04% perf perf [.] __symbols__insert
0.03% ps libc.so.6 [.] read
0.03% ps libc.so.6 [.] __close
0.03% ps [kernel.vmlinux] [k] __memcg_slab_post_alloc_hook
Miscellaneous Helper Commands
Abovely explained commands are all core functions of damo. For more convenient use of damo and debugging of DAMON or damo itself, damo supports more commands. This section explains some of those that could be useful for some cases.
damo help
Some commands including damo start and damo report access have long lists of command line options. Those makes flexible usage of damo. Because the long list makes the help message too verbose, many of the options that unessential for simple usages are hidden by default. damo help can be used for showing such hidden options.
damo args
Some commands including damo start and damo report access have long lists of available (and hidden) command line options. Those makes flexible usage of damo, but makes the usage bit complicated. For easy handling of such arguments, damo args receives such partial command line options, compile into full options, and print out in various formats including jsong and yaml. The formatted outputs allow users easily understand what full options will be generated. The json or yaml format can also be passed to relevant commands instead of the command line options. Hence, if a user prefer editing the options in json or yaml format using their personal favorite editor tools rather than damo's command line interface, they can use the ‘damo args’ outputs as a template.
damo args damon
As mentioned for damo start above, DAMON control commands including start, tune, and additionally record allows passing DAMON parameters or DAMOS specification all at once via json or yaml formats. That's for making specifying and managing complex requests easier, but writing the whole json or yaml manually could be annoying, while the partial DAMON/DAMOS parameters setup command line options are easy for simple use case. To help formatting the json or yaml input easier, damo args damon receives the partial DMAON/DAMOS parameters setup options and print out resulting json format Kdamond parameters. For example,
# damo args damon --damos_action stat --format json
prints json format DAMON parameters specification that will be result in a DAMON configuration that same to one that can be made with damo start --damos_action stat. In other words, damo start $(damo args damon --damos_action stat) will be same to damo start --damos_action stat.
The output can be saved in a file instead of printing using --out option. To read the output in more human-friendly way, damo report damon can be used.
report format
The command also supports report format, which is similar to the output of damo report damon. The format is for human users who feels json or even yaml format outputs being too verbose or difficult to read. It means it is not for machines, and therefore cannot feed to other damo commands.
Multiple kdamonds
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
damo args damon was supporting only single kdamond by default, and hence provided below ways to edit DAMON parameters for multiple kdamonds. From v2.7.5, partial DAMON parameters update command line options support multiple kdamonds use case via --nr_ctxs, --nr_schemes and --nr_targets. Read the section for more details. Below ways for handling multiple kdamonds are still be supported for users who still prefer it. If partial DAMON parameters command line options are proven to be useful and stable, below ways might be deprecated in future.
--add option of damo args damon receives a file containing output from other damo args damon execution. If the option is given, damo args damon reads the DAMON parameters from the file. Then, it further generates DAMON parameters with command line options other than --add that given to damo args damon, add the newly generated kdamond parameters to those from the file, and print the resulting parameters. Note that users can save the output to a file instead of printing, via --out option. For example, below commands will create kdamonds.json file that contains two kdamonds running migrate_hot and migrate_cold DAMOS actions respectively.
# damo args damon --damos_action migrate_hot 0 --out kdamonds.json
# damo args damon --damos_action migrate_cold 1 \
--add kdamonds.json --out kdamonds.json
--replace option of damo args damon receives a file containing output from other damo args damon execution, and the index of the kdamond in the file that the user wants to remove. If the option is given, damo args damon reads the DAMON parameters from the file, replace the kdamond parameters of the index with DAMON parameters that generated by damo args damon options except --replace, and prints resulting parameters. For example, below command will replace the kdamond for migrate_cold from the kdamonds.json file that generated above, with another kdamond for pageout instead, and save the resulting updated kdamond parameters to kdamonds.json.
# damo args damon --damos_action pageout --replace kdamonds.json 1 --out kdamonds.json
--remove option of damo args damon receives a file containing output from other damo args damon execution, and the index of the kdamond in the file that the user wants to remove. If the option is given, damo args damon reads the DAMON parameters from the file, remove the kdamond parameters of the index, and prints resulting parameters. For example, below command will remove the kdamond for pageout from the kdamonds.json file that updated above, and save the resulting single kdamond parameters to kdamonds.json.
# damo args damon --remove kdamonds.json 1 --out kdamonds.json
damo report damon with --input_file option can receive the damo args damon generated output and show the DAMON parameters in a format that more human friendly than json or yaml. --damon_params option can also be used to make the output focus on only parameters. For example, abovely generated parameters can be shown as below.
$ sudo ./damo report damon --damon_params --input_file kdamonds.json
kdamond 0
context 0
ops: paddr
target 0
region [4,294,967,296, 68,577,918,975) (59.868 GiB)
intervals: sample 5 ms, aggr 100 ms, update 1 s
nr_regions: [10, 1,000]
scheme 0
action: migrate_hot to node 0 per aggr interval
target_nid: 0
target access pattern
sz: [0 B, max]
nr_accesses: [0 %, 18,446,744,073,709,551,616 %]
age: [0 ns, max]
quotas
0 ns / 0 B / 0 B per max
priority: sz 0 %, nr_accesses 0 %, age 0 %
watermarks
metric none, interval 0 ns
0 %, 0 %, 0 %
damo dianose
Note: This is an experimental feature at the moment. Some changes could be made, or the support can be dropped in future.
damo diagnose prints the status of DAMON and any behaviors that suspicious to be bugs of DAMON. Hence, the command can be useful for investigation of DAMON bugs. When you report any issue for DAMON that not easy to be reproduced, providing the ‘damo diagnose’ output together with the issue report can be helpful.
damo version
damo version shows the version of the installed damo. The version number is constructed with three numbers. damo is doing chronological release (about once per week), so the version number means nothing but the relative time of the release. Later one would have more features and bug fixes.
damo replay (Replay Recorded Data Access Pattern)
Note: This feature is an experimental one. Some changes could be made, or the support can be dropped in future.
damo replay receives a damo record-generated data access pattern record file that specified via command line argument (./damon.data by default). Then, the command reproduces the recorded accesses in the file by making articial memory accesses. This could be useful for some types of system analysis or experiments with real-world memory access pattern.
Note that current implementation of damo replay runs on Python with single thread. Hence it might not performant enough to saturate full memory bandwidth of the system. If the record is made by workloads and/or systems that utilize memory bandwidth more than ‘damo replay’ and/or replaying systems could, and as the difference of the performance is big, the replayed accesses would be less similar to the original one. To show the real memory access performance of damo replay on specific system, users could use --test_perf option.
Write-only and CPUs-only Access Monitoring (EXPERIMENTAL)
Note: These features are extremely experimental. Many changes would be made and the support can be dropped in future.
On Linux kernel built with some hacks that exist in damon/next tree as of this writing (2025-08-03), users can do data access monitoring for only writes, or for accesses made by given set of CPUs using damo. Currently, only physical address space monitoring supports this.
Write-only Access Monitoring
Users can do write-only data access monitoring by adding --exp_ops_use_reports and --exp_ops_write_only options with value y to the usual DAMON parameters setup commands including damo start and damo tune, like below.
damo start --exp_ops_use_reports y --exp_ops_write_only y
Then all access monitoring results will silently ignore read accesses, and show only the accesses that made for writes.
Note that only physical address space monitoring is supported for now.
CPUs-only Access Monitoring
Users can monitor the data accesses made by only specific CPUs by adding --exp_ops_use_reports and --exp_ops_cpus options to the usual DAMON parameters setup commands including damo start and damo tune, like below.
damo start --exp_ops_use_reports y --exp_ops_cpus 0-3,5-7,9
--exp_ops_use_reports argument should always be y.
--exp_ops_cpus option should be given with a list of the desired CPUs. In this example, accesses that made by only CPUs of id 0, 1, 2, 3, 5, 6, 7, or 9 will be monitored and reported. The format of the cpus list is same to that for cpuset.cpus file of cgroup v2.
Note that only physical address space monitoring is supported for now.
Cautions and Plan to Drop Experimental Tag
The write-only and cpus-only monitoring features require changes in Linux kernel that not yet upstreamed. The changes are available at damon/next tree, which is for containing under-development DAMON works, as of this writing (2025-08-03). To make those upstreamed, we may need significant amount of efforts. Many changes could be made on the upstreamed version. In the worst case, we would forgive upstreaming the changes to mainline kernel, and drop the support on the damon/next tree.
The user interface for these features on damo will definitely be changed in future. At least, --exp_ part will be removed.
Finally, as of this writing (2025-08-03), the features are nearly not tested, so there could be many bugs.
So, please feel free to try and let us know bugs that you found, and if it is useful and worthy enough to make the upstream efforts. But, please don't make your long term important works depend on these.