Systemctl Coredump

- 4 mins read

A coredump is a snapshot of the memory associated with a process. A coredump is typically generated by the linux kernel when a process crashes due to an invalid operation leading to segmentation fault for example. There are other ways for generation of a coredump on demand. The signals that generate a coredump are listed in section Signals that generate a coredump.

In this document I have documented information regarding management of coredumps using systemctl-coredump. Coredump and journal logs are closely related and hence I also discuss some aspects of journalctl that may affect the management of coredumps.

Cleaning up journalctl logs

Command to rotate journalctl immediately:

sudo journalctl --rotate

To delete all rotated files:

sudo journalctl --vacuum-time=1s

Configure the limit on the number and size of journal logs by editing the following file:

/etc/systemd/journald.conf
SystemMaxUse=100M
#Followed by
$ systemctl restart systemd-journald

Effect of deletion of journalctl files on core-dump

Deletion of journalctl files (i.e removal of rotated and archived files, not rotation of current log file into an archive) have effect on the output of the coredumpctl list commad. systemd-coredump dumps core-dump related metadata in journal and once they are lost coredumpctl will not list the presence of core-dump files in spite of the fact that the core-dump files actually exist on disk.

$ sudo coredumpctl list
No coredumps found.

You can still list the coredumps using:

$ ls /var/lib/systemd/coredump/ -l

Management of systemd-coredump

Systemctl sets the system configuration to request kernel to send the coredump to a pipe using the following configuration file:

$ cat /usr/lib/sysctl.d/50-coredump.conf
kernel.core_pattern=|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %e
/usr/lib/sysctl.d/50-coredump.conf

Ensure that kernel core pattern is directed to pipe output to systemd-coredump:

$ cat /proc/sys/kernel/core_pattern
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %h %e

Configuration of systemd coredump is done in the following files:

/etc/systemd/coredump.conf
/etc/systemd/coredumpd.conf.d/*.conf

Limiting the size of coredump

To limit the size (uncompressed) of a core file use the following configuration in the systemd coredump configuration file:

ExternalSizeMax=

To limit the storage space used to store all coredumps change the following configuration

MaxUse=

Disabling Coredump

In some situations it may be required to disable generation of core-dump. For example, if a process has a large memory map (e.g suricata), it is efficient in terms of disk space and processing time to generate the large sized core-dump. In such cases to disable core-dump set following configuration in coredump.conf file:

Storage=none
ProcessSizeMax=0

The above configuration will take effect across all processes. To disable or limit core-dump generation for a specific process use the following configuration in the systemd service file for the specific process:

[Service]
LimitCORE=0

Cleaning up space used by coredump

Cleaning up journal logs (as discussed in section Cleaning up journalctl logs) does not delete the core files located at /var/lib/systemd/coredump/. This will only delete metadata related to the coredumps and coredumpctl list command will cease to list those coredumps.

Manual deletion of files in /var/lib/systemd/coredump/ is also not the correct approach to clean up space occupied by the coredumps. The proper way is to configure the systemd-coredump utility to limit the storage used by the coredumps as described in Limiting the size of coredump

Signals that generate a coredump

       Signal      Standard   Action   Comment
────────────────────────────────────────────────────────────────────────
       SIGABRT      P1990      Core    Abort signal from abort(3)
       SIGBUS       P2001      Core    Bus error (bad memory access)
       SIGFPE       P1990      Core    Floating-point exception
       SIGILL       P1990      Core    Illegal Instruction
       SIGIOT         -        Core    IOT trap. A synonym for SIGABRT
       SIGQUIT      P1990      Core    Quit from keyboard
       SIGSEGV      P1990      Core    Invalid memory reference
       SIGSYS       P2001      Core    Bad system call (SVr4);
                                       see also seccomp(2)
       SIGTRAP      P2001      Core    Trace/breakpoint trap
       SIGUNUSED      -        Core    Synonymous with SIGSYS
       SIGXCPU      P2001      Core    CPU time limit exceeded (4.2BSD);
                                       see setrlimit(2)
       SIGXFSZ      P2001      Core    File size limit exceeded (4.2BSD);
                                       see setrlimit(2)

References

  1. man systemd-coredump

  2. man coredump.conf

  3. systemd - Cleaning up coredumpctl list - Unix & Linux Stack Exchange