Orsdxy

I have a Debian (Buster) laptop with 8GB ram and 16GB swap. I'm running a very long running task. This means my laptop has been left on for the past 6 days while it churns through.

While doing this I periodically need to use my laptop as a laptop. This shouldn't be a problem, the long running task is IO bound working through stuff on a USB hard disk and doesn't take much RAM (<200MB) or CPU (<4%).

The problem is when I come back to my laptop after a few hours, it will be very sluggish and can take 30 minutes to come back to normal. This is so bad that crash-monitors flag their respective apps as having frozen (especially browser windows) and things start incorrectly crashing out.

Looking on the system monitor, of the 2.5GB used around half gets shifted into swap. I've confirmed this is the problem by removing the swap space (swapoff /dev/sda8). If I leave it without swap space it comes back to life almost instantly even after 24 hours. With swap, it's practically a brick for the first 5 minutes having been left for only 6 hours. I've confirmed that memory usage never exceeds 3GB even while I'm away.

I have tried reducing the swappiness (see also: wikipedia) to values of 10 and 0, but the problem still persists. It seems that after a day of inactivity the kernel believes the entire GUI is no longer needed and wipes it from RAM (swaps it to disk). The long running task is reading through a vast file tree and reading every file. So it might be the kernel is confused into thinking that caching would help. But on single sweep of a 2TB USB HD with ~1 billion file names, an extra GB RAM isn't going to help performance much. This is a cheap laptop with a sluggish hard drive. It simply can't load data back into RAM fast enough.

How can I tell linux to only use swap space in an emergency? I don't want to run without swap. If something unexpected happens, and the OS suddenly needs and extra few GB then I don't want tasks to get killed and would prefer start using swap. But at the moment, if I leave swap enabled, my laptop just can't be used when I need it.

The precise definition of an "emergency" might be a matter for debate. But to clarify what I mean: An emergency would be where the system is left without any other option than to swap or kill processes.

What is an emergency? - Do you really have to ask?... I hope you never find yourself in a burning building!

It's not possible for me to define everything that might constitute an emergency in this question. But for example, an emergency might be when the kernel is so pushed for memory that it has start killing processes with the OOM Killer. An emergency is NOT when the kernel thinks it can improve performance by using swap.

One fix is to make sure the memory cgroup controller is enabled (I think it is by default in even half-recent kernels, otherwise you'll need to add cgroup_enable=memory to the kernel command line). Then you can run your I/O intensive task in a cgroup with a memory limit, which also limits the amount of cache it can consume.

If you're using systemd, you can set +MemoryAccounting=yes and either MemoryHigh/MemoryMax or MemoryLimit (depeneds on if you're using cgroup v1 or v2) in the unit, or a slice containing it. If its a slice, you can use systemd-run to run the program in the slice.

Full example from one of my systems for running Firefox with a memory limit. Note this uses cgroups v2 and is set up as my user, not root (one of the advantages of v2 over v1 is that delegating this to non-root is safe, so systemd does it).

$ systemctl --user cat mozilla.slice 

# /home/anthony/.config/systemd/user/mozilla.slice

[Unit]

Description=Slice for Mozilla apps

Before=slices.target



[Slice]

MemoryAccounting=yes

MemoryHigh=5G

MemoryMax=6G



$ systemd-run --user --slice mozilla.slice --scope -- /usr/bin/firefox &

$ systemd-run --user --slice mozilla.slice --scope -- /usr/bin/thunderbird &

I found to get the user one working I had to use a slice. System one works just by putting the options in the service file (or using systemctl set-property on the service).

Here is an example service (using cgroup v1), note the last two lines. This is part of the system (pid=1) instance.

[Unit]

Description=mount S3QL filesystem

Requires=network-online.target

After=network-online.target



[Install]

WantedBy=multi-user.target



[Service]

Type=forking

User=s3ql-user

Group=s3ql-user

LimitNOFILE=20000

ExecStartPre=+/bin/sh -c 'printf "S3QL_CACHE_SIZE=%%in" $(stat -c "%%a*%%S*.90/1024" -f /srv/s3ql-cache/ | bc) > /run/local-s3ql-env'

ExecStartPre=/usr/bin/fsck.s3ql  --cachedir /srv/s3ql-cache/fs1 --authfile /etc/s3ql-authinfo  --log none «REDACTED»

EnvironmentFile=-/run/local-s3ql-env

ExecStart=/usr/bin/mount.s3ql --keep-cache --cachedir /srv/s3ql-cache/fs1 --authfile /etc/s3ql-authinfo --cachesize ${S3QL_CACHE_SIZE} --threads 4

ExecStop=/usr/bin/umount.s3ql /mnt/S3QL/

TimeoutStopSec=2m

MemoryAccounting=yes

MemoryLimit=1G

Documentation is in systemd.resource-control(5).

It seems that after a day of inactivity the kernel believes the entire GUI is no longer needed and wipes it from RAM (swaps it to disk).

The kernel is doing The Right Thing™ believing it. Why would it keep in RAM unused memory and so essentially waste it instead of using it as cache or something?

I don't think the Linux kernel is gratuitously or anticipatory swapping out pages so if it does it, that must be to store something else on RAM, thus improving performance.

If you know when you'll need to reuse your laptop in advance, you might use the at command (or crontab) to schedule a swap cleanup (swapoff -a).

Here's an idea, which I haven't tried myself (and I'm sorry I haven't the time right now to experiment with this).

Suppose you create a small VM with only 512MB memory for your background process I'm not sure if you'd want this to have any swap, your call, and switch off swap on your host system.

Is the process you're running something you've created yourself?

If so, it might be worth tweaking your code to open the files using the O_DIRECT flag, which to quote the manual page -

Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user-space buffers. The O_DIRECT flag on its own makes an effort to transfer data synchronously, but does not give the guarantees of the O_SYNC flag that data and necessary metadata are transferred. To guarantee synchronous I/O, O_SYNC must be used in addition to O_DIRECT. See NOTES below for further discussion.

Remove swap or diminish it about 20% (may vary with systems) as recently OS are not using swap anymore the same way as they did in the few years ago. It probably answer some of your question:

here --> official redhat.com

important info:

In the past, some application vendors recommended swap of a size equal to
the RAM, or even twice the RAM. Now let us imagine the above-mentioned
system with 2GB of RAM and 2GB of swap. A database on the system was by
mistake configured for a system with 5GB of RAM. Once the physical memory is
used up, swap gets used. As the swap disk is much slower than RAM, the
performance goes down, and thrashing occurs. At this point, even logins into
the system might become impossible. As more and more memory gets written to,
eventually both physical- and swap memory are completely exhausted and the
OOM killer kicks in, killing one or more processes. In our case, quite a lot
of swap is available, so the time of poor performance is long.

and

https://wiki.debian.org/Swap

small portion of link above:

Information and considerations related to the amount of swap to use:

"The recommended amount of swap space has traditionally been double the amount of
system memory. This has changed over time to one and half times system memory, both
answers are decent baselines but are becoming less and less useful answers to the
question as time passes. There are many variables about your system and intended use
that will determine the available system swap you will want to have."

You may try:

Best way to disable swap in linux

Personal note:

Since I've 6GB RAM and in all my recently Linux OS. I've never ever seem any indication of use of Swap. I determined it I must turn it off either for space (few gigs more) and because it has slowered my system sometimes.