Raspberry Pi 3 high IO

The questioner updated his question with the solution. It seems he doesn't like to create a correct answer as suggested in a comment. Here is it:

After spending a lot of time I finally found the root cause. Collectd auto enabled this on install: https://collectd.org/wiki/index.php/Plugin:RRDtool which caused the whole system to use 100% IO for that. Took me way longer to figure out than I'd like to admit. Anyway, my Pi finally seems to be fixed and be back to normal.

That automake process is not necessarily the cause of the problem, it may instead be mostly a symptom.

99% IO is now being used by [kworker/u8:0+flush-179:0] which seemingly doesn't make any sense

First, WRT iotop, the IO column shows the percentage of the process's running time that is currently spent waiting on IO.¹ So 99% for one (or some, or all) process(es) makes sense, it is just concerning.

[kworker/____] indicates a kernel thread. If anything is busy with IO, it is probably the kernel (it could by necessity mean the kernel in the context of iotop, that would take some more investigation).

As already mentioned WRT this question, automake is a component of Autotools, a suite of things used in configuring and executing a software build from source code (usually written in C and/or C++). If you've done something like this:

cd [some_source_direcory]
./configure
make [install]

That's what started that. If automake and kworker are alternating in iotop I would guess they are related to each other (ie., the kworker is doing stuff for the userland process, automake).

In any case, it may be a bit of cause for concern if you have processes constantly waiting on IO this way and your total disk access is ~32 K/s. It should be a lot faster than that.

This may be because the SD card is wearing out/has developed a defect. You should have a look at dmesg output and int /var/log/syslog to see if any IO errors are being issued.

---

1. This implies a sampling period that is presumably quite short. If you want to dive deeper I'd start with that U&L link.

Since you know the PID of the offending process (4076), you should find out what started it. Run ps -aef --forest (as root, because it's a root process) and search for the PID in the process tree.

The keyword automake suggests it could be a software package being installed from source code.

Unless you observe this behavior right after a reboot, check out your .bash_history file or whatever. It could be that you have started a software build and then forgot about it.