As suspected, something fishy was wrong.
I had a cron job that didn't exit. So I had about 300 copies of it running. And of course it was running perl, so it used quite a bit of RAM.
Strange that cron doesn't set an alarm.
My load is back to a healthy 2.0