We finally narrowed it down to a file descriptor issue - the server was sucking them all up. Eventually, it would use them all and then it would hang. More debugging of the server ensued.
We *finally* narrowed it down to sys-proctable, a C extension based on Dan Urist's Proc::ProcessTable. So, somewhere I managed to eff up my C code so that it's not letting go of a filehandle somewhere. My attempts to shift the blame to Dan failed - his code does NOT suffer from the same bug (I checked).
The really weird thing is that everything seemed be ok about 2 weeks ago. Why wasn't it failing before? I have no frikkin clue.
It's that kinda stuff that gets my stomach in knots. Now I feel like a complete knucklehead for stressing over this for the last week or so.
UPDATE: Checking the C source I see that I have 4 calls to open() and 3 calls to close(). That could be a problem...YA THINK? All better now.
all file handles (Score:1)
On my linux box, I get
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 2048
virtual memory (kbytes, -v) unlimited
Re:all file handles (Score:2)
Number of "open" calls...1..2..3..4.
Number of "close" calls...1..2..3...hmmmm..
Five minutes later I had it fixed and created a little test with an FD counter to make sure. All better now. :)