We finally narrowed it down to a file descriptor issue - the server was sucking them all up. Eventually, it would use them all and then it would hang. More debugging of the server ensued.
We *finally* narrowed it down to sys-proctable, a C extension based on Dan Urist's Proc::ProcessTable. So, somewhere I managed to eff up my C code so that it's not letting go of a filehandle somewhere. My attempts to shift the blame to Dan failed - his code does NOT suffer from the same bug (I checked).
The really weird thing is that everything seemed be ok about 2 weeks ago. Why wasn't it failing before? I have no frikkin clue.
It's that kinda stuff that gets my stomach in knots. Now I feel like a complete knucklehead for stressing over this for the last week or so.
UPDATE: Checking the C source I see that I have 4 calls to open() and 3 calls to close(). That could be a problem...YA THINK? All better now.