It chokes with an ELFCLASS error at some point during make, so it must be linking against a 32 bit lib somewhere, but I'm not sure where. Also, I rebuilt bdb as 64 bit, as Perl seemed to be a bit overzealous when it came to buiding against it. However, that didn't solve my problem, so that wasn't it.
Anyway, I've submitted an RT ticket (4317) if anyone's interested in investigating.
Update: Resubmitted to perlbug as ticket #24439.