After a bit of research, it's been determined that my Web test code is fine. I fork off 40 independent Web tests that crawl through the site and bring the database servers (Oracle) to their knees. They've also managed to take down the Groupwise server by overwhelming it with error email. It's like everything is trying to delicately sip from a firehose with predictable results. I've wreaked a lot of havoc with a wee bit of stress testing. I can't stop smiling.
We'd love to have more database servers to handle the requests, but Oracle is too expensive. We've considered Postgres, but too many people argue that our clients won't trust anything but Oracle. As a result, Oracle is fast becoming a significant bottleneck. We need to sign up enough clients that we can afford more Oracle licenses, but if we don't have the licenses, we're in danger of not being fast enough to sell our services to more clients.
For those who might think that we just need to tweak Oracle a bit, I'll agree that we need more experience with this. The amount of data we keep is only a few gigabytes, but the insane data requirements make it very difficult for us to tune the SQL. Our software routinely builds 300 to 400 line SQL queries on the fly and watching the huge number of tables, views, and snapshots that the queries span is simply mind-boggling. Caching the queries is not an option because they are different for virtually every customer on the site and often are different for just one customer because of filter changes they choose to apply. And did I mention that it's real-time data we're feeding? This is crazy stuff.