Stories
Slash Boxes
Comments
NOTE: use Perl; is on undef hiatus. You can read content, but you can't post it. More info will be forthcoming forthcomingly.

All the Perl that's Practical to Extract and Report

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More | Login | Reply
Loading... please wait.
  • At least not against a good database. (eg Oracle, PostgreSQL, etc - but MySQL would be fine.)

    The problem is that there is a trade-off between time spend preparing a query and query performance. It would not be inappropriate to think of preparing a query as a "compile and optimize this SQL" step. MySQL spends very little energy preparing, and therefore its ability to handle complex queries suffers. Most other databases put a lot of energy into preparing, and so it is very important to try to avoid recomp

    • Actually I am going to use it on a very very busy site ;) SQL preparing does not make much sense for me since our platform will usually work in the context of PostgreSQL PL/Proxy cluster *AND* the query could be dynamic enough to defeat ordinary DBI param binding.

      Actually runtime performance is the reason to choose source-filter solutions in the first place ;)

      Also, the SQL example is, well, merely an example...Filter::QuasiQuote's power reveals in the context of true DSLs ;)

      • My experience says that with a sane design you can run one of the top couple thousand busiest websites on the internet on a handful of webservers, paying only a modest amount of attention to performance of the code on your webservers.

        That same experience says that tiny mistakes in how you handle your database can cause that same site to melt unexpectedly.

        The lesson is to not worry about webserver performance, but be paranoid about database performance. Which means use placeholders properly. If you do it d

        • I must say that all you say is indeed true for an ordinary web application :)

          But unfortunately I can't use prepare+execute in my OpenResty platform in particular. Why? Because it must scale by design to serve lots of apps here in Yahoo! China and Alibaba. So it must be a cluster or something like that.

          The PL/Proxy database server requires frontend queries to be of the following form:

                select xquery('account', 'select * from posts...', 1);

          That is, the user sql query itself must be a dynam

          • Responding out of order.

            On sending multiple insert statements at once. Yes, that can be a big win because you're cutting down on round trips to the database. Each round trip takes unavoidable resources on the client, server, and network. With network latency typically being the biggest deal. However there is an upper limit to the win from that. A compromise that works fairly well is to prepare a bulk insert that inserts multiple records, thereby bypassing the prepare and reducing round trips. YMMV. But better still is to use the COPY functionality. If need be, COPY to a temp table, then run an insert from that. It is more work because you will need to create said table though, and get the datatypes right. Also with the version of DBD::Pg that I'm using you have to guess the size of buffer to use. But I think the new one automagically gets it right.

            I haven't used PgBouncer. What I've found sufficient for my needs is just having a limited number of pre-forked Apache servers, and then have proxies in front of them in accelerator mode. The children reuse their connections from request to request. Avoiding the complexities of connection pooling simplifies things and seems like a design win to me, though I grant that there are situations where you would want connection pooling.

            I have not used PL/Proxy. Glancing at the documentation it does seems that it does not support preparing queries. I guess that the resulting overhead is gained back from having multiple machines. However my immediate reaction is to think that you're not going to be able to use it with any relational features of the database. And once you give up the relational features of the database, I'd be inclined to stop pretending to be relational and look for a more exotic database that did exactly what I wanted.

            If you're doing an insert rarely, select often, then what would be really nice to have is a queryable hot backup. The technology for that exists in Oracle and is really good. It doesn't exist for postgres. $work is willing to help fund a PostgreSQL hacker to add it, but the core pg team hasn't found anyone to take it. (More precisely, they have found someone who is working on it, but he won't be tackling that part of the project any time soon...)

            • Right, preparing a bunch of insert statements first would be faster :) Merely have to deal with the last few specially :) Thanks for the tip.

              We use PgBouncer at the PL/Proxy level to cache connections to the data nodes. On the FastCGI level, a pre-forked lighttpd is used. Database connection to the PL/Proxy nodes are reused across fastcgi loops in a similar fashion as you described :)

              Well, I don't think the use of PL/Proxy necessarily means loss of relationality. Relational constraints still hold for data r

              • Tell me if a donation could help the open-sourcing decision. I'm btilly, at gmail dot com.