use Perl

• SQLite (Score:2)

  by Theory (10)

  Did you use transactions when you tested SQLite for query speed? It's a lot faster if you do. By orders of magnitude.
  
  David
  • Re:SQLite (Score:1)

    by perrin (4270)

    That would only speed things up for situations where multiple statements can be batched, right? My tests are for use as cache storage, so it's just a single read or write. I can't even keep a transaction open for reads, becasue that would block other people's writes. I did use "PRAGMA synchronous = OFF" and I see in the optimization FAQ that there are a couple of new things that have shown up since the last time I looked at it. I'll try another test soon.
• DB Benchmarks (Score:1)

  by trachtenberga (3651)

  George Schlossnagle posted some interesting performance benchmarks [schlossnagle.org] on reading from embedded databases.

  I was surprised to see how crappy SQLite was, especially in the wake of all the earlier raves I'd read about it compared to MySQL for SELECTs.

  -adam
  • Re:DB Benchmarks (Score:2)

    by Matts (1087)

    SQLite's a full database though. I suspect it would be fairly nippy if you could go straight to its btree API.
    
    It also makes a difference how you execute the query. i.e. do you cache the statement handle, do you use bound variables for the output, etc.
    • Re:DB Benchmarks (Score:1)

      by perrin (4270)

      In my tests I did use all the DBI speed tricks. There seem to be a couple of new SQLite-specific things I should try, but I suspect this (simple primary key lookups) is just a hard thing to compete with MySQL on.
      • Re:DB Benchmarks (Score:2)

        by Matts (1087)

        Probably so. If you want to send me the code I'll take a look and see if there's any obvious problems.
        • Re:DB Benchmarks (Score:2)

          by Matts (1087)

          I should clarify - I meant if there are any obvious problems in DBD::SQLite :-)
          • Re:DB Benchmarks (Score:1)

            by perrin (4270)

            What I'm doing is releasing a cache abstraction module to CPAN (tentatively named Cache::Wrapper) which will include a DBD::SQLite version. Then I'll put out my benchmark code that uses it, and people can use it for their own tuning and comparisons.
  • Re:DB Benchmarks (Score:1)

    by perrin (4270)

    I'm not quite sure I trust these, since GDBM is definitely not faster than BerkeleyDB when you use the right API.
    • Re:DB Benchmarks (Score:1)

      by trachtenberga (3651)

      Interesting. In my experience, George's stuff is usually pretty good. Since I didn't run the tests, I don't know which API he used.
• SpamAssassin (Score:1)

  by quinlan (5482)

  The one thing you're missing is the history.

  SpamAssassin originally used Any_DBM (huge mistake, but that one predates me). Of the modules that Any_DBM uses, DB_File was the best choice for stability, portability, and performance reasons and it had the same interface as Any_DBM so we've stuck with it for a few versions. We're likely to eventually deprecate (or discourage) DB_File in favor of SDBM_File which is now a better option since other changes have made its deficiencies a non-issue. SDBM_File is f
  • Re:SpamAssassin (Score:1)

    by perrin (4270)

    Okay, so compatibility is ultimately more important than speed here. That makes perfect sense to me.

    I would like to try a BerkeleyDB version. It would be much faster than SDBM_File. If I get something working, I'll send in the patch.
• Repetition. Repetition. Repetition. Repetition. (Score:1)

  by Aristotle (5147)

  That is definitely the key. Word needs to get out. People need to be told. Over and over, until they start to repeat it to others themselves.

  Good performance at macroscopic scale is all about understanding the interactions between layers of abstraction. Unfortunately that is the hardest to come by kind of knowledge, for a number of reasons. Good practices are therefore not going to be establish themselves; they need insistent advocates.
• A minor detail that could prevent BerkeleyDB (Score:2)

  by htoug (771)

  ...from being an option, is that is fails when used on an NFS filesystem. The errors are quite wierd, and do not disclose that this is the problem, so that could indicate that you have to be carefull when advocating ose of BerkeleyDB in unknown environments.
  • Re:A minor detail that could prevent BerkeleyDB (Score:1)

    by perrin (4270)

    BerkeleyDB uses shared memory for caching and communication, so it can't be used with any sort of network filesystem. It has to be used on a single machine.