What am I working on right now? Probably the Sprog project [sourceforge.net].
GnuPG key Fingerprint:
6CA8 2022 5006 70E9 2D66
AE3F 1AF1 A20A 4CC0 0851
This recent journal entry links to an article which asserts that changing data structures in a relational database is hard.
As autarch points out regardless of how you store your data, a change to the structure of the data will require a change to the code that uses it.
People are often resistant to change, even when the change is a fix to something that's broken. Often this is simply due to inertia. Sometimes, especially in the IT world, it's due to a fear of breaking other stuff.
In my $work_project we aim to develop for the current requirements without trying to second-guess future requirements. In fact just about anything I can think of that we put in because we thought we'd need it in the future, turned out to be a mistake. We all make mistakes. Hopefully we learn from our mistakes. And if we embrace change rather than fearing it, hopefully we can fix our mistakes as they become apparent.
Our database schema is moderately complex. Our PostgreSQL database has nearly 200 inter-related tables and takes up around 60GB of storage. As we add new features to the application, data structures and relationships are added and updated. Often these updates will have a ripple effect where existing bits of code need to be changed as well. Do we fear change? No, we embrace it. Our development workflow and infrastructure has been built up over time, to support frequent change.
Regression tests are obviously a critical part of how we work. We have almost 2000 test scripts which we use like this:
Another critical element of our project workflow is our 'apply_patches.pl' script. A 'patch' in this context is a
The patches are applied automatically in the development environment when we run regression tests. They are also applied automatically when 'good' code is checked out in the staging environment. Our production migration process is scripted and it also applies patches automatically. By the time a patch has made its way through developer testing and staging deployment, the chances of it going wrong in a production migration are fairly slim. And because it's all automated, we can be confident that code changes and schema changes will remain in sync.
In the three years since the project went live, nearly 450 schema patches have been deployed. Obviously many of those are trivial inserts/updates/deletes to tables of 'constants' (or 'lookup' data), but patches which add/drop/alter tables number in the hundreds.
Fear of making a mistake can slow development. Providing an infrastructure which supports rapidly fixing mistakes has significantly improved our productivity.