use Perl

• RH Linux
• MySQL database on the back end.
• All HTML and Perl source code is stored in a CVS repository. Images are not stored under CVS.
• Each developer has their own local source tree with an Apache server listening to a high-numbered port.
• We also have a staging server that we use for testing
• We have a separate production server visible to the outside world.
• HTML pages are static files.
• We write mod_perl handlers and cgi scripts that run under Apache::Registry.

At its simplest, you could imagine a release process scheduled to run every night at 1:00 am that looked something like this:
cvs update
/usr/local/apache/bin/apachectl stop
/usr/local/apache/bin/apachectl start

However, this approach is likely to prove unsatisfactory for several reasons:

• Since images are not stored under CVS, new images might not get loaded when needed.
• Some development involves creating new database tables, altering existing tables, or populating existing configuration tables with new data.
• New modules should be tested before stopping the web server. Should a module fail some of its tests during installation, the update should be rolled back so that the server can start successfully.
• Some files get committed to the repository before they are stable and/or approved. Only stable, approved files should get released.
• A related (but distinct) issue is that stable files should not necessarily be released as soon as they are stable. A simple example of such a case is a (stable) HTML file which includes a link for another HTML file which is not yet ready to be launched.

Given this description, it is possible to rough out a simple spec for The Release System (TRS):

• TRS needs to interact with CVS. This will allow releases to be customized by directory hierarchy, CVS module, and / or tag. This will also ensure that master copies will be released, rather than local copies.
• TRS needs to be able to execute simple scripts. This will allow TRS to run tests, execute arbitrary SQL, log results, etc.
• TRS needs to be able to perform a rollback should unit tests fail during release. It should only stop and start the web server if all tests pass.
• TRS needs to interact with some system that knows (1) whether a particular file is stable; and (2) whether a particular file is approved for release.
• TRS should have logging facilities that keep a record of when all changes were made, error messages and warnings that were encountered, etc.
• Although a fully automated and regularly scheduled system is nice, it is not necessary. However, a release should be possible with very little interaction on the part of the system administrator.
• TRS should be able to run the regression testing suite after release.

A few questions:

• Who has cracked this nut before?
• Can anyone recommend pointers to good articles that tackle part or all of this?
• make is certainly a candidate for some of the logic -- would anyone recommend a different installation utility?
• Parts of this sound like a Content Management System -- does anyone care to comment on the applicability of Bricolage or a similar open source CMS product?
• CVS does not seem to have as robust a system for tracking stability as I would like -- i.e. you can set tags on groups of files, as in "STABLE-06012002", but you can't flip a bit per file. (Unless you set a tag per file -- which isn't necessarily good since a file might go through various iterations of stability / instability.) Can anyone venture an opinion as to whether the per-file "stable" flag is a good idea or a cluster-f*ck?

Thanks for any and all help.