And then I found this feature comparison. CVS comes out pretty bleak... Subversion, AKA SVN, beats (or at least equals) it in nearly every item listed.
It turns out there's a Tortoise for SVN too, so I just had to try it. So I'm, in one breath, comparing the Tortoise programs as well. Well, it's at least as good as TortoiseCSV. Actually, the interface feels better in almost every respect only the diff tool (which got the name "merge"), has a bit of a more primitive feeling to it, despite having quite a bit of power under the hood, such as interactively patching the two versions.
I like the icons better, too.
Looking under the hood (I just can't resist), SVN again looks more clean, to my taste. CVS appears to have a text file per source file, holding the original file, and every diff made since then. SVN, OTOH, keeps a compressed archive per revision. I like that better, it seems more transparent to me. When comparing two files, SVN just has to extract the files from the two archives, and pass them to diff. OTOH CVS has to reconstruct a particular revision through the whole modification history of a file.
And then there's that excellent online book. Reading through it, I get more and more a warm, fuzzy feeling about Subversion, which wasn't really the case with CVS. Subversion feels like CVS rebuilt from scratch, but better, this time. That is exactly what it is.
I made up my mind, I'll stick with SVN.