Last updated 28 August 1999
This is the FAQ for CVSup. Please send comments and suggestions to cvsup-bugs@polstra.com.
CVSup is a software package for transferring and updating collections of files across a network. It consists of a server called cvsupd and a client called cvsup.
CVSup is faster (often by an order of magnitude) and more flexible than traditional network update packages.
CVSup can efficiently update any kind of file. It can even update Unix device nodes, symbolic links, and hard links. CVSup supports several different algorithms for updating various kinds of files. It tries to use the most efficient method for each file. For example, RCS files are updated using a specialized algorithm that takes advantage of their structure to greatly reduce the update time. Log files (which are changed only by appending new text at the end) are likewise updated by a special algorithm that transmits only the new text. Other text and binary files can be updated efficiently by the rsync algorithm, which is built into CVSup.
CVSup is free software, distributed under a BSD-style license. You can obtain CVSup from ftp://ftp.FreeBSD.org/pub/FreeBSD/development/CVSup/, and from FreeBSD's FTP mirror sites all over the world.
The sources subdirectory contains the full sources for CVSup. Be sure to read this before fetching the source distribution.
The binaries subdirectory contains precompiled statically-linked executables for various platforms:
Be sure to read the appropriate README file for your target platform.
Please address all mail regarding CVSup to cvsup-bugs@polstra.com.
An RCS file stores many versions of a single source file, all in one place. During the lifetime of a project, its source files evolve. Each source file starts with an initial version. Over time, changes are made to the source files to fix bugs and add features. At key moments, a programmer wish to save the current version of a source file. He can do this by checking it into its corresponding RCS file. From a given RCS file, one can later extract any desired version of the source file. This makes it easy to undo changes that turned out to be ill-advised, or to recreate an earlier release of the software.
A CVS repository is a collection of RCS files that are managed together. For example, the RCS files for all of the sources in a project would typically be stored together in a CVS repository.
Yes, try Jordan Hubbard's "cvsupit" package. You'll find it at ftp://ftp.freebsd.org/pub/FreeBSD/CVSup/cvsupit.tgz.
Make a cvsupfile that looks something like this:
*default host=cvsup3.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs
*default delete use-rel-suffix
*default tag=.
src-all
This cvsupfile will create a directory tree "/usr/src" and populate it with all of the main FreeBSD sources except the export-restricted cryptographic code. It will also create a tree "/usr/sup" containing the checkouts files that CVSup uses to maintain state between updates. If you want your source tree to be somewhere other than "/usr/src", change the "prefix=" clause. If you want the checkouts files to be located somewhere other than "/usr/sup", change the "base=" clause.
Be sure not to leave out the "tag=." clause. That tells CVSup to use checkout mode to send you the most recent version of each source file.
Simply add one more line to the end of the cvsupfile above:
cvs-crypto
Note: If you are located outside of North America, then you must also change the "host=" clause to specify a non-North American server. For example:
*default host=cvsup.internat.freebsd.org
See here
for a complete list of public CVSup servers carrying the FreeBSD
sources.
No, you don't. CVSup is capable of "adopting" your existing sources and bringing them up to date. If you think about it, your situation is the same as if you had been updating with CVSup all along but had lost your checkouts files. And CVSup uses the same technique to deal with both situations. It uses checksums to determine which revisions of the files you have currently, and then updates them in the appropriate ways to transform them into the latest versions.
Look here for the safest way to do this.
Yes, you do. The greater the distance between the revisions you have and the revisions you want to get, the greater the chance that you'll miss some important file deletions. Luckily, if you know approximately or exactly which versions of the files you are starting with, you can reduce or eliminate this risk. You do this by updating twice. First, you tell CVSup to "update" you to the versions that you already have. This won't change any of your files, but it will create a checkouts file that precisely reflects what you have currently. Second, you update again, this time telling CVSup which version you really want. On the second update, CVSup will have all the information it needs in order to know which files to delete.
This is a little tricky and there are a couple of important details we haven't mentioned yet. So we'd better give you an example. Suppose you installed FreeBSD-2.2.5, including sources, from the CD-ROM. Now you decide you want to use CVSup to track the 2.2-stable sources. For your first update only, use a cvsupfile like this:
*default host=cvsup2.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs
*default delete use-rel-suffix
src-all tag=RELENG_2_2_5_RELEASE list=cvs:RELENG_2_2
For subsequent updates, change the last line to:
src-all tag=RELENG_2_2
The unmentioned details are in this last line. First of all, how do you know what to use for the tag? The answer is, you have to consult a list. See the "Configuration" section of the CVSup chapter in the FreeBSD Handbook for a list of tags that are valid for the FreeBSD sources. The important point is that the tag for your first update should correspond to the version of the sources that you already have. And the tag for your second and subsequent updates should correspond to the version of the sources that you want to receive.
Second, what is this business with the "list" keyword? It is rarely used, but this is a situation where it is necessary. When CVSup creates or consults one of its checkouts files, it uses a filename which by default is based on the "release" and "tag" values for the collection. Specifically, the usual name of a checkouts file is "checkouts.RELEASE:TAG", where RELEASE and TAG are the "release" and "tag" settings in your cvsupfile.
In the rather special situation we are addressing here, the naming conventions for the checkouts files cause a problem. By default, our first update would produce a file named "checkouts.cvs:RELENG_2_2_5_RELEASE", while the second update would look for a file named "checkouts.cvs:RELENG_2_2". In order for the second update to benefit from the information garnered in the first, both updates must use the same checkouts file. The "list" specification in the first cvsupfile allows us to accomplish this. It overrides the default suffix in the name of the checkouts file, and forces it to have the same name as will be used in subsequent updates.
Admittedly, this is arcane. But you only have to do it once.
Follow the previous instructions, except change the src-all lines as follows. In the first update, use:
src-all tag=RELENG_2_2_5_RELEASE list=cvs
In subsequent updates, use:
src-all tag=.
Note carefully: that's a period following the tag= part.
Don't leave it out.
It doesn't make any difference. This cvsupfile:
*default host=cvsup3.freebsd.org
*default base=/usr
*default prefix=/usr
*default release=cvs
*default delete use-rel-suffix
*default tag=.
src-all
could just as well be expressed as a single line, like this:
src-all host=cvsup3.freebsd.org base=/usr prefix=/usr release=cvs delete use-rel-suffix tag=.
Using the "*default" lines can make your cvsupfile easier to read, by shortening the lines. They can also make your cvsupfile more concise if you are receiving several collections.
It gives CVSup permission to remove files on your machine. For example, suppose you have a file "foo" which you originally received using CVSup. Now the maintainer of the server host deletes "foo". When you next run CVSup, if "delete" is specified in your cvsupfile then CVSup will delete "foo" on your machine. Otherwise it will leave it alone.
Except for a few unusual applications, you should always specify "delete" in your cvsupfile.
Originally CVSup was designed to be a drop-in replacement for sup. Because of that, the defaults had to be the same whether they made sense or not.
Don't feel bad. Many people find them confusing. But they really do work.
The most common errors people make with refuse files are:
We cover these problems in the following items.
The most important thing to remember is that the patterns in refuse files are relative to the prefix, which is often not what you think of as the logical root of a particular collection. To determine the prefix, look at your cvsupfile. Does it contain something like this?
*default prefix=/some/directory
Usually it will. In that case /some/directory is the prefix.
Otherwise, the prefix is the same as the base. To determine
the base, see this.
Once you have determined your prefix, put yourself in that directory. Then figure out the relative paths of the files and/or directories you want to block, and make patterns that match them. Put these patterns into your refuse file separated by whitespace. You can put each pattern on a separate line, or put several on each line. Either way works.
OK. Suppose you use CVSup to receive the FreeBSD documentation files (the "doc-all" collection), using the doc-supfile example from /usr/share/examples/cvsup. Here's what that cvsupfile looks like, stripped of comments:
*default host=CHANGE_THIS.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default release=cvs tag=.
*default delete use-rel-suffix
*default compress
doc-all
As you can see, the prefix is /usr. Relative to there, the
entire "doc-all" collection is placed into a subdirectory named
doc, which itself contains these files and subdirectories:
FAQ/ handbook/ ru_SU.KOI8-R/
Makefile ja/ sgml/
en/ ja_JP.EUC/ share/
en_US.ISO_8859-1/ ja_JP.eucJP/ zh/
es/ ru/ zh_TW.Big5/
es_ES.ISO_8859-1/ ru_RU.KOI8-R/
Now let's suppose you're not interested in the Spanish, Japanese,
Russian, or Taiwanese versions of the documentation. So you want to
refuse the directories whose names begin with "es", "ja", "ru", and
"zh". The correct patterns relative to the prefix are:
doc/es*
doc/ja*
doc/ru*
doc/zh*
The patterns you specify must match the names of the files on the server. If the files are coming from a CVS repository (the usual case), then on the server they are RCS files. And RCS files always have names that end in ",v". Your patterns must take that into account.
For example, suppose you want to block the Makefile in the example above. The pattern "doc/Makefile" won't work, because on the server the file's name has a ",v" appended to it. The correct pattern to use is
doc/Makefile,v
or better still
doc/Makefile*
which will match the file on the server whether it is an RCS file or
not.
First you must determine your base directory.
Next you must determine your collDir, the subdirectory of base where CVSup keeps track of your collections.
Finally, you must know the name of the collection you want to restrict, for example, "doc-all".
Combine those three items with a slash after each one, and then tack "refuse" on the end. That's where to put your refuse file. For the example we've been using, it works out to
/usr/sup/doc-all/refuse
assuming you don't use the "-b" or "-c" option when you run CVSup.
Yes. Simply leave out the name of the collection and the slash that follows it when you formulate the name of the file. In the previous example, the global refuse file would be named
/usr/sup/refuse
The following instructions were kindly submitted by Alan Strassberg:
cvsup 5999/tcp # CVSup
cvsup stream tcp nowait root /usr/local/etc/plug-gw plug-gw cvsup
and send a SIGHUP to inetd.
plug-gw: port cvsup A.B.C.D -plug-to W.X.Y.Z -port cvsup
where A.B.C.D is the IP address of the internal machine,
and W.X.Y.Z is the IP address of the CVSup server.
*default host=gatekeeper.foo.com
Troubleshooting: you should be able to telnet from the internal machine to the firewall on port 5999 and see the CVS server greeting:
% telnet gatekeeper.foo.com 5999
OK 15 5 REL_15_4_2 CVSup server ready
If the greeting does not appear, run netstat -na on the
firewall and verify that it is listening on port 5999:
% netstat -na | grep 5999
...
tcp 0 0 *.5999 *.* LISTEN
Both you and your friendly server administrator need to upgrade to CVSup 15.4 or later. That will solve this problem.
CVSup upgrades an RCS file by deconstructing it on the server, sending only the pieces that have changed to the client, and reconstructing the file on the client. (It's roughly patterned after the transporter on the starship Enterprise.) CVSup then compares the MD5 checksum of the reconstructed file with that of the original on the server, to make sure that the process worked correctly.
Unfortunately, recent releases of CVS have introduced some gratuitous changes in the format of the RCS files that they write. These are simply changes in white space, which have no bearing on the logical meaning of the file. However, the result is that an RCS file constructed by the CVSup client no longer matches the original file byte-by-byte as before. Even though the reconstructed file is logically identical to the original, it does not have the same checksum. This causes older versions of CVSup to reject the updated file and use a fixup to re-transfer the entire file.
To solve this problem, CVSup 15.4 introduces a logical checksum which is used only for RCS files. Instead of blindly computing a byte-by-byte checksum over the entire file, the new checksum algorithm carefully canonicalizes the file so that irrelevant white space differences are ignored. The logical checksum should make CVSup immune to any future problems of this nature.
You are behind a firewall which is blocking attempts by the CVSup server to establish a second TCP connection to your client. Add the option "-P m" to your cvsup command line, and everything should work fine.
When you built your FreeBSD kernel, you included the undocumented "options DEBUG" in the kernel config file. Don't do that.
Yes, the statically linked FreeBSD binaries work fine under other BSD-derived operating systems. But for some of them, including BSD/OS, you have to add "@M3novm" to the command line.
CVSup is written in Modula-3, and its runtime system uses a sophisticated garbage collector which exploits hooks into the VM subsystem of the operating system to gain better interactive performance. This feature stumbles upon an incompatibility between BSD/OS and FreeBSD, causing the core dumps. The cryptic argument "@M3novm" disables the VM hooks and makes it possible to run FreeBSD binaries under other BSD-derived operating systems.
Those strange files are RCS files, and CVSup sent them to you because your cvsupfile told it to. When you ask CVSup to send you updates from a CVS repository, there are two different things you can ask for. First, you can ask for the source files to be extracted from the repository and sent to you. That is apparently what you wanted in this case. Second, you can ask for the raw RCS files (containing all versions of the sources) to be sent to you.
These two modes of operation are fundamentally different. Both work from RCS files in a CVS repository on the server host. But they use the RCS files in different ways. The first mode, called "checkout mode," extracts a particular version of the sources from the RCS files, and sends you that version. The second mode, called "CVS mode," sends you the RCS files themselves, in the same form as on the server. In the cvsupfile, the "tag" and "date" keywords control which mode is used. If either of these keywords is present, then CVSup uses checkout mode to send you a set of source files. If neither "tag" nor "date" is present, then CVSup uses CVS mode to send you the RCS files.
Assuming you want the most recent version of the sources, you simply need to add "tag=." to your supfile.
In order to update your files efficiently, CVSup needs to know what you've already got. It stores this information in files called "checkouts" files. Each time you run cvsup, it reads your checkouts files to see which files (and which revisions of them) you have. As it updates your files, it also updates the information in the checkouts files.
Confusingly, checkouts files are also sometimes referred to as "list" files.
It is not a big problem. If CVSup can't find a checkouts file that it needs, it falls back on other methods of determining which files you have. One such method is to compute checksums (MD5 file signatures) for each of your files, and use those to figure out which file revisions you have. This is perfectly safe, but it is inefficient. It slows down your update and also puts a heavier load on the server.
CVSup will detect the problem and quit, with a message suggesting that you delete the corrupted file and try again. If you follow the suggestion, it will complete your update using the fallback methods mentioned above, and recreate your checkouts file for next time.
There is just one thing that can go wrong, and it's not very serious. CVSup will only delete files that are listed in its checkouts file. Thus if you lose your checkouts file, and then a file "foo" is deleted on the server host, and then you run CVSup, your file "foo" will not be deleted.
Files are never supposed to be deleted from a CVS repository, so this isn't much of a problem in real life. But to be safe, you should run CVSup sooner rather than later, if you find that you've lost a checkouts file.
If you are careful and if you understand what is going on behind the scenes, you can make this work. CVSup was designed to allow extra locally checked-in revisions to coexist in the RCS files along with the revisions taken from the master repository on the server. Because CVSup understands the structure of RCS files, it is able to bring in new revisions from the server without disturbing your own revisions that you have checked in locally. Unfortunately, certain logistical issues make this capability awkward to use in practical situations.
To keep local revisions in your copy of the CVS repository, you have to:
We cover these topics separately below.
Simply remove the "delete" keyword from your cvsupfile. When present, this keyword gives CVSup permission to delete extra revisions from the RCS files. It also permits CVSup to delete entire RCS files, provided that it created them in the first place. By removing the "delete" keyword, you can prevent CVSup from removing extra revisions as well as entire RCS files. However, it is very important to understand the consequences of removing the "delete" keyword.
First, suppose that one or more RCS files are intentionally removed from the CVS repository at the master site. Without the "delete" keyword, CVSup will not propagate these file removals to your site. Thus you will have some RCS files in your repository that otherwise shouldn't be there.
In an ideal world, this would not be a problem. That is because, in an ideal world, one never deletes an RCS file from a CVS repository. Thus, the situation should never arise.
Unfortunately, most administrators don't manage their repositories quite so idealistically. Committers make mistakes and check files into the wrong locations. Most administrators will manually repair such errors by moving the files in the repository, rather than live with the mistake forever. As another example, in any large repository, some files eventually become completely obsolete. Eventually, developers will complain about the wasted disk space and the clutter in the repository, and the repository administrator will respond by deleting the files.
Usually, undeleted RCS files don't cause any problems, and they can be safely ignored. This is especially true if the repository administrator at the master site has taken care to mark the files "dead" on all branches with "cvs remove" a few weeks before completely removing the unwanted RCS files. Nevertheless, extra RCS files can cause problems in some cases, and you need to be aware of that.
It is difficult, because CVS chooses the revision numbers itself. One way around this is to modify CVS slightly. Before discussing that, though, the first rule of thumb is to create a new branch to hold your local changes. Don't try to check your local changes into the main branch or a branch that exists in the master repository. Doing so would make revision number collisions very likely, sooner or later.
If your local revisions are on their own branch, then the problem is reduced to ensuring that your branch has a unique revision number which will never be duplicated in the master repository. The easiest way to accomplish this is to modify CVS. The version of CVS released with FreeBSD includes such a modification. In that version, you can influence the revision numbers of branches by setting the environment variable CVS_LOCAL_BRANCH_NUM. This variable should be set to an integer value, and CVS will use that as the starting point when choosing revision numbers for new branches. By default, CVS allocates branch numbers starting with 1. So a high value such as 1000 makes a good choice. New branches at the master site will receive low revision numbers, while your own local branches will receive high revision numbers. Thus the two won't conflict.
When using this method, it is of the utmost importance that committers do not set CVS_LOCAL_BRANCH_NUM when creating branches in the master repository.
If you don't have a version of CVS that supports CVS_LOCAL_BRANCH_NUM, it is still possible to avoid conflicts with little cooperation from the master site. Simply create a branch in the master repository, declare it to be reserved for local modifications, and leave it unused in the master repository.
Make an empty directory somewhere. We'll call that directory base. Cd into base and do a mkdir -p sup/test.
Next, cd into sup/test and create a file named releases with one line in it like this:
cvs list=list.cvs prefix=prefix
In the above, replace prefix with the absolute path to some
CVS repository on your machine, e.g., /usr/cvs. If you don't
have a CVS repository, you can use any directory that has some files
in it. But RCS files are the best for testing purposes.
In the same directory as the releases file, make a file named list.cvs. It should contain a single line:
upgrade src/bin
Here, replace src/bin with the path to some reasonably-sized
subtree of your CVS repository, relative to prefix.
For example, if prefix is /usr/cvs in your
releases file, and the line in list.cvs is as above,
then the transferred subtree will be /usr/cvs/src/bin.
You have just created a CVSup collection named "test" with a release named "cvs". You can run the server like this:
cvsupd -b base
replacing base appropriately with the pathname of the
base directory that you created above. If you run it this way,
cvsupd will print its log messages to stdout. It will serve
exactly one client and then exit. To run multiple tests, you have to
restart the server each time. Alternatively, you can run the server
like this:
cvsupd -b base -C 1 -l /dev/stdout
and it will become a daemon and serve clients indefinitely until you
kill it manually. Note, it doesn't matter what your working directory
is when you start the server.
Now, create a separate empty directory where you'll run the client to receive the updates from the server. We'll call that directory dest. In dest, make a file supfile that looks like this:
*default host=localhost
*default base=.
*default release=cvs
*default delete use-rel-suffix
test
(Notice that there is a "." in "base=.".)
Make sure the server is running, and then, still in dest, run
cvsup in the usual way. The simplest command is this:
cvsup supfile
but you can add -g -L 2 to disable the GUI if you wish.
If your CVSup client was built with the GUI, press the start button. The non-GUI client starts automatically. At this point, your disk light will turn into a 50-watt beacon and your hard drive will make that noise that says, "I'm really really really busy." When it's done, you should find the updated files under your dest directory. There will also be a directory sup which is used by cvsup to record its state.
Assuming that worked, you can get arbitrarily thorough in testing the rest of it. By manipulating the source RCS files, you can add deltas and/or tags, and then make sure they propagate when you do another update. You can also try deleting RCS files entirely, and adding new ones. On the client side, you can add tag= and/or date= specifications to the supfile. Generally, if the software works at all on a given platform, it's likely to work completely. Almost all of the functionality is contained in OS-independent code.
One caveat: Doing an update all on one machine via localhost really hammers a system, particularly the disk subsystem and the network stack. It can expose OS bugs that nobody knew existed before. If you run into networking-related problems, it might help to use -P m on the cvsup command line when you do your updates.
The server configuration files used by the FreeBSD project are available via CVSup from any FreeBSD mirror site. To fetch a copy, get into an empty directory and make a file named supfile containing this text:
*default host=a.mirror.site compress
*default release=cvs tag=.
*default base=.
*default delete use-rel-suffix norsync
distrib
Replace a.mirror.site with any CVSup site
listed in the FreeBSD Handbook. Run the CVSup client using this
supfile. When it finishes, you'll find the configuration
files in the subdirectory distrib/cvsup. If you wanted to
use these configuration files directly, you'd use that subdirectory as
your base directory, like this:
cvsupd -b distrib/cvsup ...
In a real setup, you'd specify the base directory as an absolute
pathname, of course.
The basic problem is that CVSup is written in the programming language NotC. What is NotC? Well, to most people, it really doesn't matter. The important thing is that NotC is not C. Consequently, it takes a little extra work to use it.
In the case of CVSup, NotC means Modula-3. Modula-3 is an efficient compiled programming language with excellent built-in support for important features such as garbage collection and threads. For more information about Modula-3, see the Modula-3 Home Page.
Fortunately, Modula-3 ports exist for many platforms. If there is already a port for your platform of choice, then you've already cleared the biggest hurdle to building CVSup.
A second problem that many people run into is that building the Modula-3 compiler and runtime system requires a lot of virtual memory. You will need on the order of 64 MB of available virtual memory to build it. (Building CVSup itself requires less than that.) You will also need to ensure that your resource limits are set high enough. These are controlled by the "ulimit" command in sh-like shells, or by the "limit" command in csh.
Third, by default you have to have X11 on your system to build the Modula-3 runtime. It is possible to build Modula-3 without X11, but to do so you have to edit the file m3/src/m3makefile and comment out the packages that depend on X11.
No, CVSup does not currently work on Windows platforms. It relies on a number of features which are specific to POSIX-like operating systems.
No. All releases of CVSup that have ever been made public will interoperate with each other. You can use an old client with a new server, or a new client with an old server. Of course, the feature set is limited to what is supported by the older component. So it is always a good idea to upgrade both the client and the server when a new version is released.
No. Although the CVSup client can use supfiles written for SUP, the two packages are otherwise incompatible.