DragonFly kernel List (threaded) for 2004-06
a proposal for the packaging system
Sorry, this is long, I appreciate your patience.
I have been following the progress of DragonFlyBSD since the beginning and I
must tell one of the things that get me most excited about it is the prospect
of real innovations regarding package management.
However, the current discussion on the matter seems to go nowhere everytime, as
you pointed out. This is not suprising, as a packaging systems finds itself
naturally trying to satisfy everybody while having to stay simple. So guys,
everybody is right, and that's the problem.
If we want to get things a bit clearer, we must deconstruct the problem. A
packaging system as a whole is roughly made of the following parts:
1) The way packages are instaled on the filesytem. (location, pkgdb, ...)
2) The basic tools responsible for taking a package and installing it in the fs.
3) The package format
4) The port/package collection its structure, organisation and content.
5) A build system, package generating system, tools for fetching distfiles,
misc. ancillary tools, etc...
6) Package management tools providing high level functionalities based on the
"primitive" provided by point 2 and 4.
The reason it ends in a flame war everytime is that everyone talks about a
Now, serious work should be to isolate the pros and cons of each solution at
each level. Ex.: Debian may be better in the packaging format (dpkg) and
obviously fbsd ports is better with its build system (build time options).
For every improvement proposed, it has to be shown what level would be affected
and the consequences weighted on cost/benefit. Maybe someone should start a
wiki on the subject and people would submit their proposal they already exposed
in the list in this formalised way. Then, it'll be easier to take decisions.
Hastilly ditching freebsd's port collection (acting at level 4) would be
kamikaze, IMHO. We can still act at other levels, especially level 1 and 2.
In the meantime, I reread Matt's memo on packaging and obviously the bitterest
grief with current solutions is the inability to do something like install an
new version of libncurse with some funny option without breaking the
application that relie on a "normal" install of it. Now, that would be some
_innovation_, and that would definitely differenciate DBSD from any other unix.
Now the debate is wether this is achievable only through "VFS Voodoo" (level
1), or there is a way to do that with a new wizardry of the port system (level
IMHO, the second option is bound to fail like all others before, because it
amounts to try to please every user and be suitable for every use. We know this
is not achievable. The debian ports are the closest to "success" in this
approach, yet still fail. And starting from scratch is costly.
The VFS Voodoo approach needs just dealing with level 1 and 2, which are
entirely in the os and not in the port collection, in a way that would be
transparent not only to the port collection, but to the higher level tools such
as portupgrade as well.
Matt's Filesystem Overslay proposal looks daunting, but it can be implemented
in a simple, generic, and I hope, elegant way, which I'm going to try to
Filesystem Overlay for DragonflyBSD:
A special db store lists of filesystem overlay. Each list describes a single
overlay: the list names the directory or file which it applies to and a list of
replacement directory or file (must be same type). Each list has a unique
identifier, which should allow alphanumerical name.
When the executable file is runned, its environment is set up to indicate the
overlay that applies to it (read-only var) When it tries to read the overlaid
dir (let's say /usr/local/lib), the system notice the overlay, fetch the proper
list, and tries each listed dir entry in turn (order matters), and stop at the
_first_ matching one. If none match, then the actual /usr/local/lib is tried
So when you install a new version of ncurse that is not supported by your, say
midnight commander, what happens?
The installation tool (level 2) notice the already installed version of the
same port, finds in the pkgdb that midnight commander needs it. So it move the
installed package to a new directory /usr/local/prev/libncurse-1.5-xxx/...
where xxx is a local version number derived from the date, updates the pkgdb
entry of the old libncurse to match the new position, updates its name to
include the new local version number, updates the other pkgdb entries to point
to this new package name, and create a overlay list applying to /usr/local/lib
pointing to /usr/local/prev/libncurse-1.5-xxx/, and finally tags the midnight
comander executable(s) with this list.
At this stage, room has been made for the new package which can be installed
A cron job can scan through the fs and the special db to remove dangling
entries, etc... when packages have been removed.
The nice thing is that system is not limited to ports and can be used even for
surprising goals, like managing access allowed to an executable. It suffice to
put an overlay on, say, /etc/ refering listing /var/overlays/myprog/ and put a
passwd file in that latter dir with perm at 700 and this program only cannot
read /etc/passwd, because the system hit /var/overlays/myprog/passwd first and
uses its authorisations.
As a conclusion, this system ends up _creating packages_ (new versions tag) out
of known working combinations of old-new ports. This ends up being much more
natural than to trying to enforce it in the ports. Its a bit like capitalism
versus planned economy, if I dare.