Re: Packaging system effort

[ibotty <me@xxxxxxxxxx>]

Simon 'corecode' Schubert wrote:
> Thoughts about a packaging system

things i agree to are omitted, this mail is long enough even so.


> As the system should be easily usable and not just academic, easy tools
> are
> strongly needed. This includes tools for all sorts of maintenance: from
> updating
> descriptions over searching to upgrading installed packages. It is most
> desirable to also provide graphical tools (ncurses, X11, web), or at
> least
> provide infrastructure so that third parties can easily develop such.

yes, 
i vote for a very small set of simple package installation+building tools
+ an extra layer, that cares about getting packages from the network.
+ an extra layer, that cares about the novice user (X, ncurses, etc.).


> Debian's way of providing -dev packages which have been splitted off is
> always a
> highly controversial point of discussion. This is why I want to comment
> on this
> here. For a source based system as ports etc. having header files
> available is
> unavoidable, and also when using binary packages the bloat due to
> header files
> and static libs is small.

well, about 2 times the size of the binary package...

> Still there may be cases where every file
> must be
> considered, so having the possibility to prune development files if
> feasible
> might be a nice add-on. Same goes for foreign gettext language files
> etc. This
> could all be implemented via global package flags. What I'm opposing is
> the
> creation or use of additional packages for -dev headers/libs. The
> number of
> distinct packages should be kept to a minimum.

yes, a -dev flavor is best.



> As a direct conclusion the system must have strong binary package
> distribution
> support. In the past a lot of people demanded a streaming binary format
> to have
> the ability to install packages straight whilst downloading without
> having to
> wait for the whole package. 

i can think of another feature, that *might* be implemented with streaming
binary packages.
reading the changelog beforehand is really nice.
but we will nevertheless force users to have our ports tree at hand, so this
might be dumb.


> Another important aspect is a powerful build system. It should be
> possible that
> multiple packages are being built at the same time and get synchronized
> for
> installation etc. It's just PITA if you're compiling KDE or OpenOffice
> and can't
> build/install a small package like mpg123 or irssi because this might
> damage the
> package db. Existing systems handle such cases nice most of the time,
> but that's
> just luck. Pkgsrc implements locking as far as I remember.

parallel builds are needed to use multiprocessor machines (or SSI).
so this should be done.


> Display of the build progress is a nice add on for users for sure. This
> can be
> both implemented in a macroscopic way (x of y packages built) and
> microscopic
> (anybody wanna hack make for SIGINFO?).

? make with progress bar?
not really unix, huh?
saying "now building package 2/6" is enough i guess.

 
> Speaking of compilation for slow boxes: Cross compilation comes to
> mind. Is this
> needed when distcc support exists? Discussion point here.

yes, i think so. 
(speaking of cross-building to different arch)
say you are triing to port dragonfly.
you might at least want a sh and init.
bootstrapping is way easier with crossbuilds.


> It should be possible to build and install packages as an unprivileged
> user.

absolutely required!


> Sometimes local security policy or laziness of an admin demands the
> installation
> of a package into the user's home dir. A nice point would be native
> support for
> such in the packaging system. This doesn't mean that binary packages
> need to be
> relocatable into home dirs, but the system would need to provide an
> alternative
> (user home) location of package registry.

i like that, but i am not sure, it is really needed.


> An essential duty of a packaging system is the tracking of installed
> files. It
> must be an easy task to remove a package and thus all its installed
> files from
> the system. The system needs to provide collision management (same file
> installed by several packages, VFS voodoo?) and configuration file
> awareness
> (see below). Compare with portage (automatic list building) and ports
> (ugly
> manually generated plists).

dynamic plists are a requirement!


> The packaging system descriptions shouldn't consume too much space in
> general
> and inodes in specific. It's just horrible to have a myriad of small
> files and
> directories in your /usr (or whatever) wasting a big deal of inodes.
> This goes
> for end users. Package maintainers could have a different view of the
> description which could be collapsed into less files later. A possible
> approach
> could be one description file per available package and version plus
> approximately one patch file for each version (vs. patch-per-file in
> ports).

imo, cutting file count looks like premature optimization, we should better
fix the filesystem to not waste space with small files (as is done in
reiserfs on linux)
(but i agree to you separation (except patches). it makes sense.)
 

> This leads to patches. As the system should aim to be OS agnostic in
> most parts
> this also counts for patches. These should be specially crafted so that
> they at
> best don't interfere with the build process on other platforms/OSes.
> This means
> extensive use of #if defined(__MyOS__) etc.

this is ugly. maybe patches names as 'MyOS_MyArch_01.patch' or such may be
useful.
we would need #if defined(__MyOS__) frequently enough, so i guess this
should be avoided.


> This portability is the key to a close communication and development
> with the
> upstream authors. It should be policy that patches are to be written as
> cleanly
> as possible and have always to be submitted upstream. The packaging
> system might
> provide help in or even enforce this process. Having patches go in
> upstream
> reduces needed files, enhances overall acceptance of the packaging
> system and
> also provides people not using the packaging system with features and
> fixes.

this is not the goal of ports as today? uuaahh!

 
> The system should provide support for bug tracking so that users can
> easily
> check for known bugs and report new ones or add followup information for
> existing ones. As the bug tracking system should be closely integrated
> with the
> system, bugs need be associated with packages or specific package
> versions. This
> helps maintainers and committers to follow user input better than the
> GNATS/CVS
> decoupling ports is currently using.

so you do not want gnats/bugzilla or such?
i would advocate for a cooperation between package tools and bug database.
(say `pkg_bug kde-3.2`)

 
> The system should be able to bootstrap itself. This means it shouldn't
> depend on
> the system tools be included with the host OS. If the system undergoes
> changes
> the tools need to change too. As seen with ports several times having
> the tools
> in the base OS only complicates stuff and leads to legacy issues (e.g.
> tbz/tgz
> issue). Pkgsrc provides this in a nice way. Bootstrapping also means
> registering/tracking installation of the package system (and
> requirements of
> it). This seems like a chicken-and-egg problem but it can and should be
> solved.

yes, package tools in package system is best.
bootstrapping should not be a problem, as a simple (binary) tarball might be
enough.

 
> For e.g. ports a big
> show stopper
> concerning format conversion (remember pkg_info) and fast processing
> (see INDEX
> generation) is the fact that Makefiles are indeed interpreted and not
> just
> parsed. This means slowdown in processing and also problems in automatic
> conversions - you never know how creative a maintainer was in (ab-)
> using make.
> This leads me to one conclusion: Don't use an interpreted file format.
> Use a
> standardized description format which only needs to get parsed. This is
> much
> faster and more portable if multiple programs intend to work with the
> descriptions. Not having a turing-complete language to use when writing
> a
> description might at first need some change in thinking (when moving
> from ports
> or portage) and will involve the need of writing more text/data (like
> no more
> using ${variable:C/pattern/replacements/}) but will help overall
> cleanness.

i partly agree, but i would vote for both. parseable metadata, interpreted
install scripts.
but thats just me, no real argument pro that ;)


~ibotty