Re: Compatability with FreeBSD Ports

[Chris Pressey <cpressey@xxxxxxxxxxxxxxx>]

On Tue, 16 Aug 2005 01:19:44 +0200
Joerg Sonnenberger <joerg@xxxxxxxxxxxxxxxxx> wrote:

> On Mon, Aug 15, 2005 at 03:55:07PM -0700, Chris Pressey wrote:
> > That's not part of package installation, in my view.  That's part of
> > package configuration.  All pkg_add should be doing is putting the
> > software on my hard drive.  It should not be setting it up for me.
> 
> Package installation can consist of things like restarting daemons.

Then have a "daemon" class for this purpose, and have it's "upgrade"
method issue "kill -HUP ${pid}".  It still doesn't require executing
arbitrary commands.

> Look at Debian, you might or might not agree with it, but it is
> difficult to clearly cut lines. Also keep in mind that the
> INSTALL/DEINSTALL scripts are certainly not the most dangerous parts
> of a package.

But that's not a reason for them to _stay_ dangerous.

> Limiting their use by anything but policies can easily
> hurt for those cases where you really need the advanced possibilities.

Joerg, I will be happy to reconsider my position if presented with
evidence to the contrary - such is the reason for debate, after all. 
But first I would have to see such evidence.  Can you give examples of
these "advanced capabilities" and name some packages that require them?

> > > If you do bother about which commands will be executed, check the
> > > scripts first.
> > 
> > I shouldn't have to audit 'N' individual package-scripts when all
> > that is needed is to make the *one* package system running them
> > secure.
> 
> That's like saying "I don't need to audio 'N' individual userland
> programs, whenn all that is needed is to make the *one* kernel running
> them secure". It's been tried a lot, most attempts failed.

This analogy is specious because all userland programs do *different*
things, while all packages have the *same* things done to them -
"be installed", "be removed", "be upgraded".

> > One of the reasons our package tools (and many other tools) are not
> > very good is because package management is not essentially a
> > "command scripting" problem (nor are many other problems.)
> 
> bsd.pkg.mk is NOT about package management. It is about building
> packages.

When I said "our package tools" I meant the whole shebang: everything
that was derived from JKH's code, bsd.*.mk and pkg_*, in both ports and
pkgsrc.

> Don't forget this difference, it is important.

I'm quite well aware of the difference.

I have problems with multiple aspects of the package system.  Insofar as
they are seperate aspects, they are seperate problems.  But insofar as
they stem from the common philosophies under which each aspect was
designed and has evolved, they're the same problems.

To be clearer on some (not necessarily all) points, I find:

- pkg_*(1)'s methodology (specifically, directives in plists) to be
  messy and insecure;
- make(1) to be the wrong "language" for anything as complex as what
  ports/pkgsrc has grown into;
- that the two exhibit an undesirable tight coupling.

> Building is exactly nothing but scripting a bunch of commands.

Building is nigh-always already taken care of by the 3rd party package. 
Issuing "./configure && gmake" is trivial, and if that's all there was
to bsd.pkg.mk it wouldn't be ~5000 LoC.

So to say "bsd.pkg.mk" is "about building packages" is at best slightly
misleading.  Importantly, the building part itself is not where the
complexity is.  The complexity is "out there", in the filesystem ("where
shall we put the include files and libraries?") and the 3rd party
programs themselves ("where is this expecting the include files and
libraries to be?")  This part of it is clearly NOT "just scripting a
bunch of commands".  No programming language in the world can address
this complexity - only policy can do that ("include files and libraries
are always located here and always named like this.") There may,
however, be some programming languages more suited to expressing these
sorts of policies than others.

> If I can once more stress the comparision with Debian, it's the
> difference between dpkg-buildpackage and dpkg/apt-get. There are a lot
> of things to solve on the *package* management level, porting apt-get
> would be a start.

I have nothing to say about apt-get.  My criticism is of ports/pkgsrc.

> But don't mix or confuse them with with *build* level problems.

I certainly am striving not to; I've seen firsthand the mess that has
resulted from ports doing so.  Pkgsrc does not yet seem much better.

> > > I've looked at this before and make is about as good or bad as
> > > many other solutions.
> > 
> > How many of the tools that you have looked at are not essentially
> > imperative?  Anything functional, deductive, declarative, dataflow,
> > multi-paradigmal even?
> 
> No paradigm shift simplies the build problem, asI've already written.
> You can improve make e.g. when looking at ant or jam, but the
> fundamental solution is still a combination of build rules and
> dependency tree. make might not be the perfect tool for something like
> pkgsrc, but many of the difficult-to-grasp elements like lazy
> evaluation are actually needed there. I have written by own Linux
> distribution, I have started to write a build system in plain shell, I
> know Python well enough to ignore it for this. A lot of things might
> be simpler for pkgsrc when moving e.g. to jam, but it wouldn't change
> the basic complexity.

I think you are confusing design with implementation here.

*Of course* choice of tool will not simplify the underlying problem. 
*Better design* simplifies the underlying problem.  Better tools
simplify the *implementation*.  *Either one* would simplify the
resulting system.  I take issue with *both*.

Re implementation (which is what I was originally focusing on at this
point):

First you said "make is about as good or bad as many other solutions",
now you say "You can improve make e.g. when looking at ant or jam [...]
A lot of things might be simpler for pkgsrc when moving e.g. to jam".

I'm not certain what to take from this.  Which do you believe?

> There are way too many irregularities out there.

Yes, and adding make(1) syntax on top just adds more irregularity.

> > If the tools were made very simple, their evolution might even be in
> > the hands of the users.  Imagine that!
> 
> Just because a tool is easy to use, doesn't mean it is easy to
> understand.

If I claimed that, I misspoke; but I don't think I claimed that.

> Just an example from the subject I study: the integrate
> command of Computer Algreba Systems is very easy to use. You give it a
> term, the independent variable and maybe also the limits, it computes
> the closed-form of the integral if possible. The actual algorithm has
> the size of the Bible, even Mathematica and Maple needed *years* to
> fix most bugs.  Does that make the tool of symbolic integrate less
> useful?

Less useful?  No.  Easy to use?  Yes.
More expensive?  Yes.  More bug-prone?  Yes.

> Does it make the tool more complicated?

Internally?  Yes.  Externally?  No.

But I totally fail to see how this analogy applies to our discussion.

Was the point of it to show that for a system to be useful and have a
simple interface, it must be internally complex?  If it was, I think it
fails to show that.

(As an aside: assuming you're talking about the Risch Algorithm, the
implementation in Maxima appears to be only ~1000 lines of LISP code.
I don't know how many bugs it contains, nor how useful it is compared to
Mathematica's, though.)

> > Is it acceptable to you that the average developer (never mind the
> > average user!) is not expected to be able to understand the tools
> > that they use?  What is the point of the source being "open" in this
> > situation, when it's still effectively "closed" to most people?
> 
> There is a clear structure of steps taken during the processing. If
> you follow this steps, it normally becomes quite clear what the code
> is trying to achive.

Only if you possess all of the following things:

- intimate familiarity with the nature of each step (in our case, the
  individual commands;)
- intimate familiarity with the flow of execution between steps (in our
  case, how execution happens in make and sh;)
- intimate familiarity with what each symbol being worked upon
  represents (in our case, the syntax, and variable and target naming;)
- the mental capacity to visualize the cumulative effects of each step
  on the global state (in our case, the filesystem and environment;)
- the patience to trace out however many steps might be involved
  (in our case, maybe in the dozens to hundreds range per package.)

Very few people are good at all of these things.  If they were, there
would be no user/developer disparity... and there would probably be a
lot fewer bugs in the world.

> The big problems here are the various edge cases,
> which are obstacles to understand. But exactly those edge cases create
> the power of the system and allow the simple descriptive Makefiles in
> most ports/packages.

I don't follow you here.  Could you please give an example of one of
these 'edge cases'?

> > If it is acceptable to you, then I'd like to know: how do you
> > reconcile this with your position of "if you're worried that a call
> > to pkg_add might trojan your machine, you should audit the install
> > script yourself before doing it"?
> 
> I don't like code which is too complicated to understand. I agree with
> you on that. But sometimes complexity is necessary.

Well, it's hard to disagree with a statement like that.

You haven't convinced me that the complexity is necessary here, though.

(You also haven't answered my question -- again.)

> That's what documentation is for.

While documentation is critical too, I'd much rather reduce complexity
by good design and appropriate abstractions, than try to compensate for
it with documentation.

-Chris