[coyotos-dev] Change to Coyotos build passes...

[Jonathan S. Shapiro]

I'm increasingly unhappy with the Coyotos build mechanism, because it
requires far too many passes and too many of these passes are
interdependent. I'm coming to the unhappy conclusion that we need to
revisit the tree structure to simplify it.

A full build of the Coyotos tree is extremely complicated:

  CROSS TOOL BUILD
    1. Compile binutils for the target
    2. Compile a "non-hosted" gcc that will be used to compile libc
    3. Compile libc
    4. Now compile a full version of gcc
    5. Compile any appropriate "buildhost" components such as libraries
       and binaries that should live in the cross environment. These may
       have internal ordering dependencies.

       At present this includes only libgc, but I will soon add astmaker
       and bitcc to this list.

  ** For most developers, the items above will never be seen because
     they can install the cross-tool RPMs. This is good, because the
     necessary build process is really horrible.

  ACTUAL COYOTOS BUILD

     Notation: actually implemented passes are digits. Logically
     distinct passes that are merged as sub-passes are indicated with
     letters.

  1.a IDL publication.
  1.b IDL stub header generation -- these may be input to library source
        files
  2.a Library header publication -- in theory, libraries may have cross
        dependencies across their interfaces
  2.b Library compile and install.
       NOTE: In practice there is a reasonable sequential build ordering
       for libraries, so we haven't actually separated these.
  3.a Application compile and install
  3.b Publish the per-application 'map' files
  4.  System image construction (if appropriate)

This is undeniably complicated. It explains why our "default" make
target always re-builds an entire package (e.g. base) in order.

Interacting with all of this is a belief that I have held about how
builds should be conducted: one part of a system should not reach into
another part of a system to grab some internal header file unless there
is really a magical, internal interface between the parts. Such "magic"
interfaces should be firmly avoided.

My rationale for this is that you want to test in the same build
environment that your client will use, and you cannot really do this
unless you go through the effort of installing things into the output
tree and then obtain them from the output tree.

I have come to the conclusion that this is unreasonably conservative,
for several reasons:

  1. It is complicating the build.
  2. It ignores the fact that well-structured applications already
     tend to have internal subtrees

	include/
	src/
	...
     that adequately capture the "output tree compatibility" issue.
  3. In Coyotos, the non-library cross dependencies are mostly subsumed
     by IDL, which simplifies the problem quite a lot.

So:

I have in mind to do a significant re-arrangement of the tree with the
following goals:

  1. Significantly reduce the build recursion depth
  2. Impose the structure that public headers go in a reasonably
     isolated subdirectory for ease of installation.
  3. Allow cross-directory -I and -L within a single package
     so that it is not necessary to type "make install" everywhere.

     This implies that it may be necessary to add makefile
     dependencies that get input libraries built when this is necessary!


Reactions? Input?


shap