Building FreeBSD in meta mode

This doc describes the process of converting the FreeBSD build to use bmake in meta mode.

Note: while I normally talk of meta mode and using dirdeps.mk together, either can be used without the other.

The initial commit of bmake went into FreeBSD head in October 2012. The meta mode work described below is currently in a branch projects/bmake.

Why meta mode

An obvious question, with a very simple answer: to improve the build.

There are several aspects to this:

more reliable update builds

The .meta files created during a build, allow bmake much greater visibility into what happens during the creation of a target. This allows it to be more thorough when deciding if a target is out-of-date.

captured output

The .meta files capture the output from the commands that generate a target.

By default, this allows a much cleaner - and easier to read build log, and even for people who refuse to log the output of builds, the vital clues are captured when something goes wrong.

.ERROR

The .ERROR target is run when bmake hits an error, and if it was while generating a target, we can capture a copy of the relevant .meta file in a well known location.

This then becomes a simple and reliable means of spotting how and why a build failed.

tree dependencies

The .meta files capture lots of useful data, which can be used in many ways, but one of the most obvious is learning the dependencies of one directory on others.

simplify the build

This is one of the key benefits. This is mostly a benefit of using dirdeps.mk, though many individual makefiles can be simplified by use of meta mode.

The use of meta mode also allows easy automation of the data needed by dirdeps.mk, so we usually just refer to this as the meta mode build. The dirdeps.mk or meta mode build is very easy to understand.

Look at the top-level makefiles of any BSD system today, and try to understand exactly what they do. It can be a challenge indeed. Ignoring comments and blank lines, FreeBSD's top-level Makefile and Makefile.inc1 have more than 1550 lines.

The pre-meta-mode Junos build had over 5000 lines in its top-level makefiles.

By contrast, Makefile, pkgs/Makefile and pkgs/Makefile.inc which drive the Junos build today total only 117 lines.

Of course I am not counting nearly 10k autogenerated Makefile.depend* files which make it possible, but

a/ they are autogenerated and each by itself is easy to understand

b/ they are not only used for top-level builds

In fact in this model, the top-level build is no different from any other.

Not to mention that top-level builds can often be dispensed with.

We use the example of being able to make -C bin/cat in a clean tree below.

Using bmake

The first step is to be able to build FreeBSD using bmake as a replacement for /usr/bin/make. This is actually quite simple.

Conflicting modifiers

Several years ago FreeBSD make adopted :U to upper case a value, and :L to lowercase it, from OpenBSD. At about the same time I added a raft of modifiers from ODE to NetBSD's make. These included:

:@temp@string@
     This is the loop expansion mechanism from the OSF Development Envi-
     ronment (ODE) make.  Unlike .for loops expansion occurs at the time
     of reference.  Assign temp to each word in the variable and evaluate
     string.  The ODE convention is that temp should start and end with a
     period.  For example.
           ${LINKS:@.LINK.@${LN} ${TARGET} ${.LINK.}@}

     However a single character varaiable is often more readable:
           ${MAKE_PRINT_VAR_ON_ERROR:@v@$v='${$v}'${.newline}@}

:Unewval
     If the variable is undefined newval is the value.  If the variable
     is defined, the existing value is returned.  This is another ODE
     make feature.  It is handy for setting per-target CFLAGS for
     instance:
           ${_${.TARGET:T}_CFLAGS:U${DEF_CFLAGS}}
     If a value is only required if the variable is undefined, use:
           ${VAR:D:Unewval}

:Dnewval
     If the variable is defined newval is the value.

:L   The name of the variable is the value.

:P   The path of the node which has the same name as the variable is the
     value.  If no such node exists or its path is null, then the name of
     the variable is used.  In order for this modifier to work, the name
     (node) must at least have appeared on the rhs of a dependency.

At a later point NetBSD added:

:tl  Converts variable to lower-case letters.

:tu  Converts variable to upper-case letters.

While the FreeBSD ports collection uses :U and :L quite a bit, the base system does not use them at all.

Thus building FreeBSD itself using bmake is mostly a matter of adding .NOPATH statements in a couple of places; to curb its enthusiasm for finding things via .PATH.

The diff to allow bmake -jN buildworld to work was less than 300 lines (not counting comments).

meta mode

Being able to do buildworld is a useful stepping stone, but isn't very interesting.

Being able to make -j8 -C bin/cat in a freshly checked out tree, and have it correctly build all the libs etc. in one pass, is more like it. Which is where meta mode comes in. The output below shows the directories checked - only things needed by cat get built:

Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/gnu/lib/libssp/libssp_nonshared for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/xlocale for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/rpc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/rpcsvc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/csu/amd64 for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/libcompiler_rt for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/libc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/gnu/lib/libgcc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat for amd64 ...

Perhaps also interesting is that we can get the build to show us the complete tree dependency graph from any starting point. For example:

$ build-graph -C bin/cat
bin/cat.amd64: gnu/lib/libgcc.amd64 gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/xlocale.amd64 lib/csu/amd64.amd64 lib/libc.amd64 lib/libcompiler_rt.amd64 pkgs/pseudo/stage.amd64
gnu/lib/libgcc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/xlocale.amd64 lib/csu/amd64.amd64 lib/libc.amd64 pkgs/pseudo/stage.amd64
gnu/lib/libssp/libssp_nonshared.amd64: pkgs/pseudo/stage.amd64
include.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
include/rpc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
include/rpcsvc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
include/xlocale.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
lib/csu/amd64.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 pkgs/pseudo/stage.amd64
lib/libc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/rpc.amd64 include/rpcsvc.amd64 lib/csu/amd64.amd64 lib/libcompiler_rt.amd64 pkgs/pseudo/stage.amd64
lib/libcompiler_rt.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 pkgs/pseudo/stage.amd64
$

The build-graph script just leverages the debug information that dirdeps.mk can output as it examines the Makefile.depend* files.

tree dependencies

Everyone is familiar with doing make depend to capture the dependencies that a directory has on the files that it uses, to help ensure that re-running make will do the right thing.

What we need is the same level of information for the tree as a whole. As detailed in Building BSD with meta mode I've implemented that a couple of ways in the Junos build, and meta mode is the current method used.

meta files

In meta mode, for most targets bmake creates a .meta file to capture information like the expanded command used, any command output (useful for debugging), and a record of all the successful system calls that are interesting to make:

# Meta data file /var/obj/projects-bmake/amd64/bin/cat/cat.o.meta
CMD cc -O2 -pipe  -nostdinc -isystem /var/obj/projects-bmake/stage/amd64/usr/include -isystem /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2 -std=gnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -c /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
CMD ctfconvert -L VERSION cat.o
CWD /var/obj/projects-bmake/amd64/bin/cat
TARGET cat.o
-- command output --
-- filemon acquired metadata --
# filemon version 4
# Target pid 42504
# Start 1363631731.803698
V 4
F 42504 42529
E 42529 /bin/sh
R 42529 /var/run/ld-elf.so.hints
R 42529 /lib/libedit.so.7
R 42529 /lib/libncurses.so.8
R 42529 /lib/libc.so.7
S 42529 .
S 42529 /var/obj/projects-bmake/amd64/bin/cat
S 42529 /usr/bin/cc
F 42529 42530
E 42530 /usr/bin/cc
S 42530 /usr/bin/cc
S 42530 /usr/bin/cc
F 42530 42533
E 42533 /usr/bin/cc
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2
R 42533 cat.o-24d10fa0
W 42533 cat.o-24d10fa0
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/cdefs.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_null.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/types.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/endian.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/endian.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_pthreadtypes.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_stdint.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/select.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sigset.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timeval.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/syslimits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/stat.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/time.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/socket.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_iovec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sockaddr_storage.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/un.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/errno.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/runetype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/err.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/fcntl.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stddef.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdio.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdlib.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/strings.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/unistd.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/unistd.h
M 42533 '/var/obj/projects-bmake/amd64/bin/cat/cat.o-24d10fa0' 'cat.o'
X 42533 0
X 42530 0
S 42529 /usr/bin/ctfconvert
F 42529 42541
E 42541 /usr/bin/ctfconvert
R 42541 /var/run/ld-elf.so.hints
R 42541 /lib/libctf.so.2
R 42541 /usr/lib/libdwarf.so.3
R 42541 /usr/lib/libelf.so.1
R 42541 /lib/libz.so.6
R 42541 /lib/libthr.so.3
R 42541 /lib/libc.so.7
R 42541 cat.o
R 42541 cat.o
R 42541 cat.o.ctf
W 42541 cat.o.ctf
M 42541 'cat.o.ctf' 'cat.o'
X 42541 0
X 42529 0
# Stop 1363631732.322849
# Bye bye

From this we can derive not only the equivalent data of make depend, but we can derive tree based dependencies.

By that I mean that any file read from an object directory which is not .OBJDIR represents a directory which needs to be built before .CURDIR, it's that simple. Of course having some consistency in the relationship between object dirs and src dirs helps. Even when that isn't possible though we can work around it.

In the example above, a number of headers are read from /var/obj/projects-bmake/stage/amd64/usr/include/. This is where we stage headers as we build the tree.

${STAGE_OBJTOP}/usr/include

As we build libs and other things which have headers to be installed, they get staged into this directory, a long with a .dirdep file to say who put the file there. For example:

/var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h
/var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h.dirdep

the .dirdep file contains:

include/xlocale.amd64

which says it was put there by the makefile in include/xlocale while building for the machine amd64.

Thus as we read the .meta file a find a file read, we can look for the same path with .dirdep appended to find the directory we should add to our dependencies.

${STAGE_OBJTOP}/include (deprecated)

The makefile in src/include/ does not fit the normal usage pattern, and fixing it at this stage isn't an option. So rather than use our normal staging method, we let it install into a directory which only it populates, and for each directory thus created we add a .dirdep file.

We now have a stage-install.sh wrapper script which allows running install and then deposits the necessary .dirdep files. This allows all headers to be staged to ${STAGE_OBJTOP}/usr/include.

meta2deps

Originally a shell script meta2deps.sh was used to extract the useful information from .meta files. It works fine. I should say worked fine - it hasn't had any attention recently.

If Python is available though we will use meta2deps.py which produces the same result, but much more efficiently (5-10 times faster) and can gather additonal data without overhead.

Makefile.depend

After processing the .meta files for bin/cat we end up with the following in ${.MAKE.DEPENDFILE}:

# Autogenerated - do NOT edit!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        gnu/lib/libgcc \
        include \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
        lib/libcompiler_rt \

.include <dirdeps.mk>

.if ${DEP_RELDIR} == ${_DEP_RELDIR}
# local dependencies - needed for -jN in clean tree
.endif

which shows a number of interesting things.

SRCTOP

Defines the top of the src tree. This along with OBJTOP and OBJROOT (eg. OBJTOP= ${OBJROOT}${MACHINE}) allow for being able to consistently refer to things in the src and object trees.

RELDIR

Being able to trim ${SRCTOP} from the start of ${.CURDIR} provides a useful relative location within the src tree. This forms the basis of DIRDEPS.

The .dirdep files mentioned above are created by simply doing:

.dirdep:
        @echo ${RELDIR}.${MACHINE} > $@

.PARSEDIR/.PARSEFILE

NetBSD make (bmake is just a portable version of it), defines .PARSEDIR as the directory from which the current .PARSEFILE is being read. This is extremely useful to us. As is the :tA modifier:

_PARSEDIR= ${.PARSEDIR:tA}

or the absolute path to .PARSEDIR.

As you can see in the example above we can use ${_PARSEDIR} and ${SRCTOP} to derive the RELDIR of the makefile being read.

We can also use .PARSEDIR as a clue that the makefile is being run by bmake rather than FreeBSD's older make:

.if defined(.PARSEDIR)
# we are bmake ...
.endif

DEP_RELDIR

Allows dirdeps.mk to keep track of the RELDIR it is gathering/computing dependencies for, and it gets automatically updated as each Makefile.depend file is read.

DEP_MACHINE

In the example above, there are no machine qualified entries in DIRDEPS, if there were, as in pkgs/pseuod/kernel/Makefile.depend for example:

DIRDEPS = \
        include \
        include/xlocale \
        usr.sbin/config.host \

the last entry indicates that usr.sbin/config must be built for the pseuo machine host. When dirdeps.mk is reading usr.sbin/config/Makefile.depend it will have set DEP_MACHINE=host which can influence filtering and other things as well as ensuring the correct build dependencies are created.

dirdeps.mk

This is a complex makefile, and I recommend against touching it.

This is what makes everything work, and thus the complexity pays off; because everything else (especially the bits people might need to touch) can be quite simple.

bootstrapping meta mode

The easy way to bootstrap meta mode, is to use the old build with top-level makefiles running in non-meta mode, but leaf directories running in meta mode. Thus we do not rely on meta mode for any of the sequencing (that only happens if the 0th bmake is running in meta mode), but we can generate the Makefile.depend files as a consequence of building.

That was easy with the Junos, build because it hasn't used anything but the leaf directories for ages.

After fiddling for a bit trying to leverage buildworld to do this, I gave up and tried the brute force approach:

find * -name Makefile > mfile.list
egrep -v '^(contrib|crypto)/|dist/' mfile.list |
sed 's,\(.*\)/Makefile,   \1 \\,' > dirdep.list

and edit that into a simple the-lot/Makefile.depend file. By starting a build using that, bmake would try to visit every directory in the tree with a makefile. Not very smart. Especially since there are a number of directories in the tree which do not actually build any more.

With a bit of filtering, I got the tree to all build for i386, but after the last sync, and building for amd64 I decided to get a bit neater. So with a little hack (local.init.mk gets included by bsd.init.mk):

Index: share/mk/local.init.mk
===================================================================
--- share/mk/local.init.mk      (revision 242503)
+++ share/mk/local.init.mk      (working copy)
@@ -1,5 +1,16 @@

 .if defined(.PARSEDIR)
+.if make(gen-dirdeps)
+ECHODIR=:
+NO_OBJ=
+.if empty(SUBDIR)
+gen-dirdeps:
+       @echo " ${RELDIR} \\"
+.else
+gen-dirdeps: _SUBDIR
+.endif
+.endif
+
 .if ${.MAKE.MODE:Mmeta*} != ""
 .if !empty(SUBDIR) && !defined(LIB) && !defined(PROG) && ${.MAKE.MAKEFILES:M*bsd.prog.mk} == ""
 .if ${.MAKE.MODE:Mleaf*} != ""
Index: share/mk/bsd.arch.inc.mk
===================================================================
--- share/mk/bsd.arch.inc.mk    (revision 242545)
+++ share/mk/bsd.arch.inc.mk    (working copy)
@@ -2,6 +2,14 @@
 # Include the arch-specific Makefile.inc.$ARCH.  We go from most specific
 # to least specific, stopping after we get a hit.
 #
+.if make(gen-dirdeps)
+.ifndef MK_BMAKE
+.include <bsd.own.mk>
+.endif
+gen-dirdeps:
+_sd := ${SUBDIR:U}
+SUBDIR=
+.endif
 .if exists(${.CURDIR}/Makefile.${MACHINE})
 .include "Makefile.${MACHINE}"
 .elif exists(${.CURDIR}/Makefile.${MACHINE_ARCH})
@@ -9,3 +17,10 @@
 .elif exists(${.CURDIR}/Makefile.${MACHINE_CPUARCH})
 .include "Makefile.${MACHINE_CPUARCH}"
 .endif
+.if make(gen-dirdeps)
+.if !empty(SUBDIR)
+x!= echo DIRDEPS.${MACHINE}= ${SUBDIR:@d@${RELDIR}/$d@} >&2; echo
+.endif
+SUBDIR+= ${_sd}
+SUBDIR:= ${SUBDIR:O}
+.endif

We can gather a list of all leaf dirs that should build for a given machine:

make gen-dirdeps > dirdeps.list 2>&1

Which gives us output like (for amd64):

        share/info \
        lib/csu/amd64 \
        lib/libc \
        lib/libbsm \
        ...
        rescue/rescue \
DIRDEPS.amd64= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol
        sbin/adjkerntz \
        ...

which is then easy enough to post-process into a set of Makefile.depend files.

pkgs/pseudo/

The pkgs/Makefile which we use as a top-level makefile looks for subdirs undr pkgs/ and pkgs/pseudo/ that match the target to be built.

Subdirs of pkgs/pseudo/ are not expected to build anything themselves, just be place holders for Makefile.depend files (which are invariably manually maintained). Thus their makefiles do nothing - except to say not to update Makefile.depend.

From our bootstrapping exercise above we got:

pkgs/pseudo/bin/Makefile.depend
pkgs/pseudo/cddl/Makefile.depend
pkgs/pseudo/clang/Makefile.depend
pkgs/pseudo/games/Makefile.depend
pkgs/pseudo/gnu/Makefile.depend
pkgs/pseudo/include/Makefile.depend
pkgs/pseudo/kerberos5/Makefile.depend
pkgs/pseudo/lib/Makefile.depend
pkgs/pseudo/libexec/Makefile.depend
pkgs/pseudo/misc/Makefile.depend
pkgs/pseudo/sbin/Makefile.depend
pkgs/pseudo/secure/Makefile.depend
pkgs/pseudo/share/Makefile.depend
pkgs/pseudo/usr.bin/Makefile.depend
pkgs/pseudo/usr.sbin/Makefile.depend

which we simply list in pkgs/pseudo/userland/Makefile.depend:

# This file is not autogenerated - take care!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        pkgs/pseudo/bin \
        pkgs/pseudo/cddl \
        pkgs/pseudo/games \
        pkgs/pseudo/gnu \
        pkgs/pseudo/include \
        pkgs/pseudo/kerberos5 \
        pkgs/pseudo/lib \
        pkgs/pseudo/libexec \
        pkgs/pseudo/sbin \
        pkgs/pseudo/secure \
        pkgs/pseudo/share \
        pkgs/pseudo/usr.bin \
        pkgs/pseudo/usr.sbin \

.include <dirdeps.mk>

which is in turn referenced by pkgs/pseudo/the-lot/Makefile.depend:

# This file is not autogenerated - take care!

DEP_RELDIR := ${_PARSEDIR:S,${SRCTOP}/,,}

DIRDEPS = \
        pkgs/pseudo/kernel \
        pkgs/pseudo/toolchain \
        pkgs/pseudo/toolchain.host \
        pkgs/pseudo/userland \


.include <dirdeps.mk>

The compilers are built via toolchain for both the host and target machines.

machine dependent dirdeps

The DIRDEPS.amd64 lines from our bootstrap output, lead us to run a separate extraction process for each machine type:

getMdirdeps() {
    for mf in "$@"
    do
        case "$mf" in
        *pkgs/*|*~) continue;;
        esac
        d=${mf%/*}
        m=${mf##*.}
        case "$m" in
        ""|inc|orig|rej|bak|old) continue;;
        esac
        MACHINE=$m ${MAKE:-make} -C $d -f bsd.arch.inc.mk gen-dirdeps 2>&1
    done
}

which we can then add to pkgs/pseudo/sbin/Makefile.depend etc:

DIRDEPS.amd64= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol
DIRDEPS.arm= sbin/bsdlabel sbin/fdisk
DIRDEPS.i386= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol sbin/sconfig
DIRDEPS.ia64= sbin/mca
DIRDEPS.mips= sbin/bsdlabel sbin/fdisk
DIRDEPS.pc98= sbin/bsdlabel sbin/fdisk_pc98 sbin/sconfig
DIRDEPS.sparc64= sbin/bsdlabel sbin/sunlabel

DIRDEPS+= ${DIRDEPS.${MACHINE}:U}

.include <dirdeps.mk>

Note that any any directory listed in DIRDEPS.amd64 should be removed from the generic DIRDEPS - bsd.subdir.mk would have entered them and caused them to list themselves.

Also note, that the DIRDEPS.amd64 lines output by bsd.arch.inc.mk are not always leaf dirs. For instance in pkgs/pseudo/usr.sbin/Makefile.depend we originally see:

DIRDEPS = \
        usr.sbin/acpi/acpiconf \
        usr.sbin/acpi/acpidb \
        usr.sbin/acpi/acpidump \
        usr.sbin/acpi/iasl \

but we see:

DIRDEPS.amd64=  usr.sbin/acpi

which means we need to move usr.sbin/acpi/* from DIRDEPS to DIRDEPS.amd64 and any other that references usr.sbin/acpi.

pkgs/pseudo/kernel

The exception to the rule is pkgs/pseudo/kernel which actually builds a kernel in the FreeBSD way (GENERIC by default).

In the Junos build we need to build multiple kernels for multiple machines and want to automatically capture dependencies for them all. So there is a src directory that corresponds to each kernel.

In keeping with the traditional FreeBSD kernel build, this makefile just does:

# $FreeBSD: projects/bmake/pkgs/pseudo/kernel/Makefile 248288 2013-03-14 22:04:25Z sjg $

# Build the kernel ${KERNCONF}
KERNCONF?= ${KERNEL:UGENERIC}

TARGET?= ${MACHINE}
# keep this compatible with peoples expectations...
KERN_OBJDIR= ${OBJTOP}/sys/compile/${KERNCONF}
KERN_CONFDIR= ${SRCTOP}/sys/${TARGET}/conf

CONFIG= ${STAGE_HOST_OBJTOP}/usr/sbin/config

${KERNCONF}.config: .MAKE .META
        mkdir -p ${KERN_OBJDIR:H}
        (cd ${KERN_CONFDIR} && \
        ${CONFIG} ${CONFIGARGS} -d ${KERN_OBJDIR} ${KERNCONF})
        (cd ${KERN_OBJDIR} && ${.MAKE} depend)
        @touch $@

# we need to pass curdirOk=yes to meta mode, since we want .meta files
# in ${KERN_OBJDIR}
${KERNCONF}.build: .MAKE ${KERNCONF}.config
        (cd ${KERN_OBJDIR} && META_MODE="${.MAKE.MODE} curdirOk=yes" ${.MAKE})

.if ${.MAKE.LEVEL} > 0
all: ${KERNCONF}.build
.endif

UPDATE_DEPENDFILE= no

.include <bsd.prog.mk>

circular dependencies

Our goal is to be able to build the tree in a single pass. That is; visiting dirs in the correct order - once, and have the default (all) target do all that is required, such as building objects (libs, progs) installing (staging) headers, libs even progs if -DWITH_STAGING_PROG, and finally updating any dependencies.

This requires that there be no circular dependencies within the tree. Currently in head, lib/libc depends on headers from both lib/libutil and lib/msun, and since libc.so needs to be scanned when linking shared libs, we have a potential circular dependency.

In the case of lib/libutil this is trivial to avoid, since it has only two headers and they are both public, so adding:

CFLAGS+= -I${.CURDIR:H}/libutil

to lib/libc/Makefile is fine.

Things are not quite as neat for lib/msun which not only has headers in a machine specific subdir, but also has math.h in a directory which contains both public and private headers. The following in lib/libc/Makefile avoids the dependency:

MSUN_ARCH_SUBDIR != ${MAKE} -B -C ${.CURDIR:H}/msun -V ARCH_SUBDIR
# unfortunately msun/src contains both private and public headers
CFLAGS+= -I${.CURDIR:H}/msun/${MSUN_ARCH_SUBDIR} -I${.CURDIR:H}/msun/src

but is not as clean as the previous case.

Of course lib/msun/Makefile does something similar to grab headers from libc:

# Location of fpmath.h and _fpmath.h
LIBCDIR=        ${.CURDIR}/../libc
.if exists(${LIBCDIR}/${MACHINE_ARCH})
LIBC_ARCH=${MACHINE_ARCH}
.else
LIBC_ARCH=${MACHINE_CPUARCH}
.endif
CFLAGS+=        -I${.CURDIR}/src -I${LIBCDIR}/include \
        -I${LIBCDIR}/${LIBC_ARCH}

Keeping public headers separate from private headers helps ensure that the above sort of dance is harmless.

Another example is lib/libproc and lib/librtld_db each of which includes a header staged by the other - resulting in a circular dependency. The fix is simple:

Index: lib/libproc/Makefile
===================================================================
--- lib/libproc/Makefile        (revision 242545)
+++ lib/libproc/Makefile        (working copy)
@@ -14,6 +14,8 @@
 INCS=  libproc.h

 CFLAGS+=       -I${.CURDIR}
+# avoid cyclic dependency
+CFLAGS+=       -I${.CURDIR:H}/librtld_db

 .if ${MK_LIBCPLUSPLUS} != "no"
 LDADD+=                -lcxxrt

MACHINE specific depend files

Having previously built the tree for i386, and now building for amd64 we can detect directories where a single Makefile.depend is probably not ideal:

Index: usr.bin/truss/Makefile.depend
===================================================================
--- usr.bin/truss/Makefile.depend       (revision 242503)
+++ usr.bin/truss/Makefile.depend       (working copy)
@@ -8,7 +8,6 @@
        gnu/lib/libgcc \
        include \
        include/arpa \
-       include/rpc \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
@@ -18,10 +17,12 @@

 .if ${DEP_RELDIR} == ${_DEP_RELDIR}
 # local dependencies - needed for -jN in clean tree
-i386-fbsd.o: syscalls.h
-i386-fbsd.po: syscalls.h
-i386-linux.o: linux_syscalls.h
-i386-linux.po: linux_syscalls.h
+amd64-fbsd.o: syscalls.h
+amd64-fbsd.po: syscalls.h
+amd64-fbsd32.o: freebsd32_syscalls.h
+amd64-fbsd32.po: freebsd32_syscalls.h
+amd64-linux32.o: linux32_syscalls.h
+amd64-linux32.po: linux32_syscalls.h
 ioctl.o: ioctl.c
 ioctl.po: ioctl.c
 .endif

There are very few of these cases (so far), where machine specific local dependencies need to be captured. The simplest solution is for these to create Makefile.depend.${MACHINE}.

The vast majority of the tree seems to be fine with a simple Makefile.depend.

Staging conflicts

The latest version of meta.stage.mk throws an error when it detects that a different directory has already staged the file that it wants to.

Most of the cases found so far are simple - and easily fixed. For example lib/ncurses/ncursesw/Makefile inlcudes lib/ncurses/ncurses/Makefile and thus attempts to install the same headers in the same location. This is easily fixed:

Index: lib/ncurses/ncurses/Makefile
===================================================================
--- lib/ncurses/ncurses/Makefile        (revision 242503)
+++ lib/ncurses/ncurses/Makefile        (working copy)
@@ -304,6 +304,7 @@
 SYMLINKS+=     libncurses${LIB_SUFFIX}_p.a
 ${LIBDIR}/libtinfo${LIB_SUFFIX}_p.a
 .endif

+.if ${.CURDIR:T} == "ncurses"
 DOCSDIR=       ${SHAREDIR}/doc/ncurses
 DOCS=          ncurses-intro.html hackguide.html

@@ -311,6 +312,7 @@
 .PATH: ${NCURSES_DIR}/doc/html
 FILESGROUPS=   DOCS
 .endif
+.endif

 # Generated source
 .ORDER: names.c codes.c

beforeinstall

When building WITH_STAGING_PROG we want to stage pretty much everything. There are a number of makefiles however that rely on the target beforeinstall to prepare files that we would want to stage.

Rather than re-write all these makefiles, we leverage beforeinstall and set DESTDIR=${STAGE_OBJTOP} which is very similar to what we did for src/include/.

pkgs/pseudo/stage

Many of those beforeinstall targets rely on mtree having been run in DESTDIR. The makefile in pkgs/pseudo/stage ensures that this is done, and is inserted as a dependency on every directory except itself.

Ports

Making FreeBSD use bmake requires dealing with ports. The conflicting modifiers mentioned above are used quite a lot within ports. There are a few other issues too.

Quoted strings in .for loops

The following:

OPTIONS= FOO "Description of FOO" FOO_DEFAULT
.for i in ${OPTIONS}

is expected to iterate 3 times. Quoted strings isn't something NetBSD's make (hence bmake) supported. This has been added. Using bmake the above could be coded as:

.for opt description default in ${OPTIONS}

allowing a much easier to follow makefile.

Bootstrapping bmake for old versions of FreeBSD

A bigger issue for ports though is that it isn't branched and must remain buildable on the oldest supported release. Thus it is important to not use enhancements such as multiple iterators in .for loops until the oldest supported release uses bmake.

The reason being that in order to build ports on an old release, which neither has bmake nor a make which has the compatibility modifiers (:tu etc), we need ports to be able to build and install bmake for itself automatically.

This can be done with the following in Mk/bsd.ports.mk:

.if !defined(.PARSEDIR) && ${.MAKEFLAGS:M-V} == ""
.MAIN: all
# We are not bmake
# bmake-sh will invoke bmake (installing it if needed).
all ${.TARGETS}: use-bmake
.if !target(use-bmake)
use-bmake:
        +sh ${PORTSDIR}/Mk/bmake-sh ${.MAKEFLAGS} ${.TARGETS}
.endif
.else                           # Not bmake

The script Mk/bmake-sh will simply invoke bmake if it exists, and otherwise will build/install it - using a copy of Mk/*mk converted back to the old make syntax:

tports=/tmp/${USER}-ports
mkdir -p $tports/Mk
(cd $PORTSDIR/Mk &&
for f in *.mk
do
    sed -f b2fmake.sed $f > $tports/Mk/$f
done)
# install bmake
(cd $PORTSDIR/devel/bmake &&
$make BSDPORTMK=$tports/Mk/bsd.port.mk install) || exit 1

the more bmake functionality added to Mk/*mk, the more complex b2fmake.sed must be.

There is a patch in http://people.freebsd.org/~sjg/ports2bmake.tar.gz that takes care of converting ports to handle bmake.

Hack to allow old modifiers

Turns out that the way portmgr build ports cannot accommodate the approach outlined above. So a knob was added to allow bsd.port.mk to:

# tell bmake we use the old :L :U modifiers
.MAKE.FreeBSD_UL= yes

support for this will removed once 8.3 is EOL and portmgr convert ports to use :tl and :tu.

With this as part of a simple patch to bsd.port.mk, portmgr were able to get ports building with bmake, and thus close this issue.

Default -V behavior

The final issue, is extensive use of make -V FOO. The old FreeBSD make outputs a fully expanded value for FOO so:

FOO_VAR!= ${MAKE} -C foo -V VAR

is commonly used.

However, with the above, bmake will show its literal value, which might be ${DESTDIR}foo - not what is desired.

To ensure full expansion bmake -V '${FOO}' is used. In a makefile that typically needs to be coded as:

FOO_VAR!= ${MAKE} -C foo -V '$${VAR}'

But those doubled $$ get complex fast when commands get put in other variables. Thus NetBSD pkgsrc uses:

FOO_VAR!= ${MAKE} -C foo show-var VARIABLE=VAR

to avoid complexities when such things need to be used indirectly. Using this model, will work equally well regardless of make version, but only handles a single variable.

Changing -V behavior

As useful as the ability to see the literal value of a variable is for debugging (I use it a lot), it isn't a useful behavior from the build's perspective.

It would be very nice to not have to frob all the uses of make -V in the FreeBSD ports collection and elsewhere.

My preferred solution is to change the default behavior and add a debug flag to allow seeing the literal value:

$ bmake -V FOO_DIR
/opt/bin/foo
$ bmake -dV -V FOO_DIR
${DESTDIR}/opt/bin/foo

A compromise was adding a knob that could be set via FreeBSD's sys.mk (or anywhere else) that controls the default behavior. This allows both teams to see the behavior they expect:

$ FOO='${DESTDIR}${.CURDIR:H}' bmake -V FOO .MAKE.EXPAND_VARIABLES=yes
/usr/src/usr.bin
$ FOO='${DESTDIR}${.CURDIR:H}' bmake -V FOO .MAKE.EXPAND_VARIABLES=no
${DESTDIR}${.CURDIR:H}
$ FOO='${DESTDIR}${.CURDIR:H}' bmake -dV -V FOO .MAKE.EXPAND_VARIABLES=yes
${DESTDIR}${.CURDIR:H}

Funky .for loops

Currently bsd.port.mk does a rather complex dance involving nested .for loops with heavily escaped iterator variables to handle building the MLINKS list for multiple languages.

In bmake loop interators are subsituted as ${:Uvalue} which avoids lots of issues but won't work with the above. Fortunately a one line nested inline-loop does the job:

.if defined(.PARSEDIR)
# inline loops are simpler
_MLINKS=    ${_MLINKS_PREPEND} \
        ${MANLANG:S,^,man/,:S,/"",,:@m@${MLINKS:@p@${MAN${p:E}PREFIX}/$m/man${p:E}/$p${MANEXT}@}@}

.else

There are two loops there, one with iterator m for ${MANLANG} resulting in values like man man/fr etc., and another p for ${MLINKS}, thus for:

MANLANG= "" fr
MLINKS= foo.1 goo.2

we get:

/usr/local/man/man1/foo.1
/usr/local/man/man2/goo.2
/usr/local/man/fr/man1/foo.1
/usr/local/man/fr/man2/goo.2

exactly the same as the 20 or so original lines produce.

Progress

As noted above the initial import of bmake to FreeBSD happened in early October 2012.

Progress on ports stalled for some time waiting for an exp-run, but portmgr signed off on bmake early May 2013.

Stable dependencies

Being able to build in meta mode is cool, but a coversion cannot be considered complete while there is dependency churn. By that I mean certain Makefile.depend files will be observed to change without obvious cause. This is invariably caused by bugs in the makefiles.

The good news is that such churn does not necessarily affect the build results, which is handy if certain makefile bugs cannot be fixed during the transition phase.

So far I have seen little evidence of this.

Changing behavior based on clean/update build

A very common cause is a makefile using .if exists(somthing) and changing its behavior as a result. Thus the result from a clean tree build can differ from an update build. This should be fixed.

Conflicting makefiles

Whenever two (or more) makefiles try to do the same thing, eg include/arpa/Makefile and lib/libtelnet/Makefile both installing usr/include/arpa/telnet.h. In this case meta.stage.mk will throw an errror so the issue is quickly resolved.

Other cases can be more subtle. In general it is dangerous for any makefile to stomp on anything outside of its .OBJDIR.

Foreign builds

If part of the build is done with say gmake, our ability to reliably capture what it does is limited to clean tree builds. There is explicit mechanism in meta.autodep.mk to cope with this.

Current state of FreeBSD

The projects/bmake branch is tracking head and was last sync'd in Feb 2013.

The target the-lot which covers all userland, toolchains and a kernel builds in meta mode in parallel:

[Creating objdir /var/obj/projects-bmake/amd64/pkgs...]
bmake: "/b/sjg/work/FreeBSD/projects-bmake/src/pkgs/Makefile" line 105:
@ 1363627007 [2013-03-18 10:16:47] Start the-lot
--- count-makefiles ---
@ 1363627040 [2013-03-18 10:17:20] Makefiles read: total=1220 depend=1209 seconds=33
--- /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage.amd64 ---
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage for amd64 ...
--- /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage.host ---
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage for host ...
..
..
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/the-lot for amd64 ...
[Creating objdir /var/obj/projects-bmake/amd64/pkgs/pseudo/the-lot...]
Building /var/obj/projects-bmake/amd64/pkgs/pseudo/the-lot/all
--- all ---
Done!
--- /b/sjg/work/FreeBSD/projects-bmake/src/pkgs.amd64 ---
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs for amd64 ...
@ 1363638737 [2013-03-18 13:32:17] Finished the-lot seconds=11730

Environment

I should note that I generally run make via a wrapper script which first reads a file $SB/.sandbox-env to condition the environment. I have dozens of active trees for Junos, NetBSD and FreeBSD and multiple branches and this allows me use the same approach for all.

For the projects/bmake branch the key environment items are:

export SRCTOP=$SB/src
export OBJROOT=/var/obj/projects-bmake/
export OBJTOP="$OBJROOT\${MACHINE}"
export MAKESYSPATH=$SRCTOP/share/mk
export MAKEOBJDIR='${.CURDIR:S,${SRCTOP},${OBJTOP},}'
export HOST_TARGET=freebsd10-amd64
export HOST_OBJTOP="$OBJROOT$HOST_TARGET"

Not all are strictly necessary as *sys.mk can set them, but putting such logic in the makefiles constrains choices. The MAKESYSPATH can probably be skipped since bmake is built with the default of .../share/mk which will work so long as it is run within the src tree, and share/mk/local.sys.mk resolves it to an absolute path which is needed for the kernel build to work.

Note that MAKEOBJDIR value is single quoted, this defers expansion until bmake sees it.

HOST_TARGET is set by the wrapper script, based on uname output of the host machine. When building for the pseudo machine host we use ${HOST_OBJTOP} rather than ${OBJTOP}. This allows us to avoid trouble when the same tree is mounted via NFS and built from incompatible machines.

I have local.sys.mk set a number of options:

WITH_INSTALL_AS_USER= yes
WITH_AUTO_OBJ= yes
WITH_META_MODE= yes
WITH_STAGING= yes
WITH_STAGING_PROG= yes

WITH_AUTO_OBJ is important, since it causes the equivalent of make obj to be done automatically and early, so that as the makefile is read .OBJDIR and .CURDIR have their correct values which matters in the gcc build where we see:

.PATH: ../cc_tools

which is supposed to add ${.OBJDIR}/../cc_tools to .PATH, but if ${.OBJDIR} does not exist at the time that line is read, the right thing will not happen.


Author:sjg@crufty.net /* imagine something very witty here */