Building FreeBSD in meta mode

This doc describes the process of converting the FreeBSD build to use bmake in meta mode.

Note: while I normally talk of meta mode and using together, either can be used without the other.

The initial commit of bmake went into FreeBSD head in October 2012. FreeBSD 10 is the first release to use bmake by default. The meta mode work described below is currently in a branch projects/bmake.

To avoid? confusion I will use fmake when referring to the older FreeBSD make and bmake regardless of which is installed as make.

Why meta mode

An obvious question, with a very simple answer: to improve the build.

There are several aspects to this:

more reliable update builds

Provided filemon is being used, the .meta files created during a build, allow bmake much greater visibility into what happens during the creation of a target. This allows it to be more thorough when deciding if a target is out-of-date.

captured command line

Even without filemon, the .meta file captures the expanded command line used last time, and by default any change to that command line causes the target to be out-of-date.

The comparison of command lines can be controlled at multiple levels; per build, makefile, target or even line within a target script. The last is perhaps the most useful.

Any command that that references the special variable .OODATE cannot be usefully compared and so isn't. This can be leveraged by using .OODATE in such a way that it does not affect the command: ${} will expand to nothing (change the match string if .no.cmp. might actually appear ;-)

captured output

The .meta files capture the output from the commands that generate a target.

This allows a much cleaner - and easier to read build log, and even for people who refuse to log the output of builds, the vital clues are captured when something goes wrong.


The .ERROR target is run when bmake hits an error, and if it was while generating a target, .ERROR_TARGET is set to the name of the target and .ERROR_META_FILE will be set to the name of the meta file for it. Thus we can capture a copy of the relevant .meta file in a well known location.

This then becomes a simple and reliable means of spotting how and why a build failed. I have scripts that will parse that .meta file, and identify the commit that broken the build.

tree dependencies

When filemon is used, the .meta files capture lots of useful data, which can be used in many ways, but one of the most obvious is learning the dependencies of one directory on others.

simplify the build

This is one of the key benefits. This is mostly a benefit of using, though many individual makefiles can be simplified by use of meta mode.

The use of meta mode also allows easy automation of the data needed by, so we usually just refer to this as the meta mode build. The or meta mode build is very easy to understand.

Look at the top-level makefiles of any BSD system today, and try to understand exactly what they do. It can be a challenge indeed. Ignoring comments and blank lines, FreeBSD 10's top-level Makefile and Makefile.inc1 have more than 1800 lines.

The pre-meta-mode Junos build had over 5000 lines in its top-level makefiles.

By contrast, Makefile, pkgs/Makefile and pkgs/ which drive the Junos meta mode build were until recently less than 130 lines.

Of course I am not counting more than 10k autogenerated Makefile.depend* files which make it possible, but;

a/ they are autogenerated and each by itself is easy to understand

b/ they are not only used for top-level builds

In fact in this model, the top-level build is little different from any other.

Not to mention that top-level builds can often be dispensed with.

We use the example below of make -j8 -C bin/cat in a clean tree.

Using bmake

Since leverages pretty much every single feature of bmake, the first step is to be able to build FreeBSD using bmake as a replacement for /usr/bin/make. This is actually quite simple.

Conflicting modifiers

Several years ago FreeBSD make adopted :U to upper case a value, and :L to lowercase it, from OpenBSD. At about the same time I added a raft of modifiers from ODE to NetBSD's make. These included:

     This is the loop expansion mechanism from the OSF Development Envi-
     ronment (ODE) make.  Unlike .for loops expansion occurs at the time
     of reference.  Assign temp to each word in the variable and evaluate
     string.  The ODE convention is that temp should start and end with a
     period.  For example.
           ${LINKS:@.LINK.@${LN} ${TARGET} ${.LINK.}@}

     However a single character variable is often more readable:

     If the variable is undefined newval is the value.  If the variable
     is defined, the existing value is returned.  This is another ODE
     make feature.  It is handy for setting per-target CFLAGS for
     If a value is only required if the variable is undefined, use:

     If the variable is defined newval is the value.

:L   The name of the variable is the value.

:P   The path of the node which has the same name as the variable is the
     value.  If no such node exists or its path is null, then the name of
     the variable is used.  In order for this modifier to work, the name
     (node) must at least have appeared on the rhs of a dependency.

At a later point NetBSD added:

:tl  Converts variable to lower-case letters.

:tu  Converts variable to upper-case letters.

While the FreeBSD ports collection uses :U and :L quite a bit, the base system does not use them at all.

Thus building FreeBSD itself using bmake is mostly a matter of adding .NOPATH statements in a couple of places; to curb its enthusiasm for finding things via .PATH.

The diff to allow bmake -jN buildworld to work was less than 300 lines (not counting comments).

Other changes

While converting the base system to be able to use bmake, was simple, converting FreeBSD to use bmake by default was more involved.

The ports collection provided several challenges detailed in a later section.

Solutions to some issues required changes to bmake. In most cases these were made to NetBSD's make, in some cases (especially where the change was deemed short term) local changes were made to bmake in FreeBSD.

The good-will and support of developers in both projects was key to success.


While the original goal was to achieve cutover within a release cycle, this proved impractical, so a BMAKE option was added to select which make should be installed as /usr/bin/make. The knob defaults to "yes".

The top-level makefiles, have to pay attention to WITH[OUT]_BMAKE and ensure that the correct make is used.

This is a short term thing, since to keep WITHOUT_BMAKE working, requires anyone making changes to makefiles (at least src/share/mk/* and src/Makefile*) to test both ways.

job tokens

When you do make -j8, the goal is that at most 8 jobs will run in parallel. To achieve this most modern make's use a shared token pool, each sub-make waits for a token before running a job.

In fmake the token pool is managed as a named FIFO. If fmake does not find the appropriate variable in its environment, it creates and initializes the FIFO and exports its name.

In bmake the token pool is a pipe, with the open descriptors passed to sub-makes only for targets that are tagged with .MAKE. Sub-makes know that they are such due to the -J in,out arg they get. If the descriptors named are invalid, bmake behaves as for -B (compat mode).

Both designs have their benefits. The difference though can lead to problems when mixing the two.

If the first make instance is fmake, it creates its FIFO and passes -jN to sub-makes. If these are bmake the lack of -J causes them to create their own token pool - oversubscription.

Similarly if the first make is bmake and sub-makes are fmake problems can arise since bmake does not export the name of FIFO, so each sub-make will create its own. This case is less likely to happen accidentally, since -J will cause an error for fmake and thus needs to be explicitly filtered. Which implies the mixing is deliberate. An example of this is in src/Makefile when building WITHOUT_BMAKE.

.MAKE source

As noted above, bmake only passes the descriptors for its token pool if the target is tagged .MAKE:

.MAKE     Execute the commands associated with this target even if the -n
          or -t options were specified.  Normally used to mark recursive

The fmake man page says the same, but .MAKE appears not to work and thus isn't used all the places it should be.

This causes problems for bmake. While it is not a huge problem to fix the base source, this does not help all the makefiles others have written.

The solution was another local change, a knob .MAKE.ALWAYS_PASS_JOB_QUEUE to cause bmake to pass the descriptors of its token pool to all jobs.


Normally on failure during a jobs-mode build, bmake puts an error token into the job token pool. This causes the build to be abandoned immediately, which is normally exactly what is desired.

For make universe however it is desired that each branch of the build that can continue, do so. This is facilitated by the following in src/Makefile:

.if make(universe)
# we do not want a failure of one branch abort all.


Fmake runs target scripts with set -e in effect. This means that the script fails if any statement in a multi-statement command line fails. This means that the command:

cd /no/such/dir; rm -rf *

will not do any harm.

By contrast, bmake runs scripts such that the command line rather than individual statements is the unit of failure. This means that the above command should be written as:

cd /no/such/dir && rm -rf *

while fixing all the makefiles in base is feasible, fixing ports is another matter, and then there is the question of makefiles written by FreeBSD users.

The practical solution is to provide a .SHELL description that replicates the fmake behavior:

# By default bmake does *not* use set -e
# when running target scripts, this is a problem for many makefiles here.
# So define a shell that will do what FreeBSD expects.
.SHELL: name=sh \
        quiet="set -" echo="set -v" filter="set -" \
        hasErrCtl=yes check="set -e" ignore="set +e" \
        echoFlag=v errFlag=e \

meta mode

Being able to do buildworld is a useful stepping stone, but isn't very interesting.

Being able to make -j8 -C bin/cat in a freshly checked out tree, and have it correctly build all the libs etc. in one pass, is more like it. Which is where meta mode or more correctly comes in. The output below shows the directories checked - only things needed by cat get built:

Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/gnu/lib/libssp/libssp_nonshared for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/xlocale for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/rpc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/include/rpcsvc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/csu/amd64 for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/libcompiler_rt for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/lib/libc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/gnu/lib/libgcc for amd64 ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat for amd64 ...

Perhaps also interesting is that we can get the build to show us the complete tree dependency graph from any starting point. For example:

$ build-graph -C bin/cat
bin/cat.amd64: gnu/lib/libgcc.amd64 gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/xlocale.amd64 lib/csu/amd64.amd64 lib/libc.amd64 lib/libcompiler_rt.amd64 pkgs/pseudo/stage.amd64
gnu/lib/libgcc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/xlocale.amd64 lib/csu/amd64.amd64 lib/libc.amd64 pkgs/pseudo/stage.amd64
gnu/lib/libssp/libssp_nonshared.amd64: pkgs/pseudo/stage.amd64
include.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
include/rpc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
include/rpcsvc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
include/xlocale.amd64: gnu/lib/libssp/libssp_nonshared.amd64 pkgs/pseudo/stage.amd64
lib/csu/amd64.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 pkgs/pseudo/stage.amd64
lib/libc.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 include/rpc.amd64 include/rpcsvc.amd64 lib/csu/amd64.amd64 lib/libcompiler_rt.amd64 pkgs/pseudo/stage.amd64
lib/libcompiler_rt.amd64: gnu/lib/libssp/libssp_nonshared.amd64 include.amd64 pkgs/pseudo/stage.amd64

The build-graph script just leverages the debug information that can output as it examines the Makefile.depend* files.

tree dependencies

Everyone is familiar with doing make depend to capture the dependencies that a directory has on the files that it uses, to help ensure that re-running make will do the right thing.

What we need is the same level of information for the tree as a whole. As detailed in Building BSD with meta mode I've implemented that a couple of ways in the Junos build, and meta mode is the current method used.

meta files

In meta mode, for most targets bmake creates a .meta file to capture information like the expanded command used, any command output (useful for debugging), and a record of all the successful system calls that are interesting to make:

# Meta data file /var/obj/projects-bmake/amd64/bin/cat/cat.o.meta
CMD cc -O2 -pipe  -nostdinc -isystem /var/obj/projects-bmake/stage/amd64/usr/include -isystem /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2 -std=gnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -c /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
CMD ctfconvert -L VERSION cat.o
CWD /var/obj/projects-bmake/amd64/bin/cat
TARGET cat.o
-- command output --
-- filemon acquired metadata --
# filemon version 4
# Target pid 42504
# Start 1363631731.803698
V 4
F 42504 42529
E 42529 /bin/sh
R 42529 /var/run/
R 42529 /lib/
R 42529 /lib/
R 42529 /lib/
S 42529 .
S 42529 /var/obj/projects-bmake/amd64/bin/cat
S 42529 /usr/bin/cc
F 42529 42530
E 42530 /usr/bin/cc
S 42530 /usr/bin/cc
S 42530 /usr/bin/cc
F 42530 42533
E 42533 /usr/bin/cc
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/clang/3.2
R 42533 cat.o-24d10fa0
W 42533 cat.o-24d10fa0
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
R 42533 /b/sjg/work/FreeBSD/projects-bmake/src/bin/cat/cat.c
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/cdefs.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_null.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/types.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/endian.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/endian.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_types.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_pthreadtypes.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_stdint.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/select.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sigset.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timeval.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_timespec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/syslimits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/signal.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/trap.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/param.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/machine/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/x86/_align.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/limits.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/stat.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/time.h
S 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_time.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/socket.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_iovec.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/_sockaddr_storage.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/un.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/errno.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/runetype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_ctype.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/err.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/fcntl.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_locale.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stddef.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdio.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/stdlib.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/strings.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/xlocale/_string.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/unistd.h
R 42533 /var/obj/projects-bmake/stage/amd64/usr/include/sys/unistd.h
M 42533 '/var/obj/projects-bmake/amd64/bin/cat/cat.o-24d10fa0' 'cat.o'
X 42533 0
X 42530 0
S 42529 /usr/bin/ctfconvert
F 42529 42541
E 42541 /usr/bin/ctfconvert
R 42541 /var/run/
R 42541 /lib/
R 42541 /usr/lib/
R 42541 /usr/lib/
R 42541 /lib/
R 42541 /lib/
R 42541 /lib/
R 42541 cat.o
R 42541 cat.o
R 42541 cat.o.ctf
W 42541 cat.o.ctf
M 42541 'cat.o.ctf' 'cat.o'
X 42541 0
X 42529 0
# Stop 1363631732.322849
# Bye bye

Note: in the NetBSD version of filemon we don't bother recording the stat calls (S).

From this we can derive not only the equivalent data of make depend, but we can derive tree based dependencies.

By that I mean that any file read from an object directory which is not .OBJDIR represents a directory which needs to be built before .CURDIR, it's that simple.

Of course having some consistency in the relationship between object dirs and src dirs helps. Even when that isn't possible though we can work around it.

In the example above, a number of headers are read from /var/obj/projects-bmake/stage/amd64/usr/include/. This is where we stage headers as we build the tree.


As we build libs and other things which have headers to be installed, they get staged into this directory, a long with a .dirdep file to say who put the file there. For example:


the .dirdep file contains:


which says it was put there by the makefile in include/xlocale while building for the machine amd64.

Thus as we read the .meta file and find a file read, we can look for the same path with .dirdep appended to find the directory we should add to our dependencies. If no .dirdep file exists, it is presumed we can derive the src location from the objdir.

${STAGE_OBJTOP}/include (deprecated)

The makefile in src/include/ does not fit the normal usage pattern, and fixing it at this stage isn't an attractive option. So rather than use our normal staging method, we let it install into a directory which only it populates, and for each directory thus created we add a .dirdep file.

We now have a wrapper script which allows running install and then deposits the necessary .dirdep files. This allows all headers to be staged to ${STAGE_OBJTOP}/usr/include.


This is a clone device used to monitor the successful system calls (that are interesting to make) performed by a process and its descendants.

This is how bmake captures the data in the meta files For each job, bmake opens /dev/filemon and gives it the path to a temp file. After it forks, the child gives its pid to filemon. When the child exits bmake reads the syscall trace from the temp file.


Originally a shell script was used to extract the useful information from .meta files. It works fine. I should say worked fine - it hasn't had any attention recently.

If Python is available though we will use which produces the same result, but much more efficiently (5-10 times faster) and can gather additional data without overhead.


After processing the .meta files for bin/cat we end up with the following in ${.MAKE.DEPENDFILE}:

# Autogenerated - do NOT edit!


        gnu/lib/libgcc \
        include \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
        lib/libcompiler_rt \

.include <>

# local dependencies - needed for -jN in clean tree

which shows a number of interesting things.


Defines the top of the src tree. This along with OBJTOP and OBJROOT (eg. OBJTOP=${OBJROOT}${MACHINE}) allow for being able to consistently refer to things in the src and object trees.


Being able to trim ${SRCTOP} from the start of ${.CURDIR} provides a useful relative location within the src tree. This forms the basis of DIRDEPS.

The .dirdep files mentioned above are created by simply doing:

        @echo ${RELDIR}.${MACHINE} > $@


NetBSD make (bmake is just a portable version of it), defines .PARSEDIR as the directory from which the current .PARSEFILE is being read. This is extremely useful to us. As is the :tA modifier:


or the absolute path to .PARSEDIR.

As you can see in the example above we can use ${_PARSEDIR} and ${SRCTOP} to derive the RELDIR of the makefile being read.

We can also use .PARSEDIR as a clue that the makefile is being run by bmake rather than fmake:

.if defined(.PARSEDIR)
# we are bmake ...


Allows to keep track of the RELDIR it is gathering/computing dependencies for, and it gets automatically updated as each Makefile.depend file is read.


In the example above, there are no machine qualified entries in DIRDEPS, if there were, as in pkgs/pseuod/kernel/Makefile.depend for example:

        include \
        include/xlocale \
        usr.sbin/ \

the last entry indicates that usr.sbin/config must be built for the pseudo machine host. When is reading usr.sbin/config/Makefile.depend it will have set DEP_MACHINE=host which can influence filtering and other things as well as ensuring the correct build dependencies are created.

This is a complex makefile, and I recommend against touching it.

This is what makes everything work, and thus the complexity pays off; because everything else (especially the bits people might need to touch) can be quite simple.

bootstrapping meta mode

The easy way to bootstrap meta mode, is to use the old build with top-level makefiles running in non-meta mode, but leaf directories running in meta mode. Thus we do not rely on meta mode for any of the sequencing (that only happens if the 0th bmake is running in meta mode), but we can generate the Makefile.depend files as a consequence of building.

That was easy with the Junos, build because it hasn't used anything but the leaf directories for at least a decade.

After fiddling for a bit trying to leverage buildworld to do this, I gave up and tried the brute force approach:

find * -name Makefile > mfile.list
egrep -v '^(contrib|crypto)/|dist/' mfile.list |
sed 's,\(.*\)/Makefile,   \1 \\,' > dirdep.list

and edit that into a simple the-lot/Makefile.depend file. By starting a build using that, bmake would try to visit every directory in the tree with a makefile. Not very smart. Especially since there are a number of directories in the tree which do not actually build any more.

buildworld -DWITH_META_FILES

Currently we can do buildworld -DWITH_META_FILES to create meta files in a build that otherwise behaves like normal buildworld. These meta files are very useful for comparing against those in the meta mode build - especially when after a sync, things seem to be broken.

We can probably leverage these meta files for bootstrapping too though that is basically done already.

bootstrapping new content

Once the bulk of the tree can build in meta mode, dealing with new dependencies is not that big a deal.

Eg. after last sync, usr.bin/kdump needs libcapsicum and libnv:

mk -C lib/libcapsicum
mk -C lib/libnv
mk -C usr.bin/kdump

is all that is needed - of course there will now be Makefile.depend files in the new libs which need to be added to SVN.


The pkgs/Makefile which we use as a top-level makefile looks for subdirs under pkgs/ and pkgs/pseudo/ that match the target to be built.

Subdirs of pkgs/pseudo/ are not expected to build anything themselves, just be place holders for Makefile.depend files (which are invariably manually maintained). Thus their makefiles do nothing - except to say not to update Makefile.depend.

From our bootstrapping exercise above we got:


which we simply list in pkgs/pseudo/userland/Makefile.depend:

# This file is not autogenerated - take care!


        pkgs/pseudo/bin \
        pkgs/pseudo/cddl \
        pkgs/pseudo/games \
        pkgs/pseudo/gnu \
        pkgs/pseudo/include \
        pkgs/pseudo/kerberos5 \
        pkgs/pseudo/lib \
        pkgs/pseudo/libexec \
        pkgs/pseudo/sbin \
        pkgs/pseudo/secure \
        pkgs/pseudo/share \
        pkgs/pseudo/usr.bin \
        pkgs/pseudo/usr.sbin \

.include <>

which is in turn referenced by pkgs/pseudo/the-lot/Makefile.depend:

# This file is not autogenerated - take care!


        pkgs/pseudo/kernel \
        pkgs/pseudo/toolchain \
        pkgs/pseudo/ \
        pkgs/pseudo/userland \

.include <>

The compilers are built via toolchain for both the host and target machines.

For the embedded world we don't normally build compilers at all.

machine dependent dirdeps

The DIRDEPS.amd64 lines from our bootstrap output, lead us to run a separate extraction process for each machine type:

getMdirdeps() {
    for mf in "$@"
        case "$mf" in
        *pkgs/*|*~) continue;;
        case "$m" in
        ""|inc|orig|rej|bak|old) continue;;
        MACHINE=$m ${MAKE:-make} -C $d -f gen-dirdeps 2>&1

which we can then add to pkgs/pseudo/sbin/Makefile.depend etc:

DIRDEPS.amd64= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol
DIRDEPS.arm= sbin/bsdlabel sbin/fdisk
DIRDEPS.i386= sbin/bsdlabel sbin/fdisk sbin/nvmecontrol sbin/sconfig
DIRDEPS.ia64= sbin/mca
DIRDEPS.mips= sbin/bsdlabel sbin/fdisk
DIRDEPS.pc98= sbin/bsdlabel sbin/fdisk_pc98 sbin/sconfig
DIRDEPS.sparc64= sbin/bsdlabel sbin/sunlabel


.include <>

Note that any directory listed in DIRDEPS.amd64 should be removed from the generic DIRDEPS - would have entered them and caused them to list themselves.

Also note, that the DIRDEPS.amd64 lines output by are not always leaf dirs. For instance in pkgs/pseudo/usr.sbin/Makefile.depend we originally see:

        usr.sbin/acpi/acpiconf \
        usr.sbin/acpi/acpidb \
        usr.sbin/acpi/acpidump \
        usr.sbin/acpi/iasl \

but we see:

DIRDEPS.amd64=  usr.sbin/acpi

which means we need to move usr.sbin/acpi/* from DIRDEPS to DIRDEPS.amd64 and any other that references usr.sbin/acpi.

package manifests

Of course all this is moot once you have a set of manifests or mtree specs used to build the install packages/media. These simply list the applications and shared libs needed, and from those we learn the dependencies we need.

In the Junos build we produce packages that contain isofs images. These are built in pkgs/* and unlike the pkgs/pseudo/*, auto updated dependencies based on what went into the package.


The exception to the rule is pkgs/pseudo/kernel which actually builds a kernel in the FreeBSD way (GENERIC by default).

In the Junos build we need to build multiple kernels for multiple machines and want to automatically capture dependencies for them all. So there is a src directory that corresponds to each kernel.

In keeping with the traditional FreeBSD kernel build though, this makefile just does:

# $FreeBSD: projects/bmake/pkgs/pseudo/kernel/Makefile 248288 2013-03-14 22:04:25Z sjg $

# Build the kernel ${KERNCONF}

# keep this compatible with peoples expectations...

CONFIG= ${STAGE_HOST_OBJTOP}/usr/sbin/config

        mkdir -p ${KERN_OBJDIR:H}
        (cd ${KERN_CONFDIR} && \
        (cd ${KERN_OBJDIR} && ${.MAKE} depend)
        @touch $@

# we need to pass curdirOk=yes to meta mode, since we want .meta files
${KERNCONF}.build: .MAKE ${KERNCONF}.config
        (cd ${KERN_OBJDIR} && META_MODE="${.MAKE.MODE} curdirOk=yes" ${.MAKE})

.if ${.MAKE.LEVEL} > 0
all: ${KERNCONF}.build


.include <>

circular dependencies

Our goal is to be able to build the tree in a single pass. That is; visiting dirs in the correct order - once, and have the default (all) target do all that is required, such as building objects (libs, progs) installing (staging) headers, libs even progs if -DWITH_STAGING_PROG, and finally updating any dependencies.

This requires that there be no circular dependencies within the tree. Currently in head, lib/libc depends on headers from both lib/libutil and lib/msun, and since needs to be scanned when linking shared libs, we have a potential circular dependency.

In the case of lib/libutil this is trivial to avoid, since it has only two headers and they are both public, so adding:

CFLAGS+= -I${.CURDIR:H}/libutil

to lib/libc/Makefile is fine.

Things are not quite as neat for lib/msun which not only has headers in a machine specific subdir, but also has math.h in a directory which contains both public and private headers. The following in lib/libc/Makefile avoids the dependency:

# unfortunately msun/src contains both private and public headers
CFLAGS+= -I${.CURDIR:H}/msun/${MSUN_ARCH_SUBDIR} -I${.CURDIR:H}/msun/src

but is not as clean as the previous case.

Of course lib/msun/Makefile does something similar to grab headers from libc:

# Location of fpmath.h and _fpmath.h
LIBCDIR=        ${.CURDIR}/../libc
.if exists(${LIBCDIR}/${MACHINE_ARCH})
CFLAGS+=        -I${.CURDIR}/src -I${LIBCDIR}/include \

Keeping public headers separate from private headers helps ensure that the above sort of dance is harmless.

Another example is lib/libproc and lib/librtld_db each of which includes a header staged by the other - resulting in a circular dependency. The fix is simple:

Index: lib/libproc/Makefile
--- lib/libproc/Makefile        (revision 242545)
+++ lib/libproc/Makefile        (working copy)
@@ -14,6 +14,8 @@
 INCS=  libproc.h

 CFLAGS+=       -I${.CURDIR}
+# avoid cyclic dependency
+CFLAGS+=       -I${.CURDIR:H}/librtld_db

 .if ${MK_LIBCPLUSPLUS} != "no"
 LDADD+=                -lcxxrt

MACHINE specific depend files

Having previously built the tree for i386, and now building for amd64 we can detect directories where a single Makefile.depend is probably not ideal:

Index: usr.bin/truss/Makefile.depend
--- usr.bin/truss/Makefile.depend       (revision 242503)
+++ usr.bin/truss/Makefile.depend       (working copy)
@@ -8,7 +8,6 @@
        gnu/lib/libgcc \
        include \
        include/arpa \
-       include/rpc \
        include/xlocale \
        lib/${CSU_DIR} \
        lib/libc \
@@ -18,10 +17,12 @@

 .if ${DEP_RELDIR} == ${_DEP_RELDIR}
 # local dependencies - needed for -jN in clean tree
-i386-fbsd.o: syscalls.h
-i386-fbsd.po: syscalls.h
-i386-linux.o: linux_syscalls.h
-i386-linux.po: linux_syscalls.h
+amd64-fbsd.o: syscalls.h
+amd64-fbsd.po: syscalls.h
+amd64-fbsd32.o: freebsd32_syscalls.h
+amd64-fbsd32.po: freebsd32_syscalls.h
+amd64-linux32.o: linux32_syscalls.h
+amd64-linux32.po: linux32_syscalls.h
 ioctl.o: ioctl.c
 ioctl.po: ioctl.c

There are very few of these cases (so far), where machine specific local dependencies need to be captured. The simplest solution is for these to create Makefile.depend.${MACHINE}.

The vast majority of the tree seems to be fine with a simple Makefile.depend.

We configure .MAKE.DEPENDFILE_PREFERENCE such that once a Makefile.depend.${MACHINE} exists, it will be used.

We set .MAKE.DEPENDFILE_DEFAULT to the plain Makefile.depend.

Once a directory contains a machine specific depend file, we will automatically follow suit when building for other machines.

Staging conflicts

The latest version of throws an error when it detects that a different directory has already staged the file that it wants to.

Most of the cases found so far are simple - and easily fixed. For example lib/ncurses/ncursesw/Makefile includes lib/ncurses/ncurses/Makefile and thus attempts to install the same headers in the same location. This is easily fixed:

Index: lib/ncurses/ncurses/Makefile
--- lib/ncurses/ncurses/Makefile        (revision 242503)
+++ lib/ncurses/ncurses/Makefile        (working copy)
@@ -304,6 +304,7 @@
 SYMLINKS+=     libncurses${LIB_SUFFIX}_p.a

+.if ${.CURDIR:T} == "ncurses"
 DOCSDIR=       ${SHAREDIR}/doc/ncurses
 DOCS=          ncurses-intro.html hackguide.html

@@ -311,6 +312,7 @@
 .PATH: ${NCURSES_DIR}/doc/html

 # Generated source
 .ORDER: names.c codes.c


When building WITH_STAGING_PROG we want to stage pretty much everything. There are a number of makefiles however that rely on the target beforeinstall to prepare files that we would want to stage.

Rather than re-write all these makefiles, we leverage beforeinstall and set DESTDIR=${STAGE_OBJTOP} which is very similar to what we did for src/include/.


Many of those beforeinstall targets rely on mtree having been run in DESTDIR. The makefile in pkgs/pseudo/stage ensures that this is done, and is inserted as a dependency on every directory except itself.


Anyone remember porting gcc in the early days? It was a 3 stage dance.

  1. build stage1-gcc with cc
  2. build stage2-gcc with stage1-gcc
  3. build gcc with stage2-gcc

now we could do even that without breaking our single pass through the tree goal - by having separate makefiles for the three stages.

But since the whole toolchain topic needs is own overhaul, for now I'm happy to largely punt it, and even leverage the bootstrap targets from src/Makefile.inc1.


Making FreeBSD use bmake requires dealing with ports. The conflicting modifiers mentioned above are used quite a lot within ports. There are a few other issues too.

Quoted strings in .for loops

The following:

.for i in ${OPTIONS}

is expected to iterate 3 times. Quoted strings isn't something NetBSD's make (hence bmake) supported. This has been added. Using bmake the above could be coded as:

.for opt description default in ${OPTIONS}

allowing a much easier to follow makefile.

Bootstrapping bmake for old versions of FreeBSD

A bigger issue for ports though is that it isn't branched and must remain buildable on the oldest supported release. Thus it is important to not use enhancements such as multiple iterators in .for loops until the oldest supported release uses bmake.

The reason being that in order to build ports on an old release, which neither has bmake nor a make which has the compatibility modifiers (:tu etc), we need ports to be able to build and install bmake for itself automatically.

This can be done with the following in Mk/

.if !defined(.PARSEDIR) && ${.MAKEFLAGS:M-V} == ""
.MAIN: all
# We are not bmake
# bmake-sh will invoke bmake (installing it if needed).
all ${.TARGETS}: use-bmake
.if !target(use-bmake)
        +sh ${PORTSDIR}/Mk/bmake-sh ${.MAKEFLAGS} ${.TARGETS}
.else                           # Not bmake

The script Mk/bmake-sh will simply invoke bmake if it exists, and otherwise will build/install it - using a copy of Mk/*mk converted back to the old make.

I won't bore you with the details, since this approach was not adopted.

Hack to allow old modifiers

Turns out that the way portmgr build ports cannot accommodate the approach outlined above. So a knob was added to allow to:

# tell bmake we use the old :L :U modifiers
.MAKE.FreeBSD_UL= yes

support for this will removed once 8.3 is EOL and portmgr convert ports to use :tl and :tu.

With this as part of a simple patch to, portmgr were able to get ports building with bmake, and thus close this issue.

Default -V behavior

Ports uses make -V FOO extensively. The old FreeBSD make outputs a fully expanded value for FOO so:

FOO_VAR!= ${MAKE} -C foo -V VAR

is commonly used.

However, with the above, bmake will show its literal value, which might be ${DESTDIR}foo - not what is desired.

To ensure full expansion bmake -V '${FOO}' is used. In a makefile that typically needs to be coded as:

FOO_VAR!= ${MAKE} -C foo -V '$${VAR}'

But those doubled $$ get complex fast when commands get put in other variables. Thus NetBSD pkgsrc uses:

FOO_VAR!= ${MAKE} -C foo show-var VARIABLE=VAR

to avoid complexities when such things need to be used indirectly. Using this model, will work equally well regardless of make version, but only handles a single variable.

Changing -V behavior

As useful as the ability to see the literal value of a variable is for debugging (I use it a lot), it isn't a useful behavior from the build's perspective.

It would be very nice to not have to frob all the uses of make -V in the FreeBSD ports collection and elsewhere.

My preferred solution was to change the default behavior and add a debug flag to allow seeing the literal value:

$ bmake -V FOO_DIR
$ bmake -dV -V FOO_DIR

A compromise was adding a knob that could be set via FreeBSD's (or anywhere else) that controls the default behavior. This allows both teams to see the behavior they expect:


.for loop iterators

Currently does a rather complex dance involving nested .for loops with heavily escaped iterator variables to handle building the MLINKS list for multiple languages.

In bmake loop iterators are substituted as ${:Uvalue} which avoids lots of issues but won't work with the above. Fortunately a one line nested inline-loop does the job:

.if defined(.PARSEDIR)
# inline loops are simpler


There are two loops there, one with iterator m for ${MANLANG} resulting in values like man man/fr etc., and another p for ${MLINKS}, thus for:

MANLANG= "" fr
MLINKS= foo.1 goo.2

we get:


exactly the same as the 20 or so original lines produce.


As noted above the initial import of bmake to FreeBSD happened in early October 2012.

Progress on ports stalled for some time waiting for an exp-run, but portmgr signed off on bmake early May 2013.

By the time FreeBSD 10 was branched, bmake was installed as /usr/bin/make by default.

Support for fmake is being deprecated shortly. For those that have a need, there is a port.

Stable dependencies

Being able to build in meta mode is cool, but a conversion cannot be considered complete while there is dependency churn. By that I mean certain Makefile.depend files may be observed to change without obvious cause. This is invariably due to bugs in the makefiles.

The good news is that such churn does not necessarily affect the build results, which is handy if certain makefile bugs cannot be fixed during the transition phase.

So far I have seen little evidence of this in FreeBSD.

Changing behavior based on clean/update build

A very common cause is a makefile using .if exists(something) and changing its behavior as a result. Thus the result from a clean tree build can differ from an update build. This should be fixed.

Use of SRCTOP, OBJTOP, OBJROOT etc, can greatly help canonicalize references to locations within the tree which can eliminate many such issues.

Conflicting makefiles

Whenever two (or more) makefiles try to do the same thing, eg include/arpa/Makefile and lib/libtelnet/Makefile both installing usr/include/arpa/telnet.h. In this case will throw an error so the issue is quickly resolved.

Other cases can be more subtle. In general it is dangerous for any makefile to stomp on anything outside of its .OBJDIR.

Foreign builds

If part of the build is done with say gmake, our ability to reliably capture what it does is limited to clean tree builds. There is explicit mechanism in to cope with this.

Current state of FreeBSD

The projects/bmake branch is tracking head and was last sync'd in late April 2014. I still have not worked out all the kinks from the clang upgrade, and libdwarf seems to have some incompatible changes (compared to stable/10).

The target the-lot which covers all userland, toolchains and a kernel builds (or did) in meta mode in parallel:

[Creating objdir /var/obj/projects-bmake/amd64/pkgs...]
bmake: "/b/sjg/work/FreeBSD/projects-bmake/src/pkgs/Makefile" line 105:
@ 1363627007 [2013-03-18 10:16:47] Start the-lot
--- count-makefiles ---
@ 1363627040 [2013-03-18 10:17:20] Makefiles read: total=1220 depend=1209 seconds=33
--- /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage.amd64 ---
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage for amd64 ...
--- /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/ ---
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/stage for host ...
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs/pseudo/the-lot for amd64 ...
[Creating objdir /var/obj/projects-bmake/amd64/pkgs/pseudo/the-lot...]
Building /var/obj/projects-bmake/amd64/pkgs/pseudo/the-lot/all
--- all ---
--- /b/sjg/work/FreeBSD/projects-bmake/src/pkgs.amd64 ---
Checking /b/sjg/work/FreeBSD/projects-bmake/src/pkgs for amd64 ...
@ 1363638737 [2013-03-18 13:32:17] Finished the-lot seconds=11730

More recently, to support make universe which uses multiple combinations of MACHINE and MACHINE_ARCH we use the support for TARGET_SPEC in

Instead of simply using ${MACHINE} to distinguish targets we use TARGET_SPEC_VARS concatenated with , eg. ${MACHINE},${MACHINE_ARCH}


I should note that I generally run make via a wrapper script (mk) which first reads a file $SB/.sandbox-env (where SB is typically the parent dir to src/) to condition the environment. I have dozens of active trees for Junos, NetBSD and FreeBSD and multiple branches and this allows me use the same user interface for all.

For the projects/bmake branch the key environment items are:

export SRCTOP=$SB/src
export OBJROOT=/var/obj/projects-bmake/
export MAKESYSPATH=$SRCTOP/share/mk
export HOST_TARGET=freebsd10-amd64

Not all are strictly necessary as * can set them, but putting such logic in the makefiles constrains choices.

Note that MAKEOBJDIR value is single quoted, this defers expansion until bmake sees it.

HOST_TARGET is set by the wrapper script, based on uname output of the host machine. For example freebsd10-amd64 or freebsd7-i386. When building for the pseudo machine host we use ${HOST_OBJTOP} rather than ${OBJTOP}. This allows us to avoid trouble when the same tree is mounted via NFS and built from incompatible machines.

I have set a number of options:


WITH_AUTO_OBJ is important, since it causes the equivalent of make obj to be done automatically and early, so that as the makefile is read .OBJDIR and .CURDIR have their correct values which matters in the gcc build where we see:

.PATH: ../cc_tools

which is supposed to add ${.OBJDIR}/../cc_tools to .PATH, but if ${.OBJDIR} does not exist at the time that line is read, the right thing will not happen. /* imagine something very witty here */