bmake

Introduction

Many years ago, when building large software projects, I used GNU make aka gmake (or my own patched version of it), and had developed a set of macros to simplify developing complex build trees.

Since the early 90's (when my Sun workstation got zapped by a lighting strike) my main development machines, run BSD (NetBSD and FreeBSD). The BSD source tree is a good example of a large software project. In following NetBSD's -current development, I learned to use the BSD Makefiles and make(1). It was a vast improvement over anything I'd seen with gmake. I did an autoconf version of NetBSD's make that I'll refer to as bmake(1) from here on.

Since then all my new projects and many of my old ones use bmake. Because bmake.tar.gz uses GNU's autoconf and is very portable, I've skipped the effort of making the rest of my distributions support GNU make and thanks to mk-files I don't have to consider using automake which produces huge unreadable makefiles.

As noted above; bmake is derived from NetBSD's make(1), its goal is to be a portable version of same, so new features are added via imports of NetBSD's make (I'm one of the contributors to NetBSD). Thus bmake is kept in sync with NetBSD's make.

For those interested in change history, the NetBSD mirror on github is handy. Almost all meaningful changes to bmake can be seen here: https://github.com/NetBSD/src/tree/trunk/usr.bin/make

Since 2000 I've worked on Junos (a FreeBSD derived OS) which builds using bmake (in meta mode). FreeBSD 10.0 and later also use bmake (I'm a FreeBSD committer as well).

Since bmake-20121212 the distribution includes the mk/* files from my mk.tar.gz. This a collection of *.mk files (many originally derrived from early NetBSD versions) that can augment and even substitute for bsd.*.mk (use the real ones if you have them).

Including mk.tar.gz with bmake helps those who just want:

./configure
make
make install

to work as they expect.

Note: on Darwin (OS/X) or anything else with a case insensitive filesystem, the above method will fail since it cannot differentiate makefile from Makefile, resulting in infinite recursion. Actually for Darwin configure knows to disable the generation of makefile by default. Regardless, the simple expedient of using a separate directory for building in, will work:

mkdir obj
cd obj
../configure
make
make install

Since about 2003, the bmake version typically represents the date of import from NetBSD, and I avoid any code changes in bmake which are not strictly related to portability. All new features come via NetBSD make.

It may be interesting to note that I have projects where much of the tree uses the same simple Makefile:

PROG=   ${.CURDIR:T}

.include <prog.mk>

In fact you could just have a default Makefile for the whole project (in found via MAKESYSPATH) that did something like:

.if ${.CURDIR:M*lib/*}
LIB = ${.CURDIR:T:S,^lib,,}
.include <lib.mk>
.else
PROG = ${.CURDIR:T}
.include <prog.mk>
.endif

but that might be going too far ;-)

macros

The example Makefile above, perhaps suggests why I like bmake(1) The important magic is in the line:

.include <prog.mk>

Makefiles for libraries include lib.mk btw. Anyway, apart from reading a bunch of rules from the system macros (/usr/share/mk/* by default) it reads "../Makefile.inc" thus providing a hook for much magic.

In the early 90's NetBSD's bsd.*.mk were strictly targeted at building the NetBSD src tree, and were not as flexible or portable as I wanted. So I started on my own mk-files.

sys.mk

A very powerful feature of bmake results from its handling of makefiles, the following are read in this order:

sys.mk

Found via ${MAKESYSPATH} this makefile (and anything it includes) provide for initial rules and settings.

Makefile

Actually the first member of the list ${.MAKE.MAKEFILE_PREFERENCE} (default is makefile Makefile) that is found, and anything it includes, such as bsd.lib.mk.

.depend

Actually whatever ${.MAKE.DEPENDFILE} is set to, if it exists; is read, and anything it includes. This makefile is special in that bmake is told to be tollerant of stale dependencies, this same property applies to any makefile included using .dinclude.

The above provide for very elaborate arrangements like the DIRDEPS_BUILD.

posix.mk

As of 20220418 if the special target .POSIX: is encountered, as the first non-comment line of the main makefile (as specified by POSIX), bmake will try to include posix.mk to provide for POSIX compatible default rules.

.OBJDIR

Another very cool feature of bmake(1) is the built in distinction between ${.CURDIR} (where bmake was launched) and ${.OBJDIR} (where it is working).

This behavior can be confusing to those unfamiliar with it. The algorithm for finding an object dir to work in can be expressed using shell syntax:

for __objdir in ${MAKEOBJDIRPREFIX}${.CURDIR} \
      ${MAKEOBJDIR} \
      ${.CURDIR}/obj.${MACHINE} \
      ${.CURDIR}/obj \
      /usr/obj${.CURDIR} \
      ${.CURDIR}
do
        if [ -d ${__objdir} -a ${__objdir} != ${.CURDIR} ]; then
                break
        fi
done

In the simplest case, if the directory obj exists in the directory bmake was invoked in, then bmake will chdir into it before doing anything else. This helps keep src and object files separate. If OBJMACHINE is defined, then the default object dir is obj.${MACHINE}. Note that obj can be a symlink off to a separate file system, eg.:

/usr/src/bin/cat/obj -> /usr/obj/bin/cat

in either case, building for multiple architectures is easy.

Even better; if MAKEOBJDIRPREFIX is set in the environment (see below) then it is trivial to export a source tree read-only, since there is no longer a need for obj dirs (or symlinks) in the tree itself.

The only hassle with MAKEOBJDIR and MAKEOBJDIRPREFIX is that they need to be set in the environment. Actually recent versions of bmake allow the objdir to be set later, by use of .OBJDIR:, but setting MAKEOBJDIR* in the environment is simpler. This feature is used if you use mk-files and set MKOBJDIRS=auto.

MAKEOBJDIRPREFIX

If MAKEOBJDIRPREFIX is set in the environment and the directory ${MAKEOBJDIRPREFIX}${.CURDIR} (.CURDIR is set by getcwd()) exists, make(1) will chdir into it rather than look for ./obj as described above.

This simple feature allows true read-only source trees. Note that MAKEOBJDIRPREFIX is actioned before any Makefiles are read which is why it must be an environment variable.

MAKEOBJDIR

Bmake also allows variable substitutions on MAKEOBJDIR which makes it even better than MAKEOBJDIRPREFIX. For a neater result you can use something like:

export MAKEOBJDIR="\${.CURDIR:S,$SRCTOP,$OBJTOP,}"

which would result in:

$ bmake -C /usr/src/bin/cat -V SRCTOP -V OBJTOP -V .CURDIR -V .OBJDIR
/usr/src
/var/obj
/usr/src/bin/cat
/var/obj/bin/cat
$

MACHINE and MACHINE_ARCH

The variables MACHINE and MACHINE_ARCH are built into bmake. They can also be controlled via the makefiles for cross-building.

MACHINE
This is normally used to represent a specific cpu such as i386.
MACHINE_ARCH

In some cases a range of cpu's have a common architecture such as mips or m68k. MACHINE_ARCH is used to select common include dirs, toolchains etc.

Actually in FreeBSD, there can be multiple MACHINE_ARCH for a single MACHINE.

HOST_TARGET

When cross building it can be useful to differentiate things built for the host rather than the target. HOST_MACHINE and HOST_MACHINE_ARCH as well as TARGET_* can be useful.

I tend to use the pseudo MACHINE host, but use ${HOST_TARGET} for its objdir so that the same tree can be safely built by mutiple hosts without confusion.

Verifying bmake

The latest bmake should have an OpenPGP detached signature:

lrwxrwxr-x  1 sjg  wheel  21 Apr 26 16:41 bmake.tar.gz -> bmake-20210420.tar.gz
lrwxrwxr-x  1 sjg  wheel  25 Apr 26 16:41 bmake.tar.gz.asc -> bmake-20210420.tar.gz.asc

The public key for this can be found in https://www.crufty.net/ftp/pub/sjg/Crufty.pub.asc

In theory it should also be available from keyservers like pgp.mit.edu. Though I get an error...

Its full key-id is 7E228507C26D8DC164F531BFBA54C8AF755A2A99 the OpenPGP key-id is BA54C8AF755A2A99

You can try to find/import the above key using:

gpg --search-keys sigs@crufty.net

or:

gpg --receive-keys 7E228507C26D8DC164F531BFBA54C8AF755A2A99

or more reliably from the file above:

gpg --import Crufty.pub.asc

after which you should be able to:

gpg --openpgp --verify bmake.tar.gz.asc
gpg: assuming signed data in 'bmake.tar.gz'
gpg: Signature made Mon Apr 26 16:29:27 2021 PDT
gpg:                using RSA key BA54C8AF755A2A99
gpg: Good signature from "sigs (OpenPGP) <sigs@crufty.net>" [unknown]
Primary key fingerprint: 7E22 8507 C26D 8DC1 64F5  31BF BA54 C8AF 755A 2A99

Building bmake

Since bmake-20121212 a simple makefile is provided for the benefit of folk who simply want to:

$ tar zxf ~/bmake-$MAKE_VERSION.tar.gz
$ cd bmake
$ configure
$ make
# make install

where make is not bmake.

As noted earlier; the above will fail on systems like Darwin where it is necessary to separate the build from the srcs. This is approximately what bmake/boot-strap does:

$ tar zxf ~/bmake-$MAKE_VERSION.tar.gz
$ mkdir obj
$ cd obj
$ ../bmake/configure
$ make
# make install

Most of the time though, to build and install bmake I typically do:

$ cd ~/tmp
$ tar zxf ~/bmake-$MAKE_VERSION.tar.gz
$ ./bmake/boot-strap --prefix=$HOME --install-host-target -DWITH_PROG_VERSION

This gets me:

$HOME/$HOST_TARGET/bin/bmake-$MAKE_VERSION
$HOME/$HOST_TARGET/bin/bmake -> bmake-$MAKE_VERSION
$HOME/share/mk/*
$HOME/share/man/man1/bmake.1.gz

Note: the trivial makefile just leverages boot-strap.

If you already have bmake (or equivalent) available, the trivial makefile is not necessary and in fact quite anoying (see the Juniper example below).

Thus as of bmake-20181222 if you just unpack the tarball and run bmake, the Makefile will add --without-makefile to the configure command to suppress the above.

So you can just:

$ cd ~/tmp
$ tar zxf ~/bmake-$MAKE_VERSION.tar.gz

Note: if you want to keep generated files out of the src directory then you have to use MAKEOBJDIRPREFIX (or the same MAKEOBJDIR setup as boot-strap does), so that the unit-tests/Makefile.config will be generated in the correct place. Eg.:

$ mkdir /tmp/obj
$ export MAKEOBJDIRPREFIX=/tmp/obj
$ export WITH_AUTO_OBJ=1

(note: the above syntax is for a POSIX compatible shell) If you just use the default obj dir in each src directory, we'd get bmake/obj/unit-tests/Makefile.config which would not be visible in bmake/unit-tests/obj/.

Then:

$ cd bmake
$ bmake
$ bmake test
# bmake install

will just work.

At Juniper I have a set of Mercurial repos to facilitate testing new versions before importing into the official repository (we've used bmake to build Junos for over 20 years).

They are all setup in a single tree:

bmake/crufty
bmake/junos
bmake/freebsd

Each contains a bmake subdir containing the actual code. There are of course various Makefile.inc files setting all the options I use like:

WITH_PROG_VERSION=1
WITH_PROG_LINK=1
WITH_AUTO_OBJ=1
.if defined(.MAKE.PATH_FILEMON)
.MAKE.MODE += meta verbose silent=yes
.endif

The crufty repo tracks pristine bmake from upstream.

The junos repo has a couple of local tweaks.

The freebsd repo also mimics the local tweaks in FreeBSD - mostly to facilitate the transition from their older make to bmake.

The junos and freebsd repos just hg pull from the crufty one, and in each case:

mk &&
mk test &&
mk install

does all that is needed, to get the latest $HOME/$HOST_TARGET/bin/bmake-$MAKE_VERSION ready for testing.

In this setup, that trivial makefile gets in the way. With bmake-20181222 and later bmake/Makefile uses configure --without-makefile, so I don't need to specify -f Makefile.

The script mk above is part of my sb-tools collection and makes it easy to condition the environment for building a specific project. Especially useful for Emacs users.

Each new version of bmake goes through the above on several platforms (NetBSD, FreeBSD, SunOS, Linux, OS/X), and I typically spend a week building Junos with it before posting. Due to its size and complexity; the Junos build is a good torture test for make.

The example below is building bmake-20121212 on SunOS. Note that the build isn't considered successful unless the unit-tests pass.

Also note that the src directories are left untouched and a host-target specific objdir is used. Thus I can use the exact same command sequence to build bmake for all the platforms available:

$ ./bmake/boot-strap --prefix=$HOME --install-host-target -DDWITH_PROG_VERSION
NOTE: default prefix=/homes/sjg INSTALL_BIN=sunos5-sparc/bin
NOTE: reading /homes/sjg/.bmake-boot-strap.rc
Building for sunos5-sparc
checking for gcc... /usr/local/bin/gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
...
checking if diff -u works... yes
checking for MACHINE & MACHINE_ARCH...
defaults: MACHINE=sunos5, MACHINE_ARCH=sparc
Using: MACHINE=sunos5, MACHINE_ARCH=sparc
Using: MKSRC=${srcdir}/mk
Using: SHELL=/usr/xpg4/bin/sh
configure: creating ./config.status
config.status: creating makefile
config.status: creating Makefile.config
config.status: creating make-bootstrap.sh
config.status: creating unit-tests/Makefile
config.status: creating config.h

You can now run

        sh ./make-bootstrap.sh

to produce a fully functional bmake.

/usr/local/bin/gcc -c -g -O2 -I. -I/homes/sjg/tmp/bmake -DHAVE_CONFIG_H -DNEED_MAKE_LEVEL_SAFE -I/homes/sjg/tmp/bmake/missing -DMAKE_NATIVE -DUSE_META -DMAKE_VERSION="20121212" -DMACHINE="sunos5" -DMACHINE_ARCH="sparc" -D_PATH_DEFSYSPATH="/homes/sjg/share/mk" -o main.o /homes/sjg/tmp/bmake/main.c
...
/usr/local/bin/gcc -c -g -O2 -I. -I/homes/sjg/tmp/bmake -DHAVE_CONFIG_H -DNEED_MAKE_LEVEL_SAFE -I/homes/sjg/tmp/bmake/missing -DMAKE_NATIVE -DUSE_META -o meta.o /homes/sjg/tmp/bmake/meta.c
/usr/local/bin/gcc -o bmake main.o meta.o arch.o buf.o compat.o cond.o dir.o for.o getopt hash.o job.o make.o make_malloc.o parse.o sigcompat.o str.o strlist.o suff.o targ.o trace.o var.o util.o lstAppend.o lstDupl.o lstInit.o lstOpen.o lstAtEnd.o lstEnQueue.o lstInsert.o lstAtFront.o lstIsAtEnd.o lstClose.o lstFind.o lstIsEmpty.o lstRemove.o lstConcat.o lstFindFrom.o lstLast.o lstReplace.o lstFirst.o lstDatum.o lstForEach.o lstMember.o lstSucc.o lstDeQueue.o lstForEachFrom.o lstDestroy.o lstNext.o lstPrev.o stresep.o
cd /homes/sjg/tmp/bmake/unit-tests && MAKEFLAGS= /homes/sjg/tmp/sunos5-sparc/bmake -r -m / TEST_MAKE=/homes/sjg/tmp/sunos5-sparc/bmake test
/homes/sjg/tmp/sunos5-sparc/bmake -f /homes/sjg/tmp/sunos5-sparc/unit-tests/Makefile > test.out 2>&1
/usr/local/bin/diff -u /homes/sjg/tmp/bmake/unit-tests/test.exp test.out
[ -d /homes/sjg/sunos5-sparc/bin ] ||  /homes/sjg/tmp/bmake/install-sh -d -o sjg -g 705 -m 775 /homes/sjg/sunos5-sparc/bin
/homes/sjg/tmp/bmake/install-sh -c -s -o sjg -g 705 -m 555  bmake /homes/sjg/sunos5-sparc/bin/bmake-20121212
test -d /homes/sjg/sunos5-sparc/bin || /homes/sjg/tmp/bmake/install-sh -m 775 -d /homes/sjg/sunos5-sparc/bin
test -d /homes/sjg/share/man/cat1 || /homes/sjg/tmp/bmake/install-sh -m 775 -d /homes/sjg/share/man/cat1
/homes/sjg/tmp/bmake/install-sh -c -o sjg -g 705 -m 444 /homes/sjg/tmp/bmake/bmake.cat1 /homes/sjg/share/man/cat1/bmake.1
/homes/sjg/sunos5-sparc/bin/bmake -> bmake-20121212
test -d /homes/sjg/share/mk || /homes/sjg/tmp/bmake/install-sh -m 775 -d /homes/sjg/share/mk
sh /homes/sjg/tmp/bmake/mk/install-mk -v -m 644 /homes/sjg/share/mk
cp -f sys.mk auto.obj.mk autoconf.mk autodep.mk auto.dep.mk dep.mk doc.mk dpadd.mk final.mk host-target.mk host.libnames.mk inc.mk init.mk java.mk lib.mk libnames.mk libs.mk links.mk man.mk nls.mk obj.mk options.mk own.mk prlist.mk prog.mk progs.mk rst2htm.mk scripts.mk srctop.mk subdir.mk sys.clean-env.mk sys.dependfile.mk target-flags.mk warnings.mk yacc.mk dirdeps.mk gendirdeps.mk install-new.mk meta.sys.mk meta.autodep.mk meta.stage.mk meta.subdir.mk /homes/sjg/share/mk
cp -f sys/AIX.mk sys/Darwin.mk sys/Generic.mk sys/HP-UX.mk sys/IRIX.mk sys/Linux.mk sys/NetBSD.mk sys/OSF1.mk sys/OpenBSD.mk sys/SunOS.mk sys/UnixWare.mk /homes/sjg/share/mk/sys
cp -f meta2deps.py meta2deps.sh /homes/sjg/share/mk
chmod 644 sys.mk auto.obj.mk autoconf.mk autodep.mk auto.dep.mk dep.mk doc.mk dpadd.mk final.mk host-target.mk host.libnames.mk inc.mk init.mk java.mk lib.mk libnames.mk libs.mk links.mk man.mk nls.mk obj.mk options.mk own.mk prlist.mk prog.mk progs.mk rst2htm.mk scripts.mk srctop.mk subdir.mk sys.clean-env.mk sys.dependfile.mk target-flags.mk warnings.mk yacc.mk dirdeps.mk gendirdeps.mk install-new.mk meta.sys.mk meta.autodep.mk meta.stage.mk meta.subdir.mk sys/AIX.mk sys/Darwin.mk sys/Generic.mk sys/HP-UX.mk sys/IRIX.mk sys/Linux.mk sys/NetBSD.mk sys/OSF1.mk sys/OpenBSD.mk sys/SunOS.mk sys/UnixWare.mk
chmod 555 meta2deps.py meta2deps.sh

make-bootstrap.sh

Folk who just run boot-strap as above probably won't notice, but as of bmake-20100222, there is no longer a need for a native make program during the bootstraping of bmake. A simple shell script (contibuted by joerg at netbsd.org) takes the place of makefile.boot.

Unit tests

The build of bmake includes checking that its unit tests all pass. In recent years the test suite has been considerably extended (mostly by rillig at netbsd.org).

BROKEN_TESTS

Much as we try to make the test suite portable, some tests simply will not work on some platforms.

Such tests are added to BROKEN_TESTS so that they will be skipped. For example:

archive is broken on FreeBSD and possibly others.
This functionality hasn't been seriously used for at least 20 years so I just block it from TESTS ;-)

shell-ksh is broken on Darwin - the shell segfaults.

sh-flags fails if .MAKE.SHELL is ksh due to some
differences between it an sh.

And about half a dozen tests are broken on SCO_SV.

If you find a test that fails on your platform, and conclude it is due to some quirk of the OS, you can add it to BROKEN_TESTS in your environment or in the makefile, so you can check that everything else is working ok. Sending feedback would be a good idea.

meta mode

Since 2010-09 bmake supports meta mode - contributed by Juniper Networks. The filemon kernel module is currently available in NetBSD and FreeBSD. FreeBSD 10 and later use bmake.

Since bmake-20200121 on NetBSD, bmake uses filemon_ktrace which uses the fktrace(2) api. This avoids the need for a separate filemon driver. The filemon_dev interfaces is used on other platforms to interract with the filemon driver.

A version of filemon for Linux can be obtained from https://github.com/trixirt/filemon-linux.git though it could probably use some serious TLC by now.

Note: patches to adapt bmake to other tracing mechanisms (while maintaining the goals of filemon) would be most welcome.

If filemon is available the data can also be leveraged to auto capture tree dependencies which can be used by dirdeps.mk to drive the build.

While dirdeps.mk will happily use manually maintained Makefile.depend files, and is still a vast improvement over alternate means of build orchestration, being able to automate the tree dependencies is preferable.

Even without filemon, meta mode is useful for capturing errors and comparing command lines to better decide if a target is out-of-date.


Author:sjg@crufty.net
Revision:$Id: bmake.txt,v 1.18 2024/02/25 18:55:49 sjg Exp $
Copyright:Crufty.NET