bmake

Introduction

Many years ago, when building large software projects, I used GNU make (or my own patched version of it), and had developed a set of macros to simplify developing complex build trees.

Since the early 90's my main development machines, run BSD (NetBSD to be precise), and the BSD source tree is good example of a large software project. In following NetBSD's -current development, I learned to use the new BSD Makefiles and make(1), I did an autoconf version of it that I'll refer to as bmake(1) from here on.

Since then all my new projects and many of my old ones use bmake. Because bmake.tar.gz uses GNU's autoconf and is quite portable, I've skipped the effort of making the rest of my distributions support GNU make.

Bmake is derived from NetBSD's make(1), its goal is to be a portable version of same, so new features are added via imports of NetBSD's make (I'm one of the contributors to NetBSD). Thus bmake is kept in sync with NetBSD's make.

Since 2000 I've worked on Junos which builds using bmake (in meta mode). Junos is a FreeBSD derrived OS. FreeBSD 10.0 and later also use bmake (I'm a FreeBSD committer as well).

Generally speaking you also need mk.tar.gz as well, this is a portable collection of mk-files that can augment and even substitute for bsd.*.mk.

With bmake-20121212 the above mk/* files are included, to help those who just want:

./configure
make
make install

to work as they expect.

Since about 2003, the bmake version typically represents the date of import from NetBSD, and I avoid any code changes in bmake which are not strictly related to portability. All new features come via NetBSD make.

It may be interesting to note that I have projects where much of the tree uses the same simple Makefile:

PROG=   ${.CURDIR:T}

.include <prog.mk>

macros

The example Makefile above, perhaps suggests why I like bmake(1) The important magic is in the line:

.include <prog.mk>

Makefiles for libraries include lib.mk btw. Anyway, apart from reading a bunch of rules from the system macros (/usr/share/mk/* by default) it reads "../Makefile.inc" thus providing a hook for much magic.

In the early 90's bsd.*.mk were strictly targeted at building the BSD src tree, and were not as flexible or portable as I wanted. So I started on my own mk-files.

.OBJDIR

Another cool feature of bmake(1) is the built in distinction between ${.CURDIR} (where bmake was launched) and ${.OBJDIR} (where it is working).

This behavior can be confusing to those unfamiliar with it. The algorithm for finding an object dir to work in can be expressed:

for __objdir in ${MAKEOBJDIRPREFIX}${.CURDIR} \
      ${MAKEOBJDIR} \
      ${.CURDIR}/obj.${MACHINE} \
      ${.CURDIR}/obj \
      ${.CURDIR}
do
        if [ -d ${__objdir} -a ${__objdir} != ${.CURDIR} ]; then
                break
        fi
done

In the simplest case, if the directory obj exists in the directory bmake was invoked in, then bmake will chdir into it before doing anything else. This helps keep src and object files separate. If the OBJMACHINE is set then the default object dir is obj.${MACHINE}. Note that obj can be a symlink off to a separate file system, eg.:

/usr/src/bin/cat/obj -> /usr/obj/bin/cat

in either case, building for multiple architectures is easy.

Even better if MAKEOBJDIRPREFIX is set in the environment (see below) then it is trivial to export a source tree read-only, since there is no longer a need for obj dirs (or symlinks) in the tree itself.

The only hassle with MAKEOBJDIR and MAKEOBJDIRPREFIX is that they need to be set in the environment. Actually recent versions of bmake allow the objdir to be set later, by use of .OBJDIR:, but setting MAKEOBJDIR* in the environment is simpler. This feature is used if you use mk-files and set MKOBJDIRS=auto.

MAKEOBJDIRPREFIX

If MAKEOBJDIRPREFIX is set in the environment and the directory ${MAKEOBJDIRPREFIX}${.CURDIR} (.CURDIR is set by getcwd()) exists, make(1) will chdir into it rather than look for ./obj as described above. This simple feature allows true read-only source trees. This entire build tree has been built using MAKEOBJDIRPREFIX. Note that MAKEOBJDIRPREFIX is actioned before any Makefiles are read which is why it must be an environment variable.

MAKEOBJDIR

Bmake also allows variable substitutions on MAKEOBJDIR which makes it even better than MAKEOBJDIRPREFIX. For a neater result you can use something like:

export MAKEOBJDIR="\${.CURDIR:S,$SRCTOP,$OBJTOP,}"

MACHINE and MACHINE_ARCH

The variables MACHINE and MACHINE_ARCH are built into bmake. They can also be controlled via the makefiles for cross-building.

MACHINE
This is normally used to represent a specific cpu such as i386.
MACHINE_ARCH
In some cases a range of cpu's have a common architecture such as mips or m68k. MACHINE_ARCH is used to select common include dirs, toolchains etc.
HOST_TARGET

When cross building it can be useful to differentiate things build for the build host rather than the target. HOST_MACHINE and HOST_MACHINE_ARCH as well as TARGET_* can be useful.

I tend to use the pseudo MACHINE host, but use ${HOST_TARGET} for its objdir so that the same tree can be safely built by mutiple hosts without confusion.

Building bmake

The most recent versions of bmake include a simple makefile for the benfit of folk who simply want to:

$ tar zxf ~/bmake-$MAKE_VERSION.tar.gz
$ configure
$ make
# make install

However, to build and install bmake I typically do:

$ cd ~/tmp
$ tar zxf ~/bmake-$MAKE_VERSION.tar.gz
$ ./bmake/boot-strap --prefix=$HOME --install-host-target -DWITH_PROG_VERSION

This gets me:

$HOME/$HOST_TARGET/bin/bmake-$MAKE_VERSION
$HOME/$HOST_TARGET/bin/bmake -> bmake-$MAKE_VERSION
$HOME/share/mk/*
$HOME/share/man/man1/bmake.1.gz

Each new version of bmake goes through the above on several platforms (NetBSD, FreeBSD, SunOS, Linux, OS/X).

The example below is building bmake-20121212 on SunOS. Note that the build isn't considered successful unless the unit-tests pass.

Also note that the src directories are left untouched and a host-target specific objdir is used. Thus I can use the exact same command sequence to build bmake for all the platforms available:

$ ./bmake/boot-strap --prefix=$HOME --install-host-target -DDWITH_PROG_VERSION
NOTE: default prefix=/homes/sjg INSTALL_BIN=sunos5-sparc/bin
NOTE: reading /homes/sjg/.bmake-boot-strap.rc
Building for sunos5-sparc
checking for gcc... /usr/local/bin/gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
...
checking if diff -u works... yes
checking for MACHINE & MACHINE_ARCH...
defaults: MACHINE=sunos5, MACHINE_ARCH=sparc
Using: MACHINE=sunos5, MACHINE_ARCH=sparc
Using: MKSRC=${srcdir}/mk
Using: SHELL=/usr/xpg4/bin/sh
configure: creating ./config.status
config.status: creating makefile
config.status: creating Makefile.config
config.status: creating make-bootstrap.sh
config.status: creating unit-tests/Makefile
config.status: creating config.h

You can now run

        sh ./make-bootstrap.sh

to produce a fully functional bmake.

/usr/local/bin/gcc -c -g -O2 -I. -I/homes/sjg/tmp/bmake -DHAVE_CONFIG_H -DNEED_MAKE_LEVEL_SAFE -I/homes/sjg/tmp/bmake/missing -DMAKE_NATIVE -DUSE_META -DMAKE_VERSION="20121212" -DMACHINE="sunos5" -DMACHINE_ARCH="sparc" -D_PATH_DEFSYSPATH="/homes/sjg/share/mk" -o main.o /homes/sjg/tmp/bmake/main.c
...
/usr/local/bin/gcc -c -g -O2 -I. -I/homes/sjg/tmp/bmake -DHAVE_CONFIG_H -DNEED_MAKE_LEVEL_SAFE -I/homes/sjg/tmp/bmake/missing -DMAKE_NATIVE -DUSE_META -o meta.o /homes/sjg/tmp/bmake/meta.c
/usr/local/bin/gcc -o bmake main.o meta.o arch.o buf.o compat.o cond.o dir.o for.o getopt hash.o job.o make.o make_malloc.o parse.o sigcompat.o str.o strlist.o suff.o targ.o trace.o var.o util.o lstAppend.o lstDupl.o lstInit.o lstOpen.o lstAtEnd.o lstEnQueue.o lstInsert.o lstAtFront.o lstIsAtEnd.o lstClose.o lstFind.o lstIsEmpty.o lstRemove.o lstConcat.o lstFindFrom.o lstLast.o lstReplace.o lstFirst.o lstDatum.o lstForEach.o lstMember.o lstSucc.o lstDeQueue.o lstForEachFrom.o lstDestroy.o lstNext.o lstPrev.o stresep.o
cd /homes/sjg/tmp/bmake/unit-tests && MAKEFLAGS= /homes/sjg/tmp/sunos5-sparc/bmake -r -m / TEST_MAKE=/homes/sjg/tmp/sunos5-sparc/bmake test
/homes/sjg/tmp/sunos5-sparc/bmake -f /homes/sjg/tmp/sunos5-sparc/unit-tests/Makefile > test.out 2>&1
/usr/local/bin/diff -u /homes/sjg/tmp/bmake/unit-tests/test.exp test.out
[ -d /homes/sjg/sunos5-sparc/bin ] ||  /homes/sjg/tmp/bmake/install-sh -d -o sjg -g 705 -m 775 /homes/sjg/sunos5-sparc/bin
/homes/sjg/tmp/bmake/install-sh -c -s -o sjg -g 705 -m 555  bmake /homes/sjg/sunos5-sparc/bin/bmake-20121212
test -d /homes/sjg/sunos5-sparc/bin || /homes/sjg/tmp/bmake/install-sh -m 775 -d /homes/sjg/sunos5-sparc/bin
test -d /homes/sjg/share/man/cat1 || /homes/sjg/tmp/bmake/install-sh -m 775 -d /homes/sjg/share/man/cat1
/homes/sjg/tmp/bmake/install-sh -c -o sjg -g 705 -m 444 /homes/sjg/tmp/bmake/bmake.cat1 /homes/sjg/share/man/cat1/bmake.1
/homes/sjg/sunos5-sparc/bin/bmake -> bmake-20121212
test -d /homes/sjg/share/mk || /homes/sjg/tmp/bmake/install-sh -m 775 -d /homes/sjg/share/mk
sh /homes/sjg/tmp/bmake/mk/install-mk -v -m 644 /homes/sjg/share/mk
cp -f sys.mk auto.obj.mk autoconf.mk autodep.mk auto.dep.mk dep.mk doc.mk dpadd.mk final.mk host-target.mk host.libnames.mk inc.mk init.mk java.mk lib.mk libnames.mk libs.mk links.mk man.mk nls.mk obj.mk options.mk own.mk prlist.mk prog.mk progs.mk rst2htm.mk scripts.mk srctop.mk subdir.mk sys.clean-env.mk sys.dependfile.mk target-flags.mk warnings.mk yacc.mk dirdeps.mk gendirdeps.mk install-new.mk meta.sys.mk meta.autodep.mk meta.stage.mk meta.subdir.mk /homes/sjg/share/mk
cp -f sys/AIX.mk sys/Darwin.mk sys/Generic.mk sys/HP-UX.mk sys/IRIX.mk sys/Linux.mk sys/NetBSD.mk sys/OSF1.mk sys/OpenBSD.mk sys/SunOS.mk sys/UnixWare.mk /homes/sjg/share/mk/sys
cp -f meta2deps.py meta2deps.sh /homes/sjg/share/mk
chmod 644 sys.mk auto.obj.mk autoconf.mk autodep.mk auto.dep.mk dep.mk doc.mk dpadd.mk final.mk host-target.mk host.libnames.mk inc.mk init.mk java.mk lib.mk libnames.mk libs.mk links.mk man.mk nls.mk obj.mk options.mk own.mk prlist.mk prog.mk progs.mk rst2htm.mk scripts.mk srctop.mk subdir.mk sys.clean-env.mk sys.dependfile.mk target-flags.mk warnings.mk yacc.mk dirdeps.mk gendirdeps.mk install-new.mk meta.sys.mk meta.autodep.mk meta.stage.mk meta.subdir.mk sys/AIX.mk sys/Darwin.mk sys/Generic.mk sys/HP-UX.mk sys/IRIX.mk sys/Linux.mk sys/NetBSD.mk sys/OSF1.mk sys/OpenBSD.mk sys/SunOS.mk sys/UnixWare.mk
chmod 555 meta2deps.py meta2deps.sh

make-bootstrap.sh

Folk who just run boot-strap as above probably won't notice, but as of bmake-20100222, there is no longer a need for a native make program during the bootstraping of bmake. A simple shell script (contibuted by joerg at netbsd.org) takes the place of makefile.boot.

meta mode

Since 2010-09 bmake supports meta mode - contributed by Juniper Networks. The filemon kernel module is currently only available for NetBSD and FreeBSD.

A version of filemon for Linux also exists and will hopefully be released once it is deemed ready.

Even without filemon, meta mode is useful for capturing errors and allowing dirdeps.mk to drive the build.


Author:sjg@crufty.net
Revision:$Id: bmake.txt,v 1.9 2013/08/03 01:28:19 sjg Exp $
Copyright:Crufty.NET