Building NetBSD in meta mode

This doc describes building NetBSD using bmake in meta mode leveraging the work done building FreeBSD in meta mode

Junos (which is based on FreeBSD) has been built this way for a number of years, and it works very well.

Note: while I normally talk of meta mode and using dirdeps.mk together, either can be used without the other.

Introduction

There isn't a lot wrong with the NetBSD build as it is. At least for doing release builds and such. It builds in parallel ok, and can be cross-built on a number of different hosts. Thus I've not previously been seriously tempted to interfere with it.

However the prospect of doing a dev project on NetBSD was enough to prompt me to see what it would take to get the meta mode build (which is hard to live without once you've tried it ;-) working on NetBSD.

The short answer is very little. Frankly despite planning to leverage work already done for FreeBSD, I was very pleasantly surprised to find how little effort it took to get bin/cat (and everything it needs) building. Most of that time was spend finding the various places that needed to be bootstrapped to link cat.

After building for i386 I then build for something else to see what the impact on the dependencies are. For FreeBSD I found with just parameterizing CSU_DIR a single Makefile.depend worked for 99% of the tree, and most places where Makefile.depend.${MACHINE} was needed it was due to generated files being named differently.

For NetBSD some of the dependencies varied quite a bit between i386 and evbarm, but nothing that cannot be handled by a little filtering - which the meta mode infrastructure provides for.

What is this meta mode thing anyway?

Fair question.... It is a better way of building a large tree of software.

Most of this is covered in building FreeBSD in meta mode But I'll briefly rehash the story here.

My goal is to be able to start a build from anywhere in a freshly checked out tree (eg. bin/cat) and have it just work. That means building everything else in the tree needed, in the right order, in parallel. To be able to do that for multiple target machines at the same time and even multiple target operating systems at the same time. All in a single pass of the tree.

All the above is handled by using dirdeps.mk which is used by the initial make instance to compute a graph of tree dependencies from the current origin, and then go build them all in parallel.

The tree wide graph ensures that all leaf directories are visited in the correct order - no tree walks. In fact dirdeps.mk suppresses the behavior of bsd.subdir.mk.

Using dirdeps.mk we can greatly simplify the top-level build logic, which apart from share/mk/ is where we typically find a concentration of complexity.

NetBSD:

$ wc -l Makefile build.sh
     530 Makefile
    2309 build.sh
    2839 total

FreeBSD:

$ wc -l Makefile Makefile.inc1
     525 Makefile
    2196 Makefile.inc1
    2721 total

The pre-meta mode Junos build had over 5000 lines of very dense makefile at the top-level. The original top-level makefiles for the meta mode build that replaced all that was less than 300 lines!

NetBSD/FreeBSD meta mode:

$ wc -l targets/Ma* share/mk/dirdeps.mk
     172 targets/Makefile
      52 targets/Makefile.inc
      46 targets/Makefile.xtras
     644 share/mk/dirdeps.mk
     914 total

I included dirdeps.mk in the line count because that's where most of the complexity is, but of course we leverage it for all builds not just top-level ones.

Adding a new top-level target is as simple as creating a directory somewhere under targets/ and putting a Makefile.depend file there that lists the directories needed, the Makefile is often trivial - leveraging common packaging logic. These makefiles though are no more complex than any leaf makefile.

Meta files

When make is run in meta mode, it creates a .meta file for each target being built. The meta file captures information that allows make to do a much better job.

Command line

Firstly we capture the expanded command line. This allows make to compare the command it might need to run with what was done last time, if anything changed the target is out-of-date.

Now sometimes you want to suppress that behavior so there are knobs to do so for a whole target or just one line.

This feature alone makes a huge difference to build reliability. It is no longer necessary to make targets depend on every makefile that might influence them. This allows us to avoid building things when nothing relevant has changed, while ensuring we never fail to rebuild when they have.

The captured command is also invaluable for debugging.

Command output

The next thing captured is any output from the command. This is mainly for human debugging purposes.

This extremely useful when you have 1000's of users who fail to log their builds and still want to you help them understand why their build failed.

Syscall trace

The filemon module provides /dev/filemon which make will use if available.

When make runs a child, it opens /dev/filemon and gives it a temp file and the pid of the child. All successful syscalls (interesting to make) made by the child and its progeny will be recorded in the temp file. When the child exits, make appends the data to the meta file.

This data has two main uses. Firstly make itself uses it to better check if a target needs update. Secondly we can post-process the data for all sorts of purposes including extracting directory dependencies - which allows us to automate the collection of the data needed by dirdeps.mk.

For example bin/cat/Makefile.depend:

# Autogenerated - do NOT edit!

DIRDEPS = \
        external/gpl3/gcc/lib/libgcc/libgcc \
        external/gpl3/gcc/lib/libgcc/libgcc_s \
        include \
        lib/csu \
        lib/libc \
        lib/libpthread \
        sys/sys \


.include <dirdeps.mk>

.if ${DEP_RELDIR} == ${_DEP_RELDIR}
# local dependencies - needed for -jN in clean tree
.endif

As we read the files read or executed, any we see in an object directory which is not ${.OBJDIR} indicates a directory that needs to be built before ${.CURDIR}. Being able to map object dirs to src dirs is important.

The local dependencies section is where we record any dependencies involving locally generated files - also gleaned from the syscall trace. This allows us to reliably build a clean tree in parallel while skipping any explict depend step.

The syscall trace is also invaluable for debugging weird build issues. For example spotting that your supposed captive build is reading headers from /usr/include or linking libs from /usr/local/lib. Or running a perl or python binary other than the one intended. Being able to look under the covers after a build has failed is immensely useful.

meta_oodate

When make is checking if a target needs update and the normal makefile rules indicate it is up-to-date, we delve into the meta file. Here is cat.o.meta:

# Meta data file /h/NetBSD/5.X/obj/i386/bin/cat/cat.o.meta
CMD @echo '#  ' "compile " cat/cat.o
CMD /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc -O2  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare -Wno-traditional -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wno-unused-parameter     -nostdinc -I/h/NetBSD/5.X/obj/stage/i386/usr/include -nostdinc -isystem /h/NetBSD/5.X/obj/stage/i386/usr/include   -c    /h/NetBSD/5.X/src/bin/cat/cat.c
CWD /h/NetBSD/5.X/obj/i386/bin/cat
TARGET cat.o
-- command output --
#   compile  cat/cat.o

-- filemon acquired metadata --
# filemon version 4
# Target pid 2050
V 4
E 8142 /bin/sh
R 8142 /etc/ld.so.conf
R 8142 /lib/libedit.so.2
R 8142 /lib/libtermcap.so.0
R 8142 /lib/libc.so.12
X 8142 0
# Bye bye
E 5426 /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc
R 5426 /etc/ld.so.conf
R 5426 /usr/lib/libc.so.12
R 5426 /var/tmp//ccKCUD9I.s
W 5426 /var/tmp//ccKCUD9I.s
E 15146 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../libexec/gcc/i386--netbsdelf/4.1.3/cc1
R 15146 /etc/ld.so.conf
R 15146 /usr/lib/libc.so.12
R 15146 /h/NetBSD/5.X/src/bin/cat/cat.c
R 15146 /var/tmp//ccKCUD9I.s
W 15146 /var/tmp//ccKCUD9I.s
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/cdefs.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs_elf.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/param.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/null.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/inttypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stdint.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_mwgwtypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_limits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_const.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/wchar_limits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_fmtio.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/ansi.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ansi.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/endian.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian_machdep.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/byte_swap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/bswap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/fd_set.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/pthread_types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/syslimits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/signal.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/sigtypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/satypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/siginfo.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/signal.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/trap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/x86/trap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ucontext.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/mcontext.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/param.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/limits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stat.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/select.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/time.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/ctype.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/err.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/errno.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/errno.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/fcntl.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/locale.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdio.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdlib.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/strings.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/unistd.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/unistd.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
X 15146 0
E 18126 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../lib/gcc/i386--netbsdelf/4.1.3/../../../../i386--netbsdelf/bin/as
R 18126 /etc/ld.so.conf
R 18126 /usr/lib/libc.so.12
R 18126 cat.o
W 18126 cat.o
R 18126 /var/tmp//ccKCUD9I.s
X 18126 0
D 5426 /var/tmp//ccKCUD9I.s
X 5426 0
# Bye bye

I bet you didn't know it takes all those headers to compile cat.o. That's 79 headers (54 unique), not what you would expect from just looking at cat.c.

Anyway, as noted above the first thing we do is check if the command line changed, any change to CFLAGS would do it. If it did - we are done - out-of-date.

Otherwise we look at the syscall trace. This is formatted so that it is trivial to parse by even a shell script.

In fact I have a collection of shell functions for extracting useful data from meta files - like extracting the command so it can be re-run easily, spliting the command into individual words one per line so it is much easier to compare with another using diff(1). Extracting the list of files read - all resolved to absoulte paths, and so on.

In the above example it is simple, we just need to look at all the E (exec) and R (read) entries; if any of the files read/executed are newer the target is out-of-date.

Sometimes the syscall trace is more complex, so make needs to follow the F (fork) and C (chdir) syscalls by pid so it knows what the cwd of each is so if it sees a file opened by a relative path it can find it.

If a file previously written is missing (and the syscall trace does not show it being D deleted), the target is out-of-date.

Of course make will ignore some paths (eg. anything in /tmp) and you can provide it a list of path prefixes to ignore.

The ability for make to look under the covers like this allows it to do a much better job of ensuring things are re-built when they need to be.

You can use the debug flag -dM to have make tell you what it is about a meta file that made it decide the target is out-of-date.

filemon

The filemon module is currently available in NetBSD and FreeBSD and a version for Linux is available from https://github.com/trixirt/filemon-linux.git

Makefiles

Initially I only had to touch one makefile outside of share/mk and that was include/Makefile because I wanted an obj dir:

Index: include/Makefile
===================================================================
RCS file: /cvsroot/src/include/Makefile,v
retrieving revision 1.140
diff -u -p -r1.140 Makefile
--- include/Makefile    11 Dec 2013 01:24:08 -0000      1.140
+++ include/Makefile    13 May 2015 18:53:29 -0000
@@ -3,7 +3,9 @@

 # Doing a make includes builds /usr/include

+.if ${MK_DIRDEPS_BUILD:Uno} == "no"
 NOOBJ=         # defined
+.endif

 # Missing: mp.h

by contrast I had to quite heavily tweak the FreeBSD equivalent. This was a pleasant surprise.

Though since include/Makefile isn't making some of the symlinks in ${INCSDIR} that are obviously needed we had to compensate in local.final.mk (see below).

share/mk

We hook our magic into the build by introducing the notion of local.*.mk this is a very handy construct. For example:

Index: share/mk/sys.mk
===================================================================
RCS file: /cvsroot/src/share/mk/sys.mk,v
retrieving revision 1.127
diff -u -p -r1.127 sys.mk
--- share/mk/sys.mk     10 Aug 2014 05:57:31 -0000      1.127
+++ share/mk/sys.mk     13 May 2015 04:45:18 -0000
@@ -7,6 +7,8 @@

 unix?=         We run NetBSD.

+.-include "local.sys.mk"
+
 .SUFFIXES: .a .o .ln .s .S .c .cc .cpp .cxx .C .f .F .r .p .l .y .sh

 .LIBS:         .a
Index: share/mk/bsd.init.mk
===================================================================
RCS file: /cvsroot/src/share/mk/bsd.init.mk,v
retrieving revision 1.2
diff -u -p -r1.2 bsd.init.mk
--- share/mk/bsd.init.mk        28 Jul 2003 02:38:33 -0000      1.2
+++ share/mk/bsd.init.mk        13 May 2015 04:45:18 -0000
@@ -6,6 +6,8 @@
 .if !defined(_BSD_INIT_MK_)
 _BSD_INIT_MK_=1

+.-include "local.init.mk"
+
 .-include "${.CURDIR}/../Makefile.inc"
 .include <bsd.own.mk>
 .MAIN:         all

The idea is that local.*.mk are never installed to /usr/share/mk/ but the include remains as a useful customization point. Since there wasn't an obvious choice of bsd.*.mk that is included at the end of bsd.{lib,prog}.mk etc, I opted to include local.final.mk at the end of files that needed it.

Index: share/mk/bsd.kinc.mk
===================================================================
RCS file: /cvsroot/src/share/mk/bsd.kinc.mk,v
retrieving revision 1.36
diff -u -p -r1.36 bsd.kinc.mk
--- share/mk/bsd.kinc.mk        16 Mar 2006 18:43:34 -0000      1.36
+++ share/mk/bsd.kinc.mk        13 May 2015 04:45:18 -0000
@@ -82,3 +82,4 @@ incinstall::
 ##### Pull in related .mk logic
 .include <bsd.subdir.mk>
 .include <bsd.sys.mk>
+.-include "local.final.mk"
Index: share/mk/bsd.lib.mk
===================================================================
RCS file: /cvsroot/src/share/mk/bsd.lib.mk,v
retrieving revision 1.356
diff -u -p -r1.356 bsd.lib.mk
--- share/mk/bsd.lib.mk 1 Dec 2014 01:34:30 -0000       1.356
+++ share/mk/bsd.lib.mk 13 May 2015 04:45:18 -0000
@@ -857,3 +857,5 @@ LINKSMODE?= ${LIBMODE}
 .include <bsd.clean.mk>

 ${TARGETS}:    # ensure existence
+
+.-include "local.final.mk"
Index: share/mk/bsd.prog.mk
===================================================================
RCS file: /cvsroot/src/share/mk/bsd.prog.mk,v
retrieving revision 1.291
diff -u -p -r1.291 bsd.prog.mk
--- share/mk/bsd.prog.mk        1 Dec 2014 01:34:30 -0000       1.291
+++ share/mk/bsd.prog.mk        13 May 2015 04:45:18 -0000
@@ -683,3 +683,5 @@ LINKSMODE?= ${BINMODE}
 ${TARGETS}:    # ensure existence

 .endif # HOSTPROG
+
+.-include "local.final.mk"

And that is the extent of changes needed to be able to build bin/cat and the libs etc that it needs.

Obviously I added some makefiles. The following are simply copied from the latest bmake distribution:

share/mk/auto.obj.mk
share/mk/dirdeps.mk
share/mk/gendirdeps.mk
share/mk/host-target.mk
share/mk/install-new.mk
share/mk/meta.autodep.mk
share/mk/meta.stage.mk
share/mk/meta.subdir.mk
share/mk/meta.sys.mk
share/mk/meta2deps.py
share/mk/meta2deps.sh
share/mk/mkopt.sh
share/mk/options.mk
share/mk/stage-install.sh
share/mk/sys.dependfile.mk

the new files are:

share/mk/local.sys.mk
share/mk/local.init.mk
share/mk/local.final.mk

I'll cover these in some detail below.

local.sys.mk

This gets the ball rolling. We use options.mk (similar bsd.mkopt.mk in recent FreeBSD) to set the options we want - most are dependent on DIRDEPS_BUILD which defaults to "no". It also sets some variables that dirdeps.mk needs (_PARSEDIR, RELDIR and SRCTOP).

The SB_* variables are used by the make wrapper I've used for many years.

# some handy macros
_PARSEDIR= ${.PARSEDIR:tA}
_this = ${.PARSEDIR:tA}/${.PARSEFILE}
.if !defined(_TARGETS)
_TARGETS := ${.TARGETS}
.export _TARGETS
.endif
M_ListToMatch = L:@m@$${V:M$$m}@
# match against our initial targets (see above)
M_L_TARGETS = ${M_ListToMatch:S,V,_TARGETS,}
M_ListToSkip= O:u:ts::S,:,:N,g:S,^,N,
M_type = @x@(type $$x 2> /dev/null); echo;@:sh:[0]:N* found*:[@]:C,[()],,g
M_whence = ${M_type}:M/*:[1]

# this is more useful
.MAKE.EXPAND_VARIABLES= yes

OPTIONS_DEFAULT_NO= \
        META_MODE \
        DIRDEPS_BUILD \

# these all depend on DIRDEPS_BUILD
OPTIONS_DEFAULT_DEPENDENT= \
        AUTO_OBJ/DIRDEPS_BUILD \
        DIRDEPS_CACHE/DIRDEPS_BUILD \
        STAGING/DIRDEPS_BUILD \
        STAGING_PROG/DIRDEPS_BUILD

.-include "options.mk"

# since we do not install local.*.mk
# we know where we are
.if !defined(SRCTOP)
SRCTOP:= ${_PARSEDIR:H:H}
.export SRCTOP
.endif
_SRC_TOP_ = ${SRCTOP}

.if ${MK_DIRDEPS_BUILD:Uno} == "yes"
# this is what we want
.MAIN: all
.if ${.CURDIR} == ${SRCTOP}
RELDIR = .
.elif ${.CURDIR:M${SRCTOP}/*}
RELDIR := ${.CURDIR:S,${SRCTOP}/,,}
.endif

OBJROOT ?= ${SB_OBJROOT:U${MAKEOBJDIRPREFIX:U${SRCTOP:H}/obj/}}
OBJROOT := ${OBJROOT}
# unless we make TARGET_SPEC_VARS include more than MACHINE
# this will suffice
OBJTOP = ${OBJROOT}${MACHINE}

.if !defined(HOST_TARGET)
.include "host-target.mk"
.endif

We use the pseudo machine host to represent building for the build host rather than a target. But we want its objdir to be explicitly named for the actual host so a src tree can be shared and built by different hosts.

The variable HOST_TARGET gets values like netbsd5-i386 or freebsd10-amd64 etc.

HOST_OBJTOP ?= ${OBJROOT}${HOST_TARGET}

.if ${OBJTOP} == ${HOST_OBJTOP} || ${REQUESTED_MACHINE:U${MACHINE}} == "host"
MACHINE= host
.if ${TARGET_MACHINE:Uno} == ${HOST_TARGET}
# not what we want
TARGET_MACHINE= host
.endif
.endif
.if ${MACHINE} == "host"
OBJTOP := ${HOST_OBJTOP}
STAGE_MACHINE = ${HOST_TARGET}
# use host tools
USETOOLS = no
# toolchain stuff in bsd.own.mk could use work
MKMANDOC = no
.else
STAGE_MACHINE = ${MACHINE}
.endif

.if ${MK_AUTO_OBJ} == "yes"
MKOBJDIRS = auto
.-include <auto.obj.mk>
.endif

We create objdirs automatically, it needs to be done early though; before .PATH is established.

DIRDEP_USE_ENV += NOSUBDIR=1
NOSUBDIR=1
BUILD_AT_LEVEL0 = no
PYTHON ?= /usr/pkg/bin/python
# this suffices for 99%
.MAKE.DEPENDFILE_DEFAULT = ${.MAKE.DEPENDFILE_PREFIX}
# which means we need to parameterize some things
CSU_DIR.i386 = i386_elf
CSU_DIR.${MACHINE} ?= ${MACHINE}
CSU_DIR = ${CSU_DIR.${MACHINE}}
SYS_ARCH.${MACHINE} ?= ${MACHINE}
SYS_ARCH = ${SYS_ARCH.${MACHINE}}
SYS_ARCH2.i386 = x86
SYS_ARCH2.x86_64 = x86
SYS_ARCH2.evbarm = arm
SYS_ARCH2 = ${SYS_ARCH2.${MACHINE}:U}

GENDIRDEPS_FILTER_DIR_VARS = CSU_DIR SYS_ARCH SYS_ARCH2

When we build Junos we are always building for multiple machine types so using Makefile.deppend.${MACHINE} is the default. This allows the files to be updated for multiple machines at the same time without locking.

For a more general case like {Free,Net}BSD most people actually only build for a single machine at a time, and in fact a single machine independent Makefile.depend works for most of the tree. So we use that by default. When building for multiple machines we avoid the update issue by nominating one MACHINE value to update for. In this case we also need to parameterize some of the dirdeps (eg. CSU_DIR).

sys.dependfile.mk finds the most appropriate Makefile.depend* for the current ${TARGET_SPEC} (default is just ${MACHINE} but can actually be much more complex).

.-include "sys.dependfile.mk"
.-include "meta.sys.mk"
.if ${.MAKE.MODE:Mmeta*} == ""
MK_DIRDEPS_BUILD = no
.else
.if ${MK_STAGING} == "yes"
STAGE_ROOT:= ${OBJROOT}stage
STAGE_OBJTOP= ${STAGE_ROOT}/${STAGE_MACHINE}
STAGE_LIBDIR= ${STAGE_OBJTOP}${LIBDIR:U/lib}
STAGE_INCLUDEDIR= ${STAGE_OBJTOP}${INCLUDEDIR:U/usr/include}
STAGE_INCSDIR= ${STAGE_OBJTOP}${INCSDIR:U/usr/include}
SYSROOT= ${STAGE_OBJTOP}
.endif
.endif

Staging can be thought of as auto install. The big difference between staging and simply installing into a ${DESTDIR} is that when we stage a file we put a file.dirdep there too which indicates who put it there.

This allows correct dependency collection and detecting errors like multiple makefiles trying to install the same file.

MACHINE_ARCH.evbarm= earm
MACHINE_ARCH.${MACHINE} ?= ${MACHINE}
.if empty(MACHINE_ARCH)
MACHINE_ARCH := ${MACHINE_ARCH.${MACHINE}}
.else
MACHINE_ARCH ?= ${MACHINE_ARCH.${MACHINE}}
MACHINE_ARCH := ${MACHINE_ARCH}
.endif
.-include "toolchain.mk"

If TARGET_SPEC is just MACHINE when dirdeps.mk visits a leaf directory it will make MACHINE_ARCH empty, and sys.mk is expected to pick the right value based on MACHINE. The above is obviously incomplete ;-)

NOGCCERROR=

I added NOGCCERROR for current, because some prototypes were failing -Wstrict-prototypes causing errors and fixing that wasn't my current goal.

.elif ${MK_META_MODE} == "yes"
# we are doing "normal" build but we want meta files
# produced - handy for debugging
.if ${.MAKEFLAGS:U:M-B} == ""
.MAKE.MODE= meta verbose

We want to be able to do a normal release build using build.sh but get meta files produced since this can be handy for debugging. We skip this though for make -B.

ERROR_LOGDIR ?= ${SRCTOP:H}/error
meta_error_log = ${ERROR_LOGDIR}/meta-${.MAKE.PID}.log

# we are not interested in make telling us a failure happened elsewhere
.ERROR: _metaError
_metaError: .NOMETA .NOTMAIN
        -@[ "${.ERROR_META_FILE}" ] && { \
        grep -q 'failure has been detected in another branch' ${.ERROR_META_FILE} && exit 0; \
        mkdir -p ${meta_error_log:H}; \
        cp ${.ERROR_META_FILE} ${meta_error_log}; \
        echo "ERROR: log ${meta_error_log}" >&2; }; :

# these all help debugging too - if we don't get meta file
MAKE_PRINT_VAR_ON_ERROR?= \
        .CURDIR .OBJDIR \
        .MAKE .MAKE.LEVEL .MAKE.MODE \
        .TARGETS \
        .PATH \
        .MAKE.MAKEFILES \
        ${MAKE_PRINT_VAR_ON_ERROR_XTRAS}
.endif
.endif

The above just provides some clues to help explain what went wrong on failure.

local.init.mk

Generally speaking the dirdeps build works best when all actual building is done via sub-make. We also allow for the possibility of some dirs only being supported for some MACHINEs so we compute _machine_list and check if current ${MACHINE} is in the list.

Normally we would then fix bsd.{lib,prog}.mk to do nothing if _SKIP_BUILD is not empty.

.if ${MK_DIRDEPS_BUILD:Uno} == "yes"

The above allows us to disable all this when doing normal builds.

_machine_list := ${ONLY_MACHINE_LIST:U${TARGET_MACHINE:U${MACHINE}}}

.if ${.MAKE.LEVEL} == 0 && ${BUILD_AT_LEVEL0:Uno} == "no"
_SKIP_BUILD = not building at level 0
.elif ${_machine_list:M${MACHINE}} == ""
_SKIP_BUILD = ${MACHINE} not in list ${_machine_list}
.endif

.if empty(_SKIP_BUILD)
realall: beforebuild .WAIT
.endif
beforebuild: .NOTMAIN

It can be really handy to be able to ensure some activities (like stage_incs_first) run before any of the normal build activity.

.if ${CC:H} != "/usr/bin"
CPPFLAGS += -nostdinc
.endif
.if ${MK_STAGING} == "yes"
.if ${MACHINE} != "host"
DESTDIR= ${STAGE_OBJTOP}
.endif
CPPFLAGS += -I${STAGE_INCLUDEDIR}
CXXFLAGS += -I${STAGE_INCLUDEDIR}/g++
#LDFLAGS += -L${STAGE_LIBDIR}
.endif

.endif

the rest is simple enough, we set DESTDIR to the top of the stage tree (except when building for "host"). In current we thus get --sysroot=${DESTDIR} so all the right libs and includes are used. For 5.X and earlier we need to be explicit. Also in 5.X lib/libc/makelintstub relies on ${DESTDIR} to get the correct sys/syscall.h.

local.final.mk

This is easily the ugliest. It provides a shim between bsd.*.mk and meta.stage.mk as well as pulling in meta.autodep.mk to take care of updating Makefile.depend for us.

.if ${MK_DIRDEPS_BUILD:Uno} == "yes"
.if ${MK_STAGING} == "yes"

INSTALL_DIR= ${STAGE_INSTALL} -d
INSTALL_FILE= ${STAGE_INSTALL}

We use ${STAGE_INSTALL} instead of ${INSTALL} it is a script that runs the normal install command and then adds necessary .dirdep files so we maintain our staging semantics.

.if !empty(INCS)
# mimic bsd.inc.mk
staging: stage_includes
.for D F in ${INCS:O:u:@x@${INCSDIR_${x:C,/,_,g}:U${INCSDIR}}${INCSNAME_${x:C,/,_,g}:U${INCSNAME:U$x}:H:S,^,/,:N/.} $x@}
STAGE_AS_SETS += incs_${D:S,/,_,g}
STAGE_DIR.incs_${D:S,/,_,g} = ${STAGE_OBJTOP}$D
stage_as.incs_${D:S,/,_,g}: $F
STAGE_AS_${${F}:P:tA} := ${INCSNAME_${F:S,/,_,g}:U${INCSNAME:U${F}}:T}
.endfor
stage_includes stage_incs_first: ${STAGE_AS_SETS:O:u:Mincs_*:@s@stage_as.$s@}

The above is basically matching the dance that bsd.incs.mk does to install ssp/ssp.h as ${DESTDIR}/usr/include/ssp/ssp.h

.if defined(LIB)
beforebuild: stage_incs_first
.endif

Many of the libraries need their headers installed before they can build. This is an example of where beforebuild is handy.

.if ${INCSDIR} == "/usr/include/${MACHINE_CPU}"
SYMLINKS+= ${MACHINE_CPU} ${INCSDIR:H}/machine \
        machine/float.h ${INCSDIR:H}/float.h

Yes this is a layering violation, but I wanted to minimize changes to the tree (and I haven't spotted where those symlinks are supposed to be made ;-)

.endif
.endif
.if !empty(_LIBS)
staging: stage_libs
stage_libs: ${_LIBS}
SHLIB_LINK ?= ${_LIB.so}
SHLIB_NAME ?= ${_LIB.so.full}
.endif
.if !empty(FILES)
staging: stage_files
STAGE_SETS+= files
stage_files.files: ${FILES}
STAGE_DIR.files = ${STAGE_OBJTOP}${FILESDIR}
.endif
.if !empty(SYMLINKS)
staging: stage_symlinks
STAGE_SETS+= links
STAGE_SYMLINKS.links= ${SYMLINKS}
.endif
.if !empty(INCSYMLINKS)
staging: stage_symlinks
STAGE_SETS+= inclinks
STAGE_SYMLINKS.inclinks= ${INCSYMLINKS}
.endif

.if defined(PROG) && ${MK_STAGING_PROG} == "yes"
.if defined(PROGNAME)
STAGE_AS_SETS+= prog
STAGE_AS_${PROG}= ${PROGNAME}
stage_as.prog: ${PROG}
.else
STAGE_SETS+= prog
stage_files.prog: ${PROG}
staging: stage_files
.endif
STAGE_DIR.prog= ${STAGE_OBJTOP}${BINDIR}
.endif

.if ${.MAKE.LEVEL} > 0
realall: staging
.endif

.include "meta.stage.mk"
.endif

.include "meta.autodep.mk"
.endif

local.gendirdeps.mk

Sometimes we need to filter the raw data we glean from the syscall trace. For example we do this for optional functionality that would otherwise cause churn in the tree dependencies. We can have GENDIRDEPS_FILTER cause an optional dir to not be recorded, and have local.dirdeps.mk add it when needed.

# we need to filter these to reduce churn for different MACHINES

GENDIRDEPS_FILTER+= \
        Nsys/arch/${SYS_ARCH}/include* \
        Nsys/arch/${SYS_ARCH2}/include* \

.if ${RELDIR} == "lib/libc"
GENDIRDEPS_FILTER+= \
        Nexternal/gpl3/gcc/lib/libstdc++-v3/include*

.endif

We supress the above since they seem machine dependent and we want to avoid the need for Makefile.depend.${MACHINE} to the extent we can, especially for an OS that builds for so many architectures.

.if ${RELDIR:N*libgcc*:Nlib/libc} == ""
# these have explicit -I so do *not* need libpthread built first
# though due to sysroot the pthread headers will be found in stage
# root if tree already built.
GENDIRDEPS_FILTER+= Nlib/libpthread

.endif

This is an interesting case. libpthread cannot be built before libc or libgcc but those need pthread_types.h. We provide them with a direct -I so they can successfully build before it. But once the three has been built they will find that header via the stage tree, and we need to supress recording that since it would cause a cycle in the graph.

local.dirdeps.mk

# sometimes we need help

# should be set by now
DEP_MACHINE?= ${MACHINE}
DEP_RELDIR?= ${RELDIR}

.if ${DEP_RELDIR} == "include"
DIRDEPS += \
        sys/arch/${SYS_ARCH}/include \
        sys/arch/${SYS_ARCH2}/include \

The above compensates for the dependencies we supressed above.

.if ${DEP_MACHINE:M*arm*} != ""
DIRDEPS += \
        sys/arch/${SYS_ARCH2}/include/arm32 \

This is easier than thinking of something clever, perhaps setting MACHINE32 per MACHINE (which we do in Junos) but not urgent.

It is worth noting that dirdeps.mk will happily ignore bogus entires was well as dirs that lack a makefile.

.endif
.endif
.if ${DEP_RELDIR} == "lib/libc"
DIRDEPS += \
        external/gpl3/gcc/lib/libstdc++-v3/include \
        external/gpl3/gcc/lib/libstdc++-v3/include/bits \

.endif

The above is necessary on evbarm and perhaps some others, it does little harm to do it anyway.

Toolchains

I should mention that I consider building toolchains out of scope for this exercise. Building toolchains is a corner case, and should happen rarely enough to not be worth optimizing - or even integrating.

BTW for 5.X I hacked build.sh to not build the old make, since my /usr/bin/make is always the latest from current.

But in general I just use the toolchain as built by the normal build.

Circular dependencies

As noted above, our goal is to build the tree in a single pass. This means we cannot have any circular dependencies.

In NetBSD current we have to break cycles involving libpthread which is achieved by adding:

.if ${MK_DIRDEPS_BUILD:Uno} == "yes"
# avoid a circular dependency with libpthread
CPPFLAGS+= -I${SRCTOP}/lib/libpthread
.endif

to each of:

external/gpl3/gcc/lib/libgcc/Makefile.inc
lib/csu/Makefile
lib/libc/Makefile

Without the direct -I each of these needs libpthread built first so that pthread_types.h is staged, but libpthread cannot be built without all the above.

Conclusion

After only a couple of days effort I have most of userland building (400+ apps and all the libs they need) as well as building kernels. Of course I was able to re-use targets/* from FreeBSD.


Author:sjg@crufty.net
Revision:$Id: netbsd-meta-mode.txt,v 8023c5888fde 2016-10-02 22:59:09Z sjg $