Building NetBSD in meta mode

This doc describes building NetBSD using bmake in meta mode leveraging the work done building FreeBSD in meta mode

Junos (which is based on FreeBSD) has been built this way for over 10 years, and it works very well.

Note: while I normally talk of meta mode and using dirdeps.mk together, either can be used without the other.

In FreeBSD two separate controls are provided DIRDEPS_BUILD and META_MODE. Enabling DIRDEPS_BUILD implies META_MODE, but many developers use the latter even with the traditional build targets.

I first did this exercise with NetBSD in about 2015, and again in 2023, which was as much as anything a test of new a bootstrapping method. This was prompted by a failure to build NetBSD/current on an older NetBSD host.

Introduction

There isn't a lot wrong with the NetBSD build as it is. At least for doing release builds and such. It builds in parallel ok, and can be cross-built on a number of different hosts. Thus I've not previously been seriously tempted to interfere with it.

However the prospect of doing a dev project on NetBSD at work was enough to prompt me to see what it would take to get the DIRDEPS_BUILD (which is hard to live without once you've tried it ;-) working on NetBSD.

The short answer is very little. Frankly despite planning to leverage work already done for FreeBSD, I was very pleasantly surprised to find how little effort it took to get bin/cat (and everything it needs) building. Most of that time was spend finding the various places that needed to be bootstrapped to link cat. See bootstrapping below to see how this step is now much simpler.

After building for i386 I then build for something else to see what the impact on the dependencies are. For FreeBSD I found with just parameterizing CSU_DIR a single Makefile.depend worked for 99% of the tree, and most places where Makefile.depend.${MACHINE} was needed it was due to generated files being named differently.

For NetBSD some of the dependencies varied quite a bit between i386 and evbarm, but nothing that cannot be handled by a little filtering - which the meta mode infrastructure provides for.

What is this meta mode thing anyway?

Fair question.... It is a better way of building a large tree of software.

Most of this is covered in building FreeBSD in meta mode But I'll briefly rehash the story here.

My goal is to be able to start a build from anywhere in a freshly checked out tree (eg. bin/cat) and have it just work. That means building everything else in the tree needed, in the right order, in parallel. To be able to do that for multiple target machines at the same time and even multiple target operating systems at the same time. All in a single pass of the tree.

All the above is handled by using dirdeps.mk (hence the DIRDEPS_BUILD option in FreeBSD) which is used by the initial make instance to compute a graph of tree dependencies from the current origin, and then go build them all in parallel.

The tree wide graph ensures that all leaf directories are visited in the correct order - no tree walks. In fact dirdeps.mk suppresses the behavior of bsd.subdir.mk.

Using dirdeps.mk we can greatly simplify the top-level build logic, which apart from share/mk/ is where we typically find a concentration of complexity:

NetBSD (2015):

$ wc -l Makefile build.sh
     530 Makefile
    2309 build.sh
    2839 total

NetBSD (2023):

$ wc -l Makefile build.sh
     540 Makefile
    2589 build.sh
    3129 total

That's a rather modest increase in almost 10 years!

FreeBSD (2015):

$ wc -l Makefile Makefile.inc1
     525 Makefile
    2196 Makefile.inc1
    2721 total

FreeBSD (2023):

$ wc -l Makefile Makefile.inc1
     808 Makefile
    3633 Makefile.inc1
    4441 total

Much more substantial.

The pre-meta mode Junos build had over 5000 lines of very dense makefile at the top-level. The original top-level makefiles for the meta mode build that replaced all that was less than 300 lines!

NetBSD/FreeBSD meta mode (2015):

$ wc -l targets/Ma* share/mk/dirdeps.mk
     172 targets/Makefile
      52 targets/Makefile.inc
      46 targets/Makefile.xtras
     644 share/mk/dirdeps.mk
     914 total

FreeBSD meta mode (2023):

$ wc -l targets/Makefile* share/mk/dirdeps*.mk
     116 targets/Makefile
      54 targets/Makefile.inc
      77 targets/Makefile.xtras
     111 share/mk/dirdeps-options.mk
     173 share/mk/dirdeps-targets.mk
     959 share/mk/dirdeps.mk
    1490 total

I included dirdeps.mk in the line count because that's where most of the complexity is, but of course we leverage it for all builds not just top-level ones.

dirdeps-targets.mk handles the task of finding a suitable Makefile.depend under targets/ and dirdeps-options.mk handles the dependency complication introduced by dozens of optional features.

In fact targets/Makefile isn't necessary at all. The inclusion of dirdeps-targets.mk can be done from local.sys.mk which is what I did in the more recent NetBSD exercise.

With the DIRDEPS_BUILD adding a new top-level target is as simple as creating a directory somewhere under targets/ and putting a Makefile.depend file there that lists the directories needed, the Makefile is often trivial - leveraging common packaging logic. These makefiles though are no more complex than any leaf makefile.

Meta files

When make is run in meta mode, it creates a .meta file for each target being built. The meta file captures information that allows make to do a much better job.

Command line

Firstly we capture the expanded command line. This allows make to compare the command it might need to run with what was done last time, if anything changed the target is out-of-date.

Now sometimes you want to suppress that behavior so there are knobs to do so for a whole target or just one line.

This feature alone makes a huge difference to build reliability. It is no longer necessary to make targets depend on every makefile that might influence them. This allows us to avoid building things when nothing relevant has changed, while ensuring we never fail to rebuild when they have.

The captured command is also invaluable for debugging.

Command output

The next thing captured is any output from the command. This is mainly for human debugging purposes.

This extremely useful when you have 1000's of users who fail to log their builds and still want to you help them understand why their build failed.

Syscall trace

The filemon module provides /dev/filemon which make will use if available.

When make runs a child, it opens /dev/filemon and gives it a temp file and the pid of the child. All successful syscalls (interesting to make) made by the child and its progeny will be recorded in the temp file. When the child exits, make appends the data to the meta file.

This data has two main uses. Firstly make itself uses it to better check if a target needs update. Secondly we can post-process the data for all sorts of purposes including extracting directory dependencies - which allows us to automate the collection of the data needed by dirdeps.mk.

For example bin/cat/Makefile.depend:

# Autogenerated - do NOT edit!

DIRDEPS = \
        include \
        lib/libpthread \
        sys/arch/amd64/include \
        sys/arch/x86/include \
        sys/sys \


.include <dirdeps.mk>

.if ${.MAKE.LEVEL} > 0
# local dependencies - needed for -jN in clean tree
.endif

As we read the files read or executed, any we see in an object directory which is not ${.OBJDIR} indicates a directory that needs to be built before ${.CURDIR}. Being able to map object dirs to src dirs is important.

The local dependencies section is where we record any dependencies involving locally generated files - also gleaned from the syscall trace. This allows us to reliably build a clean tree in parallel while skipping any explict depend step.

The syscall trace is also invaluable for debugging weird build issues. For example spotting that your supposed captive build is reading headers from /usr/include or linking libs from /usr/local/lib. Or running a perl or python binary other than the one intended. Being able to look under the covers after a build has failed is immensely useful.

meta_oodate

When make is checking if a target needs update and the normal makefile rules indicate it is up-to-date, we delve into the meta file. Here is cat.o.meta:

# Meta data file /h/NetBSD/5.X/obj/i386/bin/cat/cat.o.meta
CMD @echo '#  ' "compile " cat/cat.o
CMD /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc -O2  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare -Wno-traditional -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wno-unused-parameter     -nostdinc -I/h/NetBSD/5.X/obj/stage/i386/usr/include -nostdinc -isystem /h/NetBSD/5.X/obj/stage/i386/usr/include   -c    /h/NetBSD/5.X/src/bin/cat/cat.c
CWD /h/NetBSD/5.X/obj/i386/bin/cat
TARGET cat.o
-- command output --
#   compile  cat/cat.o

-- filemon acquired metadata --
# filemon version 4
# Target pid 2050
V 4
E 8142 /bin/sh
R 8142 /etc/ld.so.conf
R 8142 /lib/libedit.so.2
R 8142 /lib/libtermcap.so.0
R 8142 /lib/libc.so.12
X 8142 0
# Bye bye
E 5426 /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc
R 5426 /etc/ld.so.conf
R 5426 /usr/lib/libc.so.12
R 5426 /var/tmp//ccKCUD9I.s
W 5426 /var/tmp//ccKCUD9I.s
E 15146 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../libexec/gcc/i386--netbsdelf/4.1.3/cc1
R 15146 /etc/ld.so.conf
R 15146 /usr/lib/libc.so.12
R 15146 /h/NetBSD/5.X/src/bin/cat/cat.c
R 15146 /var/tmp//ccKCUD9I.s
W 15146 /var/tmp//ccKCUD9I.s
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/cdefs.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs_elf.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/param.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/null.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/inttypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stdint.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_mwgwtypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_limits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_const.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/wchar_limits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_fmtio.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/ansi.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ansi.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/endian.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian_machdep.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/byte_swap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/bswap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/fd_set.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/pthread_types.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/syslimits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/signal.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/sigtypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/satypes.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/siginfo.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/signal.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/trap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/x86/trap.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ucontext.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/mcontext.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/param.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/limits.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stat.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/select.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/time.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/ctype.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/err.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/errno.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/errno.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/fcntl.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/locale.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdio.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdlib.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/strings.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/unistd.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/unistd.h
R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h
X 15146 0
E 18126 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../lib/gcc/i386--netbsdelf/4.1.3/../../../../i386--netbsdelf/bin/as
R 18126 /etc/ld.so.conf
R 18126 /usr/lib/libc.so.12
R 18126 cat.o
W 18126 cat.o
R 18126 /var/tmp//ccKCUD9I.s
X 18126 0
D 5426 /var/tmp//ccKCUD9I.s
X 5426 0
# Bye bye

I bet you didn't know it takes all those headers to compile cat.o. That's 79 headers (54 unique), not what you would expect from just looking at cat.c.

Anyway, as noted above the first thing we do is check if the command line changed, any change to CFLAGS would do it. If it did - we are done - out-of-date.

Otherwise we look at the syscall trace. This is formatted so that it is trivial to parse by even a shell script.

In fact I have a collection of shell functions for extracting useful data from meta files - like extracting the command so it can be re-run easily, spliting the command into individual words one per line so it is much easier to compare with another using diff(1). Extracting the list of files read - all resolved to absoulte paths, and so on.

In the above example it is simple, we just need to look at all the E (exec) and R (read) entries; if any of the files read/executed are newer the target is out-of-date.

Sometimes the syscall trace is more complex, so make needs to follow the F (fork) and C (chdir) syscalls by pid so it knows what the cwd of each is so if it sees a file opened by a relative path it can find it.

If a file previously written is missing (and the syscall trace does not show it being D deleted), the target is out-of-date.

Of course make will ignore some paths (eg. anything in /tmp) and you can provide it a list of path prefixes to ignore.

The ability for make to look under the covers like this allows it to do a much better job of ensuring things are re-built when they need to be.

You can use the debug flag -dM to have make tell you what it is about a meta file that made it decide the target is out-of-date.

filemon

The filemon module is currently available in NetBSD and FreeBSD and a version for Linux is available from https://github.com/trixirt/filemon-linux.git

filemon_ktrace

In Jan 2020 NetBSD introduced filemon_ktrace as an inteface to fktrace(2) for use by make eliminating the need for filemon(4). The old api is retained as filemon_dev for use on FreeBSD and others.

filemon_ktrace adds a little overhead in the form of an extra file descriptor per job.

With luck the filemon_ktrace model can be leveraged by other syscall trace mechanisms on other systems - especially for Linux where there is currently no good alternative.

Makefiles

Initially I only had to touch one makefile outside of share/mk and that was include/Makefile because I wanted an obj dir:

Index: include/Makefile
===================================================================
RCS file: /cvsroot/src/include/Makefile,v
retrieving revision 1.140
diff -u -p -r1.140 Makefile
--- include/Makefile    11 Dec 2013 01:24:08 -0000      1.140
+++ include/Makefile    13 May 2015 18:53:29 -0000
@@ -3,7 +3,9 @@

 # Doing a make includes builds /usr/include

+.if ${MK_DIRDEPS_BUILD:Uno} == "no"
 NOOBJ=         # defined
+.endif

 # Missing: mp.h

by contrast I had to quite heavily tweak the FreeBSD equivalent. This was a pleasant surprise.

Though since include/Makefile isn't making some of the symlinks in ${INCSDIR} that are obviously needed we had to compensate in local.final.mk (see below).

In fact for the 2023 revisit apart from the above change to include/Makefile the only existing files touched were:

share/mk/bsd.files.mk
share/mk/bsd.init.mk
share/mk/bsd.sys.mk
share/mk/sys.mk
external/gpl3/gcc/lib/libgcc/Makefile.inc

in each case only one or two lines such as:

.-include <local.dirdeps-build.mk>

local.dirdeps-build.mk is normally included by dirdeps.mk at level 1+ (building), but while bootstrapping there may not be any Makefile.depend files to include dirdeps.mk.

share/mk

We hook our magic into the build by introducing the notion of local.*.mk this is a very handy construct. For example:

Index: share/mk/sys.mk
===================================================================
RCS file: /cvsroot/src/share/mk/sys.mk,v
retrieving revision 1.127
diff -u -p -r1.127 sys.mk
--- share/mk/sys.mk     10 Aug 2014 05:57:31 -0000      1.127
+++ share/mk/sys.mk     13 May 2015 04:45:18 -0000
@@ -7,6 +7,8 @@

 unix?=         We run NetBSD.

+.-include "local.sys.mk"
+
 .SUFFIXES: .a .o .ln .s .S .c .cc .cpp .cxx .C .f .F .r .p .l .y .sh

 .LIBS:         .a
Index: share/mk/bsd.init.mk
===================================================================
RCS file: /cvsroot/src/share/mk/bsd.init.mk,v
retrieving revision 1.2
diff -u -p -r1.2 bsd.init.mk
--- share/mk/bsd.init.mk        28 Jul 2003 02:38:33 -0000      1.2
+++ share/mk/bsd.init.mk        13 May 2015 04:45:18 -0000
@@ -6,6 +6,8 @@
 .if !defined(_BSD_INIT_MK_)
 _BSD_INIT_MK_=1

+.-include "local.init.mk"
+
 .-include "${.CURDIR}/../Makefile.inc"
 .include <bsd.own.mk>
 .MAIN:         all

The idea is that local.*.mk are never installed to /usr/share/mk/ but the include remains as a useful customization point. Since there wasn't an obvious choice of bsd.*.mk that is included at the end of bsd.{lib,prog}.mk etc, I opted to include local.final.mk at the end of files that needed it.

Note: in the more recent exercise I used local.dirdeps-build.mk included from bsd.sys.mk thus minimizing changes needed.

And that is the extent of changes needed to be able to build bin/cat and the libs etc that it needs.

Obviously I added some makefiles. The following are simply copied from the latest bmake distribution:

share/mk/auto.obj.mk
share/mk/dirdeps-cache-update.mk
share/mk/dirdeps-only.mk
share/mk/dirdeps-options.mk
share/mk/dirdeps-targets.mk
share/mk/dirdeps.mk
share/mk/gendirdeps.mk
share/mk/host-target.mk
share/mk/install-new.mk
share/mk/jobs.mk
share/mk/meta.autodep.mk
share/mk/meta.stage.mk
share/mk/meta.sys.mk
share/mk/meta2deps.py
share/mk/meta2deps.sh
share/mk/options.mk
share/mk/sys.dependfile.mk
share/mk/sys.dirdeps.mk
share/mk/sys.vars.mk

the new files are:

share/mk/local.dirdeps-build.mk
share/mk/local.dirdeps-missing.mk
share/mk/local.dirdeps.mk
share/mk/local.meta.sys.mk
share/mk/local.sys.dirdeps.env.mk
share/mk/local.sys.dirdeps.mk
share/mk/local.sys.mk

I'll cover these in some detail below.

local.sys.mk

This gets the ball rolling. We use options.mk (similar to bsd.mkopt.mk in recent FreeBSD) to set the options we want - most are dependent on DIRDEPS_BUILD which defaults to "no". We then include makefiles accordingly; sys.dirdeps.mk sets up for the DIRDEPS_BUILD, meta.sys.mk deals with META_MODE and auto.obj.mk ensures .OBJDIR is created:

# initially this should be enough to enable
# META_MODE and DIRDEPS_BUILD
#
# the makefiles included here all ship with bmake
# and follow the convention of foo.mk including local.foo.mk
# to pick up customizations.
#
# some options we need to know early
OPTIONS_DEFAULT_NO += \
        DIRDEPS_BUILD \
        DIRDEPS_CACHE \

OPTIONS_DEFAULT_DEPENDENT += \
        AUTO_OBJ/DIRDEPS_BUILD \
        META_MODE/DIRDEPS_BUILD \
        STAGING/DIRDEPS_BUILD \
        STAGING_PROG/DIRDEPS_BUILD \
        STATIC_DIRDEPS_CACHE/DIRDEPS_CACHE \

.include <options.mk>

.if ${MK_DIRDEPS_BUILD} == "yes"
.include <sys.vars.mk>
.include <host-target.mk>
.include <sys.dirdeps.mk>
.endif

.if empty(MACHINE_ARCH)
MACHINE_ARCH.amd64 = x86_64
MACHINE_ARCH := ${MACHINE_ARCH.${MACHINE}:U${MACHINE}}
.endif

.if ${MK_META_MODE} == "yes"
# with filemon_ktrace we can get .meta truncated at filemon
# when build is aborted
META_MODE+= missing-filemon=yes missing-meta=yes
.include <meta.sys.mk>
.endif

.if ${MK_AUTO_OBJ} == "yes"
# if doing auto obj it is better done early
.include <auto.obj.mk>
.endif
.if ${.MAKE.LEVEL} == 0
.if ${MK_DIRDEPS_BUILD} == "yes"
.if ${.CURDIR} == ${SRCTOP}
.include <dirdeps-targets.mk>
.endif
.endif
.endif

The only bit above which is NetBSD specific is the setting of MACHINE_ARCH. sys.vars.mk provides a number of useful macros. host-target.mk sets a number of variables related to the host we are building on, including HOST_TARGET which is what we use for object dirs for the pseudo machine host when building for the build host rather than a target.

HOST_TARGET gets values like netbsd5-i386, darwin15-i386, freebsd10-amd64 etc.

sys.dirdeps.mk

This is a very recent addition, prompted by this exercise. Noting how much needed to go into local.sys.mk which was identical accross projects, I reduced it to what we see above, and added sys.dirdeps.mk. I also moved a lot of DIRDEPS_BUILD related bits from meta.sys.mk to this file. The operative bits are below. Like most of DIRDEPS_BUILD it requires a recent bmake:

# Originally DIRDEPS_BUILD and META_MODE were the same thing.
# So, much of this was done in *meta.sys.mk and local*mk
# but properly belongs here.

# Include from [local.]sys.mk - if doing DIRDEPS_BUILD
# we should not be here otherwise
MK_DIRDEPS_BUILD ?= yes
# these are all implied
MK_AUTO_OBJ ?= yes
MK_META_MODE ?= yes
MK_STAGING ?= yes

_PARSEDIR ?= ${.PARSEDIR:tA}

.-include <local.sys.dirdeps.env.mk>

.if ${.MAKE.LEVEL} == 0
# make sure dirdeps target exists and do it first
dirdeps:
# first .MAIN is what counts
.MAIN: dirdeps
.NOPATH: dirdeps
all: dirdeps .WAIT
.endif

.if empty(SRCTOP)
# fallback assumes share/mk!
SRCTOP := ${SB_SRC:U${.PARSEDIR:tA:H:H}}
.export SRCTOP
.endif

# fake SB if not using mk wrapper
.if !defined(SB)
SB := ${SRCTOP:H}
.export SB
.endif

.if empty(OBJROOT)
OBJROOT := ${SB_OBJROOT:U${MAKEOBJDIRPREFIX:U${SB}/obj}/}
.export OBJROOT
.endif

.if empty(STAGE_ROOT)
STAGE_ROOT ?= ${OBJROOT}stage
.export STAGE_ROOT
.endif

# We should be included before meta.sys.mk
# If TARGET_SPEC_VARS is other than just MACHINE
# it should be set by now.
# TARGET_SPEC must not contain any '.'s.
TARGET_SPEC_VARS ?= MACHINE

.if !target(_tspec_env_done_)
_tspec_env_done_: .NOTMAIN

.if ${TARGET_SPEC:Uno:M*,*} != ""
# deal with TARGET_SPEC from env
_tspec := ${TARGET_SPEC:S/,/ /g}
.for i in ${TARGET_SPEC_VARS:${M_RANGE:Urange}}
${TARGET_SPEC_VARS:[$i]} := ${_tspec:[$i]}
.endfor
# We need to stop that TARGET_SPEC affecting any submakes
TARGET_SPEC=
# so export but do not track
.export-env TARGET_SPEC
.export ${TARGET_SPEC_VARS}
.for v in ${TARGET_SPEC_VARS:O:u}
.if empty($v)
.undef $v
.endif
.endfor
.endif
.endif

# Now make sure we know what TARGET_SPEC is
# as we may need it to find Makefile.depend*
.if ${MACHINE:Mhost*} != ""
# host is special
TARGET_SPEC = ${MACHINE}
.else
TARGET_SPEC = ${TARGET_SPEC_VARS:@v@${$v:U}@:ts,}
.endif

.if ${TARGET_SPEC_VARS:[#]} > 1
TARGET_OBJ_SPEC ?= ${TARGET_SPEC_VARS:@v@${$v:U}@:ts.}
.else
TARGET_OBJ_SPEC ?= ${MACHINE}
.endif

MAKE_PRINT_VAR_ON_ERROR += ${TARGET_SPEC_VARS}

.if !defined(MACHINE0)
# it can be handy to know which MACHINE kicked off the build
# for example, if using Makefild.depend for multiple machines,
# allowing only MACHINE0 to update can keep things simple.
MACHINE0 := ${MACHINE}
.export MACHINE0
.endif

.if ${MACHINE} == "host"
OBJTOP = ${HOST_OBJTOP}
.elif ${MACHINE} == "host32"
OBJTOP = ${HOST_OBJTOP32}
.endif

MACHINE_OBJ.host = ${HOST_TARGET}
MACHINE_OBJ.host32 = ${HOST_TARGET32}
MACHINE_OBJ.${MACHINE} ?= ${TARGET_OBJ_SPEC}
MACHINE_OBJDIR = ${MACHINE_OBJ.${MACHINE}}
OBJTOP = ${OBJROOT}/${MACHINE_OBJDIR}

# we do not use MAKEOBJDIRPREFIX
.undef MAKEOBJDIRPREFIX
# we use this
MAKEOBJDIR ?= ${.CURDIR:S,${SRCTOP},${OBJTOP},}

STAGE_MACHINE ?= ${MACHINE_OBJDIR}
STAGE_OBJTOP ?= ${STAGE_ROOT}/${STAGE_MACHINE}
STAGE_COMMON_OBJTOP ?= ${STAGE_ROOT}/common
STAGE_HOST_OBJTOP ?= ${STAGE_ROOT}/${HOST_TARGET}
STAGE_HOST_OBJTOP32 ?= ${STAGE_ROOT}/${HOST_TARGET32}

STAGE_INCLUDEDIR ?= ${STAGE_OBJTOP}${INCLUDEDIR:U/usr/include}
STAGE_LIBDIR ?= ${STAGE_OBJTOP}${LIBDIR:U/lib}

TIME_STAMP_FMT ?= @ %s [%Y-%m-%d %T] ${:U}
DATE_TIME_STAMP ?= `date '+${TIME_STAMP_FMT}'`
TIME_STAMP ?= ${TIME_STAMP_FMT:localtime}

.if ${MK_TIME_STAMPS:Uyes} == "yes"
TRACER = ${TIME_STAMP}
ECHO_DIR = echo ${TIME_STAMP}
ECHO_TRACE = echo ${TIME_STAMP}
.endif

.if ${.CURDIR} == ${SRCTOP}
RELDIR= .
RELTOP= .
.elif ${.CURDIR:M${SRCTOP}/*}
RELDIR:= ${.CURDIR:S,${SRCTOP}/,,}
.else
RELDIR:= ${.OBJDIR:S,${OBJTOP}/,,}
.endif
RELTOP?= ${RELDIR:C,[^/]+,..,g}
RELOBJTOP?= ${RELTOP}
RELSRCTOP?= ${RELTOP}

# this does all the smarts of setting .MAKE.DEPENDFILE
.-include <sys.dependfile.mk>

.-include <local.sys.dirdeps.mk>

# check if we got anything sane
.if ${.MAKE.DEPENDFILE} == ".depend"
.undef .MAKE.DEPENDFILE
.endif
# just in case
.MAKE.DEPENDFILE ?= Makefile.depend

.if ${.MAKE.LEVEL} > 0
# Makefile.depend* also get read at level 1+
# and often refer to DEP_MACHINE etc,
# so ensure DEP_* (for TARGET_SPEC_VARS anyway) are set
.for V in ${TARGET_SPEC_VARS}
DEP_$V = ${$V}
.endfor
.endif

The SB_* variables are used by the make wrapper I've used for many years.

sys.dependfile.mk looks for an initial Makefile.depend file (actually .MAKE.DEPENDFILE_PREFIX could be set to anything, but I will stick with Makefile.depend). When a build is started in a leaf directory like bin/cat, this is the means by which DIRDEPS is initialized. This makefile also sets .MAKE.DEPENDFILE_PREFERENCE to a list of Makefile.depend patterns, the first entry being the default value for .MAKE.DEPENDFILE_DEFAULT.

When we build Junos we are always building for multiple machine types at the same time, so using Makefile.deppend.${MACHINE} is the default. This allows the files to be updated for multiple machines at the same time without locking.

For a more general case like {Free,Net}BSD most people actually only build for a single machine at a time, and in fact a single machine independent Makefile.depend works for most of the tree. So we use that by default. When building for multiple machines we avoid the update issue by nominating one MACHINE value to update for. We may also need to parameterize some of the dirdeps (eg. CSU_DIR).

The STAGE_* variables are for staging files as they are built. Staging can be thought of as auto install. The big difference between staging and simply installing into a ${DESTDIR} is that when we stage a file we put a file.dirdep there too which indicates who put it there.

This allows correct dependency collection and detecting errors like multiple makefiles trying to install the same file.

If TARGET_SPEC is just MACHINE when dirdeps.mk visits a leaf directory it will make MACHINE_ARCH empty, and sys.mk is expected to pick the right value based on MACHINE. local.dirdeps-build.mk ----------------------

This file provides glue between the normal bsd.*.mk and the DIRDEPS_BUILD bits. It is a little ugly, and much of its content would/could be integrated into bsd.*.mk.

.if !target(__${.PARSEFILE}__) __${.PARSEFILE}__: .NOTMAIN

.if ${MK_STAGING:Uno} == "yes" .if !empty(INCS) beforebuild: stage_includes STAGE_TARGETS+= stage_includes .if ${INCS:M*/} != "" incsgroups := ${INCS:H:O:u:N.} .for g s in ${incsgroups:@d@${d:S,/,_,g} $d@} STAGE_SETS+= $g STAGE_DIR.$g = ${STAGE_INCSDIR}/$s STAGE_FILES.$g = ${INCS:M$s/:S,^,${.CURDIR}/,} .endfor

The above dance allows for correctly staging headers when for example include/Makefile has ssp/ssp.h in INCS. It creates a separate set for each subdir of STAGE_INCSDIR to be populated.

.endif
.if ${INCS:N*/*} != ""
stage_incs: ${INCS:N*/*}
.endif
.endif
.if !empty(INCSYMLINKS)
STAGE_SETS+= INCS
STAGE_TARGETS+= stage_symlinks
STAGE_SYMLINKS.INCS= ${INCSYMLINKS}
.endif

.if ${MK_STAGING_PROG:Uno} == "yes"
STAGE_DIR.prog= ${STAGE_OBJTOP}${BINDIR}

.if !empty(PROG)
.if defined(PROGNAME)
STAGE_AS_SETS+= prog
STAGE_AS_${PROG}= ${PROGNAME}
stage_as.prog: ${PROG}
.else
STAGE_SETS+= prog
stage_files.prog: ${PROG}
STAGE_TARGETS+= stage_files
.endif
.endif
.endif

The above handles staging progs. The handling of libs below is likely incomplete:

.if !empty(_LIBS) && ${LIBISPRIVATE:Uno} == "no"
.if defined(SHLIBDIR) && ${SHLIBDIR} != ${LIBDIR} && ${_LIBS:Uno:M*.so.*} != ""
STAGE_SETS+= shlib
STAGE_DIR.shlib= ${STAGE_OBJTOP}${SHLIBDIR}
STAGE_FILES.shlib+= ${_LIBS:M*.so.*}
stage_files.shlib: ${_LIBS:M*.so.*}
.endif

.if target(stage_files.shlib)
stage_libs: ${_LIBS}
.if defined(DEBUG_FLAGS) && target(${SHLIB_NAME}.symbols)
stage_files.shlib: ${SHLIB_NAME}.symbols
.endif
.else
stage_libs: ${_LIBS}
.endif

.endif

.include <meta.stage.mk>
.endif

# prevent capturing local deps
.depend:
.include <meta.autodep.mk>

.endif

All in all it is not very complicated, and just works.

local.gendirdeps.mk

Sometimes we need to filter the raw data we glean from the syscall trace. For example we do this for optional functionality that would otherwise cause churn in the tree dependencies. We can have GENDIRDEPS_FILTER cause an optional dir to not be recorded, and have local.dirdeps.mk add it when needed.

# we need to filter these to reduce churn for different MACHINES

GENDIRDEPS_FILTER+= \
        Nsys/arch/${SYS_ARCH}/include* \
        Nsys/arch/${SYS_ARCH2}/include* \

.if ${RELDIR} == "lib/libc"
GENDIRDEPS_FILTER+= \
        Nexternal/gpl3/gcc/lib/libstdc++-v3/include*

.endif

We supress the above since they seem machine dependent and we want to avoid the need for Makefile.depend.${MACHINE} to the extent we can, especially for an OS that builds for so many architectures.

.if ${RELDIR:N*libgcc*:Nlib/libc} == ""
# these have explicit -I so do *not* need libpthread built first
# though due to sysroot the pthread headers will be found in stage
# root if tree already built.
GENDIRDEPS_FILTER+= Nlib/libpthread

.endif

This is an interesting case. libpthread cannot be built before libc or libgcc but those need pthread_types.h. We provide them with a direct -I so they can successfully build before it. But once the three has been built they will find that header via the stage tree, and we need to supress recording that since it would cause a cycle in the graph.

local.dirdeps-missing.mk

This makefile will be loaded by dirdeps.mk when it cannot find any Makefile.depend file in a directory. It is thus very handy for bootstrapping new dependencies:

# this is read when a dir has no Makefile.depend
# helps with bootstrapping
# we have to be careful to avoid cycles in the graph

.if ${DEP_RELDIR:Ncommon/include*:Ninclude*:Nlib/csu:Nsys/*} == ""
# things like include need nothing
DIRDEPS=
.elif ${DEP_RELDIR:Nexternal/gpl3/gcc/lib/*:Nlib/*} == ""
# libs need includes
DIRDEPS= \
        common/include/ppath \
        common/include/prop \
        common/include/rpc \
        include \
        sys/sys \
        sys/arch \

.else
# progs need libs
DIRDEPS= \
        external/gpl3/gcc/lib/libgcc/libgcc \
        external/gpl3/gcc/lib/libgcc/libgcc_s \
        lib/csu \
        lib/libc \
        lib/libpthread \
        lib/libutil \

DIRDEPS:= ${DIRDEPS:N${DEP_RELDIR}}
.endif

.include <dirdeps.mk>

The above was sufficient to get bin/cat building.

Toolchains

I should mention that I consider building toolchains out of scope for this exercise. Building toolchains is a corner case, and should happen rarely enough to not be worth optimizing - or even integrating.

BTW for 5.X I hacked build.sh to not build the old make, since my /usr/bin/make is always the latest from current.

But in general I just use the toolchain as built by the normal build.

Circular dependencies

Our goal is to build the tree in a single pass. This means we cannot have any circular dependencies.

In NetBSD current of 2015 I had to break cycles involving libpthread which is achieved by adding:

.if ${MK_DIRDEPS_BUILD:Uno} == "yes"
# avoid a circular dependency with libpthread
CPPFLAGS+= -I${SRCTOP}/lib/libpthread
.endif

to each of:

external/gpl3/gcc/lib/libgcc/Makefile.inc
lib/csu/Makefile
lib/libc/Makefile

Without the direct -I each of these needs libpthread built first so that pthread_types.h is staged, but libpthread cannot be built without all the above.

In 2023 I did not find that necessary. Though I did have to fix external/gpl3/gcc/lib/libgcc/Makefile.inc to avoid multiple variants of libgcc from trying to stage the same header:

Index: external/gpl3/gcc/lib/libgcc/Makefile.inc
===================================================================
RCS file: /cvsroot/src/external/gpl3/gcc/lib/libgcc/Makefile.inc,v
retrieving revision 1.52
diff -u -p -r1.52 Makefile.inc
--- external/gpl3/gcc/lib/libgcc/Makefile.inc   22 Jul 2022 21:59:11 -0000      1.52
+++ external/gpl3/gcc/lib/libgcc/Makefile.inc   11 May 2023 06:12:46 -0000
@@ -255,3 +255,7 @@ CPPFLAGS+=  -I${BINBACKENDOBJ}
 #.if !empty(LIBGCC_MACHINE_ARCH:Mearm*)
 COPTS.unwind-dw2.c+=   -Wno-discarded-qualifiers
 #.endif
+
+.if ${MK_STAGING} == "yes" && ${LIB:Mgcc*} != "" && ${LIB} != "gcc"
+INCS=
+.endif

Conclusion

Back in 2015 it took only a couple of days effort to have most of userland building (400+ apps and all the libs they need) as well as building kernels. Of course I was able to re-use targets/* from FreeBSD.

For the 2023 exercise it took less than a day to get bin/cat building, and I didn't do more, as I'd achieved my purpose.


Author:sjg@crufty.net
Revision:$Id: netbsd-meta-mode.txt,v 8c676d3327df 2023-05-11 23:00:35Z sjg $