This doc describes building NetBSD using bmake in meta mode leveraging the work done building FreeBSD in meta mode
Junos (which is based on FreeBSD) has been built this way since 2010 and it works very well.
We have built FreeBSD the same way for almost as long, and virtually everything discussed here has been upstreamed to FreeBSD.
Note: while I normally talk of meta mode and using dirdeps.mk together, either one can be used without the other.
In FreeBSD two separate controls are used DIRDEPS_BUILD and META_MODE. Enabling DIRDEPS_BUILD implies META_MODE, but many developers use the latter even with the traditional build targets.
I first did this exercise with NetBSD in about 2015, and again in 2023, which was as much as anything was a test of a new bootstrapping method. This was prompted by a failure to build NetBSD/current on an older NetBSD host.
In 2026 I revisted this again, and got the native (build.sh) build working with META_MODE.
Note: since bmake is just a more portable form of NetBSD's make I use the two interchangeably below.
I also tend to use mk (a wrapper around [bg]make, part of my sb-tools collection) rather than make directly. It conditions the environment, selects the appropriate version of make for the current project. It also offers some useful options which are especially handy for the DIRDEPS_BUILD.
There isn't a lot wrong with the NetBSD build as it is. At least for doing release builds and such. It builds in parallel ok, and can be cross-built on a number of different hosts. Thus I've not previously been seriously tempted to interfere with it.
However in 2015, the prospect of doing a dev project on NetBSD at work was enough to prompt me to see what it would take to get the DIRDEPS_BUILD (which is hard to live without once you've tried it ;-) working on NetBSD.
The short answer is very little. Frankly, despite planning to leverage the work already done for FreeBSD, I was very pleasantly surprised to find how little effort it took to get bin/cat (and everything it needs) building. Most of that time was spent finding the various places that needed to be bootstrapped, to link cat. See bootstrapping below to see how this step is now much simpler.
After building for i386 I then built for evbarm to see what the impact on the dependencies might be. For FreeBSD I found with just parameterizing CSU_DIR a single Makefile.depend worked for 99% of the tree, and most places where Makefile.depend.${MACHINE} was needed it was due to generated files being named differently.
For NetBSD some of the dependencies varied quite a bit between i386 and evbarm, but nothing that cannot be handled by a little filtering - which the meta mode infrastructure provides for.
Fair question.... It is a better way of building a large tree of software.
Most of this is covered in building FreeBSD in meta mode But I'll briefly rehash the story here.
The goal is to be able to start a build from anywhere in a freshly checked out tree (eg. bin/cat) and have it just work. That means building everything else in the tree needed, but only what is needed, in the right order, in parallel. To be able to do that for multiple target machines at the same time and even multiple target operating systems at the same time. All without tree walks.
The above is all handled by using dirdeps.mk (hence the DIRDEPS_BUILD option in FreeBSD) which is used by the initial make instance to compute a graph of tree dependencies from the current origin, and then go build them all in parallel.
The tree wide graph ensures that all leaf directories are visited in the correct order - no tree walks. In fact dirdeps.mk suppresses the behavior of bsd.subdir.mk.
Using dirdeps.mk we can greatly simplify the top-level build logic, which apart from share/mk/ is where we typically find a concentration of complexity:
NetBSD (2015):
$ wc -l Makefile build.sh
530 Makefile
2309 build.sh
2839 total
NetBSD (2026):
$ wc -l Makefile build.sh
536 Makefile
3037 build.sh
3573 total
That's a rather modest increase (25%) in just over 10 years!
FreeBSD (2015):
$ wc -l Makefile Makefile.inc1
525 Makefile
2196 Makefile.inc1
2721 total
FreeBSD (2026):
$ wc -l Makefile Makefile.inc1
826 Makefile
4115 Makefile.inc1
4941 total
Much more substantial (81%).
In 2010 the pre-meta mode Junos build had over 5000 lines of very dense makefiles at the top-level. The original top-level makefiles for the meta mode build that replaced all that was less than 300 lines!
NetBSD/FreeBSD meta mode (2015):
$ wc -l targets/Ma* share/mk/dirdeps.mk
172 targets/Makefile
52 targets/Makefile.inc
46 targets/Makefile.xtras
644 share/mk/dirdeps.mk
914 total
FreeBSD meta mode (2026):
$ wc -l targets/Makefile* share/mk/dirdeps*.mk
114 targets/Makefile
52 targets/Makefile.inc
76 targets/Makefile.xtras
106 share/mk/dirdeps-options.mk
173 share/mk/dirdeps-targets.mk
1022 share/mk/dirdeps.mk
1543 total
I include dirdeps.mk in the line count because that's where almost all of the complexity is, but of course we leverage it for all builds not just top-level ones, so it is well worth while.
dirdeps-targets.mk handles the task of finding a suitable Makefile.depend under targets/ and dirdeps-options.mk handles the dependency complications introduced by dozens of optional features.
In fact targets/Makefile isn't necessary at all. The inclusion of dirdeps-targets.mk can be handle by top.mk from local.sys.mk:
.if ${.MAKE.LEVEL} == 0
.if ${RELDIR} == "."
.MAKE.MAKEFILE_PREFERENCE := top.mk ${.MAKE.MAKEFILE_PREFERENCE}
.endif
top.mk will remove itself from .MAKE.MAKEFILE_PREFERENCE if needed. Normally it might be included by a top-level makefile or targets/Makefile.
With the DIRDEPS_BUILD, adding a new top-level target is as simple as creating a directory somewhere under targets/ and putting a Makefile.depend file there that lists the directories needed, the Makefile is often trivial - leveraging common packaging logic. These makefiles though are no more complex than any leaf makefile.
Pseudo targets that don't actually build anything themselves, are typicallly added under targets/pseudo/.
For example I created a target for bootstrapping, thus:
mkdir -p targets/pseudo/bs echo all: > targets/pseudo/bs/Makefile echo DIRDEPS = bin/cat > targets/pseudo/bs/Makefile.depend echo '.include <dirdeps.mk>' >> targets/pseudo/bs/Makefile.depend
I can now just:
mk bs-jobs
mk (a make wrapper - part of sb-tools) will condition the environment and run bmake. top.mk will spot the target as *-jobs and include jobs.mk which will creat the target bs-jobs to:
jobs.mk also computes a suitable value for .MAKE.JOBS. If bmake is recent, it sets .MAKE.JOBS.C=yes in which case we default JOBS_MAX ?= 1.33 which is used as a multiple of the number of CPUs available.
When make is run in meta mode, it generally creates a .meta file for each target being built. This is normally only done if ${.OBJDIR} != ${.CURDIR} - which is why we want to disable NOOBJ in include/Makefile.
The meta file captures information that allows make to do a much better job for update builds.
Firstly we capture the expanded command line. This allows make to compare the command it might need to run with what was done last time, if anything changed the target is out-of-date.
Now sometimes you want to suppress that behavior so there are knobs to do so for a whole target or just one line.
For example if a target includes variable like BUILD_UTC which might change every time, but we don't what that alone to cause the target to be out-of-date, we can add .NOMETA_CMP as a source, or include a reference like ${BUILD_UTC}${.OODATE:MNOMETA_CMP}, make knows that .OODATE is not stable and thus cannot be compared, and by adding the modifier :MNOMETA_CMP we ensure it expands to nothing, while still informing make that that line of the target script should not be compared.
This feature alone makes a huge difference to build reliability. It is no longer necessary to make targets depend on every makefile that might influence them. This allows us to avoid building things when nothing relevant has changed, while ensuring we never fail to rebuild when they have.
The captured command is also invaluable for debugging. It also allows the build log to suppress most target output, leaving us with just:
@ 1785092397 [2026-07-26 11:59:57] Checking sys/modules/autofs.amd64,x86_64 for amd64,x86_64 ... Building /var/obj/NetBSD/11/amd64.x86_64/sys/modules/autofs/ioconf.c Building /var/obj/NetBSD/11/amd64.x86_64/sys/modules/autofs/autofs.o Building /var/obj/NetBSD/11/amd64.x86_64/sys/modules/autofs/autofs_vfsops.o Building /var/obj/NetBSD/11/amd64.x86_64/sys/modules/autofs/autofs_vnops.o Building /var/obj/NetBSD/11/amd64.x86_64/sys/modules/autofs/autofs.kmod Building /var/obj/NetBSD/11/amd64.x86_64/sys/modules/autofs/stage_files.prog Checking /homes/sjg/work/NetBSD/11/src/sys/modules/autofs/Makefile.depend: ioconf.c.meta autofs.o.meta autofs_vfsops.o.meta autofs_vnops.o.meta autofs.kmod.meta stage_files.prog.meta @ 1785092400 [2026-07-26 12:00:00] Finished sys/modules/autofs.amd64,x86_64 seconds=2 meta=7 created=6
The next thing captured is any output from the command. This is mainly for human debugging purposes.
This extremely useful when you have 1000's of users who fail to log their builds and still want to you help them understand why their build failed.
The filemon module provides /dev/filemon which make will use if available.
When make runs a child, it opens /dev/filemon and gives it a temp file and the pid of the child. All successful syscalls (interesting to make) made by the child and its progeny will be recorded in the temp file. When the child exits, make appends the data to the meta file.
This data has two main uses. Firstly make itself uses it to better check if a target needs update. Secondly we can post-process the data for all sorts of purposes including extracting directory dependencies - which allows us to automate the collection of the data needed by dirdeps.mk.
For example bin/cat/Makefile.depend:
# Autogenerated - do NOT edit!
DIRDEPS = \
external/gpl3/gcc/lib/libgcc/libgcc \
external/gpl3/gcc/lib/libgcc/libgcc_s \
include \
lib/csu \
lib/libc \
lib/libpthread \
sys/sys \
.include <dirdeps.mk>
As we read the files read or executed, any object directory we see which is not under ${.OBJDIR} indicates a directory that needs to be built before ${.CURDIR}. Being able to map object dirs to src dirs is important.
Not shown is a section for local dependencies where we can record any dependencies involving locally generated files - also gleaned from the syscall trace. This allows us to reliably build a clean tree in parallel while skipping any explict depend step.
Actually with the .ORDER: bug fixed in bmake several years ago, it is rarely needed to capture any local dependencies, and the FreeBSD build disables that behavior.
The syscall trace is also invaluable for debugging weird build issues. For example spotting that your supposed captive build is reading headers from /usr/include or linking libs from /usr/local/lib. Or running a perl or python binary other than the one intended. Being able to look under the covers after a build has failed is immensely useful.
When make is checking if a target needs update and the normal makefile rules indicate it is up-to-date, make calls meta_oodate to delve deeper using the meta file. Here is cat.o.meta:
# Meta data file /h/NetBSD/5.X/obj/i386/bin/cat/cat.o.meta CMD @echo '# ' "compile " cat/cat.o CMD /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc -O2 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wno-sign-compare -Wno-traditional -Wreturn-type -Wswitch -Wshadow -Wcast-qual -Wwrite-strings -Wextra -Wno-unused-parameter -nostdinc -I/h/NetBSD/5.X/obj/stage/i386/usr/include -nostdinc -isystem /h/NetBSD/5.X/obj/stage/i386/usr/include -c /h/NetBSD/5.X/src/bin/cat/cat.c CWD /h/NetBSD/5.X/obj/i386/bin/cat TARGET cat.o -- command output -- # compile cat/cat.o -- filemon acquired metadata -- # filemon version 4 # Target pid 2050 V 4 E 8142 /bin/sh R 8142 /etc/ld.so.conf R 8142 /lib/libedit.so.2 R 8142 /lib/libtermcap.so.0 R 8142 /lib/libc.so.12 X 8142 0 # Bye bye E 5426 /var/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/i386--netbsdelf-gcc R 5426 /etc/ld.so.conf R 5426 /usr/lib/libc.so.12 R 5426 /var/tmp//ccKCUD9I.s W 5426 /var/tmp//ccKCUD9I.s E 15146 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../libexec/gcc/i386--netbsdelf/4.1.3/cc1 R 15146 /etc/ld.so.conf R 15146 /usr/lib/libc.so.12 R 15146 /h/NetBSD/5.X/src/bin/cat/cat.c R 15146 /var/tmp//ccKCUD9I.s W 15146 /var/tmp//ccKCUD9I.s R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/cdefs.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/cdefs_elf.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/param.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/null.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/inttypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stdint.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_mwgwtypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_limits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_const.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/wchar_limits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/int_fmtio.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/ansi.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ansi.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/endian.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/endian_machdep.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/byte_swap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/bswap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/bswap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/fd_set.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/pthread_types.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/syslimits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/signal.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/sigtypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/satypes.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/siginfo.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/signal.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/trap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/x86/trap.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/ucontext.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/mcontext.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/param.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/machine/limits.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/stat.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/select.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/time.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/time.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/ctype.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/err.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/errno.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/errno.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/fcntl.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/locale.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdio.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/stdlib.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/strings.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/string.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/unistd.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/unistd.h R 15146 /h/NetBSD/5.X/obj/stage/i386/usr/include/sys/featuretest.h X 15146 0 E 18126 /h/obj/NetBSD/5.X/tools/NetBSD-5.2_STABLE-i386/bin/../lib/gcc/i386--netbsdelf/4.1.3/../../../../i386--netbsdelf/bin/as R 18126 /etc/ld.so.conf R 18126 /usr/lib/libc.so.12 R 18126 cat.o W 18126 cat.o R 18126 /var/tmp//ccKCUD9I.s X 18126 0 D 5426 /var/tmp//ccKCUD9I.s X 5426 0 # Bye bye
I bet you didn't know it takes all those headers to compile cat.o. That's 79 headers (54 unique), not what you would expect from just looking at cat.c.
Anyway, as noted above, the first thing we do is check if the command line changed, any change to CFLAGS would do it. If it did - we are done - out-of-date.
Otherwise we look at the syscall trace. This is formatted so that it is trivial to parse by even a shell script.
In fact sb-tools includes meta-funcs.sh which is a collection of shell functions for extracting useful data from meta files - like the command so it can be re-run easily, spliting the command into individual words one per line so it is much easier to compare with another using diff(1). Extracting the list of files read - all resolved to absoulte paths, and so on.
In the above example it is simple, we just need to look at all the E (exec) and R (read) entries; if any of the files read/executed are newer the target is out-of-date.
Sometimes the syscall trace is more complex, so make needs to follow the F (fork) and C (chdir) syscalls by pid so it knows what the cwd of each is so if it sees a file opened by a relative path it can find it.
If a file previously written is missing (and the syscall trace does not show it being D deleted), the target is out-of-date.
Of course make will ignore some paths (eg. anything in /tmp) and you can provide it a list of path prefixes to ignore.
The ability for make to look under the covers like this allows it to do a much better job of ensuring things are re-built when they need to be.
You can use the debug flag -dM to have make tell you what it is about a meta file that made it decide the target is out-of-date.
The filemon module is currently available in NetBSD and FreeBSD and a version for Linux is available from https://github.com/trixirt/filemon-linux.git (though that needs updating)
In Jan 2020 NetBSD introduced filemon_ktrace as an inteface to fktrace(2) for use by make eliminating the need for filemon(4). The old api is retained as filemon_dev for use on FreeBSD and Linux.
filemon_ktrace adds a little overhead in the form of an extra file descriptor per job.
With luck the filemon_ktrace model can be leveraged by other syscall trace mechanisms on other systems.
Initially I only had to touch one makefile outside of share/mk and that was include/Makefile because I wanted an obj dir so we can get .meta files:
Index: include/Makefile
===================================================================
RCS file: /cvsroot/src/include/Makefile,v
retrieving revision 1.140
diff -u -p -r1.140 Makefile
--- include/Makefile 11 Dec 2013 01:24:08 -0000 1.140
+++ include/Makefile 13 May 2015 18:53:29 -0000
@@ -3,7 +3,9 @@
# Doing a make includes builds /usr/include
+.if ${MKMETA_MODE:Uno} == "no"
NOOBJ= # defined
+.endif
# Missing: mp.h
by contrast I had to quite heavily tweak the FreeBSD equivalent. This was a pleasant surprise.
In fact for the 2023 revisit apart from the above change to include/Makefile the only existing files touched were:
share/mk/bsd.files.mk share/mk/bsd.init.mk share/mk/bsd.sys.mk share/mk/sys.mk external/gpl3/gcc/lib/libgcc/Makefile.inc
in each case only one or two lines such as:
.-include <local.dirdeps-build.mk>
local.dirdeps-build.mk is normally included by dirdeps.mk at level 1+ (building), but while bootstrapping there may not be any Makefile.depend files to include dirdeps.mk.
Update 2026: actually we handle this via local.dirdeps-missing.mk. In local.sys.mk (included towards the end of sys.mk) we have:
.if ${MK_DIRDEPS_BUILD:Uno} == "yes"
.if ${.MAKE.LEVEL} == 0
.if ${RELDIR} == "."
.MAKE.MAKEFILE_PREFERENCE := top.mk ${.MAKE.MAKEFILE_PREFERENCE}
.endif
.else
# we depend on dirdeps.mk to include local.dirdeps-build.mk at the very end
.if !exists(${.MAKE.DEPENDFILE})
.MAKE.DEPENDFILE = local.dirdeps-missing.mk
.endif
.endif
.endif
and local.dirdeps-build.mk will be automatically included in most cases.
Of course, the above required a change to meta.autodep.mk to compensate.
Note: we guard all this with ${MK_DIRDEPS_BUILD:Uno} == "yes" so as to avoid affecting the native build.
I should mention that I consider building toolchains out of scope for this exercise. Building toolchains is a corner case, and should happen rarely enough to not be worth optimizing - or even integrating.
For 5.X I hacked build.sh to not build the old make, since my /usr/bin/make is always the latest from current.
For 11.x I fixed the build of nbmake to provide support for META_MODE and filemon_ktrace:
Index: tools/make/buildmake.sh.in
===================================================================
RCS file: /cvsroot/src/tools/make/buildmake.sh.in,v
retrieving revision 1.18
diff -u -p -r1.18 buildmake.sh.in
--- tools/make/buildmake.sh.in 20 Jul 2023 15:16:44 -0000 1.18
+++ tools/make/buildmake.sh.in 1 Aug 2026 15:39:38 -0000
@@ -7,6 +7,36 @@
: ${NETBSDSRCDIR:=@srcdir@/../..}
MKSRCDIR=${NETBSDSRCDIR}/usr.bin/make
+# makefiles like dirdeps.mk (comes with bmake)
+# check MAKE_VERSION to know if certain features work,
+if test -s $MKSRCDIR/VERSION; then
+ . $MKSRCDIR/VERSION
+fi
+if test "x$_MAKE_VERSION" != x; then
+ COPTS_main="-DMAKE_VERSION=\"$_MAKE_VERSION\""
+fi
+
+# Enable meta mode
+# this does not turn it on, just make it available
+COPTS="-DUSE_META"
+
+# and filemon if we can have it
+USE_FILEMON=@filemon@
+if [ x$USE_FILEMON != x ]; then
+ case "$USE_FILEMON" in
+ dev)
+ FILEMON=DEV
+ COPTS_filemon_dev="@filemon_flags@"
+ ;;
+ ktrace)
+ FILEMON=KTRACE
+ ;;
+ esac
+ COPTS_meta="-DUSE_FILEMON -DUSE_FILEMON_$FILEMON"
+ COPTS_job="$COPTS_meta"
+ XTRA_SRCS=$MKSRCDIR/filemon/filemon_$USE_FILEMON.c
+fi
+
docmd()
{
local msg=$1
@@ -23,9 +53,11 @@ docmd()
"$@" || exit 1
}
-for f in $MKSRCDIR/*.c; do
+for f in $MKSRCDIR/*.c $XTRA_SRCS; do
+ fb=`basename $f .c`
+ eval "copts=\"$COPTS \$COPTS_$fb\""
docmd "compile " "$f" @CC@ @CPPFLAGS@ @DEFS@ @CFLAGS@ @NOWARNFLAGS@ \
- -D_PATH_DEFSYSPATH=\"${NETBSDSRCDIR}/share/mk\" -c "$f"
+ -D_PATH_DEFSYSPATH=\"${NETBSDSRCDIR}/share/mk\" $copts -c "$f"
done
docmd " link " "${_TOOL_PREFIX:-nb}make" \
Index: tools/make/configure.ac
===================================================================
RCS file: /cvsroot/src/tools/make/configure.ac,v
retrieving revision 1.11
diff -u -p -r1.11 configure.ac
--- tools/make/configure.ac 20 Jul 2023 15:16:44 -0000 1.11
+++ tools/make/configure.ac 1 Aug 2026 15:39:38 -0000
@@ -50,6 +50,22 @@ AC_SEARCH_LIBS([regfree], [rx posix])
AC_CHECK_FUNCS([setenv strdup strerror strftime vsnprintf])
+dnl # We want filemon if we can
+case "`uname`" in
+FreeBSD)
+ filemon=dev
+ filemon_h=/usr/include/dev/filemon/filemon.h
+ if test -s $filemon_h; then
+ filemon_flags="-DHAVE_FILEMON_H -I`dirname $filemon_h`"
+ fi
+ ;;
+NetBSD)
+ filemon=ktrace
+ ;;
+esac
+AC_SUBST(filemon)
+AC_SUBST(filemon_flags)
+
dnl NOWARNFLAGS: compiler flags to suppress unnecessary warnings:
dnl -Wno-deprecated-declarations
dnl
This is needed to be able to build the native (build.sh) build in meta mode.
For my normal builds the make wrapper mk just finds the latest version of bmake.
Our goal is to build the tree in a single pass. This means we cannot have any circular dependencies.
In NetBSD current of 2015 I had to break cycles involving libpthread which is achieved by adding:
.if ${MK_DIRDEPS_BUILD:Uno} == "yes"
# avoid a circular dependency with libpthread
CPPFLAGS+= -I${SRCTOP}/lib/libpthread
.endif
to each of:
external/gpl3/gcc/lib/libgcc/Makefile.inc lib/csu/Makefile lib/libc/Makefile
Without the direct -I each of these needs libpthread built first so that pthread_types.h is staged, but libpthread cannot be built without all the above.
Another way to address this is to stage headers for the pseudo machine common and first visit lib/libpthread.common which would only stage its headers. The majority of headers, are machine independent so this approach generally works well. But for this exercise I'm trying to minimize changes to the tree.
In 2026 I addressed that and the need to prevent libgcc_s staging the same headers as libgcc to the same location. This was all done via local.dirdeps-build.mk.
In fact after boostrapping bin/cat the only files touched were:
Makefile build.sh external/gpl3/gcc/lib/Makefile.inc include/Makefile share/mk/bsd.init.mk share/mk/bsd.lib.mk share/mk/bsd.own.mk share/mk/bsd.prog.mk share/mk/sys.mk sys/arch/i386/stand/boot/Makefile.boot sys/arch/i386/stand/boot/biosboot/Makefile tests/net/fdpass/Makefile tools/make/buildmake.sh.in
most have already been discussed above.
The remainding diffs:
Index: Makefile
===================================================================
RCS file: /cvsroot/src/Makefile,v
retrieving revision 1.341
diff -u -p -r1.341 Makefile
--- Makefile 4 Jun 2025 06:01:59 -0000 1.341
+++ Makefile 30 Jul 2026 20:25:16 -0000
@@ -101,7 +101,7 @@
# do-obsolete: installs the obsolete sets (for the postinstall-* targets).
#
-.if ${.MAKEFLAGS:M${.CURDIR}/share/mk} == ""
+.if empty(MAKESYSPATH) && ${.MAKEFLAGS:M${.CURDIR}/share/mk} == ""
.MAKEFLAGS: -m ${.CURDIR}/share/mk
.endif
Index: build.sh
===================================================================
RCS file: /cvsroot/src/build.sh,v
retrieving revision 1.399
diff -u -p -r1.399 build.sh
--- build.sh 8 Jul 2025 12:29:40 -0000 1.399
+++ build.sh 30 Jul 2026 20:25:16 -0000
@@ -1600,7 +1600,12 @@ parseoptions()
makeenv="${makeenv} TOOLDIR MACHINE MACHINE_ARCH MAKEFLAGS"
[ -z "${BUILDID}" ] || makeenv="${makeenv} BUILDID"
[ -z "${BUILDINFO}" ] || makeenv="${makeenv} BUILDINFO"
- MAKEFLAGS="-de -m ${TOP}/share/mk ${MAKEFLAGS}"
+ MAKEFLAGS="-de ${MAKEFLAGS}"
+ if [ -z "$MAKESYSPATH" ]; then
+ MAKEFLAGS="-m ${TOP}/share/mk ${MAKEFLAGS}"
+ else
+ makeenv="${makeenv} MAKESYSPATH"
+ fi
MAKEFLAGS="${MAKEFLAGS} MKOBJDIRS=${MKOBJDIRS-yes}"
export MAKEFLAGS MACHINE MACHINE_ARCH
if [ -z "${USETOOLS}" ]; then
We want our MAKESYSPATH value left alone.
Index: external/gpl3/gcc/lib/Makefile.inc
===================================================================
RCS file: /cvsroot/src/external/gpl3/gcc/lib/Makefile.inc,v
retrieving revision 1.4
diff -u -p -r1.4 Makefile.inc
--- external/gpl3/gcc/lib/Makefile.inc 20 Apr 2016 17:18:52 -0000 1.4
+++ external/gpl3/gcc/lib/Makefile.inc 30 Jul 2026 20:25:21 -0000
@@ -4,10 +4,7 @@ GCC_MACHINE_ARCH=${MACHINE_ARCH:S/earmv5
.ifndef _EXTERNAL_GPL3_GCC_LIB_MAKEFILE_INC_
_EXTERNAL_GPL3_GCC_LIB_MAKEFILE_INC_=1
-
-.sinclude "../../Makefile.gcc_path"
-.sinclude "../../../Makefile.gcc_path"
-.sinclude "../../../../Makefile.gcc_path"
+.include "../Makefile.gcc_path"
WARNS=1
The fishing for Makefile.gcc_path is unnecessary and blows up against against autofs since none of the above are valid from the position of this Makefile.inc so we end up searching via .SYSPATH. Since Makefile.gcc_path is in the parent directory of this Makefile.inc all we need is .include "../Makefile.gcc_path".
Index: sys/arch/i386/stand/boot/Makefile.boot
===================================================================
RCS file: /cvsroot/src/sys/arch/i386/stand/boot/Makefile.boot,v
retrieving revision 1.76
diff -u -p -r1.76 Makefile.boot
--- sys/arch/i386/stand/boot/Makefile.boot 3 Jun 2023 08:52:56 -0000 1.76
+++ sys/arch/i386/stand/boot/Makefile.boot 30 Jul 2026 20:25:24 -0000
@@ -1,6 +1,7 @@
# $NetBSD: Makefile.boot,v 1.76 2023/06/03 08:52:56 lukem Exp $
-S= ${.CURDIR}/../../../../..
+#S= ${.CURDIR}/../../../../..
+S:= ${.PARSEDIR:tA:H:H:H:H}
NOMAN=
NOLIBCSANITIZER=
@@ -28,7 +29,7 @@ LIBC= # nothing
BINDIR=/usr/mdec
BINMODE=444
-.PATH: ${.CURDIR}/.. ${.CURDIR}/../../lib
+.PATH: ${.PARSEDIR} ${.PARSEDIR:H}/lib
LDFLAGS+= -nostdlib -Wl,-N -Wl,-e,boot_start
CPPFLAGS+= -I ${.CURDIR}/.. -I ${.CURDIR}/../../lib -I ${S}/lib/libsa
the above is not strictly needed, just the needless use of ../../.. offends me ;-)
Index: sys/arch/i386/stand/boot/biosboot/Makefile =================================================================== RCS file: /cvsroot/src/sys/arch/i386/stand/boot/biosboot/Makefile,v retrieving revision 1.3 diff -u -p -r1.3 Makefile --- sys/arch/i386/stand/boot/biosboot/Makefile 11 Dec 2005 12:17:48 -0000 1.3 +++ sys/arch/i386/stand/boot/biosboot/Makefile 30 Jul 2026 20:25:24 -0000 @@ -2,4 +2,4 @@ PROG= boot -.include <../Makefile.boot> +.include "../Makefile.boot"
Another case where using "" rather than <> is more efficient.
Index: tests/net/fdpass/Makefile
===================================================================
RCS file: /cvsroot/src/tests/net/fdpass/Makefile,v
retrieving revision 1.4
diff -u -p -r1.4 Makefile
--- tests/net/fdpass/Makefile 13 Sep 2012 21:13:34 -0000 1.4
+++ tests/net/fdpass/Makefile 30 Jul 2026 20:25:26 -0000
@@ -23,7 +23,7 @@ BINDIR.fdpass32= ${TESTSDIR}
fdpass64.c fdpass32.c: fdpass.c
- ln -s ${.CURDIR}/fdpass.c ${.TARGET}
+ ln -sf ${.CURDIR}/fdpass.c ${.TARGET}
CLEANFILES += fdpass64.c fdpass32.c
For some reason in the native build this directory is visted more than once and the above target is run again (meta mode sometimes looks too deep). Using ln -sf was just an expedient fix.
Now that the dust as settled we and run make -dM to can see why:
mk --machine 1,amd64 -DWITHOUT_DIRDEPS_BUILD MAKEOBJDIRPREFIX=/h/obj/NetBSD/meta -DWITHOUT_AUTO_OBJ -dM -C tests/net/fdpass MKVERBOSE=0 Skipping meta for filesbuild: .PHONY Skipping meta for beforebuild: no commands Skipping meta for .WAIT_3: .PHONY /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64.c.meta:11: file '/bin/ln' is newer than the target... Building /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64.c /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64.meta: missing files: /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64.ctf... Building /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64 # link fdpass/fdpass64 Skipping meta for .WAIT_4: .PHONY /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32.c.meta:11: file '/bin/ln' is newer than the target... Building /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32.c /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32.meta: missing files: /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32.ctf... Building /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32 # link fdpass/fdpass32 Skipping meta for realall: .PHONY Skipping meta for subdir-all: .PHONY Skipping meta for all: .PHONY Skipping meta for .END: .SPECIAL
so because fdpass64.c and fdpass32.c are symlinks to tests/net/fdpass/fdpass.c which hasn't changed since 2012, no surprise that /bin/ln is newer. We can tell bmake to ignore paths in /bin etc by adding the following to local.meta.sys.mk:
.MAKE.META.IGNORE_PATHS += /bin /sbin /usr/bin /usr/sbin /usr/pkg /lib
Now we can revert that makefile change and we get:
Skipping meta for filesbuild: .PHONY Skipping meta for beforebuild: no commands Skipping meta for .WAIT_3: .PHONY /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64.meta: missing files: /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64.ctf... Building /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass64 Skipping meta for .WAIT_4: .PHONY /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32.meta: missing files: /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32.ctf... Building /h/obj/NetBSD/meta/homes/sjg/work/NetBSD/11/src/tests/net/fdpass/fdpass32 Skipping meta for realall: .PHONY Skipping meta for subdir-all: .PHONY Skipping meta for all: .PHONY Skipping meta for .END: .SPECIAL
The missing fdpass64.ctf is interesting. The meta file clearly shows it created:
E 29673 /h/obj/NetBSD/tools/NetBSD-11.0_RC6-x86_64/bin/nbctfmerge R 29673 /usr/lib/librt.so.1 R 29673 /usr/lib/libz.so.1 R 29673 /usr/lib/libpthread.so.1 R 29673 /usr/lib/libc.so.12 R 29673 fdpass64.o R 29673 fdpass64.o R 29673 fdpass64 R 29673 fdpass64.ctf W 29673 fdpass64.ctf X 29673 0
but nothing shows it being removed, and it is indeed missing afterwards.
Interestingly one cannot usefullly ktrace make while it is using fktrace(2). We have to disable use of that to get a useful ktrace:
ktrace -i -f /tmp/kt mk --machine 1,amd64 -C tests/net/fdpass
-DWITHOUT_DIRDEPS_BUILD MAKEOBJDIRPREFIX=/h/obj/NetBSD/meta
-DWITHOUT_AUTO_OBJ -dM META_MODE_XTRAS=nofilemon
kdump -E -f /tmp/kd -m 78 > /tmp/kd
grep nbctfmerge /tmp/kd
..
..
12959 12959 nbctfmerge 0.479872861 CALL open(0x782a2df12000,0x602,0x81ed)
12959 12959 nbctfmerge 0.479874468 NAMI "fdpass64.ctf"
12959 12959 nbctfmerge 0.482359775 RET open 6
12959 12959 nbctfmerge 0.482362010 CALL __fstat50(6,0x7f7fff17db50)
12959 12959 nbctfmerge 0.482363546 RET __fstat50 0
..
..
12959 12959 nbctfmerge 0.482840466 CALL ftruncate(6,0,0)
12959 12959 nbctfmerge 0.482845215 RET ftruncate 0
12959 12959 nbctfmerge 0.482848707 CALL lseek(6,0,0,0)
12959 12959 nbctfmerge 0.482850383 RET lseek 0
12959 12959 nbctfmerge 0.482879575 CALL write(6,0x782a2de28000,0x6a08)
12959 12959 nbctfmerge 0.482923014 GIO fd 6 wrote 4088 bytes
..
12959 12959 nbctfmerge 0.482934957 GIO fd 6 wrote 2616 bytes
12959 12959 nbctfmerge 0.482936563 RET write 27144/0x6a08
12959 12959 nbctfmerge 0.482949064 CALL close(5)
12959 12959 nbctfmerge 0.482950880 RET close 0
12959 12959 nbctfmerge 0.482952276 CALL close(6)
12959 12959 nbctfmerge 0.482955489 RET close 0
12959 12959 nbctfmerge 0.482958003 CALL __posix_rename(0x782a2df12000,0x7f7fff17ed3e)
12959 12959 nbctfmerge 0.482959958 NAMI "fdpass64.ctf"
12959 12959 nbctfmerge 0.482964707 NAMI "fdpass64"
12959 12959 nbctfmerge 0.484119680 RET __posix_rename 0
..
that looks suspiciously like a rename that did not get recorded.
Yes, filemon_ktrace.c needs an entry for SYS__posix_rename.
Now we see the rename in the meta file:
E 14982 /h/obj/NetBSD/tools/NetBSD-11.0_RC6-x86_64/bin/nbctfmerge R 14982 /usr/lib/librt.so.1 R 14982 /usr/lib/libz.so.1 R 14982 /usr/lib/libpthread.so.1 R 14982 /usr/lib/libc.so.12 R 14982 fdpass64.o R 14982 fdpass64.o R 14982 fdpass64 R 14982 fdpass64.ctf W 14982 fdpass64.ctf M 14982 'fdpass64.ctf' 'fdpass64' X 14982 0
And meta_oodate is happy:
mk --machine 1,amd64 -DWITHOUT_DIRDEPS_BUILD MAKEOBJDIRPREFIX=/h/obj/NetBSD/meta -DWITHOUT_AUTO_OBJ -dM -C tests/net/fdpass MKVERBOSE=0 Skipping meta for filesbuild: .PHONY Skipping meta for beforebuild: no commands Skipping meta for .WAIT_3: .PHONY Skipping meta for .WAIT_4: .PHONY Skipping meta for realall: .PHONY Skipping meta for subdir-all: .PHONY Skipping meta for all: .PHONY Skipping meta for .END: .SPECIAL
And of course, we generated the following:
bin/cat/Makefile.depend common/include/ppath/Makefile.depend common/include/prop/Makefile.depend common/include/rpc/Makefile.depend external/gpl3/gcc/lib/libgcc/libgcc/Makefile.depend external/gpl3/gcc/lib/libgcc/libgcc_s/Makefile.depend external/gpl3/gcc/lib/libstdc++-v3/include/Makefile.depend external/gpl3/gcc/lib/libstdc++-v3/include/bits/Makefile.depend external/gpl3/gcc/usr.bin/include/Makefile.depend include/Makefile.depend include/rpc/Makefile.depend lib/csu/Makefile.depend lib/libc/Makefile.depend lib/libpthread/Makefile.depend lib/libutil/Makefile.depend sys/arch/Makefile.depend sys/arch/amd64/include/Makefile.depend sys/arch/x86/include/Makefile.depend sys/sys/Makefile.depend
For example, bin/cat/Makefile.depend:
# Autogenerated - do NOT edit!
DIRDEPS = \
external/gpl3/gcc/lib/libgcc/libgcc \
external/gpl3/gcc/lib/libgcc/libgcc_s \
include \
lib/csu \
lib/libc \
lib/libpthread \
sys/sys \
.include <dirdeps.mk>
There is at at least on x86 specific entry sys/arch/x86/include which you do not see. It is suppressed by local.gendirdeps.mk and added automatically by local.dirdeps.mk so that the one Makefile.depend can work for many different architectures.
Back in 2015 it took only a couple of days effort to have most of userland building (400+ apps and all the libs they need) as well as building kernels. Of course I was able to re-use targets/* from FreeBSD.
For the 2023 exercise it took less than a day to get bin/cat building, and I didn't do more, as I'd achieved my purpose.
For the 2026 revisit I reduced the number of changes to the tree to a minimum.
| Author: | sjg@crufty.net |
|---|---|
| Revision: | $Id: netbsd-meta-mode.txt,v cf9539b1820d 2026-07-31 01:24:56Z sjg $ |