DIRDEPS for gmake

This is a somewhat crude approximation of dirdeps for gmake. We cannot handle all the bells and whistles that bmake can but we can still manage the basics to handle orchestrating a large build correctly.

DIRDEPS in brief

The concept is quite simple. We locate a Makefile.depend* (actually GNUmakefile.depend* for gmake) in each directory to be made.

Each such makefile sets DIRDEPS to a list of paths relative to SRCTOP that need to be build before the current directory. And then includes dirdeps.gmk to do the work.

This happens recursively and in the process we build a tree dependency graph which we hand over to make to build in the optimal order, in parallel with no risk of race conditions - provided DIRDEPS are accurate.

In the bmake version a Makefile.depend file looks like:

# Autogenerated - do NOT edit!

DIRDEPS = \
        lib/sjg \


.include <dirdeps.mk>

for gmake it will be a bit more crude. For a start we'll assume that all GNUmakefile.depend* are manually maintained.

There is a vpath PATTERN DIRLIST directive, than sounds like we could have:

vpath %.gmk ${GMAKESYSPATH}

so that:

include dirdeps.gmk

should just work, but it doesn't. Instead we need:

include ${GMKSYSDIR}/dirdeps.gmk

where GMKSYSDIR is either set in environment or by sys.gmk which we can cause to be included before each makefile by setting something like MAKEFILES=${SRCTOP}/gmk/sys.gmk.

Since I tend to use a wrapper around make for conditioning the environment, that's simple.

We could also run gmake with -I ${GMKSYSDIR} which is easy enough with the mk wrapper, but that's a bit ugly for anyone not using it.

Challenges

Since gmake lacks many of the string manipulation features of bmake (many of which I added over the last 20 years) we will have to rely on more manual coding.

No TARGET_SPEC_VARS

For example if TARGET_SPEC_VARS needs to be more than just the default MACHINE, then things like TARGET_SPEC and DEP_TARGET_SPEC need to be set manually in say local.sys.gmk since we cannot auto generate them from TARGET_SPEC_VARS.

Further, since we cannot expect sys.gmk to be able to decompose a TARGET_SPEC tupple into useful variables.... (gmake seems incapable of dealing with , in variables), we have to ensure that MACHINE at least, is split out and passed by dirdeps.gmk.

Ok, I guess one could use $(shell echo "${var}" | tr ',' ' ') but is it too much to expect something like $(subst \,, ,${var}) to work?

No .for loop

At first glance the lack of a .for loop appeared daunting. But to cut a long story short; using nested macros can get the job done.

Having constructed the fully qualified list of dirdeps for the current directory eg.:

${SRCTOP}/${RELDIR}.${TARGET_SPEC}

We can turn that into a list of qualified depend files to try and include:

${SRCTOP}/${RELDIR}/${DEPENDFILE_PREFIX}.${TARGET_SPEC}

We use the following to do the actual including of the next depend file while setting DEP_RELDIR and DEP_TARGET_SPEC appropriately. We want a qualified depend file if it exists, otherwise an unqualified one should do:

define include-depend
        $(eval DEP_TARGET_SPEC=$(subst .,,$(suffix $1)))
        $(eval DEP_RELDIR=$(patsubst %/,%,$(subst ${SRCTOP}/,,$(dir $1))))
        $(if ${DEBUG_DIRDEPS},$(info Looking for $1 DEP_TARGET_SPEC=${DEP_TARGET_SPEC} DEP_RELDIR=${DEP_RELDIR}),)
        $(eval -include $(shell test -s $1 && echo $1 || echo $(basename $1)))
endef

invoked via:

$(eval $(foreach dep,${_more_depends},$(call include-depend,${dep})))

That's actually not too shabby.

Target tracking

Gmake does not appear to provide a means of knowing if a target has been defined, so we need to do that ourselves, otherwise we can get lots of warnings about duplicate targets. The following does the trick:

# we have to do target tracking ourselves to avoid complaints
define if-new-target
        $(eval t=_${1}_target)
        $(if ${$t},,$1)
        $(eval $t=1)
endef

invoked thus:

# add dependencies to the graph
${_this_dirdep}: ${_dirdeps}
# now which ones have not been made targets yet?
_newdeps := $(sort $(foreach dep,${_dirdeps},$(call if-new-target,${dep})))
ifneq "${_newdeps}" ""
${_newdeps}:
        $(call build-dirdep)
endif

Limited pattern matching

We want to be able to:

mk -j8 -f dirdeps.gmk some/dir.host other/dir.i386

which should behave exactly as if we had a GNUmakefile.depend file containing:

DIRDEPS= \
        some/dir.host \
        other/dir.i386 \

include ${GMKSYSDIR}/dirdeps.gmk

Unfortunately gmake's filter does not seem able to match a pattern like %/% so we have to resort to a shell script:

# (shell is needed because filter cannot do %/%)
_dirdeps := $(sort $(shell for f in ${MAKECMDGOALS}; do echo "$$f"; done | grep /))

and we had to use grep since case "$$f" in */*) echo $$f;; esac caused a parsing error - the ) closed the $(shell.

Inefficient, but it's a one-off and it works.

Automated obj dirs

Another cool feature missing in gmake is the ability to control the directory where files will be generated. But we can compensate by having dirdeps.gmk create the necessary objdir and telling gmake to chdir there and with the desired src dir in VPATH.

This of course relies on our previously mentioned sys.gmk to deal with correctly setting _CURDIR, _OBJDIR and RELDIR to the desired values. The lack of elif or elifeq etc make that messier than is desirable but it works - see example below.

Example

The example below (available in http://www.crufty.net/ftp/pub/sjg/gmk-tests.tar.gz ) consists of:

tests/GNUmakefile.inc
tests/lib/GNUmakefile.inc
tests/lib/a/GNUmakefile
tests/lib/a/GNUmakefile.depend
tests/lib/a/GNUmakefile.depend.host
tests/lib/b/GNUmakefile
tests/lib/b/GNUmakefile.depend
tests/lib/b/GNUmakefile.depend.host
tests/lib/d/GNUmakefile
tests/lib/d/GNUmakefile.depend
tests/lib/e/GNUmakefile
tests/lib/e/GNUmakefile.depend
tests/prog/GNUmakefile
tests/prog/GNUmakefile.depend
tests/tools/GNUmakefile.inc
tests/tools/tool/GNUmakefile
tests/tools/tool/GNUmakefile.depend.host

of which the following all list tests/tools/tool.host as a dependency:

tests/lib/a/GNUmakefile.depend
tests/lib/e/GNUmakefile.depend
tests/prog/GNUmakefile.depend

and tests/lib/a depends on tests/lib/b and tests/lib/e, while tests/lib/b depends on tests/lib/d and tests/lib/e.

Since tests/tools/tool/GNUmakefile.depend.host is:

DIRDEPS = \
        tests/lib/a \

include ${GMKSYSDIR}/dirdeps.gmk

it needs tests/lib/a built for host. note that tests/lib/a/GNUmakefile.depend.host is just:

DIRDEPS = \
        tests/lib/b \

include ${GMKSYSDIR}/dirdeps.gmk

(so tests/lib/b also needs to build for host) while tests/lib/a/GNUmakefile.depend is:

DIRDEPS = \
        tests/tools/tool.host \
        tests/lib/b \
        tests/lib/e

include ${GMKSYSDIR}/dirdeps.gmk

Also note that there is no GNUmakefile.depend.host in tests/lib/b so we will re-use its GNUmakefile.depend.

Finally note that all of the GNUmakefile end up including tests/GNUmakefile.inc which has just:

all:

# sys.gmk already set this
_CURDIR ?= ${CURDIR}

PROG ?= $(notdir ${_CURDIR})
THING ?= ${PROG}

ifneq "${MAKELEVEL}" "0"
all: realbuild

realbuild:
        @echo Building ${CURDIR}/${THING}
        @echo Finished making ${RELDIR} for ${TARGET_SPEC}
endif

so we don't actually build anything, we just claim to.

As we can see below, everything get's built in the optimal order and nothing is visited more than once:

$ mk-i386 -C tests/prog
gmake: Entering directory '/h/sjg/work/gmk/tests/prog'
Checking /h/sjg/work/gmk/tests/lib/b for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a
Finished making tests/lib/b for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'
Checking /h/sjg/work/gmk/tests/lib/a for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a/liba.a
Finished making tests/lib/a for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a'
Checking /h/sjg/work/gmk/tests/tools/tool for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool/tool
Finished making tests/tools/tool for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool'
Checking /h/sjg/work/gmk/tests/lib/d for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/d'
Building /h/sjg/work/gmk/obj/i386/tests/lib/d/libd.a
Finished making tests/lib/d for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/d'
Checking /h/sjg/work/gmk/tests/lib/e for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/e'
Building /h/sjg/work/gmk/obj/i386/tests/lib/e/libe.a
Finished making tests/lib/e for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/e'
Checking /h/sjg/work/gmk/tests/lib/b for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/b'
Building /h/sjg/work/gmk/obj/i386/tests/lib/b/libb.a
Finished making tests/lib/b for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/b'
Checking /h/sjg/work/gmk/tests/lib/a for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/a'
Building /h/sjg/work/gmk/obj/i386/tests/lib/a/liba.a
Finished making tests/lib/a for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/a'
Checking /h/sjg/work/gmk/tests/prog for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/prog'
Building /h/sjg/work/gmk/obj/i386/tests/prog/prog
Finished making tests/prog for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/prog'
gmake: Leaving directory '/h/sjg/work/gmk/tests/prog'
$

and it all works just as well in parallel (see below).

Also if we change the origin, the graph changes accordingly - only that which is needed is built:

$ mk-amd64 -j8 -C tests/lib/b
gmake: Entering directory '/h/sjg/work/gmk/tests/lib/b'
Checking /h/sjg/work/gmk/tests/lib/d for amd64 ...
Checking /h/sjg/work/gmk/tests/lib/b for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d'
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'
Building /h/sjg/work/gmk/obj/amd64/tests/lib/d/libd.a
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a
Finished making tests/lib/d for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d'
Finished making tests/lib/b for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'
Checking /h/sjg/work/gmk/tests/lib/a for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a/liba.a
Finished making tests/lib/a for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a'
Checking /h/sjg/work/gmk/tests/tools/tool for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool/tool
Finished making tests/tools/tool for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool'
Checking /h/sjg/work/gmk/tests/lib/e for amd64 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e'
Building /h/sjg/work/gmk/obj/amd64/tests/lib/e/libe.a
Finished making tests/lib/e for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e'
Checking /h/sjg/work/gmk/tests/lib/b for amd64 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b'
Building /h/sjg/work/gmk/obj/amd64/tests/lib/b/libb.a
Finished making tests/lib/b for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b'
gmake: Leaving directory '/h/sjg/work/gmk/tests/lib/b'
$

Note too that when building for the pseudo machine host we use an objdir named for HOST_TARGET (set by mk) which helps ensure that a tree shared via NFS with lots of different machines will just work.

The mk tool is reading .sandbox-env to condition the environment and identify the top of the tree (sandbox):

export SRCTOP=$SB
export GMKSYSDIR=$SB/gmk
export MAKEFILES=$GMKSYSDIR/sys.gmk
REAL_MAKE=gmake
SB_PATH=$PATH

The MAKEFILES=$GMKSYSDIR/sys.gmk is key.

You can run the tests as:

$ mk -j8 -f gmk/dirdeps.gmk tests/prog.i386 tests/prog.amd64

and it will build tests/prog for both i386 and amd64 but all the host bits will still be built only once.

If we add DEBUG_DIRDEPS=1 we can get to see all the plumbing in action:

$ mk DEBUG_DIRDEPS=1 -j8 -f gmk/dirdeps.gmk tests/prog.i386 tests/prog.amd64
INCLUDED_FROM=/h/sjg/work/gmk/gmk/local.sys.gmk
dirdeps:  /h/sjg/work/gmk/tests/prog.amd64  /h/sjg/work/gmk/tests/prog.i386
_more_depends= /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.amd64  /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.i386
Looking for /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.amd64 DEP_TARGET_SPEC=amd64 DEP_RELDIR=tests/prog
-including /h/sjg/work/gmk/tests/prog/GNUmakefile.depend.inc ...
dirdeps: DEP_RELDIR=tests/prog DEP_TARGET_SPEC=amd64 DIRDEPS=tests/tools/tool.host tests/lib/a tests/lib/e
/h/sjg/work/gmk/tests/prog.amd64:  /h/sjg/work/gmk/tests/tools/tool.host  /h/sjg/work/gmk/tests/lib/a.amd64  /h/sjg/work/gmk/tests/lib/e.amd64
_more_depends= /h/sjg/work/gmk/tests/tools/tool/GNUmakefile.depend.host  /h/sjg/work/gmk/tests/lib/a/GNUmakefile.depend.amd64  /h/sjg/work/gmk/tests/lib/e/GNUmakefile.depend.amd64
Looking for /h/sjg/work/gmk/tests/tools/tool/GNUmakefile.depend.host DEP_TARGET_SPEC=host DEP_RELDIR=tests/tools/tool
-including /h/sjg/work/gmk/tests/tools/tool/GNUmakefile.depend.inc ...
dirdeps: DEP_RELDIR=tests/tools/tool DEP_TARGET_SPEC=host DIRDEPS=tests/lib/a
/h/sjg/work/gmk/tests/tools/tool.host:  /h/sjg/work/gmk/tests/lib/a.host
...

and so on.

It is easier to read without that noise though:

$ mk -j8 -f gmk/dirdeps.gmk tests/prog.i386 tests/prog.amd64
Checking /h/sjg/work/gmk/tests/lib/b for host ...
Checking /h/sjg/work/gmk/tests/lib/d for amd64 ...
Checking /h/sjg/work/gmk/tests/lib/d for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/d'
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d'
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'

Note that since tests/lib/d and tests/lib/b (for host) do not have any dependencies (empty DIRDEPS) they get built first.

Building /h/sjg/work/gmk/obj/i386/tests/lib/d/libd.a
Building /h/sjg/work/gmk/obj/amd64/tests/lib/d/libd.a
Finished making tests/lib/d for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/d'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a
Finished making tests/lib/d for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/d'
Finished making tests/lib/b for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'
Checking /h/sjg/work/gmk/tests/lib/a for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a/liba.a
Finished making tests/lib/a for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/a'
Checking /h/sjg/work/gmk/tests/tools/tool for host ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool'
Building /h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool/tool
Finished making tests/tools/tool for host
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/freebsd11-amd64/tests/tools/tool'
Checking /h/sjg/work/gmk/tests/lib/e for amd64 ...
Checking /h/sjg/work/gmk/tests/lib/e for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e'
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/e'
Building /h/sjg/work/gmk/obj/amd64/tests/lib/e/libe.a
Building /h/sjg/work/gmk/obj/i386/tests/lib/e/libe.a
Finished making tests/lib/e for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/e'
Finished making tests/lib/e for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/e'
Checking /h/sjg/work/gmk/tests/lib/b for amd64 ...
Checking /h/sjg/work/gmk/tests/lib/b for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b'
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/b'
Building /h/sjg/work/gmk/obj/amd64/tests/lib/b/libb.a
Building /h/sjg/work/gmk/obj/i386/tests/lib/b/libb.a
Finished making tests/lib/b for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/b'
Finished making tests/lib/b for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/b'
Checking /h/sjg/work/gmk/tests/lib/a for amd64 ...
Checking /h/sjg/work/gmk/tests/lib/a for i386 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/lib/a'
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/lib/a'
Building /h/sjg/work/gmk/obj/amd64/tests/lib/a/liba.a
Building /h/sjg/work/gmk/obj/i386/tests/lib/a/liba.a
Finished making tests/lib/a for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/lib/a'
Finished making tests/lib/a for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/lib/a'
Checking /h/sjg/work/gmk/tests/prog for i386 ...
Checking /h/sjg/work/gmk/tests/prog for amd64 ...
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/i386/tests/prog'
gmake[1]: Entering directory '/h/sjg/work/gmk/obj/amd64/tests/prog'
Building /h/sjg/work/gmk/obj/i386/tests/prog/prog
Building /h/sjg/work/gmk/obj/amd64/tests/prog/prog
Finished making tests/prog for i386
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/i386/tests/prog'
Finished making tests/prog for amd64
gmake[1]: Leaving directory '/h/sjg/work/gmk/obj/amd64/tests/prog'
gmake: Nothing to be done for 'tests/prog.amd64'.
$

dirdeps-targets.gmk

This makefile handles the logic normally found in a top-level makefile to decide what to build.

For any given target dirdeps-targets.gmk looks for a directory of that name under ${SRCTOP}/targets/ and anything else in DIRDEPS_TARGETS_DIRS, that contains a GNUmakefile.depend* file.

Building on our example above:

$ cat targets/tests/GNUmakefile.depend
DIRDEPS= tests/prog

include ${GMKSYSDIR}/dirdeps.gmk

is all we need to:

$ mk -f gmk/dirdeps-targets.gmk tests
Checking /homes/sjg/work/gmk/tests/lib/b for host ...
gmake[1]: Entering directory '/homes/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b'
Building /homes/sjg/work/gmk/obj/freebsd11-amd64/tests/lib/b/libb.a
Finished making tests/lib/b for host
..
..
gmake[1]: Entering directory '/homes/sjg/work/gmk/obj/amd64/tests/prog'
Building /homes/sjg/work/gmk/obj/amd64/tests/prog/prog
Finished making tests/prog for amd64
gmake[1]: Leaving directory '/homes/sjg/work/gmk/obj/amd64/tests/prog'
$

Conclusion

This really just a proof of concept.

I haven't used gmake seriously for over 20 years, so it is quite possible others could improve on my implementation.

The real dirdeps (with bmake) works incredibly well. This implementation provides only the basic functionality but is far more capable that I expected would be possible.

A key missing feature is the ability to filter DIRDEPS in weird and wonderful ways, and this implementation cannot automatically adapt to different requirements by simply teaking TARGET_SPEC_VARS.

Filtering I address by including local.dirdeps-filter.gmk at the appropriate point. The implementation is left as an exercise for the suitably motivated reader ;-)

If interested; you can download this version from http://www.crufty.net/ftp/pub/sjg/gmk.tar.gz

Consider dirdeps.gmk and sys.gmk as the key bits, the rest are really just examples so I could exercise dirdeps.gmk

Improvements are always welcome.


Author:sjg@crufty.net
Revision:$Id: gmake-dirdeps.txt,v b6d0bf4dd35e 2020-07-28 23:36:09Z sjg $