Sandbox tools

As far as I know, the concept of a sandbox and the tools mksb and mk come from the OSF Development Environment (ODE).

The scripts described here implement the idea but not necessarily in a compatible manner.

Introduction

A sandbox is simply a software source tree with a special file .sandbox-env which serves both to mark the location of the sandbox (the variable SB is set to the directory where .sandbox-env was found) and provide a set of environment variables for it.

Below we describe some of the tools that work within this ecosystem. I've been using this setup for over 15 years and find it very useful.

mk

This is the most commonly used tool. In a nutshell its job is to condition the environment and run ${REAL_MAKE:-make}.

In this implementation it is sourced by both mksb and workon to handle environment setup in a consistent manner, each sets MYNAME to its canonical name.

The following files are read (if they exist) in this order:

$Mydir/sb-env.rc
$Mydir/$MYNAME.rc
$HOME/.sandboxrc

Any of the above can contribute hook functions to control the behavior. For example:

. $Mydir/os.sh
. $Mydir/test_opt.h

# set test_L to -L or -h
test_opt L -h

add_hooks mk_target_machine_hooks mk_target_machine_xtra

mk_target_machine_xtra() {
    case "$MACHINE" in
    host32) OBJTOP=$OBJROOT/$HOST_TARGET32;;
    host)  OBJTOP=$OBJROOT/$HOST_TARGET;;
    esac
    :
}

add_hooks ${MYNAME}_init_hooks sb_project

# canonicalize SB_PROJECT
sb_project() {
    _p=${1:-$SB_PROJECT}

    case "$_p" in
    netbsd*|NetBSD*)
        SB_PROJECT=NetBSD
        # other setup....
        ;;
    esac
    :
}

results in each app calling sb_project early and for mk calling mk_target_machine_xtra from mk_target_machine.

The last : just ensures a happy return code (return 0 would work too), since by default run_hooks stops if any hook returns non-zero.

After the global rc files have been read we run:

run_hooks ${MYNAME}_begin_hooks

Next mk searches upwards from the current directory for the file .sandbox-env. If found, its location defines SB. If not found mk throws an error.

Once SB is known, it reads (if they exist):

$SB/../.sandboxrc
$SB/.sandboxrc

We then run:

run_hooks ${MYNAME}_init_hooks

and then read:

$SB/.sandbox-env

After all the rc files have been read, we run more hook functions:

run_hooks ${MYNAME}_setup_hooks
run_hooks ${MYNAME}_finish_hooks

Some apps run others as well. For example mk runs mk_cmd_hooks before mk_finish_hooks

Finally mk runs the command it has been told to (normally ${REAL_MAKE:-make}).

options

With the following exceptions, anything on the mk command line is expected to be consumed by $REAL_MAKE.

If any of these options are used, they must appear before anything else:

--doc
Calls the __doc function. The default implementation in sb-funcs.sh simply displays the documentation embedded in the script.

--help

Calls the __help function. The default implementation runs any ${MYNAME}_help_hooks, if none found it calls ${MYNAME}_help if it exists and if all else fails calls __doc.

If an argument is provided it is treated as a topic and refines the functions we look for.

See mk --doc for details.

--machine [MAKELEVEL,]MACHINE[,...]

Calls mk_target_machine to set MACHINE to the value provided.

If the tupple provided starts with a numeric it is used to set MAKELEVEL and the remainer used for MACHINE, and if it is also a tupple we set TARGET_SPEC to the full tupple while MACHINE is set to just the first element.

It also sets REQUESTED_MACHINE so that the build can tell the difference between this case and a default value of MACHINE.

The same result happens if mk is invoked as mk-$MACHINE.

mksb

The ev (.sandbox-env) file can be created manually of course. Or it can be created by a tool; mksb which can also handle initial check-out of the source tree.

Using mksb we can hide details like the SCM used by a given project, or the URL of the repository, etc. This allows those to be changed over time, without the need for users (or robots) to change their operations. When you have 1000's of developers and dozens of projects that's very handy.

Note that mksb itself is just an engine, it is told the SB_PROJECT that a sandbox is being created for and a set of site specific hook functions do the actual work.

Apart from running lots of hooks, mksb provides a number of functions to help populate .sandbox-env:

expShellVars VAR ...

Simply adds VAR=value; export VAR

expShellDefVars VAR ...

Like expShellVars but adds VAR=${VAR:-value}; export VAR

expShellLiteralVars VAR ...

Like expShellVars but uses single quotes. This is useful for cases like: MAKEOBJDIR='${.CURDIR:S,${SRCTOP},${OBJTOP},}' which is to be interpreted by 'bmake' rather than the shell.

The hooks are run in several stages so that earlier ones can influence later ones:

run_hooks mksb_begin_hooks
run_hooks mksb_pre_create_hooks

now we actually create $SB, then:

run_hooks mksb_init_hooks
run_hooks mksb_setup_hooks

and initialize .sandbox-env, then:

run_hooks mksb_env_init_hooks
run_hooks mksb_env_setup_hooks
run_hooks mksb_env_hooks
run_hooks mksb_env_finish_hooks

at this point .sandbox-env is generally complete and read by mksb, then we run:

run_hooks mksb_checkout_init_hooks
run_hooks mksb_checkout_setup_hooks
run_hooks mksb_checkout_hooks
run_hooks mksb_checkout_finish_hooks
run_hooks mksb_finish_hooks

options

Because mksb is a much more complicated script than say mk it uses eval_args.sh to process its command line.

This means that the --help option can be given an arg to refine the information returned.

--help

Behaves as described for mk.

--help=topic

Will look for help on $topic, either hooks: mksb_help_${topic}_hooks, help_${topic}_hooks or a function: mksb_help_${topic}, help_${topic}

--expShellVars=VAR --expShellDefVars=VAR --expShellLiteralVars=VAR

As described above for expShellVars etc. Assumes VAR=value already appeared on the command line.

sb-opt.sh

This can be included by mksb.rc and provides handling for options like --sb-opt-KNOB={yes,no}.

It creates a file sbopt-KNOB.inc and arranges for $SB/.sandboxrc to include it.

If the arg is yes then sbopt-KNOB.inc will contain export WITH_KNOB=1 and export WITHOUT_KNOB=1 for no.

This works in conjunction with our options.mk or FreeBSD's bsd.mkopt.mk to set MK_KNOB={yes,no}

sb-opt.sh can be used at any time to update sbopt-*.inc.

workon

This is also an optional tool, though I find it very handy.

$ workon NetBSD/current

will find the sandbox called NetBSD/current chdir into it, set the environment up (as per mk) and run my shell. I can then make use of variables like $SB and others setup by .sandbox-env.

I have mksb export the sandbox name in a variable that will be included in my shell prompt:

tty_tag() {
        # I use this in the xterm title bar (among other things)
        TTY_TAG="[\$SB_NAME]"
        expShellVars TTY_TAG
}

add_hooks mksb_env_finish_hooks tty_tag

You can also use it as a one command thing:

$ workon FreeBSD/current command

will do as above, but instead of running my shell, it will run command and exit when it does.

Sandbox environment

There is only one variable that every sandbox has; SB which is set by tools like mk when they find the magic .sandbox-env marker. Everything else is totally up to you. Ok we also set SB_NAME to the basename of SB.

For projects that use bmake, I typically have:

export OBJROOT=$SB/obj/
export SRCTOP=$SB/src
export OBJTOP='${OBJROOT}${MACHINE}'
export MAKEOBJDIR='${.CURDIR:S,${SRCTOP},${OBJTOP},}'
export MAKESYSPATH=$SB/src/share/mk

Note that OBJTOP and MAKEOBJDIR are single quoted so that they are seen like that by bmake.

This allows us to ensure that each build uses the correct share/mk/*.mk files and we get nice neat object dirs outside of the src tree without the ugliness that MAKEOBJDIRPREFIX results in.

Such a setup even works well when shared via NFS. Though in that case we might make SB_OBJROOT point to a local filesystem for better performance.

Download

A simple but useful implementation of the above can be found at

http://www.crufty.net/ftp/pub/sjg/sb-tools-min.tar.gz


Author:sjg@crufty.net
Revision:$Id: sb-tools.txt,v c9eb56451528 2017-08-25 05:24:27Z sjg $
Copyright:Crufty.NET