a Signing Server

Private keys for signing packages, binaries, anything; need to be kept private.

We want to be able to achieve that, while having a widely available and scalable signing service.

Signing Server

The solution is a signing server.

For the sake of efficiency we want to keep the traffic involved to a minimum. Clients send a digest (eg. SHA256) of something to be signed, and receive back a PEM encoded signature.

You might declaim but that would let me sign anything! and the answer is yes it would. So does any model that allows me access to the private key. The crucial difference is that when someone leaves the building, they cannot take the keys with them.

In some cases, it would make sense to require authenticated connections, but in most cases simple ACLs suffice.

Desirable features include:


The protocol is designed to be pure 7bit ascii and easily implemented.

The client sends a hexdigest with trailing newline as in:


it receives back a response which could be an error:

ERROR: not enough data

it is a bad idea to send only a SHA1 digest to a signer using SHA256.

The client can send more data in a slightly different format:

user=sjg path=/tmp/motd hash=0c1bc52c50016933679b0980ccff3680e5831162

If all goes well, we get back a signature:

#set: sig_ext=.esig
Any random header the server is told to send
A common setup might be the distinguished name of the signer

The #set: line cannot be mistaken for PEM data, and allows the server to inform the client of the preferred file extension. Thus if we were signing /tmp/the the signature would be saved to /tmp/the.esig.

Because it is entirely possible that the signing server is swamped by requests, the client will retry several times - with delays after the first retry:

ERROR: 54: Connection reset by peer. retrying...

if it runs out of retries it will exit with a bad status. In any reasonable implementation, there will be multiple servers per key and thus each retry may hit another server.

There is provision for a fallback IP address for the case that DNS lookups are failing, but the network is otherwise ok.

Server implementation

The server is implemented as a number of Python classes. A configuration file provides the key to use, and the name of the Signer class to use it:

SigExt = .esig
SigHeader= ECDSA p256 sha256
TrustAnchor= rootCA.pem
CRL= ${TrustAnchor:R}.crl

The variable references in the config can use many of the same modifiers documented in bmake(1), in the example above CRL=rootCA.crl.


The modules conf.py and vars.py provide support for the above configuration. They will be used by the client sign.py too if available.

As of 20220525, conf.py provides .include, .-include and simple conditionals - again following the syntax of bmake(1).

For example, the config for the Signer instance I use for my packages contains:

ListenPort= 12345
OpenPGPKeyid= BA54C8AF755A2A99
syslogFacility= local0.debug
syslogIdent= Crufty.NET
children= 2
.include "openpgp.inc"

As with bmake(1) the "openpgp.inc" will be looked for first in the directory where the above was found. It contains:

Signer= OpenSSLSigner
Hash= fakesha256
SetKnobs= OpenPGPKeyid=${OpenPGPKeyid}
SigningKey= ${config_dir:H}/keys/${OpenPGPKeyid}.sec.pem
TrustAnchor= ${config_dir:H}/keys/${OpenPGPKeyid}.pub.asc
Certs= ${TrustAnchor}

The client for the above is openpgp-sign.py and since it leverages sign.py it will automatically look for openpgp-sign.cf in the same directory. In that file I have:

.include "conf/crufty.cf"

which means I can simply run:

openpgp-sign.py sigs.tar.gz

to get sigs.tar.gz.asc.


This is a base class for a generic server. It needs to be extended to be useful.

After initial setup (whatever the sub-class wants - such as loading the private key) and listening on the configured port, it gets rid of any controlling tty and forks N children (one per cpu works well). These children accept(2) connections from clients and service them.

Since each transaction takes only a few ms, avoiding the overhead of continually forking, or starting threads keeps the overhead to a minium, at the same time as constraining the resource consumption.

This class covers the startup and shutdown of the server pool.


A sub-class of PoolServer, provides the basic signing protocol. It loads an appropriate Signer class based on the key and signature it has been configured to produce.

It can enforce network based access controls.


A mostly abstract base class for doing a signing operation. It should be extended to do something useful.


This is the most commonly used Signer. It interfaces to a C module which provides two generic methods load_key and sign_digest which in turn simply use OpenSSL (0.9.9 or later) APIs.

We add a directory to sys.path corresponding to a host-target derrived from os.uname() eg. netbsd5-i386 or freebsd10-amd64 which just happens to be were the compiled C module is placed on such platforms. The Python api does not provide access to uname -p so src/Makefile takes this into account.

Note: the latest version of config file allows for doing:

uname_p != uname -p

ie. run uname -p and assign its output to uname_p, but it isn't necessary.

This Signer can handle any key type that OpenSSL supports.

C module

The real work is done in signer.c which uses the OpenSSL APIs. It can be built into a standalone signer applictaion for testing.

There are two interfaces to Python.

Originally SWIG (http://www.swig.org/) was used. An interface spec signer.i is fed to swig that generates signer_wrap.c and signer.py. This works very well, but did not support Python3.

Using Cython (http://www.cython.org/) we can support Python 3.x and 2.x and the inteface ossl.pyx is easier to read. We use it to produce two modules ossl2.so and ossl3.so and OpenSSLSigner will use the one appropriate to the version of Python.

Pre-generated C code is provided.

Building requires bmake or you can translate cython.mk to work for gmake.


Takes advantage of Python's duck typing. It loads itself with the data provided, and provides digest() and hexdigest() methods that simply return that same data.

This allows the signing server to implement signature methods that do not normally lend themselves to an efficient client server model.

That is; the signature methods typically want a hash object passed to them rather than a hash value. For efficiency, we want the client to send us a hash. So what the signer usually ends up getting is in fact a hash of a hash. This is not always desirable - FakeHash is the fix.


This is an interface to external signing tools. It is not intended for high volume use.

For each message received it saves the input in a temp file, and creates a temp file for the signature, then runs the configured command line which can refer to {infile} and {outfile}. The content of the outfile is returned to caller.

Client implementation

It was a goal that the client could be implemented as a shell script, certainly the verification of signatures can be done using nothing more than the openssl binary.


Per the protocol above, for each file, it produces a hexdigest to send to the server (with trailing newline).

It receives back a PEM encoded signature, and (normally) a clue as to which file extension to use.

It can also ask the server for the certificate chain needed to verify the signatures being produced.

The client is designed to know as little as possible about the signing operation. It really only needs to be told the url to use and whether to send a SHA1 or SHA256 digest.

The hash sent and the processing done by the server, need to produce a signature acceptible to whatever is expected to verify it.

The client will look at the name it was invoked with and see if a configuration file exists with that name. That is sign.py will look for sign.cf and openpgp-sign.py will look for openpgp-sign.cf.

Additional configuration files can be passed via the -c option. This makes it easy to configure hash choice, and server url.

Note: the client will attempt to import the conf modules to get loadConfig but if that fails a simplified implementation which does not support variable references will be used. The fallback reader will ignore any of the fancy features it cannot support:

Warning: ignoring: arch != uname -p
Warning: ignoring: .include "openpgp.inc"

It is thus possible to construct the configuration file for a Signer such that at least some portion of it can be used by the client.

data requests

The client can send the server a number of requests for data other than a signature:


Return the certificate data needed to verify a signature. This is the value of the servers Certs setting.
Return any CRL data the server has.
Return the TrustAnchor associated with the signatures. This is typically the Root CA certificate.


Extends the SignClient class in sign.py to generate OpenPGP compatable signatures.

The corresponding server instance must be configured to report the keyid (and use FakeHash):

Signer= OpenSSLSigner
Hash= fakesha256

The keyid ends up in the signature and allows a verifier to find the correct public key.

Note: OpenSSLSigner knows nothing about OpenPGP, it is just generating an RSA signature. All the work of putting the signature into OpenPGP packets is done by the client.

Use convert-gpg-seckey-to-pem to convert a private key generated by gpg or another OpenPGP client to PEM format so it can be loaded by OpenSSLSigner.


Extends the SignClient class in sign.py to generate Linux IMA signatures.


As the name might suggest, this client is for controlling the signing server. It is used by the rc_sigs script for stopping, and checking status of servers.

It prefixes its requests with signctl:, to ensure they are not mistaken for signing requests. The only one the server actually cares about is shutdown.

When a server instance receives the shutdown request, it sets a flag and exits with with a status that ensures the main process knows.

The main server process will signal any other children, but for simplicity, signctl will repeat the shutdown request until it gets a connection failure. This will shutdown all the server instances as rapidly as possible.

SSH proxy

If only authenticated singing requests are desired. The signing server can listen to localhost only, and an SSH sub-agent SignProxy used.

For example in sshd_config:

Subsystem sign-rsa /opt/sigs/SignProxy.py -c /opt/sigs/conf/rsa2k.cf -L local0.debug

where /opt/sigs/conf/rsa2k.cf is the config for an instance of the signing server, that includes the port it listens on.

You can also run SignProxy.py via a symlink and it will pickup a config based on the name it was invoked as so:

Subsystem sign-rsa /opt/sigs/sign-rsa-proxy

Will read its config from /opt/sigs/sign-rsa-proxy.cf

Apart from the extra overhead that SSH introduces, this model reduces the load balancing effect of multiple servers, since the SignAgent keeps a connection to server open much longer than the normal sign client.


Run as an SSH sub-agent, this class opens a connection to the signing server when it has work for it, and sends the responses back to its client.

After a period of idleness it disconnects from the server. After an extended period of idleness, it shuts down. The next request from the client (SignAgent) will cause it to be restarted.


Run on client host by the user, and with access to SSH_AUTH_SOCK. It creates a private unix domain socket, which it listens to for singing requests. It indicates the socket it is listening on, which should then be used as the url given to sign.

When it gets a request it (if needed) opens a connection to the specified url - which leads to SignProxy. Sends and gets the response which it passes back to the caller.

There is only one thread, so if multiple sign clients attempt requests they will be serviced sequentially.

Signing Keys

Signing keys should be used for a limited period. Some signing methods leak information about the key used.

One way to achieve this, is to use new keys at regular intervals; annually, quarterly or even monthly depending on the load.

In most cases an annual key turnover would suffice. This makes it easy for a pool of servers to support multiple keys identified by the port the server is listening on.

By arranging for the signing server for a given key to listen on a port which maps to its key (eg. last 2 digits of the port map to year or month depending on the turnover cycle), it is simple to ensure that the same key is used for an entire build, even when spaning the key turnover period. This leaves at least three digits of the port number to indicate the type of key or signature method.

We can use the common telephone keypad mapping of letters to numbers to lend some determinism to the port numbers. For example:

D (Development):        3
E (ECDSA):              3
P (Production):         7
R (RSA):                7

End with 2 digits to indicate the era of the key (be it month, year etc) and its simple enough to coordinate port numbers.



contains everything needed to run the OpenSSLSigner based SignServerPool, assuming you have bmake available.

It is a pointer to the latest sigs-YYMMDD.tar.gz archive.


contains pre-built _signer.so for various systems ( freebsd10-amd64 freebsd7-i386 linux3-x86_64 netbsd5-i386 netbsd6-amd64 ) this is for the benfit of those that want to quickly try it out, you should really build it yourself.


After unpacking sigs, you can unpack sigs-signer-so.tar.gz in the sigs/ directory, or build _signer.so (see src/Makefile for instuctions).

There is an example.cf:

# this signer can handle any key type OpenSSL can
# use hash from client directly rather than hash it again
# the signing key
# cert chain to verify above
# optional header
SigHeader= /C=US/ST=California/O=Crufty.NET/CN=Crufty2013/emailAddress=root@crufty.net
# hint to client the extension to use
SigExt= .sig
# a desired PEM tag
# the port to listen on - rc_sigs feeds this same config to signctl.py
# one child per cpu works well

The start/stop script rc_sigs will look for config files in ./conf/ else . and default to starting an instance of the server for each config it sees:

$ ./rc_sigs start
SignServerPool.py conf/rsa2k.cf
$ ./rc_sigs status
conf/rsa2k.cf: running
$ ./rc_sigs stop
shutdown SignServerPool.py conf/rsa2k.cf
$ ./rc_sigs check
restarting SignServerPool.py conf/rsa2k.cf
$ ./rc_sigs restart
shutdown SignServerPool.py conf/rsa2k.cf
restarting SignServerPool.py conf/rsa2k.cf

The check operation is handy to run from cron(8), since it will restart the servers if for any reason they are not running.

Author:sjg@crufty.net /* imagine something very witty here */
Revision:$Id: signing-server.txt,v 1.27 2023/11/02 03:44:08 sjg Exp $