a Signing Server

Private keys for signing packages, binaries, anything; need to be kept private.

We want to be able to achieve that, while having a widely available and scalable signing service.

Signing Server

The solution is a signing server.

For the sake of efficiency we want to keep the traffic involved to a minimum. Clients send a digest (eg. SHA256) of something to be signed, and receive back a PEM encoded signature.

You might declaim but that would let me sign anything! and the answer is yes it would. So does any model that allows me access to the private key. The crucial difference is that when someone leaves the building, they cannot take the keys with them.

In some cases, it would make sense to require authenticated connections, but in most cases simple ACLs should suffice.

Desirable features include:

Protocol

The protocol is designed to be pure 7bit ascii and easily implemented.

The client sends a hexdigest with trailing newline as in:

0c1bc52c50016933679b0980ccff3680e5831162

it receives back a response which could be an error:

ERROR: not enough data

it is a bad idea to send only a SHA1 digest to a signer using SHA256.

The client can send more data in a slightly different format:

user=sjg path=/etc/motd hash=0c1bc52c50016933679b0980ccff3680e5831162

If all goes well we get back a signature:

#set: sig_ext=.esig
Any random header the server is told to send
A common setup might be the distinguished name of the signer
-----BEGIN EC SIGNATURE-----
MEQCIE+0i0ORmYnxfEsfbeKo2IjBS68FmPEm5us3CH8NpxggAiA1CsZaL8J3+FZr
3Z6HfqfA0YEXYvFjngqdk9CTbVNNSQ==
-----END EC SIGNATURE-----

The #set: line cannot be mistaken for PEM data, and allows the server to inform the client of the preferred file extension. Thus if we were signing /tmp/the the signature would be saved to /tmp/the.esig.

Because it is entirely possible that the signing server is swamped by requests, the client will retry several times - with delays after the first retry:

ERROR: 54: Connection reset by peer. retrying...

if it runs out of retries it will exit with a bad status. In any reasonable implementation, there will be multiple servers per key and thus each retry may hit another server.

Server implementation

The server is implemented as a number of Python classes. A configuration file provides the key to use, and the name of the Signer class to use it:

Signer=OpenSSLSigner
SigningKey=ec2012_key.pem
Certs=ec2012_certs.pem
ListenPort=32012
PEMTag= EC SIGNATURE
SigExt = .esig
SigHeader= ECDSA p256 sha256
TrustAnchor= rootCA.pem
CRL= ${TrustAnchor:R}.crl
logFacility=local0.info
Hash=sha256
allow_ctl= 127.0.0.1
allow_nets= 192.168.0.0/16 172.0.0.0/8

The variable references in the config can use many of the same modifiers documented in bmake(1), in the example above CRL=rootCA.crl.

PoolServer

This is a base class for the server. After initial setup such as loading the private key and listening on the configured port, we get rid of any controlling tty and fork N children (one per cpu works well). These children accept(2) connections from clients and service them.

Since each transaction takes only a few ms, avoiding the overhead of continually forking, or starting threads keeps the overhead to a minium, at the same time as constraining the resource consumption.

This class covers the startup and shutdown of the server pool.

SignServerPool

A subclass of PoolServer, provides the basic signing protocol. It loads an appropriate Signer class based on the key and signature it has been configured to produce.

It can enforce network based access controls.

Signer

A mostly abstract base class for doing a signing operation. It should be extended to do something useful.

OpenSSLSigner

This class interfaces to a C module which provides two generic methods load_key and sign_digest which in turn simply use OpenSSL (0.9.9 or later) APIs.

We add a directory to sys.path corresponding to a host-target derrived from os.uname() eg. netbsd5-i386 or freebsd10-amd64 which just happens to be were the compiled C module is placed on such platforms. The Python api does not provide access to uname -p so src/Makefile takes this into account.

This Signer can thus handle any key type that OpenSSL supports.

C module

The real work is done in signer.c which uses the OpenSSL APIs. It can be built into a standalone signer applictaion for testing.

There are two interfaces to Python.

Originally SWIG (http://www.swig.org/) was used. An interface spec signer.i is fed to swig that generates signer_wrap.c and signer.py. This works very well, but does not support Python3.

Using Cython (http://www.cython.org/) we can support Python 3.x and 2.x and the inteface ossl.pyx is easier to read. We use it to produce two modules ossl2.so and ossl3.so and OpenSSLSigner will use the one appropriate to the version of Python.

Pre-generated C code is provided.

Building requires bmake or you can translate cython.mk to work for gmake.

FakeHash

Takes advantage of Python's duck typing. It loads itself with the data provided, and provides digest() and hexdigest() methods that simply return that same data.

This allows the signing server to implement signature methods that do not normally lend themselves to an efficient client server model.

That is; the signature methods typically want a hash object passed to them rather than a hash value. For efficiency, we want the client to send us a hash. So what the signer usually ends up getting is in fact a hash of a hash. This is not always desirable - FakeHash is the fix.

ExternalSigner

This is an interface to external signing tools. It is not intended for high volume use.

For each message received it saves the input in a temp file, and creates a temp file for the signature, then runs the configured command line which can refer to {infile} and {outfile}. The content of the outfile is returned to caller.

Client implementation

It was a goal that the client could be implemented as a shell script, certainly the verification of signatures can be done using nothing more than the openssl binary.

sign

Per the protocol above, for each file, it produces a hexdigest to send to the server (with trailing newline).

It receives back a PEM encoded signature, and (normally) a clue as to which file extension to use.

It can also ask the server for the certificate chain needed to verify the signatures being produced.

The client is designed to know as little as possible about the signing operation. It really only needs to be told the url to use and whether to send a SHA1 or SHA256 digest. Use of SHA1 only matters if using FakeHash, otherwise the default SHA256 is fine.

The client will look at the name it was invoked with and see if a configuration file exists with that name.

Note: the client config reader does not support variable references.

data requests

The client can send the server a number of requests for data other than a signature:

certs:

Return the certificate data needed to verify a signature. This is the value of the servers Certs setting.
crl:
Return any CRL data the server has.
ta:
Return the TrustAnchor associated with the signatures. This is typically the Root CA certificate.

openpgp-sign

Extends the SignClient class in sign.py to generate OpenPGP compatable signatures.

ima-sign

Extends the SignClient class in sign.py to generate Linux IMA signatures.

SSH proxy

If only authenticated singing requests are desired. The signing server can listen to localhost only, and an SSH sub-agent SignProxy used.

For example in sshd_config:

Subsystem sign-rsa /opt/sigs/SignProxy.py -c /opt/sigs/conf/rsa2k.cf -L local0.debug

where /opt/sigs/conf/rsa2k.cf is the config for an instance of the signing server, that includes the port it listens on.

Apart from the extra overhead that SSH introduces, this model reduces the load balancing effect of multiple servers, since the SignAgent keeps a connection to server open much longer than the normal sign client.

SignProxy

Run as an SSH sub-agent, this class opens a connection to the signing server when it has work for it, and sends the responses back to its client.

After a period of idleness it disconnects from the server. After an extended period of idleness, it shuts down. The next request from the client (SignAgent) will cause it to be restarted.

SignAgent

Run on client host by the user, and with access to SSH_AUTH_SOCK. It creates a private unix domain socket, which it listens to for singing requests. It indicates the socket it is listening on, which should then be used as the url given to sign.

When it gets a request it (if needed) opens a connection to the specified url - which leads to SignProxy. Sends and gets the response which it passes back to the caller.

There is only one thread, so if multiple sign clients attempt requests they will be serviced sequentially.

Signing Keys

Signing keys should be used for a limited period. Some signing methods leak information about the key used.

One way to achieve this, is to use new keys at regular intervals; annually, quarterly or even monthly depending on the load.

In most cases an annual key turnover would suffice. This makes it easy for a pool of servers to support multiple keys identified by the port the server is listening on.

By arranging for the signing server for a given key to listen on a port which maps to its key (eg. last 2 digits of the port map to year or month depending on the turnover cycle), it is simple to ensure that the same key is used for an entire build, even when spaning the key turnover period. This leaves at least three digits of the port number to indicate the type of key or signature method.

We can use the common telephone keypad mapping of letters to numbers to lend some determinism to the port numbers. For example:

D (Development):        3
E (ECDSA):              3
P (Production):         7
R (RSA):                7

End with 2 digits to indicate the era of the key (be it month, year etc) and its simple enough to coordinate port numbers.

Distribution

http://www.crufty.net/ftp/pub/sjg/sigs.tar.gz

contains everything needed to run the OpenSSLSigner based SignServerPool, assuming you have bmake available.

It is a pointer to the latest sigs-YYMMDD.tar.gz archive.

http://www.crufty.net/ftp/pub/sjg/sigs-signer-so.tar.gz

contains pre-built _signer.so for various systems ( freebsd10-amd64 freebsd7-i386 linux3-x86_64 netbsd5-i386 netbsd6-amd64 ) this is for the benfit of those that want to quickly try it out, you should really build it yourself.

Setup

After unpacking sigs, you can unpack sigs-signer-so.tar.gz in the sigs/ directory, or build _signer.so (see src/Makefile for instuctions).

There is an example.cf:

# this signer can handle any key type OpenSSL can
Signer=OpenSSLSigner
# use hash from client directly rather than hash it again
Hash=fakeSHA1
# the signing key
SigningKey=keys/rsa2k2013.key
# cert chain to verify above
Certs=certs/rsa2k2013.certs
# optional header
SigHeader= /C=US/ST=California/O=Crufty.NET/CN=Crufty2013/emailAddress=root@crufty.net
# hint to client the extension to use
SigExt= .sig
# a desired PEM tag
PEMTag= RSA SIGNATURE
# the port to listen on - rc_sigs feeds this same config to signctl.py
ListenPort=17713
# one child per cpu works well
children=2
syslogFacility=local0.info

The start/stop script rc_sigs will look for config files in ./conf/ else . and default to starting an instance of the server for each config it sees:

$ ./rc_sigs start
SignServerPool.py conf/rsa2k.cf
$ ./rc_sigs status
conf/rsa2k.cf: running
$ ./rc_sigs stop
shutdown SignServerPool.py conf/rsa2k.cf
$ ./rc_sigs check
restarting SignServerPool.py conf/rsa2k.cf
$ ./rc_sigs restart
shutdown SignServerPool.py conf/rsa2k.cf
restarting SignServerPool.py conf/rsa2k.cf
$

The check operation is handy to run from cron(8), since it will restart the servers if for any reason they are not running.


Author:sjg@crufty.net /* imagine something very witty here */
Revision:$Id: signing-server.txt,v 1.19 2017/10/28 19:24:50 sjg Exp $