Avoid race condition (#86)

This solves the following error when building this image.
rm: can't remove '/root/.gnupg/S.gpg-agent.ssh': No such file or directory

This is likely due to a race condition when removing a directory that
gpg-agent and dirmanger care about.