Speed up the container CI job

[abbiemery]

Currently it takes ages and really slows things down. We could implement caching to speed this up.

[tpoliaw]

Container job is down to around 1m30s and most of that is reading from and writing to the cache, I don't think we're going to get much faster while still using github's runners and storage