Gzip Support and cache-control headers

[SaulDoesCode]

It would be very nice to see a golang web framework handle gzip natively and do it well. There are various libraries and middlewares out there which purport a well adjusted and performant solution to serving gzipped content.

I've made a fairly primitive project some time ago, which provides an echo instance with the ability to read, monitor, cache and recache various static assets either within a specified folder or definitively located elsewhere; with this it would determine an asset's compress-ability and compress it thusly, serving it, thereby, only when a client's Accept-Encoding header indicates gzip support.

Desireable Features In a Gzipping Sollution:

• - Cache Gzipped Assets:

  • In memory, with coffer potentially.
    • watched assets, should, on changes, (asynchronously) re-compress/update a memory/disk backed asset.ext(.gz)
  • On Disk, like some nginx/apache configs permit.
    • eg. route localhost/asset.txt -> first looks for - > ./assets/.cache/asset.txt.gz
  • Client Side, by setting and maintaining the correct cache-control and etag headers.

• - Compress at the highest level when caching, but automatically scale down with dynamic content.

---

Air does not seem to handle caching headers, (to my knowledge, I may be wrong), as all my coffer cached assets seem to be missing their cache-control headers. While it would be possible to deliberately add such caching headers, it would certainly also have been nice had the air sub systems managed cache headers when it detected that the higher level user has not set any or has not deliberately disabled cache-control.

[SaulDoesCode]

In my fork, I've managed to get some basic gzip caching working with coffer, however, there are many improvements, or in this case most likely, overhauls, to be made; but it works, more or less.

coffer.go

var (
 // Compressable - list of compressable file types, append to it if needed
 Compressable = []string{"", ".txt", ".htm", ".html", ".css", ".toml", ".php", ".js", ".json", ".md", ".mdown", ".xml", ".svg", ".go", ".cgi", ".py", ".pl", ".aspx", ".asp"}
)

for _, cext := range Compressable {
	if ext == cext {
		var buf bytes.Buffer
		gz := gzip.NewWriter(&buf)
		if _, err := gz.Write(b); err == nil {
			gz.Flush()
		}
		gz.Close()
		c.assets[name].compressed = buf.Bytes()
		c.assets[name].isCompressed = true
		break
	}
}

response.go ~L628 or circa L615 (in the original)

if a, err := theCoffer.asset(filename); err != nil {
	return err
} else if a != nil {
	if a.isCompressed && strings.Contains(r.request.Request.Header.Get("accept-encoding"), "gzip") {
		r.SetHeader("content-encoding", "gzip")
		r.SetHeader("vary", "accept-encoding")
		c = bytes.NewReader(a.compressed)
	} else {
		c = bytes.NewReader(a.content)
	}
	ct = a.mimeType
	et = a.checksum[:]
	mt = a.modTime
}

[aofei]

In fact, Air handles the caching headers. However, yeah, it doesn't actively add the cache-control to the response headers (it's too diverse, isn't it?). Therefore, the best solution at present is to let a Gas be responsible for adding the cache-control headers. See air-example/blog:

air.STATIC(
	"/assets",
	air.AssetRoot,
	func(next air.Handler) air.Handler {
		return func(req *air.Request, res *air.Response) error {
			res.Headers["cache-control"] = &air.Header{
				Name:   "cache-control",
				Values: []string{"max-age=3600"},
			}

			return next(req, res)
		}
	},
)

[aofei]

For caching gzipped asset files, I don't think it's worthwhile to let Air produce extra files while it's running (since supporting ACME). After all, those asset files are not that big. So definitely in-memory.

For gzipping dynamic content (yeah, that's what I want), I think we can try to wrap the http.ResponseWriter. What do you think?

Then,

I'm thinking about... Do you think we need to introduce a new component (perhaps called "compressor") to Air to support Gzip natively? I mean, it seems that these features we are discussing are not suitable for adding to the minifier component.

[SaulDoesCode]

First Minify then Gzip, it could work why not, it's just another level of making something smaller, though if the user is not using coffer with this then minifying and gzipping every time is costly, it's just not going to be all that great to scale with so it would be best to make this optional.

But at the same time, I had a hard time getting plain gorilla and http middleware to work, and by adding gzip preferably in air itself or at least as a gas then it makes the whole air ecosystem a bit more self-contained and whole, if that makes sense. But of course, you should do as you see fit, this is your project don't let others sway you in ways you don't want to go.

Although I probably won't build anything that's going to have to handle 10 000+ requests per second, I still think it would be best to approach this problem with scale in mind. By disk caching gzipped assets to avoid extra compute overhead, and potentially introducing a multi-threaded compression pool or semaphore the server might be able to handle that many requests and still save a lot of traffic (which can cost more than just plain compute on some hosts).

Granted, most asset folders will never be much larger than maybe 20mb, a few styles, javscripts, and maybe an image. Just imagine you are building something which is intended serving a milliard of small to mid-range (<200kb) compressible assets, like svg images, in that case if you've over 2GB of small assets it would be impractical to hold all of them in memory and more impractical still to compress each one again when it is requested. By having a disk cache such use cases may be catered for, I know it's overkill and probably won't happen but it's worth a shot, cause it's a nice feature, I think.

But ultimately, yes, adding a few extra checks and steps inside .Write could make all the difference, turn air apps into smart servers that cache and compress auto-magically.

Sorry for the delay, am cooking, will reply when possible.

[aofei]

Haha, have a good meal. I have some ideas, let me sort it out and I'll reply you tomorrow. It's getting dark, I should go to sleep now.