Return a robots.txt if (and only if) host isn't production

[dontlaugh]

We want crawlers to ignore our devs sites, which are still on the public internet.

If Cro can do this by matching the Host request header, that's preferred. However, if you need some extra runtime config, I can do that on the server for you.

If the request's host != "raku.org", we need to return the following at the path /robots.txt

User-agent: *
Disallow: /

Otherwise, we can return 404.

The docs site does this correctly, and fwiw it's more important for the docs site since there's lots more content to index.

Compare: https://docs.raku.org/robots.txt with https://dev.docs.raku.org/robots.txt

[librasteve]

on my todo list

(btw - is dev.raku.org now decoupled from production raku.org ?)

[dontlaugh]

is dev.raku.org now decoupled from production raku.org

No, they still both track main, unfortunately. You'll need to test locally. I will try to find a solution here that generalizes to the other websites. I need to integrate some of the automation that has been checked in but not productionized in our infra code. Then we can do this.

[dontlaugh]

Also: strictly speaking this doesn't really depend on #262, since it's for robots and not humans.

[librasteve]

ok, since Air does not (yet) support any robots.txt, I will try to do what you say above and also, when the url IS production, then to produce:

# Allow all search engines to crawl everything
User-agent: *
Disallow:

(noting the empty Disallow: which I believe means allow everything)

[librasteve]

I want to make this change at the Air module level (so it applies to all HARC stack sites) ... currently I am working on externalising the config defaults and I think that I need to add a config line like domain: "raku.org" so I can compare the actual url to this production url. This line to go in a .air.yaml file in the project root alongside .cro.yaml. This config is loaded by Air::Base::Site.build() whereas our production entry point will be Air::Base::Site.start() so should be all done by the time you see it. [Note to self - review the Dockerfile.] Gonna make this generic for http|https and /|www.