[Bug]: Setup check for `X-Robot-Tag` expects exact match, resulting in a warning

[stavros-k]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I have set the following tags using treafik (which I can see in the response headers)

x-robots-tag: none,noarchive,nosnippet,notranslate,noimageindex,nofollow,noindex

Yet nextcloud still shows

The "X-Robots-Tag" HTTP header is not set to "noindex, nofollow". This is a potential security or privacy risk, as it is recommended to adjust this setting accordingly.

Steps to reproduce

1. Set required X-Robot-Tag plus some extra.
2. Go to Overview
3. See the warning

Expected behavior

Warning to only show when headers are lesser than the recommended

Installation method

Community Docker image

Operating system

Other

PHP engine version

PHP 8.1

Web server

Nginx

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Updated to a major version (ex. 22.2.3 to 23.0.1)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

root@a0e913b615b7:/# occ config:list system
{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "10.10.10.254",
            "cloud.DOMAIN.TLD"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/cloud.DOMAIN.TLD\/",
        "overwriteprotocol": "https",
        "overwritehost": "cloud.DOMAIN.TLD",
        "dbtype": "pgsql",
        "version": "26.0.0.11",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "updater.release.channel": "stable",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "logfile": "\/config\/log\/nextcloud\/nextcloud.log",
        "logdateformat": "D d\/m\/Y H:i:s",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "data-fingerprint": "6619a1ae23dec248f717298ba30bd687",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "LOGIN",
        "mail_sendmailmode": "pipe",
        "mail_smtpauth": 1,
        "mail_smtpsecure": "tls",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "30,60",
        "versions_retention_obligation": "30,60",
        "enable_previews": true,
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "preview_max_memory": "1024",
        "jpeg_quality": "60",
        "default_phone_region": "GR",
        "activity_expire_days": "90",
        "app_install_overwrite": [
            "previewgenerator",
            "breezedark",
            "bookmarks",
            "bruteforcesettings",
            "imageconverter",
            "files_antivirus",
            "news",
            "timemanager"
        ],
        "enabledPreviewProviders": [
            "OC\\Preview\\Image",
            "OC\\Preview\\MP3",
            "OC\\Preview\\TXT",
            "OC\\Preview\\PDF",
            "OC\\Preview\\Movie",
            "OC\\Preview\\Photoshop",
            "OC\\Preview\\TIFF",
            "OC\\Preview\\SVG",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\HEIC"
        ]
    }
}

List of activated Apps

Enabled:
  - activity: 2.18.0
  - admin_audit: 1.16.0
  - bookmarks: 13.0.1
  - breezedark: 25.0.1
  - bruteforcesettings: 2.6.0
  - calendar: 4.3.1
  - circles: 26.0.0
  - cloud_federation_api: 1.9.0
  - comments: 1.16.0
  - contacts: 5.2.0
  - contactsinteraction: 1.7.0
  - dashboard: 7.6.0
  - dav: 1.25.0
  - deck: 1.9.0
  - federatedfilesharing: 1.16.0
  - federation: 1.16.0
  - files: 1.21.1
  - files_antivirus: 4.0.3
  - files_automatedtagging: 1.16.1
  - files_pdfviewer: 2.7.0
  - files_rightclick: 1.5.0
  - files_sharing: 1.18.0
  - files_trashbin: 1.16.0
  - files_versions: 1.19.1
  - firstrunwizard: 2.15.0
  - imageconverter: 1.3.4
  - logreader: 2.11.0
  - lookup_server_connector: 1.14.0
  - mail: 3.0.2
  - news: 21.1.0
  - nextcloud_announcements: 1.15.0
  - notes: 4.7.2
  - notifications: 2.14.0
  - oauth2: 1.14.0
  - password_policy: 1.16.0
  - photos: 2.2.0
  - previewgenerator: 5.2.1
  - privacy: 1.10.0
  - provisioning_api: 1.16.0
  - recognize: 3.7.0
  - recommendations: 1.5.0
  - related_resources: 1.1.0-alpha1
  - richdocuments: 8.0.0
  - serverinfo: 1.16.0
  - settings: 1.8.0
  - sharebymail: 1.16.0
  - support: 1.9.0
  - survey_client: 1.14.0
  - suspicious_login: 4.4.0
  - systemtags: 1.16.0
  - tasks: 0.14.5
  - text: 3.7.2
  - theming: 2.1.1
  - twofactor_backupcodes: 1.15.0
  - twofactor_totp: 8.0.0-alpha.0
  - updatenotification: 1.16.0
  - user_status: 1.6.0
  - viewer: 1.10.0
  - weather_status: 1.6.0
  - workflowengine: 2.8.0
Disabled:
  - encryption: 2.14.0 (installed 2.8.1)
  - files_external: 1.18.0 (installed 1.17.0)
  - onlyoffice: 7.8.0 (installed 7.8.0)
  - user_ldap: 1.16.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[szaimen]

Cc @MichaIng

[MichaIng]

Not related to our recent change, so I removed 26-feedback tag.

Same request was done here: nextcloud/docker#1218
Since it makes more sense to discuss it here on the server repo, I'm closing the issue at the Docker repo and copying the relevant part of my comment here:

All the other header values you added are obsolete if noindex is set. So they are not "doing more" but should be removed to save the additional bytes transferred and to avoid possible confusion of other search engines which do not understand them.

So while the extras do most likely not hurt, they are all redundant/unnecessary and at best ballast. And while I do not think so, there is also a chance that those, since not supported by all search engines, may even confuse other search engines and in worst case lead to the essential noindex being ignored. So IMO it is correct to explicitly accept only noindex, nofollow, as failsafe value.

[smileyxy]

modify the X-Robots-Tag in nextcloud\config\nginx\site-confs
"add_header X-Robots-Tag "noindex, nofollow" always;"
It should be possible to eliminate this warning.

[hrenard]

Shouldn't none be equivalent to noindex, nofollow and not show the warning ?

[MichaIng]

It is not equivalent: #36689

[schneidr]

So IMO it is correct to explicitly accept only noindex, nofollow, as failsafe value

I disagree. While this might be true for the values noted in the OP I am using the additional value noai, which is not covered by noindex or nofollow. It should only be checked if the wanted values are set, not if they are set exclusively.

[MichaIng]

Is there some "official" documentation about this header value? I could only find it being respected by the img2dataset tool, which used to massively cause trouble for image-related websites like DeviantArt, not to forget the personal and legal concerns: https://github.com/rom1504/img2dataset/pull/249/files

However, checking the docs of the option which is not set by default to 4 header values, suggests that any of them is interpreted as denial: https://github.com/rom1504/img2dataset/blob/6adc82b/README.md?plain=1#L169
So for this tool at least, noindex implies noai (as long as users of this tool do not override/empty the options default).

If you know a tool/service/case where it is different, and noai is additionally required, let me know, and I will adjust the check when I find time.