fix: transliterate non-Latin titles in URL slugs

[ahmedqasid]

fix: transliterate non-Latin titles in URL slugs

Summary

• Pure-Arabic (and other non-Latin) titles were getting stripped by slugify.Slugify and collapsed to the literal "topic" fallback, so on a live site every Arabic question ended up at /questions/<id>/topic.
• Mirrors the existing convertChinese pre-step pattern in pkg/htmltext/htmltext.go using github.com/mozillazg/go-unidecode (same author as go-pinyin already in the repo, to minimise new-dep friction).
• Triggers only when the title contains non-Latin/non-CJK letters. Latin-only and Chinese-only inputs short-circuit so existing slugs stay byte-identical (tests pin this).

The fix

UrlTitle() now runs convertNonLatin after convertChinese. The detector skips ASCII, Latin-1 Supplement, Latin Extended, and CJK (which is handled by the existing pinyin step), so emoji / punctuation / symbols still flow into clearEmoji + slugify unchanged. Only when there are non-Latin letters present does it pay the unidecode cost.

Example: كيف حالك → kyf-hlk (was topic).

Live deployment / real-world verification

This patch has been running in production on ask.namasoft.com (an Apache Answer instance we operate) since deployment, built directly from this branch via docker compose build. The site has Arabic-language questions, so the fix exercises the affected code path on every page load.

Sample question URL on the deployed instance:

https://ask.namasoft.com/questions/10010000000000115

Click the link and you'll see the slug is the transliterated Arabic title rather than topic. No data migration was needed since url_title is computed on every request from Title and never persisted (see "Why this is safe to ship" below).

Admin-configurable

The transliteration is gated by a package-level atomic.Bool (default on, since the current behavior is objectively broken for affected users):

• htmltext.SetTransliterateNonLatin(enabled bool)
• htmltext.IsTransliterateNonLatinEnabled() bool

This is deliberately the minimum surface needed to satisfy "the setting must be readable from UrlTitle()". A follow-up PR can add the admin UI section (Non-Latin Languages Handling) that calls SetTransliterateNonLatin on save and on startup, without having to re-plumb every htmltext.UrlTitle call site through context.Context.

Why this is safe to ship

• url_title is not a persisted column. It's not on the Question entity in internal/entity/question_entity.go, no migration has ever added/dropped it, and every call site (internal/service/content/question_service.go, revision_service.go, vote_service.go, etc.) recomputes it from Title at response-build time via htmltext.UrlTitle(...).
• That means the fix is read-only: existing rows light up with correct slugs on the next request, with no migration and no data rewrite.
• Rollback is just redeploying the prior image; nothing on disk changes.

Test coverage

pkg/htmltext/htmltext_test.go:

• TestUrlTitleTable (table-driven): empty, pure Latin (unchanged), pure Chinese (unchanged — pins existing pinyin behavior), pure Arabic, mixed Latin+Arabic, emoji-only (still collapses to topic as before), very long Arabic (exercises cutLongTitle's 150-byte cap and UTF-8 boundary safety).
• TestUrlTitleTransliterationToggle: with the toggle off, Arabic collapses to topic (pre-fix behavior); with it on, transliterates.
• Existing TestUrlTitle left untouched.

Test plan for reviewers:

• go test ./pkg/htmltext/... — all pass locally
• Click the live sample URL above and confirm slug is transliterated, not topic
• Verify Chinese / Latin behavior is byte-identical to main (covered by table tests)

Out of scope (intentionally)

• No admin UI / site setting plumbing in this PR — see Admin-configurable above. Happy to do the React Non-Latin Languages Handling admin page + SiteType + service / controller / migration in a follow-up if maintainers want it.
• No change to the "topic" empty-result fallback.
• No plugin interface for slug generation — mirrored the existing convertChinese pre-step pattern instead.

Issues / discussion

I didn't find an existing upstream issue covering this — happy to be pointed at one if there is.

🤖 Generated with Claude Code

[LinkinStars]

I think this PR needs a clearer scope statement, because in its current form it does more than fix Arabic-only titles.

By adding go-unidecode before slugify, it changes slug generation for many non-Latin scripts, not just Arabic. For example, on my local reproduction:

• مرحبا hello changed from hello to mrhb-hello
• ไทย ไทย changed from topic to aithy-aithy
• こんにちは changed from topic to konnichiha
• 안녕하세요 changed from topic to annyeonghaseyo

So this is not only an Arabic fix. It changes the behavior for Thai, Japanese, Korean, Hebrew, Cyrillic, and other scripts as well. I think the PR description and tests should reflect that broader impact explicitly.

The second concern is about transliteration quality. What this PR introduces is a generic ASCII approximation, not linguistically correct multi-language romanization. That may be acceptable as a pragmatic fallback to avoid collapsing to topic, but it is different from saying the generated slug is “correct” for each language. For example, こんにちは -> konnichiha is machine-acceptable as an ASCII slug, but it is not necessarily the natural or expected romanization users would want. The same concern applies to other languages as well.

If the goal of this PR is “avoid empty/topic fallback for non-Latin titles by generating a usable ASCII slug”, then I think that should be stated much more explicitly in the PR description and test coverage.