Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/scripts/i18n/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,22 @@ Configure explicit paths in `translation-config.json` → `chunked_files`, or re
on `auto_chunk` (default: body ≥ 10k chars and ≥ 4 `##` sections) to auto-enable
`heading_sections`.

Changelog `<Update description="…">` dates are **derived from English** and
localized automatically (`ja`/`zh`: `YYYY年M月D日`, `ko`: `YYYY년 M월 D일`) after
each block is translated or re-serialized — English month names should not
remain in translated files.

During a chunked run the script:

1. Parses English into blocks (intro + each `##` section).
2. Compares each block’s hash to `translationBlockHashes` in the target frontmatter.
3. Translates only pending blocks (plus frontmatter when needed).
4. Checkpoints after every block so a failed run can resume.

`translationBlockHashes` keys are written in **descending semver order** for
changelog (`v0.25.1` before `v0.25.0`), matching the canonical `<Update>`
sequence. Long pages use **English document order** for section hashes.

Example:

```bash
Expand Down
276 changes: 253 additions & 23 deletions .github/scripts/i18n/chunked-translate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,20 @@ export function parseFrontmatterAndBody(content: string): { frontmatter: string;
return { frontmatter: `${match[1]}\n`, body: match[2] };
}

function parseBlockHashEntry(
line: string
): { label: string; hash: string } | null {
const quoted = line.match(/^\s{2}("(?:\\.|[^"\\])*"):\s*"?([a-f0-9]{8})"?\s*$/);
if (quoted) {
return { label: JSON.parse(quoted[1]!) as string, hash: quoted[2]! };
}
Comment on lines +99 to +102

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verifies that the current regex path can feed invalid JSON-escaped labels into JSON.parse.
node - <<'NODE'
const line = '  "v0.25.1\\q": deadbeef';
const quoted = line.match(/^\s{2}("(?:\\.|[^"\\])*"):\s*"?([a-f0-9]{8})"?\s*$/);
if (!quoted) {
  console.log("Regex did not match test input.");
  process.exit(0);
}
try {
  JSON.parse(quoted[1]);
  console.log("No exception (unexpected for this test).");
} catch (e) {
  console.log("Throws as expected:", e.message);
}
NODE

Repository: Comfy-Org/docs

Length of output: 141


🏁 Script executed:

git ls-files | grep -i "chunked-translate"

Repository: Comfy-Org/docs

Length of output: 100


🏁 Script executed:

head -n 120 .github/scripts/i18n/chunked-translate.ts | tail -n +90

Repository: Comfy-Org/docs

Length of output: 1175


🏁 Script executed:

grep -n "parseBlockHashEntry" .github/scripts/i18n/chunked-translate.ts

Repository: Comfy-Org/docs

Length of output: 193


🏁 Script executed:

sed -n '96,130p' .github/scripts/i18n/chunked-translate.ts

Repository: Comfy-Org/docs

Length of output: 1105


🏁 Script executed:

sed -n '110,160p' .github/scripts/i18n/chunked-translate.ts

Repository: Comfy-Org/docs

Length of output: 1387


🏁 Script executed:

git log --oneline --all -S "JSON.parse" -- .github/scripts/i18n/chunked-translate.ts | head -5

Repository: Comfy-Org/docs

Length of output: 144


🏁 Script executed:

sed -n '115,150p' .github/scripts/i18n/chunked-translate.ts

Repository: Comfy-Org/docs

Length of output: 977


Guard JSON.parse against malformed escaped labels—prevent translation kaboom.

At line 101, JSON.parse can throw when the regex matches a quoted key with invalid escape sequences (e.g., \q). The malformed label slips past the regex but crashes on parse, derailing the entire translation run. Wrap the return in try-catch to gracefully return null instead, staying true to the function's null-handling contract already in place.

Suggested fix
   if (quoted) {
-    return { label: JSON.parse(quoted[1]!) as string, hash: quoted[2]! };
+    try {
+      return { label: JSON.parse(quoted[1]!) as string, hash: quoted[2]! };
+    } catch {
+      return null;
+    }
   }
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const quoted = line.match(/^\s{2}("(?:\\.|[^"\\])*"):\s*"?([a-f0-9]{8})"?\s*$/);
if (quoted) {
return { label: JSON.parse(quoted[1]!) as string, hash: quoted[2]! };
}
const quoted = line.match(/^\s{2}("(?:\\.|[^"\\])*"):\s*"?([a-f0-9]{8})"?\s*$/);
if (quoted) {
try {
return { label: JSON.parse(quoted[1]!) as string, hash: quoted[2]! };
} catch {
return null;
}
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/scripts/i18n/chunked-translate.ts around lines 99 - 102, The
JSON.parse call on quoted[1] at line 101 can throw an error when the regex
matches a quoted key with invalid escape sequences, causing the entire
translation run to fail. Wrap the return statement in a try-catch block so that
if JSON.parse throws an exception, the function gracefully returns null instead
of crashing, maintaining consistency with the function's existing null-handling
contract.

const plain = line.match(/^\s{2}([^:]+):\s*"?([a-f0-9]{8})"?\s*$/);
if (plain) {
return { label: plain[1]!.trim(), hash: plain[2]! };
}
return null;
}

/** Parse `translationBlockHashes` YAML map from frontmatter body. */
export function parseBlockHashesFromFrontmatter(fmBody: string): Record<string, string> {
const out: Record<string, string> = {};
Expand All @@ -104,27 +118,64 @@ export function parseBlockHashesFromFrontmatter(fmBody: string): Record<string,
continue;
}
if (inMap) {
const quoted = line.match(/^\s{2}("(?:\\.|[^"\\])*"):\s*"?([a-f0-9]{8})"?\s*$/);
if (quoted) {
out[JSON.parse(quoted[1]!)] = quoted[2]!;
const entry = parseBlockHashEntry(line);
if (entry) {
out[entry.label] = entry.hash;
continue;
}
const plain = line.match(/^\s{2}([^:]+):\s*"?([a-f0-9]{8})"?\s*$/);
if (plain) {
out[plain[1]!.trim()] = plain[2]!;
if (/^[A-Za-z_][\w-]*:/.test(line)) {
inMap = false;
}
}
}
return out;
}

/** Block labels in frontmatter file order (not sorted). */
export function parseBlockHashLabelOrderFromFrontmatter(fmBody: string): string[] {
const labels: string[] = [];
const lines = fmBody.split("\n");
let inMap = false;
for (const line of lines) {
if (/^translationBlockHashes:\s*$/.test(line)) {
inMap = true;
continue;
}
if (inMap) {
const entry = parseBlockHashEntry(line);
if (entry) {
labels.push(entry.label);
continue;
}
if (/^[A-Za-z_][\w-]*:/.test(line)) {
inMap = false;
}
}
}
return out;
return labels;
}

export function formatBlockHashesYaml(blockHashes: Record<string, string>): string {
export function blockHashLabelOrderDrifts(
enLabels: string[],
storedLabels: string[]
): boolean {
if (enLabels.length !== storedLabels.length) return true;
return enLabels.some((label, i) => label !== storedLabels[i]);
}

export function formatBlockHashesYaml(
blockHashes: Record<string, string>,
labelOrder: string[]
): string {
const lines = ["translationBlockHashes:"];
for (const label of Object.keys(blockHashes).sort()) {
const seen = new Set<string>();
for (const label of labelOrder) {
if (!(label in blockHashes) || seen.has(label)) continue;
lines.push(` ${JSON.stringify(label)}: ${blockHashes[label]}`);
seen.add(label);
}
for (const label of Object.keys(blockHashes)) {
if (seen.has(label)) continue;
lines.push(` ${JSON.stringify(label)}: ${blockHashes[label]}`);
}
return lines.join("\n");
Expand Down Expand Up @@ -177,12 +228,13 @@ export function setChunkedTranslationMeta(
content: string,
fileHash: string,
enPath: string,
blockHashes: Record<string, string>
blockHashes: Record<string, string>,
labelOrder: string[]
): string {
const metaLines = [
`translationSourceHash: ${fileHash}`,
`translationFrom: ${enPath}`,
formatBlockHashesYaml(blockHashes),
formatBlockHashesYaml(blockHashes, labelOrder),
];
const metaBlock = metaLines.join("\n");

Expand Down Expand Up @@ -332,8 +384,159 @@ export function parseUpdateBlocks(body: string): ContentBlock[] {
return blocks;
}

/** Parse ComfyUI changelog label like `v0.25.1` into numeric segments. */
export function parseChangelogVersion(label: string): number[] {
const m = label.match(/^v?(\d+(?:\.\d+)*)/i);
if (!m) return [];
return m[1]!.split(".").map((n) => Number(n));
}

/** Descending semver compare for changelog `<Update label="…">` values (newest first). */
export function compareChangelogVersionsDesc(a: string, b: string): number {
const pa = parseChangelogVersion(a);
const pb = parseChangelogVersion(b);
const len = Math.max(pa.length, pb.length);
for (let i = 0; i < len; i++) {
const da = pa[i] ?? 0;
const db = pb[i] ?? 0;
if (da !== db) return db - da;
}
return b.localeCompare(a);
}

export function canonicalChangelogLabelOrder(labels: string[]): string[] {
return [...labels].sort(compareChangelogVersionsDesc);
}

export function sortUpdateBlocksByVersion(blocks: ContentBlock[]): ContentBlock[] {
return [...blocks].sort((a, b) => compareChangelogVersionsDesc(a.label, b.label));
}

export function orderSlotsForStrategy(strategy: ChunkStrategy, slots: BlockSlot[]): BlockSlot[] {
if (strategy !== "update_blocks") return slots;
return [...slots].sort((a, b) => compareChangelogVersionsDesc(a.label, b.label));
}

export function canonicalBlockLabelOrder(
strategy: ChunkStrategy,
labels: string[]
): string[] {
return strategy === "update_blocks" ? canonicalChangelogLabelOrder(labels) : labels;
}

export function changelogLabelHash(blocks: ContentBlock[]): string {
return blockHash(blocks.map((b) => b.label).join("|"));
const labels = canonicalChangelogLabelOrder(blocks.map((b) => b.label));
return blockHash(labels.join("|"));
}

const EN_MONTH_NAMES = [
"January",
"February",
"March",
"April",
"May",
"June",
"July",
"August",
"September",
"October",
"November",
"December",
] as const;

const EN_DATE_DESC_RE =
/^(January|February|March|April|May|June|July|August|September|October|November|December)\s+(\d{1,2}),\s+(\d{4})$/;

export function parseEnglishChangelogDate(
date: string
): { year: number; month: number; day: number } | null {
const m = date.trim().match(EN_DATE_DESC_RE);
if (!m) return null;
const month = EN_MONTH_NAMES.indexOf(m[1] as (typeof EN_MONTH_NAMES)[number]) + 1;
if (month <= 0) return null;
return { year: Number(m[3]), month, day: Number(m[2]) };
}

/** Localized `<Update description="…">` date for ja / ko / zh. */
export function formatChangelogDateForLang(
year: number,
month: number,
day: number,
lang: string
): string {
switch (lang) {
case "ja":
case "zh":
return `${year}年${month}月${day}日`;
case "ko":
return `${year}년 ${month}월 ${day}일`;
default:
return `${EN_MONTH_NAMES[month - 1]} ${day}, ${year}`;
}
}

export function localizeEnglishChangelogDate(date: string, lang: string): string {
const parsed = parseEnglishChangelogDate(date);
if (!parsed || lang === "en") return date;
return formatChangelogDateForLang(parsed.year, parsed.month, parsed.day, lang);
}

export function extractUpdateDescription(updateBlock: string): string | null {
return updateBlock.match(/<Update\s+[^>]*description="([^"]+)"/)?.[1] ?? null;
}

/** Replace description with locale date derived from the English source block. */
export function syncUpdateBlockDescription(
translatedBlock: string,
enBlock: ContentBlock,
langCode: string
): string {
const enDesc = extractUpdateDescription(enBlock.content);
if (!enDesc || langCode === "en") return translatedBlock;
const localized = localizeEnglishChangelogDate(enDesc, langCode);
if (localized === enDesc) return translatedBlock;
return translatedBlock.replace(
/(<Update\s+[^>]*description=")([^"]+)(")/,
`$1${localized}$3`
);
}

export function applyChangelogBlockLocalizations(
slots: BlockSlot[],
enBlocks: ContentBlock[],
langCode: string
): BlockSlot[] {
if (langCode === "en") return slots;
const enByLabel = new Map(enBlocks.map((b) => [b.label, b]));
return slots.map((s) => {
const enBlock = enByLabel.get(s.label);
if (!s.content || !enBlock) return s;
return {
label: s.label,
content: syncUpdateBlockDescription(s.content, enBlock, langCode),
};
});
}

export function hasChangelogDateDrift(
enContent: string,
targetContent: string,
langCode: string
): boolean {
if (langCode === "en") return false;
const enDoc = parseDocument(enContent, "update_blocks");
const targetDoc = parseDocument(targetContent, "update_blocks");
const targetByLabel = new Map(targetDoc.blocks.map((b) => [b.label, b]));
for (const enBlock of enDoc.blocks) {
const enDesc = extractUpdateDescription(enBlock.content);
if (!enDesc) continue;
const expected = localizeEnglishChangelogDate(enDesc, langCode);
const targetBlock = targetByLabel.get(enBlock.label);
if (!targetBlock) continue;
const actual = extractUpdateDescription(targetBlock.content) ?? "";
if (actual !== expected) return true;
}
return false;
}

// ---------------------------------------------------------------------------
Expand All @@ -352,21 +555,36 @@ export function serializeChunkedDocument(
slots: BlockSlot[],
fileHash: string,
enRel: string,
blockHashes: Record<string, string>
blockHashes: Record<string, string>,
strategy: ChunkStrategy
): string {
const body = slots
const ordered = orderSlotsForStrategy(strategy, slots);
const body = ordered
.map((s) => s.content)
.filter((c): c is string => c !== null)
.join("\n\n");
const labelOrder = ordered.map((s) => s.label);
const raw = `${frontmatter}\n${body}\n`;
return setChunkedTranslationMeta(raw, fileHash, enRel, blockHashes);
return setChunkedTranslationMeta(raw, fileHash, enRel, blockHashes, labelOrder);
}

/** Rebuild changelog body with `<Update>` blocks in descending semver order. */
export function serializeUpdateBlocksDocument(
frontmatter: string,
blocks: ContentBlock[]
): string {
const body = sortUpdateBlocksByVersion(blocks)
.map((b) => b.content)
.join("\n\n");
return `${frontmatter}\n${body}\n`;
}

export function getSectionSyncStatus(
enContent: string,
existingContent: string,
strategy: ChunkStrategy,
force: boolean
force: boolean,
langCode?: string
): SectionSyncStatus {
const enDoc = parseDocument(enContent, strategy);
const enHashes = documentBlockHashes(enDoc.blocks);
Expand All @@ -389,22 +607,33 @@ export function getSectionSyncStatus(

const existingDoc = parseDocument(existingContent, strategy);

const existingFmBody = parseFrontmatterAndBody(existingContent).frontmatter
.replace(/^---\n/, "")
.replace(/\n---$/, "");
const storedLabels = parseBlockHashLabelOrderFromFrontmatter(existingFmBody);
const enLabels = canonicalBlockLabelOrder(
strategy,
enDoc.blocks.map((b) => b.label)
);
const hasOrderDrift = blockHashLabelOrderDrifts(enLabels, storedLabels);

if (strategy === "update_blocks") {
const existingLabels = new Set(existingDoc.blocks.map((b) => b.label));
const pendingBlocks = enDoc.blocks
.filter((b) => !existingLabels.has(b.label))
.map((b) => b.label);
const hasDateDrift = langCode
? hasChangelogDateDrift(enContent, existingContent, langCode)
: false;
return {
upToDate: pendingBlocks.length === 0,
upToDate: pendingBlocks.length === 0 && !hasOrderDrift && !hasDateDrift,
pendingBlocks,
needsFrontmatter: existingDoc.blocks.length === 0,
needsReserialize: pendingBlocks.length === 0 && (hasOrderDrift || hasDateDrift),
};
}

const existingFm = parseFrontmatterAndBody(existingContent).frontmatter;
const storedHashes = parseBlockHashesFromFrontmatter(
existingFm.replace(/^---\n/, "").replace(/\n---$/, "")
);
const storedHashes = parseBlockHashesFromFrontmatter(existingFmBody);

const pendingBlocks = enDoc.blocks
.filter((b) => storedHashes[b.label] !== enHashes[b.label])
Expand All @@ -413,10 +642,11 @@ export function getSectionSyncStatus(
const hasStructureDrift = Object.keys(storedHashes).some((k) => !(k in enHashes));

return {
upToDate: pendingBlocks.length === 0 && !hasStructureDrift,
upToDate: pendingBlocks.length === 0 && !hasStructureDrift && !hasOrderDrift,
pendingBlocks,
needsFrontmatter: Object.keys(storedHashes).length === 0,
needsReserialize: hasStructureDrift && pendingBlocks.length === 0,
needsReserialize:
pendingBlocks.length === 0 && (hasStructureDrift || hasOrderDrift),
};
}

Expand Down
Loading
Loading