After updating to v3.1, large repo build takes 3 hours

[anaclumos]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Prerequisites

• I'm using the latest version of Docusaurus.
• I have tried the npm run clear or yarn clear command.
• I have tried rm -rf node_modules yarn.lock package-lock.json and re-installing packages.
• I have tried creating a repro with https://new.docusaurus.io.
• I have read the console error message carefully (if applicable).

Description

~/Downloads/www  on   main *3 +9 !1  qqqq                                                                                                               128 ✘  took 56s   base   at 12:56:46 PM 
$ $npm_execpath run prepare-to-launch && $npm_execpath run scc && $npm_execpath run format && git add . && git commit -m 'wrote something' && git push && $npm_execpath run build && $npm_execpath redirects && until $npm_execpath ship; do :; done
$ $npm_execpath run clear && $npm_execpath run sanitize && $npm_execpath run process-blog && $npm_execpath run process-docs && $npm_execpath run backlinks && $npm_execpath run figcaption && $npm_execpath run readme
$ docusaurus clear && rm -rf 'blog' && rm -rf 'docs' && rm -rf '**/*.config.js' && rm -rf '**/*.config.js.map' && rm -f 'docusaurus.config.js.map' && rm -f 'docusaurus.config.js' && rm -rf 'i18n /**/*.md' && cp tools/안녕.md i18n/ko/docusaurus-plugin-content-docs/current/Hey.md && rm -rf 'i18n /**/*.png' && rm -rf 'i18n /**/*.svg' && rm -rf 'i18n /**/*.jpg' && rm -rf 'i18n /**/*.jpeg'
[SUCCESS] Removed the Webpack persistent cache folder at "/Users/cho/Downloads/www/node_modules/.cache".
$ python3 tools/sanitize.py
Found 2289 MD and MDX files.
Replaced 0 hex marks.
$ python3 tools/process-blog.py
$ python3 tools/process-docs.py
Replaced 10642 wikilinks.
$ python3 tools/process-backlinks.py
Found 4552 MD files.
Wrote 2860 files with 8283 mentions to backlinks.ts.
Wrote 2276 filenames to filenames.ts.
$ python3 tools/img-alt-to-figcaption.py
Found 2324 MD and MDX files.
Replaced 1320 alt texts.
$ cp tools/README.src.md README.md && printf "\n\n## Last updated \n\n$(date)\n" >> README.md
$ printf '\n## Stats\n' >> README.md && printf '\n```\n' >> README.md && scc . >> README.md && printf '```\n' >> README.md
$ prettier --log-level silent --config .prettierrc -w '**/*.{ts,tsx,json,md,mdx,css,scss,html,yml,yaml,mts,mjs,cts,cjs,js,jsx,xml}'
[main 478a064d] wrote something
 9 files changed, 13 insertions(+), 6 deletions(-)
 create mode 100644 Research/assets/F6FE2E.png
Enumerating objects: 30, done.
Counting objects: 100% (30/30), done.
Delta compression using up to 12 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (16/16), 1.43 MiB | 33.99 MiB/s, done.
Total 16 (delta 6), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (6/6), completed with 6 local objects.
To https://github.com/anaclumos/extracranial.git
   f316c4bd..478a064d  main -> main
$ NODE_OPTIONS="--max-old-space-size=16384" docusaurus build
[INFO] Website will be built for all these locales:
- en
- ko
[INFO] [en] Creating an optimized production build...

✔ Client
  Compiled successfully in 1.30m

✔ Server
  Compiled successfully in 4.46m

[SUCCESS] Generated static files in "build".
[INFO] [ko] Creating an optimized production build...

✔ Client
  Compiled successfully in 1.35m

✔ Server
  Compiled successfully in 4.43m

[SUCCESS] Generated static files in "build/ko".
[INFO] Use `npm run serve` command to test your build locally.
$ cp _redirects build/_redirects
$ wrangler pages deploy ./build --commit-dirty=true --project-name=memex
🌎  Uploading... (16403/16403)

✨ Success! Uploaded 8158 files (8245 already uploaded) (155.03 sec)

✨ Uploading _redirects
✨ Deployment complete! Take a peek over at https://25a4b06d.memex.pages.dev
⌛ Done in 12618.98s.

    ~/Downloads/www  on   main *3 !2                                                                                                                     ✔  took 3h 30m 19s   base   at 04:48:29 PM

On the last line, take a look at 3h 30m 19s. Even though the client and server was compiled in ~4m, it just hangs there forever, and the node process takes ~7GB of RAM. In previous versions it went up to ~14GB; was there any change in how docusaurus limit RAM usage in sacrifice of compilation speed?

Reproducible demo

https://github.com/anaclumos/extracranial

Steps to reproduce

1. Run all-in-one:build

Expected behavior

Compiles relatively fast, preferably under 30 minutes

Actual behavior

Takes 3 hours to build.

Your environment

• Public source code:
• Public site URL:
• Docusaurus version used:
• Environment name and version (e.g. Chrome 89, Node.js 16.4):
• Operating system and version (e.g. Ubuntu 20.04.2 LTS):

Self-service

• I'd be willing to fix this bug myself.

[slorber]

Hey

No we didn't change anything recently that could lead to such a significant difference.

But your report is not clear enough.

What was the version of Docusaurus you used before exactly?

How long did it take to build previously?

Can you replicate this only on your computer, or also on CI such as GitHub Actions?

What was the upgrade PR?

Are we even sure it's Docusaurus fault? Your log shows that Done in 12618.98s.. Please show us the time it takes executing only the Docusaurus build command, building just one language for example, and nothing else.

How comes you are reporting using Node.js 16.4 while Docusaurus v3.0 requires Node 18?

[ravilach]

First off, huge fan of Docusaurus. Wanted to comment along. Might be tangential to this, we also were increasing a 2x increase in build times upgrading from Docusaurus 3.0.1 to 3.1. We ended up downgrading back to 3.0.1. We leverage our own CI Solution, Harness CI Enterprise.

3.0.1 Builds: 8-9 mins
3.1 Builds: 17-20 mins.

We are wanting to dig in a little further if anyone on Docusaurus Project Side can weigh in On Broken Anchors feature [https://github.com//pull/9528]. If that feature has to build a list of the anchors, on larger sites if that step takes time. We tried configuring onBrokenMarkdownLinks to Ignore but I believe the process still runs, just. not producing or throwing the output. Potentially moving Ignore to not execute at all?

The big increase comes between Server Compile and the "done" hook.

[success] [webpackbar] Server: Compiled successfully in 7.36m
[SUCCESS] Generated static files in "build".
[INFO] Use `npm run serve` command to test your build locally.
Done in 983.84s.

Node Build Version: 18.19.0

Thanks for a great project!

[anaclumos]

Hey

No we didn't change anything recently that could lead to such a significant difference.

But your report is not clear enough.

What was the version of Docusaurus you used before exactly?

Was using 3.0.1.

How long did it take to build previously?

It took under 30 minutes.

Can you replicate this only on your computer, or also on CI such as GitHub Actions?

My Docusaurus site is pretty big and doesn't fit on CI machines. RAM usage used to spike to 14GB the sealing process, and all CI machines crashed at this point.

Are we even sure it's Docusaurus fault? Your log shows that Done in 12618.98s.. Please show us the time it takes executing only the Docusaurus build command, building just one language for example, and nothing else.

I am sure. Every other script finishes under 1 minute, and the it's only the Docusaurus build step that hangs.

How comes you are reporting using Node.js 16.4 while Docusaurus v3.0 requires Node 18?

I am using v18.17.1. Where did you get this information, may I ask?

[ravilach]

Hey
No we didn't change anything recently that could lead to such a significant difference.
But your report is not clear enough.
What was the version of Docusaurus you used before exactly?

Was using 3.0.1.

How long did it take to build previously?

It took under 30 minutes.

Can you replicate this only on your computer, or also on CI such as GitHub Actions?

My Docusaurus site is pretty big and doesn't fit on CI machines. RAM usage used to spike to 14GB the sealing process, and all CI machines crashed at this point.

Are we even sure it's Docusaurus fault? Your log shows that Done in 12618.98s.. Please show us the time it takes executing only the Docusaurus build command, building just one language for example, and nothing else.

I am sure. Every other script finishes under 1 minute, and the it's only the Docusaurus build step that hangs.

How comes you are reporting using Node.js 16.4 while Docusaurus v3.0 requires Node 18?

I am using v18.17.1. Where did you get this information, may I ask?

+1 on the sealing process, where the resource usage/time seems to spike for us also. Anything added to that process from 3.0.1 -> 3.1, e.g On Broken Anchors? Thanks!

[OzakIOne]

If it's coming from the brokenAnchor you may try to put onBrokenAnchors in docusaurus.config file to ignore

Maybe you can disable it in your CI but still have a build process somewhere that you run manually / every few times to check for broken links / anchors

[andrewgbell]

Just to add we've recently rolled back from 3.1 to 3.0.1 for this exact same issue (we also have a large site). Normally would take approx 45 mins to build, and with 3.1 moves to just over 2 hours.

However, maybe of interest, when we initially rolled back we updated our package-lock.json and noticed the build times stayed the same (close to 2 hours). Reverting to the original package-lock.json prior to our 3.1 upgrade that we used when originally on 3.0.1, the build went back to 45 mins.

I've just tried it again, and when using 3.0.1, and building without a package-lock.json to use the latest dependencies, the build time more than doubles.

As an aside, onBrokenAnchors: "ignore", made no difference for us (and we also fixed all the broken anchors).

[ravilach]

If it's coming from the brokenAnchor you may try to put onBrokenAnchors in docusaurus.config file to ignore

Maybe you can disable it in your CI but still have a build process somewhere that you run manually / every few times to check for broken links / anchors

Thanks @OzakIOne it's a great feature. Curious, we noticed the same behavior with ignore. Does the onBrokenAnchor always run but does not display results if ignore. If does not run, can weed that out.

[slorber]

@andrewgbell it looks like the build-time increase is not related to the 3.1 upgrade, but rather the upgrade of a transitive dependency that has a perf regression.

It would be super helpful for me to be able see/run that upgrade myself and study the package-lock.json diff.

Can someone share a site / branch that build faster in 3.0.1, and where I could reproduce the build time regression by upgrading.

---

Does the onBrokenAnchor always run but does not display results if ignore. If does not run, can weed that out.

@ravilach I'd recommend to try turning off both onBrokenLinks: "ignore" and onBrokenLinks: "ignore", because we only "bypass" the broken link checker if both are ignored atm.

I'll try to optimize that better in the future, but in the meantime the code looks like this:

  if (onBrokenLinks === 'ignore' && onBrokenAnchors === 'ignore') {
    return;
  }

  const brokenLinks = getBrokenLinks({
    routes,
    collectedLinks: normalizeCollectedLinks(collectedLinks),
  });

  reportBrokenLinks({brokenLinks, onBrokenLinks, onBrokenAnchors});

---

Note: is this possible that you encounter longer build times only due to cache eviction.

We use Webpack with persistent caching and on rebuilds it's supposed to rebuild faster.

It may be possible that your site builds longer simply because the caches were empty?

In this case I suggest trying to run docusaurus clear && docusaurus build on your "fast branch" and see if it becomes slower to build.

[slorber]

@anaclumos I tried using your repo before the upgrade (https://github.com/anaclumos/extracranial/tree/f144432acdfff55d741a1dbc568ae0b51dd052fe) but the usage of Bun package manager makes it inconvenient to troubleshoot.

First when I run bun install on your repo with latest Bun version, it seems to resolve to newer versions of Docusaurus dependency ranges, and modify your bun.lockb file:

Then, the binary format of the lockfile makes it super inconvenient to inspect and diff.

Maybe I could try using the exact same version of Bun you are using, and it would not upgrade? For now I'm unable to troubleshoot this using your repo.

[andrewgbell]

@andrewgbell it looks like the build-time increase is not related to the 3.1 upgrade, but rather the upgrade of a transitive dependency that has a perf regression.

It would be super helpful for me to be able see/run that upgrade myself and study the package-lock.json diff.

Can someone share a site / branch that build faster in 3.0.1, and where I could reproduce the build time regression by upgrading.

Unfortunately the repo isn't (yet) open source, but I can share the package-lock.json's from both runs if any use and potentially the build log files if there's anything particular you need?

[slorber]

Unfortunately the repo isn't (yet) open source, but I can share the package-lock.json's from both runs if any use and potentially the build log files if there's anything particular you need?

@andrewgbell I'd have to run this locally myself, partially upgrading some libs in a dichotomic way to find out which transitive dep cause the problem. I doubt seeing a diff will be enough to identify the problem unfortunately, I need to run the code.

[ravilach]

@andrewgbell it looks like the build-time increase is not related to the 3.1 upgrade, but rather the upgrade of a transitive dependency that has a perf regression.

It would be super helpful for me to be able see/run that upgrade myself and study the package-lock.json diff.

Can someone share a site / branch that build faster in 3.0.1, and where I could reproduce the build time regression by upgrading.

Does the onBrokenAnchor always run but does not display results if ignore. If does not run, can weed that out.

@ravilach I'd recommend to try turning off both onBrokenLinks: "ignore" and onBrokenLinks: "ignore", because we only "bypass" the broken link checker if both are ignored atm.

I'll try to optimize that better in the future, but in the meantime the code looks like this:

  if (onBrokenLinks === 'ignore' && onBrokenAnchors === 'ignore') {
    return;
  }

  const brokenLinks = getBrokenLinks({
    routes,
    collectedLinks: normalizeCollectedLinks(collectedLinks),
  });

  reportBrokenLinks({brokenLinks, onBrokenLinks, onBrokenAnchors});

Note: is this possible that you encounter longer build times only due to cache eviction.

We use Webpack with persistent caching and on rebuilds it's supposed to rebuild faster.

It may be possible that your site builds longer simply because the caches were empty?

In this case I suggest trying to run docusaurus clear && docusaurus build on your "fast branch" and see if it becomes slower to build.

Unfortunately the repo isn't (yet) open source, but I can share the package-lock.json's from both runs if any use and potentially the build log files if there's anything particular you need?

@andrewgbell I'd have to run this locally myself, partially upgrading some libs in a dichotomic way to find out which transitive dep cause the problem. I doubt seeing a diff will be enough to identify the problem unfortunately, I need to run the code.

Ours is Open Source: https://github.com/harness/developer-hub if that helps. Currently on DS 3.0.1.

Here is the yarn.lock from the 3.1 upgrade: https://github.com/harness/developer-hub/blob/7b5fbafc4036f61d30e094362a67204cc573cf7a/yarn.lock

[andrewgbell]

@andrewgbell it looks like the build-time increase is not related to the 3.1 upgrade, but rather the upgrade of a transitive dependency that has a perf regression.
It would be super helpful for me to be able see/run that upgrade myself and study the package-lock.json diff.
Can someone share a site / branch that build faster in 3.0.1, and where I could reproduce the build time regression by upgrading.

Unfortunately the repo isn't (yet) open source, but I can share the package-lock.json's from both runs if any use and potentially the build log files if there's anything particular you need?

@slorber If you need another repo, let me know as I can invite you into our org.