Legacy Wiki.js GraphQL performance when adding content

Legacy notes for the old Wiki.js deployment, kept for reference after the wiki migration to llm-wiki/.

Why it slows down

• Page tree rebuild: On each page create or update, Wiki.js rebuilds the entire page tree in memory. It does not use streaming for this, so with many pages the process can use a lot of RAM and block other requests (Wiki.js discussion #5602).
• Node.js heap: The app runs on Node.js. Under heavy GraphQL write load, the default V8 heap limit can be hit, leading to slowness or crashes.
• Database: With database search enabled, the DB can also become a bottleneck as the corpus grows.

So slowdown when “adding content via the GraphQL endpoint” is often a combination of page-tree rebuild cost and memory pressure, not necessarily a classic memory leak (though repeated rebuilds without enough GC can look like one).

Is this a memory leak? Can we fix it?

It's design-driven memory growth, not a classic leak. A traditional leak is a bug where references are never released. Here, Wiki.js intentionally rebuilds the entire page tree in memory on every create/update; that allocation is large and may not be fully reclaimed by GC before the next rebuild. So memory grows with each write and only drops after a restart. The behavior looks like a leak but comes from how the app is built.

What we can do (without changing Wiki.js itself):

• Restart when it gets slow — Clears state and heap; you already saw sync get much faster after a restart.
• Throttle writes — Delays between mutations give GC time and reduce peak load (your sync script already uses 5 s in prod, 100 ms local).
• Give Node more heap — NODE_OPTIONS=--max-old-space-size=2048 so the process doesn't hit the default limit as quickly.
• Restart periodically — e.g. restart the wiki container after a big sync or on a schedule if you do heavy GraphQL often.

What would require an upstream change: A real "fix" would be in requarks/wiki: e.g. incremental or streaming page-tree updates instead of a full rebuild every time. The maintainers are aware of the limitation (discussion #5602). We can't patch that from our repo; we use the official Docker image. So for now the practical approach is the mitigations below.

Mitigations

1. Give Wiki.js more memory (production and local)

Increase the Node.js heap for the Wiki.js container so it can handle large page-tree rebuilds.

Production (DigitalOcean droplet)
If you control the Wiki.js run (e.g. docker-compose or docker run), add:

# In the wiki service environment
environment:
  NODE_OPTIONS: "--max-old-space-size=2048"

So the wiki container runs with a 2 GB heap. For a 2 GB droplet, 2048 is reasonable; for 4 GB, you can try 3072. Leave headroom for the OS and Postgres.

Local dev (dev/wikijs)
Heap size is configurable via .env:

# In dev/wikijs/.env — default is 2048 (MB). For a large import (e.g. full Synergetics Dictionary), increase:
WIKI_NODE_MAX_OLD_SPACE_SIZE=8192

Then restart so the wiki container picks it up: docker compose up -d (or docker compose restart wiki). Set it back to 2048 or remove the line after the import if you want to free host memory.

2. Rate-limit client requests (script/code that calls GraphQL)

If you add many pages in a loop (e.g. from a script using the pages.create mutation):

• Add a delay between mutations (e.g. 1–2 seconds per page, or 500 ms). That spreads out page-tree rebuilds and gives the server time to GC.
• Process in small batches (e.g. 50 or 100 pages), then pause longer (e.g. 10–30 s) between batches.
• Avoid parallel bursts of many concurrent mutations; prefer sequential or low concurrency.

Example pattern (pseudo):

for each page:
  call GraphQL pages.create(...)
  sleep 1.5   # seconds
  if (count % 50 == 0): sleep 15   # longer pause every 50 pages

This reduces the chance of overloading the single rebuild process and keeps the wiki responsive.

3. Docker memory limit (production)

Ensure the wiki container is allowed enough RAM so it isn’t OOM-killed:

# wiki service
deploy:
  resources:
    limits:
      memory: 2G

Adjust to match your droplet size. The container should have at least as much as max-old-space-size plus some overhead.

4. Restart the wiki container (quick win)

If sync has gotten slow after a lot of GraphQL writes, restarting the wiki container often restores speed: it clears in-memory state and resets the Node heap. For local dev: cd dev/wikijs && docker compose restart. On production, restart the wiki container (e.g. docker restart wiki). No need to restart the DB.

5. Large import: more memory and “pause” page tree?

Increase memory: For a large one-off sync (e.g. full Synergetics Dictionary), raise the Node heap so each rebuild has room. In dev/wikijs/.env set:

WIKI_NODE_MAX_OLD_SPACE_SIZE=8192

Restart the wiki container, run your sync (e.g. elixir scripts/sync_cards_to_buckyverse.exs --local), then optionally set it back to 2048 or remove the line.

Pause page tree rebuild until sync is complete: Wiki.js 2.x does not support this. The app rebuilds the page tree on every single create/update; there is no setting or API to defer or batch rebuilds. So you cannot “pause” rebuilding until the import is done. Options in practice:

• Sync locally with a bigger heap (above), throttle the script (your 100 ms delay with --local is good), and restart the wiki once or twice if it gets slow mid-run. When done, push the DB to production with dev/wikijs/scripts/push-to-prod.sh so you don’t have to run the same long sync on prod.
• Use a Git storage backend for the dictionary (if you can point Wiki.js at a repo that contains the markdown). Then you push content to Git and trigger one sync in Wiki.js; the tree is rebuilt once after that sync instead of after every page. That would require changing how the dictionary content is fed (Git instead of GraphQL).
• Wiki.js 3.x (when stable) is expected to avoid full tree rebuilds on every write; upgrading later would address this at the source.

6. Find large pages for removal (Synergetics Dictionary)

Old Synergetics Dictionary imports can include huge index pages (e.g. A.md per letter) that make page-tree rebuilds slower. To list pages by content size and find candidates for removal:

Local dev (after pull-from-prod.sh):

cd dev/wikijs
./scripts/list-large-pages.sh 50                    # top 50 by size
./scripts/list-large-pages.sh 30 synergetics        # top 30 with path containing 'synergetics'

The script queries the pages table and orders by length(content). Remove large index pages via the Wiki.js UI (delete page) or GraphQL pages.delete if they’re redundant (e.g. replaced by per-card pages). Fewer large pages reduce rebuild cost.

7. Large payloads (413 Payload Too Large)

If you hit 413 when saving very long pages via GraphQL, the request body is too large for the reverse proxy or the app. Options:

• Chunk the content into smaller pages, or
• Increase client/nginx body size limits if you control the server (e.g. client_max_body_size in nginx).

Summary

Wiki.js’s page-tree rebuild on every write is an upstream design constraint; the above steps reduce its impact and the risk of slowdowns or out-of-memory when adding content via the GraphQL endpoint.