# Severe slowdown on _temp with k8s-novolume hooks (v0.13.0) and actions/setup-go

[mcammisa78]

Why are you starting this discussion?

Bug

What GitHub Actions topic or product is this about?

ARC (Actions Runner Controller)

Discussion Details

Severe slowdown on _temp with k8s-novolume hooks (v0.13.0) and actions/setup-go

Environment

• Controller: actions-runner-controller
• Runner type: kubernetes with type: novolume
• Container hooks: ACTIONS_RUNNER_CONTAINER_HOOKS=/home/runner/k8s-novolume/index.js
• Hooks version: k8s-novolume 0.13.0
• Filesystem in pods: overlay filesystem, no PVC, ephemeral storage only
• Language/stack: Go, using:
  • actions/setup-go@v5
  • go test + coverage
  • actions/upload-artifact@v4
  • dorny/test-reporter@v2

There are two pods involved per job:

• Runner pod (GitHub Runner)
• Workflow pod (where the job actually runs, /__w/...)

---

Symptoms

During the job, each step logs the container hook:

Run '/home/runner/k8s-novolume/index.js'

(node:643494) [DEP0005] DeprecationWarning: Buffer() is deprecated ...

For some steps (especially later ones like actions/upload-artifact@v4 and dorny/test-reporter@v2), the workflow appears to hang for tens of minutes while this Node process is running, even though the actual user commands (e.g. go test) have already completed.

Example around actions/upload-artifact@v4:

Run actions/upload-artifact@v4
Run '/home/runner/k8s-novolume/index.js'

(node:643494) [DEP0005] DeprecationWarning: Buffer() is deprecated ...
GITHUB_EVENT_PATH /github/workflow/event.json does not exist
GITHUB_EVENT_PATH /github/workflow/event.json does not exist
GITHUB_EVENT_PATH /github/workflow/event.json does not exist
GITHUB_EVENT_PATH /github/workflow/event.json does not exist
Multiple search paths detected. Calculating the least common ancestor of all paths
The least common ancestor is /__w/CI-hub-templates/CI-hub-templates. This will be the root directory of the artifact
...
Artifact go-test-results has been successfully uploaded!

And then again for dorny/test-reporter@v2:

Run dorny/test-reporter@v2
Run '/home/runner/k8s-novolume/index.js'

(node:867916) [DEP0005] DeprecationWarning: Buffer() is deprecated ...

The overall job runtime was ~50+ minutes, with a large portion of time spent around these hook invocations.

---

What index.js is doing (find + stat on _temp)

In the workflow pod

From the workflow pod (as root):

ps -ef

Shows:

root      260513       0  0 10:25 ?        00:00:00 sh -c cd /__w/_temp && find . -not -path '*/_runner_hook_responses*' -exec stat -c '%b %n' {} \;
root      260519  260513 12 10:25 ?        00:00:00 find . -not -path */_runner_hook_responses* -exec stat -c %b %n {} ;

In the runner pod

From the runner pod (as runner):

ps -ef

Shows:

runner    573192  418946  0 10:35 ?        00:00:00 sh -c cd /home/runner/_work/_temp && find . -not -path '*/_runner_hook_responses*' -exec stat -c '%b %n' {} \;
runner    573193  573192  5 10:35 ?        00:00:01 find . -not -path */_runner_hook_responses* -exec stat -c %b %n {} ;

So index.js is:

• cd-ing into _temp in each pod, and

• running:

  find . -not -path '*/_runner_hook_responses*' -exec stat -c '%b %n' {} \;

This looks like a disk-usage / quota check for novolume mode, but it becomes very expensive when _temp is large.

---

Size of _temp and I/O characteristics

In the workflow pod under /__w/_temp:

cd /__w/_temp
du -sh .
# 309M    .

find . | wc -l
# 14026

So _temp has ~300MB and ~14k files.

I/O on the underlying storage is actually quite fast (tested in the runner pod):

cd /home/runner/_work

dd if=/dev/zero of=./io-test.bin bs=1M count=1024 oflag=direct
# 1073741824 bytes (1.1 GB) copied, 1.47938 s, 726 MB/s

dd if=./io-test.bin of=/dev/null bs=1M
# 1073741824 bytes (1.1 GB) copied, 3.39752 s, 316 MB/s

So the underlying disk is not the bottleneck. The expensive part is the repeated find/stat over a large _temp tree.

---

How actions/setup-go fills _temp

The Go setup step logs:

/usr/bin/tar xz --warning=no-unknown-keyword --overwrite \
  -C /__w/_temp/6b6aec57-c121-4e8c-b914-f258b6092b81 \
  -f /__w/_temp/84ad1d42-cbfe-4249-b723-04bc0be91383

Successfully extracted go to /__w/_temp/6b6aec57-c121-4e8c-b914-f258b6092b81
Adding to the cache ...
Successfully cached go to /__w/_tool/go/1.21.8/x64
Added go to the path
Successfully set up Go version 1.21.8

So actions/setup-go@v5:

• extracts the Go toolchain into /__w/_temp/<guid>/...
• caches it under /__w/_tool/go/1.21.8/x64
• leaves additional data in _temp as part of normal operation

This explains why _temp grows significantly during the job.

---

Key experiment: deleting go setup tmp dir under _temp makes the job fast

While a job was in a “slow” phase (around upload-artifact / test-reporter, with index.js active), I tried manually cleaning _temp in both pods.

In the runner pod

rm -rf /home/runner/_work/_temp/<go-setup-guid-folder>

In the workflow pod

rm -rf /__w/_temp/<go-setup-guid-folder>

After doing this:

• the ongoing step completed within minutes instead of tens of minutes,
• the rest of the workflow also finished quickly.

So:

• With _temp ~300MB / 14k files → index.js spends a long time scanning _temp with find/stat.
• With _temp emptied → the same hooks complete very quickly and the job runtime drops dramatically.

This strongly suggests the disk-usage scan in _temp is the main bottleneck in novolume mode for this scenario.

---

Workflow action (for context)

The main composite action used for tests:

name: Go Unit Test
description: Run Go unit tests with coverage and JUnit reporting.

inputs:
  go-mod-path:
    description: 'Path to the go.mod file. Default is the root of the repository.'
    required: false
    default: './'
  go-dependencies-to-install:
    description: 'Bash commands to install system dependencies.'
    required: false
    default: ""
  go-test-script:
    description: 'Custom Go test script to run.'
    required: false
    default: 'go test -v ./...'

runs:
  using: "composite"
  steps:
    - name: Install system dependencies
      run: |
        apt-get update && apt-get install -y curl jq git
        curl -L https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -o yq
        chmod +x yq && mv yq /usr/local/bin/yq
      shell: bash

    - name: Mark repository as safe
      run: git config --global --add safe.directory $GITHUB_WORKSPACE
      shell: bash

    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Check go.mod presence and Go version match
      id: check_go_version
      shell: bash
      run: |
        GO_MOD_PATH_INPUT="${{ inputs.go-mod-path }}"
        echo "GO_MOD_PATH input: ${GO_MOD_PATH_INPUT}"

        if [ -f "go.mod" ]; then
          echo "✅ go.mod found at root."
          GO_MOD_PATH=$(pwd)
        elif [ -n "${GO_MOD_PATH_INPUT}" ] && [ -f "${GO_MOD_PATH_INPUT}/go.mod" ]; then
          echo "✅ go.mod found at custom path: ${GO_MOD_PATH_INPUT}"
          cd "${GO_MOD_PATH_INPUT}"
          GO_MOD_PATH=$(pwd)
        else
          echo "❌ go.mod not found."
          exit 1
        fi
        echo "go-mod-dir=${GO_MOD_PATH}" >> $GITHUB_OUTPUT

    - name: Setup Go Dependencies
      if: ${{ inputs.go-dependencies-to-install != '' }}
      uses: ALM/CIHub/.github/actions/[email protected]
      with:
        commands: ${{ inputs.go-dependencies-to-install }}

    - name: Set up Go Version
      uses: actions/setup-go@v5
      with:
        go-version-file: ${{ steps.check_go_version.outputs.go-mod-dir }}/go.mod

    - name: Run Go Tests with Coverage
      shell: bash
      run: |
        mkdir -p coverage reports
        go install github.com/jstemmer/go-junit-report@latest
        export PATH="$HOME/go/bin:$PATH"

        read -ra GO_TEST_CMD <<< "${{ inputs.go-test-script }}"
        GO_TEST_CMD+=("-cover" "-coverprofile=coverage.out")

        echo "▶️ Running: ${GO_TEST_CMD[@]}"
        "${GO_TEST_CMD[@]}" 2>&1 | tee test_output.txt || true
        TEST_EXIT_CODE=${PIPESTATUS[0]}

        go-junit-report < test_output.txt > reports/junit.xml
        go tool cover -html=coverage.out -o coverage/coverage.html
        go tool cover -func=coverage.out > coverage/coverage.txt

        exit $TEST_EXIT_CODE

    - name: Upload Go test results
      if: always()
      uses: actions/upload-artifact@v4
      with:
        name: go-test-results
        path: |
          coverage/coverage.html
          coverage/coverage.txt
          coverage.out
          reports/junit.xml

    - name: Upload JUnit test results
      if: success()
      uses: dorny/test-reporter@v2
      with:
        name: Go Unit Tests
        path: reports/junit.xml
        reporter: java-junit

---

Questions / Requests

1. Is it expected that the k8s-novolume hooks (v0.13.0) run:

   cd /__w/_temp      # or /home/runner/_work/_temp
   find . -not -path '*/_runner_hook_responses*' -exec stat -c '%b %n' {} \;

   on every hook invocation?

2. Is there any configuration / environment variable to:

   • disable or relax this disk-usage scanning, or
   • limit it to a smaller directory, or
   • reduce its frequency?

3. Since actions/setup-go and other actions naturally populate _temp with many files, this scan becomes a major bottleneck in novolume mode.
   Is there a recommended way to:

   • point the hooks to a different directory than _temp, or
   • split _temp and _work onto different mounts/paths specifically for the hook logic?

Right now, the only reliable workaround I have found is to manually contents of _temp in both pods during the job, which immediately speeds up the workflow, probably there is a "copy" of that temp dir between runner pod and pod workload. Obviously this is not ideal, so any guidance or improvements around disk-usage checks in the k8s-novolume hooks would be very helpful.

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐