Pushing application fails with BuildpackCompileFailed on Ubuntu Noble stemcell

[dimivel]

Current behavior

We started Ubuntu "noble" stemcell evaluation with this validation pipeline:
https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/noble-stemcell-validation

We also managed to successfully deploy CF and run smoke tests.

CF Acceptance tests run is failing when pushing java-buildpack application with the following error message:

cf push CATS-5-APP-a67b86fe48359dc9 -b java_buildpack -p assets/java -m 1024M -f assets/java/manifest.yml

  Pushing app CATS-5-APP-a67b86fe48359dc9 to org CATS-5-ORG-3b3cda5f08c9ece6 / space CATS-5-SPACE-20160fd7cf0c50a3 as CATS-5-USER-3da78c81c22328e9...
  Applying manifest file assets/java/manifest.yml...

  Updating with these attributes...
    ---
    applications:
  + - name: CATS-5-APP-a67b86fe48359dc9
      path: /tmp/build/33ac16d1/cf-acceptance-tests/assets/java
      memory: 1024M
  +   default-route: true
  +   buildpacks:
  +   - java_buildpack
  +   env:
  +     JAVA_OPTS: -Djava.security.egd=file:///dev/urandom
  Manifest applied
  Packaging files to upload...

Waiting for API to complete processing files...

  Staging app and tracing logs...
     Downloading java_buildpack...
     Downloaded java_buildpack (247.4K)
     Cell a6736615-7775-4211-9041-f79ce23c5668 creating container for instance f776eff6-a927-4d17-8d14-65bb9059d968
     Security group rules were updated
     Cell a6736615-7775-4211-9041-f79ce23c5668 successfully created container for instance f776eff6-a927-4d17-8d14-65bb9059d968
     Downloading app package...
     Downloaded app package (1.7K)
  BuildpackCompileFailed - App staging failed in the buildpack compile phase
  FAILED

cf logs --recent CATS-5-APP-a67b86fe48359dc9

Retrieving logs for app CATS-5-APP-a67b86fe48359dc9 in org CATS-5-ORG-3b3cda5f08c9ece6 / space CATS-5-SPACE-20160fd7cf0c50a3 as CATS-5-USER-3da78c81c22328e9...

     2025-01-20T09:22:31.63+0000 [API/1] OUT Added process: "web"
     2025-01-20T09:22:31.65+0000 [API/1] OUT Created app with guid 6994111d-63f5-4c8f-9a55-5e7273f244af
     2025-01-20T09:22:31.68+0000 [API/1] OUT Applied manifest to app with guid 6994111d-63f5-4c8f-9a55-5e7273f244af (---
     2025-01-20T09:22:31.68+0000 [API/1] OUT applications:
     2025-01-20T09:22:31.68+0000 [API/1] OUT - name: CATS-5-APP-a67b86fe48359dc9
     2025-01-20T09:22:31.68+0000 [API/1] OUT path: "/tmp/build/33ac16d1/cf-acceptance-tests/assets/java"
     2025-01-20T09:22:31.68+0000 [API/1] OUT memory: 1024M
     2025-01-20T09:22:31.68+0000 [API/1] OUT default-route: true
     2025-01-20T09:22:31.68+0000 [API/1] OUT buildpacks:
     2025-01-20T09:22:31.69+0000 [API/1] OUT - java_buildpack
     2025-01-20T09:22:31.69+0000 [API/1] OUT env: "[PRIVATE DATA HIDDEN]"
     2025-01-20T09:22:31.69+0000 [API/1] OUT )
     2025-01-20T09:22:38.07+0000 [API/1] OUT Uploading app package for app with guid 6994111d-63f5-4c8f-9a55-5e7273f244af
     2025-01-20T09:22:41.33+0000 [API/1] OUT Creating build for app with guid 6994111d-63f5-4c8f-9a55-5e7273f244af
     2025-01-20T09:22:42.10+0000 [STG/0] OUT Downloading java_buildpack...
     2025-01-20T09:22:42.16+0000 [STG/0] OUT Downloaded java_buildpack (247.4K)
     2025-01-20T09:22:42.16+0000 [STG/0] OUT Cell a6736615-7775-4211-9041-f79ce23c5668 creating container for instance f776eff6-a927-4d17-8d14-65bb9059d968
     2025-01-20T09:22:42.54+0000 [STG/0] OUT Security group rules were updated
     2025-01-20T09:22:42.56+0000 [STG/0] OUT Cell a6736615-7775-4211-9041-f79ce23c5668 successfully created container for instance f776eff6-a927-4d17-8d14-65bb9059d968
     2025-01-20T09:22:43.15+0000 [STG/0] OUT Downloading app package...
     2025-01-20T09:22:43.18+0000 [STG/0] OUT Downloaded app package (1.7K)
     2025-01-20T09:22:43.38+0000 [STG/0] ERR Failed to compile droplet: Failed to run finalize script: exit status 6
     2025-01-20T09:22:43.44+0000 [STG/0] OUT Exit status 223
     2025-01-20T09:22:44.00+0000 [STG/0] OUT Cell a6736615-7775-4211-9041-f79ce23c5668 stopping instance f776eff6-a927-4d17-8d14-65bb9059d968
     2025-01-20T09:22:44.00+0000 [STG/0] OUT Cell a6736615-7775-4211-9041-f79ce23c5668 destroying container for instance f776eff6-a927-4d17-8d14-65bb9059d968
     2025-01-20T09:22:44.04+0000 [API/0] ERR Failed to stage build: staging failed
     2025-01-20T09:22:44.42+0000 [STG/0] OUT Cell a6736615-7775-4211-9041-f79ce23c5668 successfully destroyed container for instance f776eff6-a927-4d17-8d14-65bb9059d968

Expected behavior

Successfully push and start application with java-buildpack on Ubuntu Noble stemcell

Steps to Reproduce

The compilation failure is reproducible in the "Noble" stemcell validation pipeline:
https://concourse.wg-ard.ci.cloudfoundry.org/teams/main/pipelines/noble-stemcell-validation/jobs/run-cats/builds/1

[beyhan]

@dimivel Are you running the test with the latest stated of the buildpack from the main branch? Could you try to run the test with the latest released version. If you are using manifest file you can specify the release tag with buildpack: https://github.com/cloudfoundry/java-buildpack.git@4.74.0 or if you are using the CLI cf push -b https://github.com/cloudfoundry/java-buildpack.git@4.74.0. I suspect a general issue introduced by e5c8eb3 which is not yet included in the latest 4.74.0 version and not a Noble stemcell issue.

[dimivel]

hi @beyhan,
The test are running in general with the latest released version 7.47.0.

root@1e31a47f8ef2:/home/bbl/bbl-state/cf-acceptance-tests# cf buildpacks
Getting buildpacks as admin...

position   name                    stack        enabled   locked   state   filename
1          staticfile_buildpack    cflinuxfs4   true      false    READY   staticfile_buildpack-cflinuxfs4-v1.6.18.zip
2          java_buildpack          cflinuxfs4   true      false    READY   java-buildpack-cflinuxfs4-v4.74.0.zip

cf push CATS-5-APP-fee7153f770293a3 -b java_buildpack -p assets/java -m 1024M -f assets/java/manifest.yml
Pushing app CATS-5-APP-fee7153f770293a3 to org system / space dev-spave as admin...
Applying manifest file assets/java/manifest.yml...

Updating with these attributes...
  ---
  applications:
  - name: CATS-5-APP-fee7153f770293a3
    path: /home/bbl/bbl-state/cf-acceptance-tests/assets/java
    memory: 1024M
+   default-route: true
    buildpacks:
+   - java_buildpack
-   - https://github.com/cloudfoundry/java-buildpack.git@v4.74.0
    env:
      JAVA_OPTS: -Djava.security.egd=file:///dev/urandom
Manifest applied
Packaging files to upload...
Uploading files...
 1.68 KiB / 1.68 KiB [==============================================================================================================================================================================================================================================] 100.00% 1s

Waiting for API to complete processing files...

Staging app and tracing logs...
   Downloading java_buildpack...
   Downloaded java_buildpack
   Cell 526fb1a0-0c65-43c5-98b1-c111040d7171 creating container for instance 18a3bd80-2020-4849-a8f2-9faa53a91132
   Security group rules were updated
   Cell 526fb1a0-0c65-43c5-98b1-c111040d7171 successfully created container for instance 18a3bd80-2020-4849-a8f2-9faa53a91132
   Downloading app package...
   Downloaded app package (1.7K)
   Failed to compile droplet: Failed to run finalize script: exit status 6
   Exit status 223
   Cell 526fb1a0-0c65-43c5-98b1-c111040d7171 stopping instance 18a3bd80-2020-4849-a8f2-9faa53a91132
   Cell 526fb1a0-0c65-43c5-98b1-c111040d7171 destroying container for instance 18a3bd80-2020-4849-a8f2-9faa53a91132
   Cell 526fb1a0-0c65-43c5-98b1-c111040d7171 successfully destroyed container for instance 18a3bd80-2020-4849-a8f2-9faa53a91132
BuildpackCompileFailed - App staging failed in the buildpack compile phase

It seems to be a Noble stemcell issue.

[ramonskie]

can we get any logs on the compile phase?
as Failed to compile droplet: Failed to run finalize script: exit status 6 is to generic.

[beyhan]

@dimivel you could try with https://github.com/cloudfoundry/java-buildpack/blob/main/docs/debugging-the-buildpack.md?plain=1#L11-L82 to get Debug logs from the staging process.

[dimivel]

hi @beyhan,
I'll collect Debug logs based on the guidance and provide feedback.

Thanks!
Dimitar

[dimivel]

hi @beyhan ,
I tried to adjust the log level to debug and push the application again. There is no dump of additional logs.

root@9f3a3c752ad2:/home/bbl/cf-acceptance-tests# cf set-env CATS-8-APP-a057160d46a3dbe3 JBP_LOG_LEVEL DEBUG
Setting env variable JBP_LOG_LEVEL for app CATS-8-APP-a057160d46a3dbe3 in org system / space my-new-space as admin...
OK

TIP: Use 'cf restage CATS-8-APP-a057160d46a3dbe3' to ensure your env variable changes take effect.
root@9f3a3c752ad2:/home/bbl/cf-acceptance-tests# cf restage CATS-8-APP-a057160d46a3dbe3
This action will cause app downtime.

Restaging app CATS-8-APP-a057160d46a3dbe3 in org system / space my-new-space as admin...

Staging app and tracing logs...
   Downloading java_buildpack...
   Downloaded java_buildpack
   Cell 3d15a1d5-94a0-43ac-8e59-54e4a272c1ab creating container for instance 1df8c26d-ec28-4e5f-ab1e-33ac37b29b23
   Security group rules were updated
   Cell 3d15a1d5-94a0-43ac-8e59-54e4a272c1ab successfully created container for instance 1df8c26d-ec28-4e5f-ab1e-33ac37b29b23
   Downloading app package...
Error staging application: BuildpackCompileFailed - App staging failed in the buildpack compile phase
FAILED
root@9f3a3c752ad2:/home/bbl/cf-acceptance-tests# cf push CATS-8-APP-a057160d46a3dbe3 -b java_buildpack -m 1024M -p assets/spring-sleuth/spring-sleuth.jar -t 120
Pushing app CATS-8-APP-a057160d46a3dbe3 to org system / space my-new-space as admin...
Packaging files to upload...
Uploading files...
 863.97 KiB / 863.97 KiB [=======================================================================================================================================================================] 100.00% 1s

Waiting for API to complete processing files...

Staging app and tracing logs...
   Downloading java_buildpack...
   Downloaded java_buildpack
   Cell dfad2c13-4e11-4850-ada7-255eebae4940 creating container for instance 642be72c-87ea-414f-9efc-e7aeb33836f7
   Security group rules were updated
   Cell dfad2c13-4e11-4850-ada7-255eebae4940 successfully created container for instance 642be72c-87ea-414f-9efc-e7aeb33836f7
   Downloading app package...
BuildpackCompileFailed - App staging failed in the buildpack compile phase
FAILED
root@9f3a3c752ad2:/home/bbl/cf-acceptance-tests#

[beyhan]

I debugged the issue by using the ubuntu-noble-stemcell image and following this guide to debug a stemcell locally. What I did is:

git clone git@github.com:cloudfoundry/java-buildpack/
git clone git@github.com:cloudfoundry/cf-acceptance-tests.git

docker pull ghcr.io/cloudfoundry/ubuntu-noble-stemcell:0.26
docker run -it -v ./java-buildpack:/mnt/java-buildpack -v ./cf-acceptance-tests/assets:/mnt/debug-apps ghcr.io/cloudfoundry/ubuntu-noble-stemcell bash

// inside the container
cd /mnt/debug-apps
export VCAP_APPLICATION="{}"
export VCAP_SERVICES="{}"
export PORT=8080
JBP_LOG_LEVEL=DEBUG ../java-buildpack/bin/compile java/

The error you get by executing this is:

<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>jvmkill/noble/x86_64/index.yml</Key><RequestId>8B82KG9E3SFA8TPZ</RequestId><HostId>8F2xnhlZU5DmGYEZkluGxAVq6+uNZgnu+8Q4hqnYQ+QaC9571G2aZObri6J24lQdcNrT4cwK+qw=</HostId></Error>
[DownloadCache]                  DEBUG Response headers: {"date"=>["Fri, 14 Feb 2025 08:22:28 GMT"], "content-type"=>["application/xml"], "transfer-encoding"=>["chunked"], "connection"=>["keep-alive"], "x-amz-request-id"=>["8B82KG9E3SFA8TPZ"], "x-amz-id-2"=>["8F2xnhlZU5DmGYEZkluGxAVq6+uNZgnu+8Q4hqnYQ+QaC9571G2aZObri6J24lQdcNrT4cwK+qw="], "cf-cache-status"=>["HIT"], "age"=>["75"], "expires"=>["Fri, 14 Feb 2025 12:22:28 GMT"], "cache-control"=>["public, max-age=14400"], "vary"=>["Accept-Encoding"], "server"=>["cloudflare"], "cf-ray"=>["911ba68c8984bbdd-FRA"], "content-encoding"=>["gzip"]}
[DownloadCache]                  DEBUG Response status: 404
[DownloadCache]                  WARN  Request failure 5, retrying after 60s.  Failure: 404 Not Found
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Key>jvmkill/noble/x86_64/index.yml</Key><RequestId>8B82KG9E3SFA8TPZ</RequestId><HostId>8F2xnhlZU5DmGYEZkluGxAVq6+uNZgnu+8Q4hqnYQ+QaC9571G2aZObri6J24lQdcNrT4cwK+qw=</HostId></Error>
^C/mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:171:in `sleep': Interrupt
	from /mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:171:in `rescue in attempt_update'
	from /mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:162:in `attempt_update'
	from /mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:356:in `block in update'
	from /tmp/java-buildpack/ruby/lib/ruby/3.2.0/net/http.rb:1238:in `start'
	from /tmp/java-buildpack/ruby/lib/ruby/3.2.0/net/http.rb:687:in `start'
	from /mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:352:in `update'
	from /mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:268:in `from_mutable_cache'
	from /mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:72:in `get'
	from /mnt/java-buildpack/lib/java_buildpack/repository/repository_index.rb:41:in `initialize'
	from /mnt/java-buildpack/lib/java_buildpack/repository/configured_item.rb:63:in `new'
	from /mnt/java-buildpack/lib/java_buildpack/repository/configured_item.rb:63:in `index'
	from /mnt/java-buildpack/lib/java_buildpack/repository/configured_item.rb:48:in `find_item'
	from /mnt/java-buildpack/lib/java_buildpack/component/versioned_dependency_component.rb:41:in `initialize'
	from /mnt/java-buildpack/lib/java_buildpack/jre/open_jdk_like.rb:41:in `new'
	from /mnt/java-buildpack/lib/java_buildpack/jre/open_jdk_like.rb:41:in `sub_components'
	from /mnt/java-buildpack/lib/java_buildpack/component/modular_component.rb:40:in `initialize'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:204:in `new'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:204:in `block in instantiate'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:188:in `map'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:188:in `instantiate'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:159:in `instantiate_components'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:153:in `initialize'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:278:in `new'
	from /mnt/java-buildpack/lib/java_buildpack/buildpack.rb:278:in `with_buildpack'
	from /mnt/java-buildpack/bin/ruby-run:31:in `<main>'
/mnt/java-buildpack/lib/java_buildpack/util/cache/download_cache.rb:150:in `block in attempt': 404 Not Found (JavaBuildpack::Util::Cache::InferredNetworkFailure)

The buildpack is trying to download a dependency from https://java-buildpack.cloudfoundry.org/jvmkill/noble/x86_64/index.yml which doesn't exist. While in my setup there is internet connectivity there is an issue at the moment open for Noble regarding DNS issue inside the staging container. I expect after solving the DNS issue to face the missing dependency issue. I'm not sure how the dependencies list in https://java-buildpack.cloudfoundry.org/ get updated. I can see that there are dependencies for previous stemcell versions like Jammy but nothing for Noble. @pivotal-david-osullivan, @anthonydahanne is publishing those dependencies part of the buildpack release process?

[pivotal-david-osullivan]

It's not part of the release process itself but a separate dependency-updater one.

However the kill agent is compiled per stack as opposed to most other deps which are simply copied under each supported stack as there is no tie to the underlying OS.

The JVM Kill Agent will need to be updated & deployed for Noble before the buildpack can support the stack

[beyhan]

Hi @pivotal-david-osullivan,

thank you for the insights. Based on your explanation I think my test wasn't fully correct because I executed the Java Buildpack in the Noble Stemcell docker container that is why the buildpack tries to download the JVM Kill Agent for Nobel. In a real CF environment this will be executed on a Noble Stemcell but inside of a container with the Jammy (cflinuxfs4) stack which means the JVM Kill Agent for Jammy will be required. Based on this I'm assuming that the root cause for this issue is the missing DNS resolution discussed in cloudfoundry/diego-release#987 where Java Buildpack fails to download required dependencies.

[jochenehret]

We've applied the latest fixes for the BOSH DNS issue: cloudfoundry/diego-release#987 (comment)

Now the CATs are mostly green, including the Java buildpack tests. I think we can close this issue and track cloudfoundry/diego-release#987 instead.