rubygems test fail with latest dependencies

[Segaja]

Running rake test on version v3.3.21 I get this result:

Loaded suite /usr/lib/ruby/gems/3.0.0/gems/rake-13.0.6/lib/rake/rake_test_loader
Started
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...........................................................F
===============================================================================
Failure: test_build_extensions(TestGemExtBuilder): <"/build/rubygems/src/rubygems-3.3.21/tmp/test_rubygems_20220925-1122-f7jhor/gemhome/gems/a-2/lib/a.rb"> was expected to exist
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_builder.rb:140:in `test_build_extensions'
     137:     assert_path_exist @spec.gem_build_complete_path
     138:     assert_path_exist File.join @spec.extension_dir, "gem_make.out"
     139:     assert_path_exist File.join @spec.extension_dir, "a.rb"
  => 140:     assert_path_exist File.join @spec.gem_dir, "lib", "a.rb"
     141:     assert_path_exist File.join @spec.gem_dir, "lib", "a", "b.rb"
     142:   end
     143:
===============================================================================
......F
===============================================================================
Failure: test_build_extensions_with_gemhome_with_space(TestGemExtBuilder): <"/build/rubygems/src/rubygems-3.3.21/tmp/test_rubygems_20220925-1122-4fif2k/gem home/gems/a-2/lib/a.rb"> was expected to exist
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_builder.rb:140:in `test_build_extensions'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_builder.rb:153:in `test_build_extensions_with_gemhome_with_space'
     150:     @spec = util_spec "a"
     151:     @builder = Gem::Ext::Builder.new @spec, ""
     152:
  => 153:     test_build_extensions
     154:   end
     155:
     156:   def test_build_extensions_install_ext_only
===============================================================================
.....P
===============================================================================
Pending: test_build_cdylib(TestGemExtCargoBuilder): cargo not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:169:in `skip_unsupported_platforms!'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:45:in `test_build_cdylib'
     42:   end
     43:
     44:   def test_build_cdylib
  => 45:     skip_unsupported_platforms!
     46:     setup_rust_gem "rust_ruby_example"
     47:
     48:     output = []
===============================================================================
P
===============================================================================
Pending: test_build_dev_profile(TestGemExtCargoBuilder): cargo not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:169:in `skip_unsupported_platforms!'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:69:in `test_build_dev_profile'
     66:   end
     67:
     68:   def test_build_dev_profile
  => 69:     skip_unsupported_platforms!
     70:     setup_rust_gem "rust_ruby_example"
     71:
     72:     output = []
===============================================================================
P
===============================================================================
Pending: test_build_fail(TestGemExtCargoBuilder): cargo not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:169:in `skip_unsupported_platforms!'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:94:in `test_build_fail'
     91:   end
     92:
     93:   def test_build_fail
  => 94:     skip_unsupported_platforms!
     95:     setup_rust_gem "rust_ruby_example"
     96:
     97:     output = []
===============================================================================
P
===============================================================================
Pending: test_build_staticlib(TestGemExtCargoBuilder): cargo not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:169:in `skip_unsupported_platforms!'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:26:in `test_build_staticlib'
     23:   end
     24:
     25:   def test_build_staticlib
  => 26:     skip_unsupported_platforms!
     27:     setup_rust_gem "rust_ruby_example"
     28:
     29:     content = @fixture_dir.join("Cargo.toml").read.gsub("cdylib", "staticlib")
===============================================================================
P
===============================================================================
Pending: test_custom_name(TestGemExtCargoBuilder): cargo not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:169:in `skip_unsupported_platforms!'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:140:in `test_custom_name'
     137:   end
     138:
     139:   def test_custom_name
  => 140:     skip_unsupported_platforms!
     141:     setup_rust_gem "custom_name"
     142:
     143:     Dir.chdir @ext do
===============================================================================
P
===============================================================================
Pending: test_full_integration(TestGemExtCargoBuilder): cargo not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:169:in `skip_unsupported_platforms!'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cargo_builder.rb:116:in `test_full_integration'
     113:   end
     114:
     115:   def test_full_integration
  => 116:     skip_unsupported_platforms!
     117:     setup_rust_gem "rust_ruby_example"
     118:
     119:     require "open3"
===============================================================================
......................P
===============================================================================
Pending: test_self_build(TestGemExtCmakeBuilder): cmake not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cmake_builder.rb:18:in `rescue in setup'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cmake_builder.rb:14:in `setup'
===============================================================================
P
===============================================================================
Pending: test_self_build_fail(TestGemExtCmakeBuilder): cmake not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cmake_builder.rb:18:in `rescue in setup'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cmake_builder.rb:14:in `setup'
===============================================================================
P
===============================================================================
Pending: test_self_build_has_makefile(TestGemExtCmakeBuilder): cmake not present
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cmake_builder.rb:18:in `rescue in setup'
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_gem_ext_cmake_builder.rb:14:in `setup'
===============================================================================
...............................................................................
...............................................................................
...............................................................................
...............................................................................
..................................................................F
===============================================================================
Failure: test_realworld_default_gem(TestGemRequire): <""> was expected to not be empty.
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_require.rb:456:in `test_realworld_default_gem'
     453:     RUBY
     454:     output = Gem::Util.popen(*ruby_with_rubygems_in_load_path, "-e", cmd).strip
     455:     assert $?.success?
  => 456:     refute_empty output
     457:   end
     458:
     459:   def test_realworld_upgraded_default_gem
===============================================================================
F
===============================================================================
Failure: test_realworld_upgraded_default_gem(TestGemRequire): <false> is not true.
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_require.rb:475:in `test_realworld_upgraded_default_gem'
     472:     File.write(path, code)
     473:
     474:     output = Gem::Util.popen({ "GEM_HOME" => @gemhome }, *ruby_with_rubygems_in_load_path, path).strip
  => 475:     assert $?.success?
     476:     refute_empty output
     477:     assert_equal "999.99.9", output.lines[0].chomp
     478:     # Make sure only files from the newer json gem are loaded, and no files from the default json gem
===============================================================================
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...............................................................................
...................................................F
===============================================================================
Failure: test_manifest_is_up_to_date(TestProjectSanity):
  Expected Manifest.txt to be up to date, but it's not. Run `rake update_manifest` to sync it.
  <false> is not true.
/build/rubygems/src/rubygems-3.3.21/test/rubygems/test_project_sanity.rb:12:in `test_manifest_is_up_to_date'
      9:
     10:     _, status = Open3.capture2e("rake check_manifest")
     11:
  => 12:     assert status.success?, "Expected Manifest.txt to be up to date, but it's not. Run `rake update_manifest` to sync it."
     13:   end
     14:
     15:   def test_require_rubygems_package
===============================================================================
.............................
Finished in 26.353848462 seconds.
-------------------------------------------------------------------------------
2306 tests, 6331 assertions, 5 failures, 0 errors, 9 pendings, 0 omissions, 0 notifications
99.3929% passed
-------------------------------------------------------------------------------
87.50 tests/s, 240.23 assertions/s
rake aborted!
Command failed with status (1)

Tasks: TOP => test
(See full trace by running task with --trace)

[simi]

@Segaja is there more info about your env? Is there any way to share steps to reproduce in in isolated environment like docker? I can't reproduce locally.

$ git checkout v3.3.21
$ rake test 

Loaded suite /home/retro/.gem/ruby/3.1.2/gems/rake-13.0.6/lib/rake/rake_test_loader
Started
..............................................................................................................................................................................................
....................................................................................................P
==============================================================================================================================================================================================
Pending: test_operating_system_customizing_default_dir(GemTest): Temporary pending custom default_dir test
/home/retro/code/work/oss/rubygems/test/rubygems/test_rubygems.rb:48:in `rescue in test_operating_system_customizing_default_dir'
/home/retro/code/work/oss/rubygems/test/rubygems/test_rubygems.rb:45:in `test_operating_system_customizing_default_dir'
     42:       "require \"rubygems\"; puts Gem::Specification.stubs.map(&:full_name)",
     43:       { :err => [:child, :out] }
     44:     ).strip
  => 45:     begin
     46:       assert_empty output
     47:     rescue Test::Unit::AssertionFailedError
     48:       pend "Temporary pending custom default_dir test"
==============================================================================================================================================================================================
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
...................................................................................................................
Finished in 153.651654711 seconds.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2306 tests, 6358 assertions, 0 failures, 0 errors, 1 pendings, 0 omissions, 0 notifications
99.9566% passed
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
15.01 tests/s, 41.38 assertions/s

[Segaja]

it is a clean chroot in a ruby 3.0.4 setup. (sorry i just realize that is not "latest" :)

[simi]

same with 3.0.4

Loaded suite /home/retro/.gem/ruby/3.0.4/gems/rake-13.0.6/lib/rake/rake_test_loader
Started
..............................................................................................................................................................................................
....................................................................................................P
==============================================================================================================================================================================================
Pending: test_operating_system_customizing_default_dir(GemTest): Temporary pending custom default_dir test
/home/retro/code/work/oss/rubygems/test/rubygems/test_rubygems.rb:48:in `rescue in test_operating_system_customizing_default_dir'
/home/retro/code/work/oss/rubygems/test/rubygems/test_rubygems.rb:45:in `test_operating_system_customizing_default_dir'
     42:       "require \"rubygems\"; puts Gem::Specification.stubs.map(&:full_name)",
     43:       { :err => [:child, :out] }
     44:     ).strip
  => 45:     begin
     46:       assert_empty output
     47:     rescue Test::Unit::AssertionFailedError
     48:       pend "Temporary pending custom default_dir test"
==============================================================================================================================================================================================
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
..............................................................................................................................................................................................
...................................................................................................................
Finished in 142.566131683 seconds.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2306 tests, 6358 assertions, 0 failures, 0 errors, 1 pendings, 0 omissions, 0 notifications
99.9566% passed
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16.17 tests/s, 44.60 assertions/s

[Segaja]

Is there any command I can run in the chroot before the tests to give you more informations?

[simi]

@Segaja not sure. You'll need to check why the test is failing on your own or share steps to reproduce in some isolated environment like docker to make it easy to reproduce for me (or other maintainers).

[Segaja]

@simi maybe this information helps:

+ ruby --version
ruby 3.0.4p208 (2022-04-12 revision 3fa771dded) [x86_64-linux]
+ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 3.3.21
  - RUBY VERSION: 3.0.4 (2022-04-12 patchlevel 208) [x86_64-linux]
  - INSTALLATION DIRECTORY: /usr/lib/ruby/gems/3.0.0
  - USER INSTALLATION DIRECTORY: /build/.local/share/gem/ruby/3.0.0
  - RUBY EXECUTABLE: /usr/bin/ruby
  - GIT EXECUTABLE: /usr/bin/git
  - EXECUTABLE DIRECTORY: /usr/bin
  - SPEC CACHE DIRECTORY: /build/.local/share/gem/specs
  - SYSTEM CONFIGURATION DIRECTORY: /etc
  - RUBYGEMS PLATFORMS:
     - ruby
     - x86_64-linux
  - GEM PATHS:
     - /usr/lib/ruby/gems/3.0.0
     - /build/.local/share/gem/ruby/3.0.0
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :backtrace => true
     - :bulk_threshold => 1000
     - "gem" => "--user-install"
  - REMOTE SOURCES:
     - https://rubygems.org/
  - SHELL PATH:
     - /build/.local/share/gem/ruby/3.0.0/bin
     - /usr/local/sbin
     - /usr/local/bin
     - /usr/bin
     - /usr/bin/site_perl
     - /usr/bin/vendor_perl
     - /usr/bin/core_perl
+ gem list --local
abbrev (default: 0.1.0)
base64 (default: 0.1.1)
benchmark (default: 0.2.0)
bigdecimal (default: 3.1.2)
bundler (default: 2.3.22)
cgi (default: 0.3.2)
csv (default: 3.2.5)
date (default: 3.2.2)
dbm (default: 1.1.0)
debug (default: 0.2.1)
delegate (default: 0.2.0)
did_you_mean (default: 1.6.1)
digest (default: 3.1.0)
drb (default: 2.1.0)
english (default: 0.7.1)
erb (default: 2.2.3)
etc (default: 1.3.0)
fcntl (default: 1.0.1)
fiddle (default: 1.1.0)
fileutils (default: 1.6.0)
find (default: 0.1.1)
forwardable (default: 1.3.2)
gdbm (default: 2.1.0)
getoptlong (default: 0.1.1)
io-console (default: 0.5.11)
io-nonblock (default: 0.1.0)
io-wait (default: 0.2.3)
ipaddr (default: 1.2.4)
irb (default: 1.4.1)
json (default: 2.6.2)
logger (default: 1.5.1)
matrix (default: 0.3.1)
minitest (5.16.3)
mutex_m (default: 0.1.1)
net-ftp (default: 0.1.2)
net-http (default: 0.2.2)
net-imap (default: 0.1.1)
net-pop (default: 0.1.1)
net-protocol (default: 0.1.1)
net-smtp (default: 0.2.1)
nkf (default: 0.1.0)
observer (default: 0.1.1)
open-uri (default: 0.2.0)
open3 (default: 0.1.1)
openssl (default: 2.2.1)
optparse (default: 0.1.1)
ostruct (default: 0.3.1)
pathname (default: 0.1.0)
power_assert (2.0.1)
pp (default: 0.2.1)
prettyprint (default: 0.1.1)
prime (default: 0.1.2)
pstore (default: 0.1.1)
psych (default: 4.0.4)
racc (default: 1.6.0)
rake (13.0.6)
rdoc (default: 6.4.0)
readline (default: 0.0.2)
readline-ext (default: 0.1.1)
reline (default: 0.3.1)
resolv (default: 0.2.1)
resolv-replace (default: 0.1.0)
rexml (3.2.5)
rinda (default: 0.1.1)
ruby2_keywords (0.0.5)
securerandom (default: 0.1.0)
set (default: 1.0.1)
shellwords (default: 0.1.0)
singleton (default: 0.1.1)
stringio (default: 3.0.2)
strscan (default: 3.0.1)
syslog (default: 0.1.0)
tempfile (default: 0.1.1)
test-unit (3.5.3)
time (default: 0.2.0)
timeout (default: 0.1.1)
tmpdir (default: 0.1.2)
tracer (default: 0.1.1)
tsort (default: 0.1.0)
un (default: 0.1.0)
uri (default: 0.11.0)
weakref (default: 0.1.1)
webrick (1.7.0)
yaml (default: 0.1.1)
zlib (default: 2.0.0)

[simi]

The only unusual thingie I see is "gem" => "--user-install". Can you try to run tests without this config? Maybe we rely on some default configuration not present thanks to this option.

[Segaja]

i found a way to reproduce it in docker. Here is the Dockerfile:

FROM archlinux

RUN echo -e "[community-testing]\nInclude = /etc/pacman.d/mirrorlist" >> /etc/pacman.conf \
    && useradd --create-home build \
    && pacman --sync --refresh --sysupgrade --noconfirm --needed \
       # cargo \
       community-testing/ruby \
       community-testing/ruby-cgi \
       community-testing/ruby-io-nonblock \
       community-testing/ruby-json \
       # cmake \
       devtools \
       gcc \
       git \
       make \
       ruby-rake \
       ruby-webrick

USER build
WORKDIR /home/build

RUN git clone https://github.com/rubygems/rubygems.git \
    && cd rubygems \
    && git checkout "v3.3.21"

WORKDIR /home/build/rubygems

RUN rake test --trace

[Segaja]

I solved the last error about the "Run rake update_manifest to sync it" error.

[Segaja]

@simi any more ideas about this?

[Segaja]

The only unusual thingie I see is "gem" => "--user-install". Can you try to run tests without this config? Maybe we rely on some default configuration not present thanks to this option.

I tried it without that setting but had no effect on the result.

I also noticed that both tests are working with the json gem. Maybe somehow that is the problem. In my environment json is in 2.6.2 and not 2.5.1. Can this cause issues?

[simi]

docker build .
Sending build context to Docker daemon   2.56kB
Step 1/7 : FROM archlinux
 ---> 7b17a3e650d6
Step 2/7 : RUN echo -e "[community-testing]\nInclude = /etc/pacman.d/mirrorlist" >> /etc/pacman.conf     && useradd --create-home build     && pacman --sync --refresh --sysupgrade --noconfirm --needed        community-testing/ruby        community-testing/ruby-cgi        community-testing/ruby-io-nonblock        community-testing/ruby-json        devtools        gcc        git        make        ruby-rake        ruby-webrick
 ---> Running in e561ca19260e
:: Synchronizing package databases...
 core downloading...
 extra downloading...
 community downloading...
 community-testing downloading...
error: target not found: ruby
error: target not found: ruby-cgi
error: target not found: ruby-io-nonblock
error: target not found: ruby-json
The command '/bin/sh -c echo -e "[community-testing]\nInclude = /etc/pacman.d/mirrorlist" >> /etc/pacman.conf     && useradd --create-home build     && pacman --sync --refresh --sysupgrade --noconfirm --needed        community-testing/ruby        community-testing/ruby-cgi        community-testing/ruby-io-nonblock        community-testing/ruby-json        devtools        gcc        git        make        ruby-rake        ruby-webrick' returned a non-zero code: 1

Is there anything I do wrong?

[Segaja]

Sorry. Small changes:

FROM archlinux

RUN echo -e "[community-testing]\nInclude = /etc/pacman.d/mirrorlist" > /etc/pacman.conf.new \
    && cat /etc/pacman.conf >> /etc/pacman.conf.new \
    && mv /etc/pacman.conf.new /etc/pacman.conf \
    && useradd --create-home build \
    && pacman --sync --refresh --sysupgrade --noconfirm --needed \
       # cargo \
       ruby \
       ruby-cgi \
       ruby-io-nonblock \
       ruby-json \
       # cmake \
       devtools \
       gcc \
       git \
       make \
       ruby-rake \
       ruby-webrick \
    && rm -f /etc/gemrc

USER build
WORKDIR /home/build

RUN git clone https://github.com/rubygems/rubygems.git \
    && cd rubygems \
    && git checkout "v3.3.23"

WORKDIR /home/build/rubygems

RUN rake test --trace