[RFC 0197] package sets by-name

[quantenzitrone]

A new location and a unified interface for package sets in nixpkgs.

rendered

[Sigmanificient]

First time interacting with an RFC, i hope i am not making comment that are too dumb 😅

[Sigmanificient]

thanks for adding trees ❤️

[Sigmanificient]

i think a drawback from idea 3 is that we end up with attrpath in directory names

[SigmaSquadron]

I was discussing this privately with @Sigmanificient earlier today, and he helped me organise my thoughts on this.

[SigmaSquadron]

I think our implementation plan should be separated into four stages:

1. Create interfaces for package sets that uses the same functions as by-name to callPackage packages from sharded directories, with additional extensions for package set versioning.
2. Update nixpkgs-vet to expand the existing linting checks to the new package sets.
3. Migrate all package sets into the new interface when it makes sense to do so, or move the children of smaller sets into by-name as top-level packages.
4. Add new rules to nixpkgs-merge-bot to allow unprivileged merging to package sets.

We can't move into the next stage before the previous one is complete, so let's not think about the later stages just now.

[SigmaSquadron]

(Answering because this applies to idea 1)

I never thought it made sense for functions and derivation builders to be inside package sets. I think our Go infrastructure has the right idea: move them to the top-level, and version them in the attribute name. fishPlugins.buildFishPlugin can just be moved to the top-level, while python313Packages.buildPythonApplication becomes buildPython313Application on the top-level. We can deal with this at stage 3, where we migrate everything over.

[7c6f434c]

This proposal is awkward with Python where you have CPython and PyPy and then language version-dialects within them, and even worse for Common Lisp where you have same base language but multiple compatible implementations with their completely own versioning.

[quantenzitrone]

i also think keeping functions in package sets is fine

[SigmaSquadron]

(Answering because this applies to idea 1)

I generally think it's a good idea to think of Nixpkgs as having three types of package infrastructure systems:

1. Top-level packages, which nowadays go to by-name;
2. Simple package sets, like fishPlugins, which are easier to implement in this new interface as we can simply call them as we do by-name packages and shove the resulting set into fishPlugins from the top-level.
3. Versioned package sets, like python3Packages, which are by far the most complex. I think we should absolutely get the input of the primary maintainers of these sets here before we move forward. There are many concerns regarding updating versioned sets, and default aliases, as well as edge cases like python3.pkgs and python3Packages. In the end, having something that is no worse to maintain as it was before would be ideal.

[quantenzitrone]

4. Auto generated package sets like haskellPackages.

[quantenzitrone]

i want handle both 2 and 3 with this RFC
actually i want to handle all package sets in this RFC

[SigmaSquadron]

That seems good to me. As mentioned in #197 (comment), this is conditional on the approval of the current versioned package set maintainers.

[infinisil]

In addition to the comments, also a meta-comment: While it's really great to create RFC documents like these, we don't need the lengthy RFC process anymore to get them approved, because if Nixpkgs committers can't agree on it, we can escalate to the SC-appointed Nixpkgs core team, which has the authority to make a call.

Probably still beneficial for Nixpkgs to keep discussions for such changes outside the Nixpkgs repo itself due to the noise there, but otherwise I'd suggest to PR this to an adr/¹ directory in Nixpkgs.

Footnotes

1. From https://adr.github.io/, which is very similar ↩

[7c6f434c]

What is a package set? There are some that do not have a direct top-level name. I think in general some places of the text is written for a package set that comes with its set of expressions, while there are cases where meaningfully sets share a large fraction of the expressions.

[quantenzitrone]

i included a definition of package sets
does it suffice?

[7c6f434c]

By the way, lispPackages treated as sbclPackages but also needed for ccl.pkgs and ecl.pkgs seems even worse semantically than override luajit = lua5_3 or the other way round that current attempts to minimise all-packages.nix at any cost promote. So this should include pins from the beginning.

[quantenzitrone]

Guys I think we need to decide on and merge #83 first.
Especially the foo.pkgs vs fooPackages part.
I've always thought of package sets as top level attributes like fooPackages, because that is what is exposed on https://search.nixos.org, but it seems that is not always the case.

[SigmaSquadron]

#83 is unfortunately abandoned. I don't think we should depend on it, but rather replace it.

[7c6f434c]

I think #192 has chances and also is needed (on its own or incorporated) for this one to be good?

[quantenzitrone]

Why do we need versioned python, ghc and ocaml packages but not different package sets for other languages like C? Why are there no {clang{18,19,20,22},gcc{13,14,15},tinyc,arocc,hcc}Packages?

edit: The out path depends on the version of the interpreter:

• For Python: $out/lib/python${version}/site-packages/${pname}
• For Ocaml: $out/lib/ocaml/${version}/site-lib/${pname}

[7c6f434c]

Why do we need versioned python, ghc and ocaml packages but not different package sets for other languages like C?

• The recent stream of PRs setting --std=gnu17 is addressing a similar issue for C in a different way
• For SBCL we do sometimes have to use stumpwm = sbcl_<previous_version>.pkgs.stumpwm somewhere, but on the balance we don't have a full top-level name for previous-version sbclPackages.

[quantenzitrone]

If we don't incorporate a system for nested package sets/package sets nested in attribute sets, how do we deal with them?

• these could probably be moved to the top level:
  • emacsPackages.*Packages
  • ocaml-ng.ocamlPackages*
  • haskell.packages.{,native-bignum.}*
  • chickenPackages.chickenEggs
• vscode-extensions could probably be flattened
• ocamlPackages*.janeStreet could maybe be merged into ocamlPackages*
• kdePackages.{sources,gear,framework,plasma}?
• vimPlugins.nvim-treesitter-{,legacy-}parsers?
• linuxKernels.**?
• luaPackages.luaLib?
• androidenv.**?
• nginxModules.*?

[qweered]

Somewhat related — I've been working on lib.mkPackageVariants, which solves a complementary problem: managing multiple variants of a single package (versions + feature configurations).

Currently packages like openssl, python, perl, etc. each reinvent their own multi-version handling with ad-hoc patterns. mkPackageVariants standardizes this into a shared utility — you get a variants.nix (version/config metadata), a generic.nix (curried builder), and a thin package.nix wrapper. The default variant is the package itself, and all variants are accessible via passthru (openssl.v1_1, openssl.v3_6, openssl.oqs, etc.).

1. by-name compatible: mkPackageVariants returns a single derivation (the default variant) from package.nix, so it works natively with by-name auto-discovery. The openssl migration in the PR already lives in pkgs/by-name/op/openssl/. Versioned aliases (openssl_1_1, openssl_3, etc.) go in all-packages.nix as backward-compat references.

2. Complements package sets: This RFC handles package ecosystems (pythonPackages, etc.) where multiple packages share infrastructure. mkPackageVariants handles the orthogonal case of one package with multiple versions/configs. They compose naturally — a package set member could itself use mkPackageVariants for its versioning.

3. Reduces all-packages.nix clutter: Instead of N top-level entries per versioned package (e.g., openssl_1_1, openssl_3, openssl_3_5, openssl_3_6, openssl_oqs, openssl_legacy), there's one by-name entry with variants in passthru and lightweight aliases for backward compat.

4. Grouped definitions: Packages using mkPackageVariants naturally share files across versions (patches, generic builder), which aligns with RFC 197's grouped definition support. The package.nix + variants.nix + generic.nix convention could become a standard pattern for this.

[quantenzitrone]

Alternative:
have versioned package sets be in a package-set-set
e.g.

• ocamlPackageSets.ocaml_5_4
• lixPackageSets.lix_2_93 (this is how it currently is)
• pythonPackageSets.python_3_13

This could also be used as a nice way of handling versioned packages
e.g.

• ocamlVersions.ocaml_5_4, ocamlVersions.ocaml_4_14_unsafe_string
• nixVersions.nix_2_32 (this is how it currently is)
• pythonVersions.python_3_13, pythonVersions.pypy_3_10
• ffmpegVersions.ffmpeg_7, ffmpegVersions.ffmpeg_7_headless

yay (👍) or nay (👎)?

[Eveeifyeve]

What is the problem that could happen with doing both?

[qweered]

The problem is that we not only have versions, but variants like cmakeMinimal, I'll try to come up a solution to passthru problem in my pr. Also ocamlPackageSets.ocaml_5_4 is much longer than ocaml.v5_4

[Eveeifyeve]

I agree on the shorter version, however the main question is how can we make sure users don't get confused, especially beginners? My only thoughts are to just produce a error that tells you what is available if it's evals and it's incorrect. Like ocaml.test which doesn't exist.