<aside> ❓

If you are reading this from Notion, some things are highlighted:

• Blue — this is information provided by the template for RFCs.
• Yellow — this is a TODO and should be discussed.
• Red — this is a critical problem and should be resolved during discussion. </aside>

Summary

Nixpkgs contains non-trivial amounts of generated, rather than handwritten code. There is an update script (usually in Python) for each. We want to start systematizing to make it easier to maintain and use. We could do plenty of future work to build upon this, but we stop here for now to avoid needing to change any tools (Nix, Hydra, etc.).

This RFC standardizes only updater scripts, it doesn’t touch almost any Nix code and many things previous RFC 109 tried to do. This means that this RFC tries to create a single library for all updaters, mainly to improve the maintainability of hundreds of random scripts.

Terms

• the library — the package that is described by this RFC. It is named “library” because update scripts will use it as a library for implementing most of the boring logic, and only ecosystem-specific details will be implemented by the updater script.
• An updater script — this is what users would run to add/update plugins in specific ecosystems.
• An input — list of references to the outputs. As an example, for Vim plugins, this would be a CSV table with GitHub URLs and other data.
• An output — what is generated at the end. This can be for example a plugin or an extension.
• An entry — a single unit of what updater updates. For Vim plugins this would be a plugin, for Gnome extensions this would be an extension.

Motivation

Why generate packages?

Many ecosystems have central package repositories (e.g. pypi.org, npmjs.com, luarocks.org, etc) that are parsable and we can easily get everything we need to know about every existing package. This allows us to automatically generate a list of all packages for each ecosystem.

Or, if we don’t want to fill nixpkgs with unused packages, we can store a list of package names in nixpkgs and then generate a .nix code for all of them. This is what update scripts currently do: they get a list of all entries and then prefetch all required info for building the output.

This is much more desirable and maintainable than packaging every single package manually. Especially if the packaging process is almost identical for each entry.

Why create this RFC?

The current approach is to create an updater script, which receives a list of names of entries and returns (usually) a .nix file with all packages updated to the latest version. As you can imagine, having (probably) hundreds of independent random scripts is not a good solution at all. Sometimes such scripts are really dated and poorly written, they may have obvious and obscure bugs.

As an example, we could take pluginupdate.py. I created https://github.com/NixOS/nixpkgs/pull/336137, so I dug deep into the code and found a dozen bugs, a huge amount of suboptimal code, and a few major issues. This doesn’t mean the original version was poorly written, it probably was a small and straightforward script, to which some features were added one at a time for years. As usual in software, this resulted in ugly Frankenstein, which is no one's fault. A more specific description of problems in pluginupdate.py is described in the “Prior art” section.

Some of the pros of creating a standardized ecosystem for updaters are:

• Reuse of code: There are a ton of update scripts in nixpkgs currently, and almost every single one duplicates the logic of each other, but in an entirely different way.