That's definitely word salad of a title.

Why?

I've seen local development environment drift everywhere I've worked, and frequently between my own machines. This can be as simple as slightly different system libraries installed to major versions of platforms like Python or Node. The problem gets solved in a variety of ways - virtual environments, nodeenv, /etc/alternatives, but none of those are unified and require care and feeding from everyone on the team, individually.

With how many repos a given team might have these days, and how each one may differ, knowing the precise steps to setting up a development environment is frequently maintained in a wiki like Confluence or Notion. But, in the same way that tooling about a repo has moved closer to the repo (things like git hooks or linter configuration), what if we can get the environment documented with code, as well?

This is not a new idea. Vagrant is 15 years old, at this point. Thin-client development is one of the theoretical solutions to the problem. But, every developer I've ever worked with has slightly different workflows and tools that they like using, so I'd prefer to enable that as much as possible, while ensuring the things that do matter are kept constant.

Goals

Describe a mechanism for cloning a repository and having all of the development dependencies and tools ready and available, without polluting your global system.

For this post in particular, describe how I handle automatically switching between environments in a terminal.

Credit

The majority of the implementation here I picked up from CommonLawFeature. I felt it needed to get documented. Thanks, CLF!

TL;DR

On a Linux (non-NixOS) system:

1. Install Nix
2. In ~/.config/nix/nix.conf, add:

experimental-features = nix-command flakes

3. Install direnv and configure direnv for your shell
4. (Optional) Install Starship and configure it for your shell
5. In any folder you wish to have an automated environment in, run:

nix flake new -t 'github:numtide/devshell#toml' .

6. In those places, use the packages variable in devshell.toml to install any packages you need for that particular environment

Okay, obviously that's a vast oversimplification, so let's break it down.

Explanation

Process

When changing to a new folder, direnv loads the .envrc file which loads .env and .env.local if they exist, invokes nix-direnv, watches the devshell.toml file, and then expects to utilize flakes for the next step.

The flake is set up to build a shell using devshell, which tries to create the minimal environment necessary for a project to ensure that additional environment variables aren't loaded that might complicate the project. It also ensures that the packages available in the environment are only the ones specified by either the flake.nix, devshell.toml, or any other Nix tools the project might be using.

The shell loads, Starship picks up that you have a certain environment, and changes the prompt to show the environment so you always have a sense of where you are.

"Simple!"

Tools

Nix (& Flakes)

Nix is a declarative tool for reproducible environments and builds. This guide barely touches some of its powers, and while its configuration language is complex, if you can, I recommend looking into it more.

Flakes are a convenient way of working with the dependencies within a given project, and make it very easy to help define what is necessary for a dev environment, including dev tools that shouldn't be included in a build of the project. If your whole team is utilizing Nix, then it makes it very easy to swap to a new project and ensure that everything is using the toolchains expected and correct package versions. Those of you who work in Python or JavaScript across many repos might know particular pain here, and I've yet to see an environment where there wasn't at least a bit of version mismatch between developers.

What that leads to is a declarative way to easily specify packages, versions, toolchains, and more, all while not modifying your global state and providing lots of tools for build / environment configuration.

There's also home-manager, which I haven't had a chance to explore either, for dealing with the "I just got a new computer / wiped my old one and now I'm trying to recreate all of my environments" problem.

devshell Flake Template

So the nix flake new -t ... command is copying a template from a particular repository. In my case, Numtide's devshell template is a good starting place, and you can browse the templates here.

In the case of the toml template, it adds a simple flake.nix, shell.nix, and devshell.toml file. You can do all the usual things with flakes that you would expect, and the toml file makes it easy to add new packages or commands.

direnv

direnv is a tool that does one thing and one thing well - load and unload environments based on the folder you are in. Once it hooks into your shell, it looks for a .envrc file in the folder you're in, and if one exists and has been allowed, will execute loading the environment, as specified.

I mentioned "has been allowed;" direnv does not load arbitrary code without you explicitly approving it. It does mean that for any change you make to the .envrc, you must re-approve it, but that does prevent something from changing it out from under you in a malicious way.

nix-direnv

In short, a convenience tool for caching and preventing certain kinds of garbage collection of Nix packages. Gets utilized automatically by the flake invocation.

devshell

devshell seeks to build the smallest possible environment that can be utilized for a project. It has some support for different languages out of the box that need a bit more environment configuration, and can be used both for CI / CD processes as well as to build in-project shell commands or run background services, as needed.

I have found the devs to be responsive to questions and helpful, and while the documentation is a bit sparse, it's a relatively simple tool that does exactly what I need it to.

🤔 An Aside on C++

devshell does not officially support C++ currently. There is a request for enhancement but last I checked, there were other priorities. You can set it up to work, and I will try to document that in a separate post, but you will run into some issues, especially if you have to deal with xorg at all.

My current workaround for this has been to add the following to my devshell.toml:

imports = ["language.c"]
includes = [
	# String names of any necessary libraries for linking, but not necessary to include as packages
]
...
[[env]]
name = "CMAKE_PREFIX_PATH"
eval = "$DEVSHELL_DIR"

I will note that this may not be the best solution, may encounter issues when dealing with graphical applications that you're trying to link against and don't have all the dependencies included for, or Other Issues(TM) that might summon Cthulhu. You have been warned.

Starship

Starship is a cross-shell prompt that dynamically changes content based on the environment that is loaded in the shell and the folder that you are in. I find it invaluable for things like, "What version of {Python|Node|CMake} is this project using?" or "What {kubernetes cluster|AWS profile} will I be touching if I run a command?" It is not necessary for the process of reproducible environments or automated switching. It's just helpful and I tend to be visual.

I recommend glancing down the configuration page for Starship, as it supports a ton of different environments and tools, and the way that each gets configured or shown is also highly configurable.

And, for those of you who remember the configuration headaches of the syntax for configuring your $PS1, Starship offers much simpler ways of describing how it should look.

Other Thoughts / Alternatives

I considered a whole section about my exploration of using layout uv alongside use flake in the .envrc, but I think it takes away from the rest of this. I will say that you can utilize devshell.toml to specify the Python version you want, and then utilize virtual envs in exactly the same way that you're used to with requirements.in and requirements.txt files.

Other people do work inside of Docker containers, with the project folder mounted into the container. Vagrant is a tool I've seen one person use, and there's been a slew of other tools in recent years that do some form of "reproducible dev environments," like devenv and devbox, both built off of Nix in different ways.

Examples

JDK Environment

For my environment working on the Breath of the Wild Decompilation project, since I was running into some small environment differences with compilers and such, and I was toying with running Ghidra via Nix (and doing some nonsense with Python 3 before they added official support), I had the following:

imports = ["language/rust"]

[devshell]
packages = [
	"jdk",
	"gradle",
	"cmake",
	"ccache",
	"xdelta",
	"clang_16",
	"glibc",
	"haskellPackages.itanium-abi"
]

[[env]]
name = "JAVA_HOME"
eval = "$(dirname $(dirname $(which java)))/lib/openjdk"

[[env]]
# Needed for building plugins
name = "GHIDRA_INSTALL_DIR"
eval = "$(pwd)/ghidra"

[language.rust]
# No options, but here if needed

[[commands]]
name = "ghidra"
command = "$PRJ_ROOT/ghidra/ghidraRun"
help = "Launch Ghidra, the open-source decompiler"

You can see I had to do some nesting with dirname for getting the Java home. Again, this was because I was trying to do some work with plugins in particular, and this made it easy to do, within Nix.

Game Dev Project in C++

There's a project where I was working on rebuilding a quite old codebase. One of the challenges with that if your system is not NixOS deals with graphics drivers. There is a "shim" called NixGL which I found necessary to get things working, and I will include my devshell.toml here, complete with the comments where I was trying to figure out what needed to go where.

imports = ["language.c"]

[language.c]
compiler = "pkgs.clang"
includes = [
	"xorg.xorgproto",
	"xorg.libX11",
	"xorg.libX11.dev",
	"xorg.libXrender",
	"xorg.libXrender.dev",
	"xorg.libXres",
	"xorg.libXres.dev",
]

libraries = [
#	"xorg.xorgproto",
	"xorg.libX11",
#	"xorg.libX11.dev",
#	"xorg.libXrender",
#	"xorg.libXrender.dev",
#	"xorg.libXres",
#	"xorg.libXres.dev",
]

[devshell]
packages = [
	"pkg-config",
	"cmake",
	"mesa.dev",
	"SDL2",
	"SDL2.dev",
	"SDL2_ttf",
	"lua",
	"libatomic_ops",
	"libatomic_ops.dev",
	"libglvnd",
	"libglvnd.dev",
	"glm",
	"glew",
	"glew.dev",
	"enet",
	"yaml-cpp",
	"libpng12.out",
	"libpng12.dev",
	"minizip"
]

[[env]]
name = "CMAKE_PREFIX_PATH"
eval = "$DEVSHELL_DIR"

I'm certain it's not all of the dependencies, and it's still pulling some from the system, but this is a much better state and made it easy for me to work on different systems.