This post is about trying to do something seemingly simple. You have an unusual system that you want to write code for. You even have a cross compiler! But how can you get this working with nix? Although the answer turned out to be simple, the journey to getting there was super long and undocumented. I hope this post saves you a lot of time!

My Motivations Link to heading

I have a deep love for an operating system called Risc Os. It is a really old operating system made by Acorn Computers Ltd. for their Archimedes range of computers and you can still run it on Raspberry Pi! I love it for a few reasons:

  1. It’s so simple
  2. It was one of the first things I nerded out over after seeing at the national computing museum in Bletchley Park
  3. Programming it in C helped is where pretty much all of my C knowledge comes from

But programming it is not easy. It predates every programming convention we have. There are no good text editors on it, and compiling is a pain!

But there is a cross compiler! The only issue is that when you try to build it, it becomes obvious that you need a seriously old version of ubuntu on X86_64 linux (it’s a patched GCC 4.7.4) and these days, I sport an Arm Mac which won’t even build a compiler that old!

Can we nix it? Link to heading

Well of course we can. I cannot think of a better use of a flake! And then we can use something like the rosetta builder from Mac OS to cross build software! I thought that would be easy, but my god it was not! Nix was a dream, but building old versions of GCC is a task I would not wish upon my worst enemies! But it is at this point we fast forward because this is the point I assume you are reading this. You have a cross toolchain built in nix (maybe because you have followed a great tutorial like this one). And that’s all well and good…. but it’s a bit annoying.

Every time you want to build something, you have to manually invoke the correct compiler (e.g. arm-unknown-riscos-gcc), or you have to manually pass the correct --target to ./configure or whatever. What you want is something more like a pkgsCross."your platform".callPackage ... that does all of that for you! You want a custom cross-stdenv!

How C compilers work in Nix Link to heading

So if you follow the tutorial I linked, you will have noticed that the entire compiler, and tools, was built in one package. Why? The reason is that if you do as I did instead, and build binutils and GCC separately, when you finally try to invoke your cross GCC, it will weirdly try to use ar and as instead of arm-unkonwn...-ar and arm-unknown...-as. My initial solution was to build a wrapper for GCC (Nix has some nice tools for this) that passed the -B argument and told GCC where the correct Binutils was, but this was when doubt started to grown in my mind. It felt a bit hack-y!

What you are meant to do is wrap a C compiler with pkgs.wrapCCWith. The documentation of which seems to not have been written! Fair enough! Let me give you a snippet from my flake:

wrappedGcc = pkgs.wrapCCWith {
  cc = riscosTools.rogcc;
  bintools = wrappedBintools;
};

You will also want wrappedBintools:

wrappedBintools = pkgs.wrapBintoolsWith {
  bintools = riscosTools.robinutils;
  libc = null;
  coreutils = pkgs.coreutils;
};

Now the magic begins! If you stick that in a shell and run nix develop you will get your cross toolchain and it will all be working together… or will it?

The answer is no. No it won’t. That would be too easy. If you look through the code in nixpkgs:/pkgs/build-support/cc-wrapper, you might see why. Of course, if we want to cross compile with nix, we have to tell nix that we are cross compiling!

Telling Nix we want to cross compile Link to heading

The reason nix failed to wrap our compiler is because pkgs is defined with three really important variables:

  1. buildPlatform – The platform we are building on
  2. targetPlatform – The platform we are building for
  3. hostPlatform – The platform which can run the stuff we are building

So, if you are anything like me, you will have followed right up until 3. The concept of a host platform is really strange but it will make sense once I tell you this. The host platform is only important when we are building compilers (which we are).

For example! I might want to use linux to build GCC that runs on mac but outputs code for Z80 CPUs. In which case:

  • build = linux
  • target = Z80
  • host = darwin

When you aren’t cross-building then this is simple, all three are your system. If you load nixpkgs into nix repl, then you can take a look at pkgs.stdenv.[host,target,build]Platform and confirm that they are all the same.

But what about when cross compiling? I shall specify, I do not mean building a cross compiler, I mean using a cross compiler to build a package. Well then host = target != build. And this also makes sense! If we build a normal package, say pkgs.pkgsCross.mingwW64.hello, then our buildPlatform is our system’s platform (linux or darwin), we are targeting windows. What about the host platform? Well for the hello package, it makes no sense to ask given that it isn’t a compiler. If we build pkgs.pkgsCross.mingwW64.gcc, then host = target. Because we would build a GCC that can run on windows. It is not the compiler we are using to build this package set!

The hidden package set Link to heading

So at this point, you may have realised something a bit strange, given that the various host, target and build platforms are defined for each package set in nix, where do the cross compilers live? They can’t live in the normal pkgs for your system because the cross compiler’s targret differs from the package set’s target. But they can’t live in pkgsCross because the compiler’s host (your system) differs from the package set’s. Sadly it matters! The reason being because for wrapCC to correctly find your compiler and binutils, it needs to work out the compiler prefix (arm-unknown-riscos- in my case) which it will only do if host != build. The answer is a strange one and can be found in nixpkgs:pkgs/stdenv/cross/default.nix. If you read this file closely, it starts with your normal stdenv, then it sets targetPlatform = crossSystem (so finally host != build), then builds the cross stdenv where it uses the compiler from the previous stage!

So how can we modify or extend this package set? I have no idea! It seems completely inaccessible! But we can always try to mimic the same trick!

The final solution Link to heading

I shall now walk you through the solution I came up with. You can find the whole flake here.

We start by importing pkgs and defining our cross system:

pkgs = import nixpkgs {
  system = "x86_64-linux";
};

crossSystem = {
  config = "arm-unknown-none-elf";
  targetPrefix = "arm-unknown-riscos-";
  isCross = true;
  isStatic = true;
};

riscosPkgs = import nixpkgs {
  system = "x86_64-linux";
  inherit crossSystem;
};

So far, pretty normal for cross compiling. I then have a section where I build the cross toolchain, I’ll leave it out, but the important things I define are riscosTools.rogcc (my cross GCC), and riscosTools.robinutils, my cross binutils. We move onto the trick:

buildSystem = import nixpkgs {
  system = "x86_64-linux";
  config.replaceStdenv = { pkgs }: pkgs.stdenv.override { 
	targetPlatform = riscosPkgs.stdenv.targetPlatform // { config = "arm-unknown-riscos" ;};
  };
};

Here I define a new package set! I set the target platform with my target platform and then override config. Why? arm-unknown-riscos is not a valid nix platform and so it will complain. I have to use arm-unknown-none-elf. But annoyingly, wrapCCWith uses config to get the prefix for the compiler. I do a cheeky override here purely so it picks up the correct compiler.

wrappedBintools = buildSystem.wrapBintoolsWith {
  bintools = riscosTools.robinutils;
  libc = null;
  coreutils = buildSystem.coreutils;
};

wrappedGcc = buildSystem.wrapCCWith {
  cc = riscosTools.rogcc;
  bintools = wrappedBintools;
};

I now use the buildSystem to call wrapBintoolsWith and wrapCCWith. Finally! It will pick up the correct prefix.

riscosPkgs = import nixpkgs {
  system = "x86_64-linux";
  inherit crossSystem;
  config.replaceCrossStdenv = {buildPackages, baseStdenv }:
    (buildPackages.overrideCC baseStdenv wrappedGcc).override {
      extraNativeBuildInputs = [riscosTools.elf2aif];
    };
}

I can now define the package set!

So does this work? Amazingly it does! I defined the following flake:

{
  description = "A new application for RISC OS";

  inputs = {
    utils.url = "github:numtide/flake-utils";

    # Point to your toolchain flake. This can be a local path or a git URL.
    riscos-nix.url = "path:../.";
  };

  outputs = { self, utils, riscos-nix }:
    utils.lib.eachDefaultSystem (system:
      let
        pkgs = riscos-nix.riscosPkgs;
      in
      {
        packages.default = pkgs.callPackage ./default.nix {};
      });
}

And default.nix:

{pkgs, lib}:

pkgs.stdenv.mkDerivation {
  pname = "hello-riscos";
  version = "1.0";

  src = ./.;
  hardeningDisable = [ "all" ];

  buildPhase = ''
    echo CC=$CC
    $CC hello.c -o hello
    elf2aif hello
  '';

  # The phase to install the executable into the Nix store
  installPhase = ''
    mkdir -p $out/bin
    cp hello $out/bin
  '';
}

and look what happened:

james> file result/bin/hello
result/bin/hello: RISC OS AIF executable

Who knew file knows about Risc OS!

That concludes things! I hope this saves you a lot of time!