nixos/netbird-server: init module

[Tom-Hubrecht]

Description of changes

Adds a package for netbird dashboard and creates a module for self-hosting the netbird management server.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.11 Release Notes (or backporting 23.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[RaitoBezarius]

Do you have to consistently recreate it or could something like rsync with appropriate flags could update the new files?

[RaitoBezarius]

(explain why this is needed also)

[Tom-Hubrecht]

It is needed because some configuration values, e.g. the auth endpoint are hardcoded in the web-ui and replaced from env varibles with envsubst. I suppose that we could also pass those variables to the dashboard package and set them at build time.

[oddlama]

It's quite unfortunate that they require envsubst instead of accessing the environment variables directly... I don't like copying each time but it seems like this is currently necessary - unless upstream changes the process.

On that note: In netbirdio/dashboard#242 one of the maintainers (?) noted that they would accept contributions to make this process easier to host without docker. I imagine that a simple PR to upstream that replaces this envsubst by just accessing process.env.SomeVar in their config.ts would solve this for us and for them.

[Tom-Hubrecht]

I mean, we can also do it ourselves with substituteAll while building the dashboard, but we need to decide if we want to rebuild the dashboard as a whole every time an option change, which would take some time

[oddlama]

IMO an upstream solution seems to be aligned with their goals anyway, so only if they refuse I would start resorting to alternatives.

[RaitoBezarius]

How close is the dashboard the original package? Should it be a passthru derivation of pkgs.netbird ?

i.e. pkgs.netbird.dashboard ?

[misuzu]

There should be a meta.doc, explaining how to actually use this module. There are a lot of options and it's not clear what a working "production"-like configuration should look like. A NixOS test is also desirable.

[Tom-Hubrecht]

I have added a bit of documentation and also reduced the amount of settings that the end user has to provide by default. I don't have the time right now to write a complicated test, as I believe we need to setup multiple nodes, one being an sso and try adding nodes to the network in the test and until the start of september I will be a bit busy.

[oddlama]

Could you add a nixos tests that verifies that this is working as intended?

[oddlama]

It's quite unfortunate that they require envsubst instead of accessing the environment variables directly... I don't like copying each time but it seems like this is currently necessary - unless upstream changes the process.

On that note: In netbirdio/dashboard#242 one of the maintainers (?) noted that they would accept contributions to make this process easier to host without docker. I imagine that a simple PR to upstream that replaces this envsubst by just accessing process.env.SomeVar in their config.ts would solve this for us and for them.

[jvanbruegge]

I just stumbled upon this after being 90% done with setting this up for myself.
One thing that would be nice is being able to pass also the turn password via a file. This is how I did that:

services.coturn = mkIf cfg.createCoturn {
  enable = true;
  lt-cred-mech = true;
  no-cli = true;
  extraConfig = ''
    fingerprint
    no-software-attribute
    external-ip=${domain}
  '';
};

systemd.services.coturn.serviceConfig = mkIf cfg.createCoturn {
  EnvironmentFile = "/run/secrets/netbird";
  ExecStart = lib.mkForce ''
    ${pkgs.coturn}/bin/turnserver -c /run/coturn/turnserver.cfg --user="''${COTURN_USER}:''${COTURN_PASSWORD}"
  '';
};

Also, the createNginx option could optionally accept a listen config, for people that use a seperate reverse proxy

[jvanbruegge]

Additionally there could be a user defined secrets file here. Is a bit more convenient to have all the variables in one file

[jvanbruegge]

As I said earlier, it is possible to pass this via the command like. Also the config should probably set external-ip to NETBIRD_DOMAIN

[Tom-Hubrecht]

external-ip expects an ip address, which is not what NETBIRD_DOMAIN contains

[jvanbruegge]

Still works as expected though. I tested this with coturn behind two layers of NAT

[jvanbruegge]

For comparison, I had to vendor this module and adapt it to fit my needs like this: https://github.com/jvanbruegge/server-config/blob/master/modules/netbird-server.nix

In the nginx config, only the webserver directive is used, the other two grpc endpoints are already handled by HAProxy

[PatrickDaG]

I took the module and reworked it a bit as well as updated everything to the newest versions:

• https://git.lel.lol/patrick/nix-config/src/branch/master/modules/netbird-server.nix
• https://git.lel.lol/patrick/nix-config/src/branch/master/modules/netbird-dashboard.nix

There are still a few thing missing in this draft most notably:

• systemd hardening
• secret management
• better documentation
  But I wanted to post the draft here as a suggestion for further work.

Notable things I've changed

Split modules

as far as I see there is no reason that the dashboard and the server need to be on the same host.
I don't even think the different parts of the server, the management API and the signal server need to be on the same host
so to be completely flexible we might even want to to change those.

dashboard templating

I've changed the dashboard templating to use a derivation so it's only done once at build time and not on every restart.

Used RFC 42 style settings

Instead of this weird back and forth from Environment to templated json I've directly used json. I think this enables user to better configure the server to their need but it also forces us to always keep the defaults updated and might make configuring harder as the json values are pretty undocumented.

OIDC autoconfig

Turns out netbird is able to automatically get the oidc config from the AutoConfigEndpoint. It does so automatically and overwrites any values set in the config so we can just skip those.

Things I'm still unsure about

secret management

Some of those secrets are not a secret at all. Namely, the dashboard client secret is templated into the statically served frontend and thus open for anyone who can reach the webpage. For this reason I would oppose exposing this secret as a file option to not give user a false sense of security but make them very aware that if their Identity provider depends on this being secret that they should be careful.
Gladly, this secret is not needed most of the time, as OIDC supports public clients which should not need a secret.
As for the other client secret I'm not sure. I think netbird support using different IDP clients for different things, but my IDP does not support such a setup so I can't test if there is a need or use for the other secrets to be actually secret, in which case we should support them being read from files.

IDP management

netbird has an IDP manager which can be used to read additional information from the IDP in a way that standart oidc does not support. But this only work if you use one of their supported IDPs, which I do not so again I'm unable to check if my config works for this case of if we need additional things.

device Authorization

Same thing my IDP does no provide any device authorization so I can't test my config.

Nginx setup

So far I've only exposed the nginx config for serving the frontend. You also may need additional config if you use nginx as a reverse proxy to correctly route all the other parts, management and signal. But as I don't want to assume people use nginx as a reverse proxy I think it would be better to just document what endpoints need to be reachable and let people set it up themselves.

[Tom-Hubrecht]

So, I rewrote the whole thing, thanks a lot for your work @PatrickDaG

Split modules

Each part is now activable independently :

• signal service
• management service
• static dashboard
• coturn server

This split configuration also allows to setup nginx only for the parts that we want @jvanbruegge, the use of certificates for the coturn server is also independent now.

There is also the option to enable everything at once with services.netbird.server.enable

Secrets

To use secrets in the management config, there is nifty function https://github.com/NixOS/nixpkgs/blob/master/nixos/lib/utils.nix#L174 so we can either pass a value directly as a string that will end up in the store, or we can instead pass { _secret = "/path/to/secret"; } and the secret will be subtituted in the config at runtime.

I added the option to pass the turn user password as a secret as well, using the same mechanism that the coturn module uses for the auth-secret

[PatrickDaG]

Thanks for the work. Looks really good.
I've found some things I feel like we could improve. If you don't feel like it, I can also do it in the next week or so.

[PatrickDaG]

@Tom-Hubrecht What's your status for this? Do you still have time and motivation to work on it? If not, I can offer to help out to get this merged.

[Tom-Hubrecht]

@Tom-Hubrecht What's your status for this? Do you still have time and motivation to work on it? If not, I can offer to help out to get this merged.

I'll have time friday to address your comments, and then we should be good to go

[PatrickDaG]

Looks great. Thanks for all the work.

[RaitoBezarius]

It's excellent work, almost idiomatic.

There's only one slight problem, it's the usage of the enableNginx pattern.
AFAIK the good way to do this is to provide a nginx subconfiguration fragment which reuse the NGINX types (look in other modules how they do it) and once you do something like services.netbird.dashboard.nginx.enable = true; or if the subfragment is non-null, you make use of NGINX.

This way, we can easily separate webservers implementation from the rest.

I do not deem this a blocker for this enormous PR and this is future work for anyone who care about other webservers and this ecosystem.

[mannp]

Wow, thank you to all involved :) going to give this try 👍🏻

[srounce]

@Tom-Hubrecht I'm currently using this module and I'm curious why a Coturn submodule was added when a Coturn module already existed in NixOS? If I'm already using the other one should I point to that instead or should I run an instance from each module?

[Tom-Hubrecht]

@Tom-Hubrecht I'm currently using this module and I'm curious why a Coturn submodule was added when a Coturn module already existed in NixOS? If I'm already using the other one should I point to that instead or should I run an instance from each module?

Because the current module does not allow specifying secrets so I had to create an overlay, under the hood it uses services.coturn so the options should merge