2024-11-07 Setting Up a Mesh VPN with Nebula

Jing-Ying Chen

Jing-Ying Chen

— 3 min read
To run applications on top of databases, it is crucial to consider high availability, even though for a self-hosted home lab. It is not that useful if the replica servers are all in the same place too. So I am looking for a secure (mesh) VPN that can span across geological locations.

I would like to run the simple webapp built previously on a database with proper replication, or better yet high availability. Currently there are many popular alternatives (Tailscale, NetMaker, Nebula, etc). I'd like to try Nebula to see if it fits, installation and performance wise.

Setup

Search "nebula" and "vpn", visit the GitHub site, and follow the Getting started (quickly) guide:

mkdir nebula
cd nebula

I have a mixture of old and new Ubuntu boxes, so I will just download the binaries here. For me, I downloaded the netbula-linux-amd64.tar.gz one:

wget https://github.com/slackhq/.../nebula-linux-amd64.tar.gz
tar zxvf netbula-linux-amd64.tar.gz
./nebula -version

I already have some VPSs as well as machines near by, so I run

./netbula-cert ca -name "Pointegrity"

If you have jq installed, you can run

$ ./nebula-cert print -json -path ca.crt | jq .details
{
  "curve": "CURVE25519",
  "groups": [],
  "ips": [],
  "isCa": true,
  "issuer": "",
  "name": "Pointegrity",
  "notAfter": "2025-11-07T02:26:08Z",
  "notBefore": "2024-11-07T02:26:08Z",
  "publicKey": "...",
  "subnets": []
}

Then create certificates as suggested (adapt for your own setup):

./nebula-cert sign -name "vps1" -ip "192.168.111.1/24"
./nebula-cert sign -name "vps2" -ip "192.168.111.2/24"

./nebula-cert sign -name "laptop" -ip "192.168.111.10/24" -groups "laptop"
./nebula-cert sign -name "lan1" -ip "192.168.111.11/24" -groups "lan"
./nebula-cert sign -name "lan2" -ip "192.168.111.12/24" -groups "lan"

Check some certificates:

$ ./nebula-cert print -json -path vps1.crt | jq .details
{
  "curve": "CURVE25519",
  "groups": [],
  "ips": [
    "192.168.111.1/24"
  ],
  "isCa": false,
  "issuer": "...",
  "name": "vps1",
  "notAfter": "2025-11-07T02:26:07Z",
  "notBefore": "2024-11-07T06:20:40Z",
  "publicKey": "...",
  "subnets": []
}

Download the file config.yml from example configuration and modify it (I choose to place the key files in current folder rather than /etc/nebula for experiment purposes)

...
pki:
  ca: .../ca.crt
  cert: .../<host>.crt
  key: .../<host>.key
...
static_host_map:
  "192.168.111.1": ["x.x.x.x:4242"]
  "192.168.111.2": ["x.x.x.x:4242"]
...
lighthouse:
  am_lighthouse: false # or true
  hosts: # or empty if true above
    - "192.168.111.1"
    - "192.168.111.2"
...
firewall:
  inbound:
    # Allow icmp between any nebula hosts
    - port: any
      proto: icmp
      host: any

    # Allow tcp/443 from any host with BOTH laptop and lan group
    - port: 443
      proto: tcp
      groups:
        - laptop
        - lan

    # remove others

Then copy the files to corresponding hosts. For example:

scp config.yml ca.crt vps1.crt vps1.key <user>@<vps1 host>:/.../nebula

And modify config.yml for the host, notably the key files, whether am_lighthouse is true, and the list of posts.

Before launching, make sure the firewall of the lighthouse hosts do not block UDP port 4242. Then launch with the command:

sudo ./nebula -config /.../config.yml

You can check the interface nebula1 to see if it is up:

# or ip addr show dev nebula1
$ ifconfig nebula1
nebula1: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1300
        inet 192.168.111.1  netmask 255.255.255.0  destination 192.168.111.1
        inet6 fe80::379f:b974:ad42:e7f0  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7  bytes 336 (336.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

After launching the second host, see if you can ping from each host. For example, I ping from lighthouse 192.168.111.1 to another lighthouse 192.168.111.2:

$ ping 192.168.111.2
PING 192.168.111.2 (192.168.111.2) 56(84) bytes of data.
INFO[0557] Handshake message sent
...
INFO[0557] Handshake message received
...vpnIp=192.168.111.2

It also works for hosts behind NAT on a local LAN.

systemd Service

People also want to start Nebula as a systemd service with a non-root user. The service given below is based on this:

[Unit]
Description=Nebula
Wants=basic.target
After=basic.target network.target
# Before=sshd.service

[Service]
ExecStartPre=/usr/local/bin/nebula -test -config /etc/nebula/config.yml
ExecStart=/usr/local/bin/nebula -config /etc/nebula/config.yml
ExecReload=/bin/kill -HUP $MAINPID

RuntimeDirectory=nebula
ConfigurationDirectory=nebula
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
ProtectControlGroups=true
ProtectHome=true
ProtectKernelTunables=true
ProtectSystem=full
User=nebula
Group=nebula

SyslogIdentifier=nebula

Restart=always
RestartSec=2
TimeoutStopSec=5
StartLimitInterval=0
LimitNOFILE=131072

Nice=-1

[Install]
WantedBy=multi-user.target
Make sure you have the nebula user and group created, or use the non-root user you want. Also, it is advisable to refer to the official guide and place configuration and key files in /etc/nebula. You probably also want to place the executable nebula in /usr/local/bin. Since I encountered errors when using my user folder from the systemd service file.

Read more