I Have a Problem (Two Really)

As of writing this post, my VPS still has its SSH port open. Although this is very common and generally safe since no one can access it without my private key but I feel uneasy leaving such an important port open to the internet.

But I know of a solution - Tailscale. Tailscale is a network service that can connect all my devices (laptop, phone and VPS) in a private network like a VPN.

My second problem is that I don’t know what is happening in my VPS. I know which applications run on it for the most part but I don’t know if one has crashed, how much CPU, RAM, or storage it is using or if my VPS is about to run out of storage altogether.

Beszel solves this problem for me. Beszel is a lightweight server monitoring platform that includes a friendly web UI, gives me a dashboard view of all the resource usage in my VPS including Docker statistics.

I am going to use Tailscale to allow access to my Beszel UI in the private network without opening any ports or setting up DNS or subdomains and I will also be closing my ssh port from the Hetzner firewall.

Now let’s understand how Tailscale works before setting both of these up

How Does Tailscale Work

It creates a private mesh network (called a tailnet) between my devices using WireGuard

First sign up and install the clients on each device (VPS, laptop, phone)

After authentication you get a stable private IP

it is as simple as that but there are questions which I had on how does all of this work

Questions

  1. How does WireGuard fit into all of this?
  2. Where does the private IP come from?
  3. How can it accept inbound requests with no inbound ports open?

Answers

How Does WireGuard Fit into All of This?

WireGuard is a VPN protocol - each device generates public/private key pair, each device knows the public keys of the peers it’s allowed to talk to and few more things and it runs on UDP

Where Does the Private IP come From?

When I use the app, it generates a key pair and registers with a Tailscale coordination server under my account. The coordination server assigns the private IP, keeps track of the virtual network (tailnet), and sends the details of all other devices in the network.

How Can It Accept Inbound Requests with no Inbound Ports Open?

Whenever two devices want to talk to each other, they first contact each other’s through Tailscale’s relay servers called DERP (Designated Encrypted Relay for Packets). DERP is used by Tailscale to pass encrypted data between your devices when a direct connection cannot be made.

Each device maintains a persistent outbound connection with a nearby DERP server at all times. Since the connection is over TCP (port 443) it is never blocked by any firewall unless manually set up.

The devices exchange their info (like public IP/port), and both sides then try to punch a direct UDP path through the NAT. This basically means that both devices using the info DERP helped them exchange, send a UDP packet at each other at roughly the same time.

NAT here stands for Network Address Translation.

Each device’s outbound packet creates a temporary mapping in their NAT table that says “expect a reply from this address”. When the other device sends a packet and it arrives at the NAT, it matches the mapping created so it gets let through even though there were no open ports.

This technique is called NAT hole punching and is used to establish a direct peer-to-peer connection between two devices when any one of them is behind restrictive routers or firewalls.

After this point the two devices can communicate with each other over a WireGuard encrypted, peer-to-peer connection without any DERP server in the middle.

What else does DERP do for us here?

Now if in the previous step the direct connection has failed, the devices just fall back to use the existing outbound connection with the DERP relay and use it to communicate.

DERP then becomes a switchboard operator from the early telephone days. Each devices writes a packet to the outbound connection with DERP with the other device’s public key as the destination and DERP looks up which outbound connection the other device has with it and forwards the packet out from there.

DERP still preserves the end-to-end encryption as it is still relaying the WireGuard encrypted packets.

More Questions

If Tailscale Uses UDP Then How Does SSH Work with TCP Reliability?

The SSH client still does use TCP to connect, but it connects to the Tailscale private IP of the remote device. From there the traffic hits the tailscale0 virtual interface which will use WireGuard to encrypt the entire TCP payload inside a UDP packet and that UDP packet is sent over the internet. The remote server’s tailscale0 interface again with WireGuard decrypts the payload and sends the original TCP packet up to the kernel. So the client and remote server think they are communicating via TCP.

Tailscale and Beszel

Now that we have gone deep into that rabbit hole, let’s climb back up and actually do what we set out to accomplish.

Setting up Tailscale on Our Device and VPS

I am basically just going to follow the official instructions found here .

  1. Go to tailscale.com and select Get Started.

Tailscale homepage

  1. Sign up with Google (or any other option)

Tailscale Signup Page

  1. Set up the account

Tailscale Profile setup

  1. Download and install the Tailscale app I am downloading the mac version

Tailscale Download

I am going to install it on my phone as well and log in with the same Google account.

  1. Installing on the VPS

    SSH into the VPS

    curl -fsSL https://tailscale.com/install.sh | sh

    sudo tailscale up

Then we just log in using the same account as before and done

  1. Setup SSH on the VPS We need to tell our VPS that tailscale will managing the SSH connections for it from now

    sudo tailscale set --ssh

Tailscale Console

I have also removed the inbound rule allowing my SSH port from the firewall.

Setting up Beszel over Tailscale

  1. SSH into the VPS

    ssh fazil@hetzner-tailscale

  2. Create a directory for beszel

    mkdir -p ~/beszel && cd ~/beszel

  3. Create Docker compose for the Beszel Hub and paste in the official docker-compose.yml

I have changed the host port to bind 127.0.0.1 IP to the host, this is to prevent Beszel from being reachable from the public interface of the VPS

services:
  beszel:
    image: henrygd/beszel
    container_name: beszel
    restart: unless-stopped
    environment:
      - APP_URL=http://localhost:8090
    ports:
      - 127.0.0.1:8090:8090
    volumes:
      - ./beszel_data:/beszel_data

  1. Start the Beszel Hub

    docker compose up -d beszel

  2. Expose it privately over the tailnet:

    sudo tailscale serve --bg localhost:8090

  3. Open the Beszel URL and create your admin account.

  4. In the Beszel UI, click “Add System.”

    Beszel Systems and copy the compose file for the agent that Beszel gives you

  5. Create a directory for the agent

    mkdir agents && cd agents

  6. Create the compose file and paste the copied code

    nano docker-compose.yml

  7. Start the Beszel agent

    docker compose up -d

Now Beszel is up and running and we can see the status of our VPS from anywhere in the world

Beszel Monitoring 1

Beszel Monitoring 2

Final Thoughts

My SSH port is now closed to the internet, Beszel is accessible only through my Tailnet, and I have a much better view of what’s happening on my VPS.

I learned a lot while setting all of this up, and even more while writing this post. If you found it useful, I’d really appreciate it if you shared it with someone who might be interested.

If I’ve explained something incorrectly or missed an important detail, feel free to reach out to me at syedfazil539@gmail.com . I’d be happy to correct it and learn something new.