Homelab security with open source tools

Informatec Digital » Resources » Homelab security with open source tools

Setting up a homelab today is like having a small data center at home with services 100% under your control : private cloud, home automation, backups, multimedia, even generative AI. But as soon as you start opening ports, exposing services, or connecting IoT devices, the question arises naturally: how do I keep all this secure without spending a fortune and using open-source tools?

If you already have a Synology or QNAP NAS, a server with Proxmox, or even a simple mini PC running Docker, this content will be perfect for you. We'll cover how to secure a home lab with open-source software , what alternatives you have to exposing services directly to the internet, how to segment your network, how to access it with a mesh VPN (Tailscale, NetBird, ZeroTier), what to use to protect passwords, backups, security cameras, and your personal cloud, and how to fit all these pieces together without going crazy.

What exactly is a homelab and why does safety matter so much?

A modern homelab is no longer just "an old PC acting as a server," but an ecosystem of self-managed services: cloud computing, multimedia, home automation, and AI running 24/7. Thanks to increasingly refined open-source projects, it's easy to set up something at home that looks suspiciously like the infrastructure of a small business.

In many cases, the heart of the home lab is a NAS (Synology, QNAP, TrueNAS, OpenMediaVault, etc.) or a hypervisor like Proxmox VE, accompanied by Docker or Kubernetes managed with Portainer, Rancher, or other orchestration layers. On top of that foundation, you deploy Plex or Jellyfin, Nextcloud, Home Assistant, AI apps, monitoring dashboards, and a thousand other things.

The problem arises when you start exposing services to the outside world using the NAS's reverse proxy, opening ports on the router without much thought, or connecting dozens of IoT devices without segmenting the network . Suddenly, what was once a fun project becomes a very tempting attack surface. And if you also store family photos, sensitive documents, or access to your bank account, you can imagine the risk.

The good news is that the open source ecosystem offers everything needed to set up a secure homelab, accessible from outside and with best practices very close to those of professional environments, but without recurring costs or with free plans sufficient for a home infrastructure.

Homelab foundation: virtualization, containers, and secure storage

Security begins long before thinking about VPNs or tunnels. A solid foundation involves choosing the right hypervisor, how you manage containers, and how you store data to minimize risks and facilitate backups and restorations.

In the container realm, options like Portainer or Rancher simplify managing Docker and Kubernetes through a web interface, minimizing the hassle of the command line. Portainer is ideal if you only need to manage Docker or a small cluster, while Rancher feels more natural if you're more experienced with Kubernetes, such as K3s or multiple nodes.

If you're looking for something that lets you install services with a single click, projects like CasaOS, Runtipi, or Cosmos function as a kind of self-hosted "app store ." They're very useful for beginners, although it's best not to overuse them to continue understanding what's being deployed and which ports are being opened.

In the realm of virtual machines and serious storage, a typical setup is to use Proxmox VE as the primary hypervisor and a NAS based on TrueNAS or OpenMediaVault as the storage backend. With ZFS, snapshots, and replication, you can better isolate services, run tests on lab VMs, and maintain consistent copies of your critical data.

A realistic example: a QNAP TS-253E with RAID 1 disks and a 16 TB external drive for general backups provides a centralized location for Docker volumes, ISOs, backups, and media libraries . On top of this, Proxmox or the NAS's own operating system hosts containers and VMs with separate services, so that a failure in one part doesn't bring down the rest of the system.

Network segmentation and isolation: first line of defense

Before considering exposing Overseerr, Plex, or the *arrs, it's advisable to organize your internal network. One of the best practices, both in businesses and at home, is to segment the network into different zones with specific subnets based on the type of device and its level of trust.

A very practical design in homelab is to separate at least four segments: a LAN for reliable equipment (personal PCs, some critical devices), a guest network for visitors, an IoT network for "suspicious" gadgets, and, if you have many SBC-type devices, a specific segment just for them, isolated but accessible via static routes.

For example, you can define something like this:

• LAN – 192.168.1.0/24: trusted teams, without internal restrictions.
• GUEST – 192.168.2.0/24Guest Wi-Fi, devices isolated from each other and with restricted internet access.
• IoT – 192.168.3.0/24: smart plugs, LED strips, air purifiers, smart speakers, LoRa controllers….
• SBC – 192.168.4.0/24Raspberry Pi, BeagleBone and other boards, connected only by cable, with controlled access.

The main router (or an advanced neutral router) enforces firewall policies across networks, preventing IoT devices from freely accessing your NAS or computers , and hindering the guest network from scanning your home lab. From the LAN, you can access everything else, and from the SBC segment, you can act as a router to specific areas using well-defined static routes.

This type of design has another advantage: when some teams also participate in virtual private networks like Tailscale , it is much easier to decide what is exposed through the VPN and what remains completely locked in a local segment with no direct exit.

Secure remote access: Mesh VPN, tunnels, and reverse proxies

One of the most common mistakes in homelabs is directly exposing services like Plex, Overseerr, Sonarr, Radarr, or the NAS's own admin panel through the built-in reverse proxy and a couple of router rules. It's convenient, yes, but it also opens the door to brute-force attacks, zero-day exploits, and mass scans.

If you're the only one using those services, the most sensible option is to keep them private and access them only through a VPN . Instead of setting up a traditional VPN like OpenVPN or WireGuard with manual configurations, mesh solutions are becoming increasingly common, greatly simplifying the process.

In many homelabs, the ideal scenario is to leave only one service exposed for third-party use (for example, Overseerr so your friends can request multimedia content ) and keep the *arrs, the Docker admin panel, and the rest of the services accessible only via VPN. This reduces the attack surface and forces access to any sensitive information through an encrypted tunnel.

When you do need to expose something publicly (a website, a blog, or a service that must be accessible without a VPN), solutions like Cloudflare Tunnels or open-source alternatives like NetBird with its reverse proxy functionality come into play . The latter is shaping up to be an interesting replacement for Cloudflare Tunnels for those already using NetBird as their private network.

NetBird and other open source reverse proxies focused on security

NetBird began as a WireGuard-based virtual private network solution, and over time it has expanded its functionality to include an open-source reverse proxy capable of exposing internal services without the need for external proprietary tunnels. For those with a homelab where services sometimes need to be public, this significantly reduces reliance on third parties.

Among the most interesting features of NetBird's reverse proxy are automatic TLS support with Let's Encrypt certificates , so you don't have to struggle with manual renewals or complex Nginx or Traefik configurations for each service you add.

At the authentication level, the proxy allows you to choose between several options: SSO integrated with your identity provider, password authentication, PIN, or even unprotected public mode (which you should only use for services truly intended for the general public). This flexibility helps tailor each endpoint to the associated risk.

Furthermore, NetBird's routing capabilities are quite powerful: you can perform route-based routing , for example, sending /api to one specific service and /docs to a different one, as long as they are accessible within the NetBird network. And it doesn't stop at a single proxy; it's designed to scale with multiple nodes if your homelab grows larger.

As an alternative or complement, many homelab installations still rely on reverse proxies like Traefik, Nginx Proxy Manager, or Caddy , which also offer Let's Encrypt integration, advanced routing, and additional authentication. The key is to avoid leaving services exposed "without protection," always using a well-configured proxy with HTTPS and clear access rules.

Open source security and video surveillance cameras in a homelab

Another typical use case is setting up a home security camera system using open-source software , for example, to monitor the home of retired parents or a second residence. Here, security is twofold: protecting access to the cameras and avoiding dependence on third-party cloud services.

If you already have Blink cameras or other IP cameras, the first thing to do is check how accessible they are through open-source solutions. Some brands allow access to RTSP or HTTP streaming, while others are very closed and only work with their cloud-based app. Depending on this, you'll be able to integrate more or fewer elements into your home lab.

Among the most popular open-source projects for video surveillance are options like ZoneMinder, MotionEye, and Frigate (the latter being especially popular when integrating cameras with Home Assistant and requiring AI-powered person or object detection). All of them allow continuous or event-driven recording, alerts, and centralized management of multiple cameras.

For this system to be truly secure, ideally the cameras should reside on the IoT network, without direct access to the LAN , and the server running the video surveillance software should be responsible for collecting the images, storing them securely on your NAS, and exposing the interface only through the LAN or via VPN.

If you want your family members to be able to view the cameras from outside your home, you can combine Home Assistant or the video surveillance system itself with a mesh VPN like Tailscale or a reverse proxy like NetBird or Traefik, protected with strong authentication. This prevents you from opening essential ports like 80 or 554 (RTSP) to the outside world.

Services for everyday use: personal cloud, photos, multimedia and AI

Beyond pure security, one of the reasons to go to the trouble of setting up a homelab is to stop relying on Google Drive, Google Photos, Netflix, and similar services and bring all those services to your own infrastructure. The interesting thing is that many of these tools can be integrated relatively easily and securely.

For file storage and synchronization, Nextcloud is the de facto standard, supporting files, calendars, contacts, notes, and collaborative editing using Collabora or ONLYOFFICE. If you're looking for something lighter or with a different focus, projects like Seafile, Filestash, ownCloud, or Pydio Cells offer viable alternatives.

In the realm of personal photos and videos, tools like Immich, PhotoPrism, and LibrePhotos allow you to deploy a fairly decent clone of Google Photos , complete with facial recognition, automatic tagging, and content search. These applications tend to be resource-intensive, so it's advisable to install them on a server with a GPU or at least a good CPU and fast storage.

For multimedia in general, the combination of Jellyfin as a media center, Navidrome for streaming music, and Audiobookshelf for audiobooks and podcasts covers virtually the entire spectrum of home entertainment. Jellyfin has established itself as the free alternative to Plex/Emby, without licenses or restrictions on basic features.

If you'd like to go further, the homelab is an ideal place to experiment with generative AI and LLMs locally. Projects like Ollama simplify the download and execution of models such as Llama, Gemma, or DeepSeek , and they also expose an API compatible with OpenAI's, making it easy to integrate chatbots into other applications.

To interact with these models from your browser, you have interfaces like Open WebUI, Lobe Chat, or Anse , which support both local models and external services and add features such as history, workspaces, and RAGs. And if you want to go a step further and build complex agents or flows, tools like Flowise, Dify, or Cheshire-Cat allow you to design AI pipelines with nodes, memories, and external tools.

Home automation, IoT and automation: power and risks in the same play

Home automation is another fundamental aspect of the modern homelab. Thanks to open-source projects, you can integrate light bulbs, plugs, sensors, televisions, air purifiers, or LoRa controllers into a single panel, create complex automations, and even link it to your local AI system.

The undisputed king in this area is Home Assistant, which acts as a centralized automation platform from which almost any IoT device on the market can be controlled. It can be deployed on a Raspberry Pi, a Proxmox VM, or even in containers, and integrates seamlessly with the segmented networks mentioned earlier.

For more "flow-based" automations or integrations between services and APIs, Node-RED and n8n stand out , allowing the creation of visual pipelines by combining triggers, transformations, and actions. Other tools like Activepieces or Huginn focus more on "agent-type" automations, reacting to external events such as RSS feeds, emails, or website changes.

A good security practice here is to place all IoT devices on the IoT network, with controlled access and minimal internet connectivity . Home Assistant, which can be on the LAN or the SBC segment, is allowed to communicate with them, but not vice versa. This way, if a device becomes vulnerable, it cannot pivot to your NAS or your personal computers.

To access Home Assistant from outside your network, instead of opening its port to the outside, the ideal solution is to use a mesh VPN like Tailscale or a solution like NetBird , or expose it with a reverse proxy protected by strong authentication and valid TLS certificates. The goal is to ensure it's never left "raw" on the internet with only a simple password as the barrier.

Monitoring, analytics, and incident response

As your homelab grows, it becomes very useful to set up a monitoring and observability system that alerts you when something goes wrong , in addition to attractive dashboards for viewing the overall status of your infrastructure. It's not just about being a techie: it greatly helps in detecting early failures and potential security incidents.

The classic combination is Prometheus as a metrics collector and Grafana as a dashboard engine . With this, you can monitor everything from CPU and memory load on your VMs to NAS disk space and the status of home automation services. Many homelab projects already include exporters ready for integration with Prometheus.

If you want something more plug-and-play, Netdata offers full-stack monitoring with almost no configuration , while Glances provides a quick overview via terminal or web. To check if your services are accessible and receive alerts when they go down, tools like Uptime Kuma are simple and very effective in home environments.

It also makes sense to set up self-hosted web analytics for your personal websites or projects without relying on Google Analytics. Solutions like Plausible, Umami, Matomo, or Openpanel allow you to collect traffic statistics while respecting privacy. And if you're interested in business analytics on your own databases, Metabase, Redash, or PostHog offer a powerful range of options.

For those who want to take security to the next level, there are SOC-level projects like Wazuh, OpenCTI, TheHive, and Cortex , designed for intrusion detection, analysis of indicators of compromise, and incident management. They are more advanced and perhaps a bit overkill for a small homelab, but they work very well in laboratory and training environments.

Passwords, secrets, and backups: what you can't do without

None of the above makes sense if you don't take care of two basic pillars: secure password and secret management, and a decent backup strategy . Many home labs fail right here, and it's where it hurts the most when something goes wrong.

On the password side, you have the option of setting up your own manager with tools like Bitwarden, Vaultwarden, KeeWeb, or Passbolt . Vaultwarden, in particular, is a lightweight implementation of the Bitwarden server, perfect for home labs, allowing you to use the official clients and keep your entire vault at home.

When it comes to backups, it's ideal to use tools that offer encryption, deduplication, and efficient use of space . Restic, BorgBackup, Kopia, Duplicati, and Rclone fit this profile perfectly and can be used on local drives, your NAS, or storage providers like S3, Backblaze, and similar services.

A common saying in the community is that if you don't have a backup, you don't have a homelab . The sensible approach is to automate regular backups of your critical data (Proxmox configurations, Docker volumes, service databases, photos, personal documents) to another disk or even another physical location, combining NAS snapshots with file-level or block-level backups.

Furthermore, it's worthwhile to have an internal wiki with documentation of your infrastructure . Projects like BookStack, Wiki.js, or Docmost allow you to keep track of how your network is segmented, which services are deployed, internal credentials, restoration scripts, and so on. This "source of truth" saves you a lot of trouble when you need to modify something months later.

How to choose where to start and avoid the new toy syndrome

With so many open-source options available, it's easy to fall into the trap of wanting to install absolutely everything and end up with a chaotic, insecure, and difficult-to-maintain homelab . The key is to prioritize and proceed in phases, ensuring security from day one.

The first step is to decide what you need to solve right now. If your main problem is backups and photos, it makes a lot of sense to start with a well-configured NAS (TrueNAS, OpenMediaVault, or your QNAP/Synology), Nextcloud for your personal cloud, and Immich for photos , all behind a VPN or secure proxy.

If your interest lies in AI and experimentation, you can focus on setting up Ollama with an interface like Open WebUI , taking advantage of a decent GPU if you have one. From there, you can add Flowise or Dify to build more complex agents or flows within your homelab.

For a home automation-oriented approach, it makes a lot of sense to use Home Assistant as the central component and a Tailscale mesh network for secure remote access. Later on, you can integrate Node-RED or n8n, and surround everything with good network segmentation that keeps IoT devices well contained.

Whichever path is chosen, it is advisable to establish a minimum base of security and observability: a clear and tested backup scheme, and a couple of simple monitoring tools (for example, BorgBackup or Resti for backups, and Uptime Kuma or Grafana+Prometheus to know what is going down and when).

With all this in mind, a homelab based on open source software can become a very powerful yet secure platform for your personal services: from private cloud and security cameras to local AI and home automation, as long as you combine network segmentation, remote access via VPN or well-configured proxies, careful password management and automated backups , instead of leaving services open to the world without protection.