Setting up multiple websites with docker compose

It's the Easter weekend and I've booked the whole off next week of as well, so i need a project to work on 🤓

I've been meaning to restructure the web server for awhile now and this seems as good a time as any. Let's see how badly I can mess things up.

Set up Docker

As this post is to serve as my reminder for when I have to redo this in the future I'll assume we're starting with a clean system. So follow the Post install setup then Install Docker.

The only extra command that I need to run is to create a custom user-defined network. Once connected to a user-defined network, containers can communicate with each other using container names.

docker network create myDockerNetwork

Set up Local Storage

My setup for the server assumes that it will be compromised so the only things I bring down from it are logs. With that the files on my local system are considered the source of truth, which means that I can if I need to overwrite the server versions to return to a known good state.

On my local system I have a ProductionServer folder in that is the following structure:-

ProductionServer
├── docker/
└── sites/
    ├── default/
    │   ├── config/
    │   ├── logs/
    │   └── public/
    └── mort8088.com/
        ├── config/
        ├── logs/
        └── public/

Build the docker-compose.yml

The one and only file to stand up all the services I need, the docker-compose.yml lives in the docker folder

we'll start out with the network:-

networks:
	myDockerNetwork:
		external: true

Add the Proxy Server

The first service we need is the proxy service this will deal with all the incoming traffic and any SSL connections that need negotiating.

services:
  nginx-proxy:
    image: nginxproxy/nginx-proxy:1.7
    container_name: nginx-proxy
    restart: always
    ports:
      - "80:80"
      - "443:443"
    environment:
      - ENABLE_IPV6=true
      - TRUST_DOWNSTREAM_PROXY=true
      - DEFAULT_HOST=default.local
    volumes:
      - /etc/nginx/certs:/etc/nginx/certs:ro
      - /etc/nginx/vhost.d:/etc/nginx/vhost.d
      - /usr/share/nginx/html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - myDockerNetwork

Add Let's Encrypt

Next we want to have SSL certificats for the domains we host so we'll be adding the automated ACME SSL certificate generation for Nginx-proxy.

letsencrypt:
  image: nginxproxy/acme-companion:2.5
  container_name: nginx-letsencrypt
  restart: always
  environment:
    - DEFAULT_EMAIL=%adminEmailAdress%
    - nginx_PROXY_CONTAINER=nginx-proxy
  volumes:
    - /etc/nginx/certs:/etc/nginx/certs
    - /etc/nginx/vhost.d:/etc/nginx/vhost.d
    - /usr/share/nginx/html:/usr/share/nginx/html
    - /var/run/docker.sock:/var/run/docker.sock:ro
  depends_on:
    - nginx-proxy
  networks:
    - myDockerNetwork

Add The Default Site

The default site is the first container that needs some extra configuration outside of the compose file.

default_site:
  image: nginx:alpine
  container_name: default_site
  restart: unless-stopped
  environment:
    - VIRTUAL_HOST=default.local
  volumes:
    - ~/sites/default/public:/usr/share/nginx/html:ro
    - ~/sites/default/logs:/var/log/nginx
    - ~/sites/default/config:/etc/nginx/conf.d
  networks:
    - myDockerNetwork

As this is effectively a holding site for any domain that comes to the server that doesn't yet have a site set up for it as well as handling any web requests to the ip address. We'll add in a basic set of files for it to serve.

In the public folder create an index.html and a 404.html file. Put whatever content you want in them and add any supporting files they may reference.

Next up is the Nginx configuration file, create a default.conf in the config folder with the following content

server {
  listen 80 default_server;
  listen [::]:80 default_server;
  
  server_name _;
  
  set_real_ip_from %NetworkSubnetMask%;
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;
  
  root /usr/share/nginx/html;
  index index.html index.htm;
  
  access_log /var/log/nginx/access.log realip;
  error_log /var/log/nginx/error.log;
  error_page 404 /404.html;
  
  location / {
    try_files $uri $uri/ =404;
  }
  
  location = /404.html {
    internal;
  }

  autoindex off;
}

With this there is one other file to add to the config folder but this is just to format the access log file we specified realip in the access_log line so now we need the format file.

Create a file called 00-logformat.conf in the config folder with the following in it:-

log_format realip '$http_x_forwarded_for - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"';

This is the last part I needed so that the logs actually had the requesting client IP and not the Proxy IP.

Add a Virtual Host

With the domains that you want to serve the set up is largely the same as the default server with a few exceptions.

I have a second server block to redirect requests to the shorter https://mort8088.com

server {
  listen 80;
  server_name www.mort8088.com;

  return 301 https://mort8088.com$request_uri;
}

Automating Sync

Because the files on my local system are the source of truth for the server I need a way to pull down any logs from the server and then upload new files and remove any that aren't supposed to be there.

It's scripting time! Create a file in your local ~/bin folder and open it for editing. Start the bash script in the normal way and then we need to set up some variables:-

rsync options

Download -avz --update

• -a: Archive mode – preserves permissions, timestamps, symbolic links, etc.
• -v: Verbose – shows progress/details during transfer.
• -z: Compress – reduces data size during transfer.
• --update: Skip files that are newer on the destination.(probably not needed)

Copies files efficiently while preserving attributes, compressing data, and avoiding overwriting newer files.

For each site that I want the logs for I add a line like this changing out the %domainName% for the appropriate folder name. For a new site just add a new line.

rsync $RSYNC_DOWN_OPTIONS "$USER@$HOST:$REMOTE_SITES/%domainName%/logs/" "$LOCAL_SITES/%domainName%/logs/"

Upload -avz --delete

• -a: Archive mode – preserves file permissions, timestamps, symbolic links, etc.
• -v: Verbose – displays detailed output during transfer.
• -z: Compress – compresses data during transfer for efficiency.
• --delete: Deletes files from the destination that no longer exist in the source.

Synchronises source to destination exactly, preserving attributes, compressing data, and removing extraneous files from the destination.

Because we know there are only two folders we need to maintain the upload block should never need updating

# Sync from local to server.
echo "Syncing from local to server..."
echo "Sites Up..."
rsync $RSYNC_UP_OPTIONS "$LOCAL_SITES/" "$USER@$HOST:$REMOTE_SITES/"
echo "Docker Up..."
rsync $RSYNC_UP_OPTIONS "$LOCAL_DOCKER/" "$USER@$HOST:$REMOTE_DOCKER/"

If anyone manages to upload anything to your public folder it'll get deleted on your next update, if they some how modify your docker-compose file it'll be replaced with your known good file.

You could add a cron-job that once a week runs an update script on the server that stops all your containers pulls updates and restarts them.

Things that go wrong

When you're sorting things out with your set-up don't keep restarting the full docker stack I kept messing things up and ended up hitting the rate limit with Let's Encrypt on more than one occasion. Just restart the container you've changed.

While I was working out the sync script I didn't think about what I was doing and wiped out my local changes by pulling the whole server down before uploading and --update didn't help because the server time was different to my local time.

The reason I had to add set_real_ip_from, real_ip_header & real_ip_recursive to the server config and to put in a custom format for the access log was because the log only had the proxy IP address which is useless for knowing where the request came from. I do plan on rolling my own bad actor blocker at some point.

Conclusion

Using Docker to run a reverse proxy in front of multiple containerised Nginx websites offers several key benefits:

1. Centralised Routing with the Reverse Proxy The reverse proxy handles all incoming traffic and routes it to the correct back-end container this simplifies DNS and public-facing configuration because only one container needs to listen to ports 80/443.
2. Automatic HTTPS from Let’s Encrypt The proxy handles the TLS certificates automatically, so there's no need to manage certificates inside each container.
3. Simplified Networking All containers can communicate over the custom docker network, each site container can remain internal, improving security.
4. Scalability You can add or remove site containers easily by editing the docker-compose.yml and restarting. The proxy config is dynamic so no need to edit anything.
5. Security The proxy acts as a single controlled ingress point, allowing you to apply firewall rules or rate limiting.
6. Clean Separation of Concerns Reverse proxy handles TLS, routing and caching where as the site containers only serve content.
7. Consistent Deployment and Recovery The whole stack is defined in Docker Compose making it reproducible, portable, and easy to rebuild or migrate.

My next job is to investigate more options for securing the servers from bad actors. But until then i am confident that a "If they breach me a I'll burn it to the ground" approach is one I'm happy with.