docker-apps/_instructions/jailmaker.txt

Helpful tools
-------------
# Install PuTTY and Pageant for easy copying of files/folders from Windows to NAS server

Jailmaker
---------
# Create datasets
/mnt/SSD1/docker/data (replace SSD1 with name of pool)
/mnt/SSD1/docker/stacks
/mnt/SSD1/jailmaker

install jailmaker
-----------------
# Open shell
cd /mnt/SSD1
git clone https://github.com/Jip-Hop/jailmaker.git
cd /mnt/SSD1/jailmaker
nano ~/.zshrc (assuming shell is zsh)
# add the following alias:
alias jlmkr="sudo -E '/mnt/SSD1/jailmaker/jlmkr.py'"
# execute the contents of .zshrc
source /root/.zshrc
jlmkr list
# should return "No jails."
# Go to System Settings -> Advanced -> Init/Shutdown Scripts -> Add
Description: start jailmaker
Type: Command
Command: /mnt/SSD1/jailmaker/jlmkr.py startup
When: Post Init
Enabled: <checked>
Timeout: 30

Nvidia GPU
----------
# If you want to pass through Nvidia GPU and the truenas scale version is Electric Eel:
# Go to Apps->Configuration - click down arrow and select Settings
# Update the following settings:
#  Install NVIDIA Drivers: <checked>
# Click Save

install docker
--------------
# Go to https://github.com/Jip-Hop/jailmaker
# Navigate to templates/docker/config
# Select then entire config file and copy to clipboard
# Open shell and enter
jlmkr create
# Enter "y" at the prompt: Do you wish to create a jail from a config template? [y/N]
# nano will open
# Right click to paste config copied earlier
Press ctrl-y to scroll to top
# Change gpu_passthrough_nvidia to "1" to install NVIDIA Container Toolkit during initial setup (if a want to pass through an Nvidia card)
gpu_passthrough_nvidia=1
# Ensure that "systemd_nspawn_user_args" is set according to the interface setup of your truenas
# e.g. if bridge, change to:
systemd_nspawn_user_args=--network-bridge=br1 (br1 is the name of the targeted interface)
    --resolv-conf=bind-host
    --system-call-filter='add_key keyctl bpf'
# or, if no bridge, change to:
systemd_nspawn_user_args=--network-macvlan=enp7s0 (enp7s0 is the name of the targeted interface)
    --resolv-conf=bind-host
    --system-call-filter='add_key keyctl bpf'
# Edit config file further as needed
# Save and exit
Enter jail name: docker
Enter "y" at the prompt: Do you want to start this jail now (when create is done)? [Y/n]
jlmkr list
expected result:
NAME   RUNNING STARTUP GPU_INTEL GPU_NVIDIA OS     VERSION ADDRESSES
docker True    False   False     False      debian 12      10.0.0.162… (ip address might be diffrent)
# If it fails to start: ----
jlmkr edit docker
# Fix the issue in the config file (probably network issue)
# Restart docker
jlmkr restart docker
# end of if it fails to start ----
jlmkr shell docker # open docker shell
apt install nano

container static ip for macvlan
-------------------------------
jlmkr shell docker # open docker shell
nano /etc/systemd/network/mv-dhcp.network
# Comment out existing entries under network with #, set DHCP to false and specify Address and Gateway, e.g.:
[Network]
#DHCP=yes
#LinkLocalAddressing=ipv6
DHCP=false
Address=10.0.0.61/24  # should be ip you want for docker container
Gateway=10.0.0.2      # should be default gateway

reboot 
jlmkr shell docker
ip a (verify that ip is correct)

container static ip for bridge
------------------------------
https://github.com/Jip-Hop/jailmaker/blob/main/docs/network.md
nano /etc/systemd/network/80-container-host0.network
# Comment out existing entries under network with #, set DHCP to false and specify Address, Gateway and LinkLocalAddressing=no, e.g.:
[Network]
#DHCP=yes
#LinkLocalAddressing=ipv6
#LLDP=yes
#EmitLLDP=customer-bridge
DHCP=false
Address=10.0.0.61/24
# comment Gateway line if this is not the default interface
Gateway=10.0.0.2
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge

systemctl restart systemd-networkd
systemctl status systemd-networkd
ip a (verify that ip is correct)

Making docker ip address static using router/dhcp server
--------------------------------------------------------
# NB! the static ip address setting in network files explained above is the preferred method as docker sometimes contact dhcp server with what appears to be an ipv6 mac instead of ipv4 mac.
jlmkr shell docker (open docker shell)
ip a
# Note the mac address of mv-enp7s0@if2 (enp7s0 is the interface specified in the config file)
# Set up dhcp server to issue same ip for this mac (static ip)
reboot docker jail
jlmkr shell docker
ip a (verify that ip is correct)
# If not, release and renew IP as follows
dhclient -v -r mv-enp7s0  (where mv- is the prefix defined in /etc/systemd/network/mv-dhcp.network under [Match]; enp7s0 is the host i/f docker is tied to) 
dhclient -v mv-enp7s0

Additional Bridge Interface
---------------------------
# https://github.com/Jip-Hop/jailmaker/discussions/179#discussioncomment-9499289
# Create extra interfaces and join them to host bridges manually with systemd-nspwan 

The --network-veth-extra argument instructs system-nspawn to create an addition linked interface between the host and jail and uses a syntax of
--network-veth-extra=ve-docker-1:vee-eth1 #Adds virtual link between host and jail with ve-docker-1 i/f being created in the host and vee-eth1 being created in the jail
or
--network-veth-extra=ve-docker-1 #Adds virtual link between host and jail with ve-docker-1 i/f being created in the host and ve-docker-1 being created in the jail, i.e. uses the same name

# However, for network connectivity, we need to connect to a bridge on the host.
# To create bridge interface on the host (truenas):
# Go to Network
# Select physical i/f that we will bridge to, e.g. ens2f0, and click on edit (pen icon)
# Note the IP address assigned to the interface
# Uncheck DHCP if selected and remove aliases if any
# Save
# DO NOT CLICK ON "Test Changes" OR "Revert Changes" YET, as we will add bridge first
# Click on Add
Type: Bridge
Name: br1
Description: Link to Docker - 192
DHCP: <unchecked>
Autoconfigure IPv6: <unchecked>
Bridge Members: ens2f0
Aliases: Add
IP Address: <same as the IP that was assigned to ens2f0>
# Save
# Now click on "Test Changes"

# Next, we will edit docker config file to configure additional virtual ethernet link and bridge connection
jlmkr edit docker
# Add --network-veth-extra argument to systemd_nspawn_user_args, similar to the following:
systemd_nspawn_user_args=--network-macvlan=enp7s0
        --network-veth-extra=ve-docker-1:vee-eth1  # new addition, NB! jail interface name must start with vee-
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        ...

# In order for this virtual link to connect to the outside, we need to link ve-docker-1 to br1 on the host (truenas).
# This is done with the ExecStartPost post commands in the initialisation script.
# Add post_start_hook stanza, similar to the following: (NB! this is separate to the pre-existing pre_start_hook stanza, i.e. it is standing by itself)
# Script to run on the HOST after starting the jail
post_start_hook=#!/usr/bin/bash
        set -euo pipefail
        echo 'POST_START_HOOK'
        ip link set dev ve-docker-1 master br1    # links ve-docker-1 to br1 on host
        ip link set dev ve-docker-1 up            # activates ve-docker-1 on host

# Save and exit
jlmkr restart docker
# Verify in host/truenas that ve-docker-1 interface was created
jlmkr shell docker
ip a
# Verify that vee-eth1@##### interface was created

# Add static ip as network config for new network
nano /etc/systemd/network/vee-dhcp.network
# Comment out old settings under [Network] and add ip address and gateway
# -------------- snip --------------
[Match]
Virtualization=container
Name=vee-*

[Network]
#DHCP=yes
#LinkLocalAddressing=ipv6
#Bridge=br0
DHCP=false
Address=192.168.2.6/24
# uncomment following line if this is the default interface
#Gateway=192.168.2.1

[DHCPv4]
UseDNS=true
UseTimezone=true
# -------------- snip --------------
systemctl restart systemd-networkd
systemctl status systemd-networkd
ip a  #verify that ip is correct
# Update dhcp and dns servers with IP config of vee-eth1@##### interface

Add bindings for docker container
---------------------------------
exit # out of docker shell
jlmkr edit docker
# Add the following args to "systemd_nspawn_user_args": 
--bind='/mnt/SSD1/docker/data:/mnt/data'
--bind='/mnt/SSD1/docker/stacks:/opt/stacks'

# systemd_nspawn_user_args should reflect the following (example shows bindings added - other args might differ):
systemd_nspawn_user_args=--network-macvlan=enp7s0  # this will update /etc/systemd/network/mv-dhcp.network
        --resolv-conf=bind-host
        --system-call-filter='add_key keyctl bpf'
        --bind='/mnt/SSD1/docker/data:/mnt/data'       # new addition
        --bind='/mnt/SSD1/docker/stacks:/opt/stacks'   # new addition
        --bind='/mnt/stpool1/NData1/Media:/mnt/media'  # new addition
        --bind='/mnt/stpool1/Downloads:/mnt/downloads' # new addition
        --bind='/mnt/stpool1/Shared_Data:/mnt/shared'  # new addition
        --bind='/mnt/stpool1/apps/pgadmin/storage/stuurmcp_telkomsa.net:/mnt/migrate'   # new addition

# The last 3 bindings are for audio/video, shared and migrate folders respectively. Refer to jellyfin and database restore instructions for usage

jlmkr restart docker

Install Dockge
--------------
jlmkr shell docker
nano ~/install_dockge.sh
# Copy script below OR go to https://dockge.kuma.pet and copy installation script from there to install_dockge.sh:

# ----- snip -----
# Create directories that store your stacks and store Dockge's stack
mkdir -p /opt/stacks /opt/dockge
cd /opt/dockge

# Download your compose.yaml
curl "https://dockge.kuma.pet/compose.yaml?port=5001&stacksPath=%2Fopt%2Fstacks" --output compose.yaml

# Start the Server
docker compose up -d

# If you are using docker-compose V1 or Podman
# docker-compose up -d
# ----- snip -----

# Save and exit
sh ~/install_dockge.sh
# Dockge should install and start

# Login to Dockge
# Go to http://10.0.0.61:5001 (10.0.0.61 is ip of docker container)
# Enter admin as username and choose desired password (write down password)

Set jail to startup when system is started
------------------------------------------
exit #out of docker shell
jlmkr edit docker
# Change first line to:
startup=1

Enabling dockge secure login
============================
# Install traefik using newly installed Dockge. Refer to traefik installation instructions.
# Install a second app and configure it to use traefik
# Troubleshoot any issues with secure access to the app and/or traefik dashboard
# This will ensure correct installation and working of traefik once you have success with secure accessing the new app
# Open docker shell:
jlmkr shell docker
cd /opt/dockge
# Make backup of compose.yaml
cp -v compose.yaml compose.yaml.bak
nano compose.yaml
# Under server.dockge
# Comment out ports stanza, e.g.:
#    ports:
#      - 5001:5001
# Add the following under server.dockge
    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik-net1
      - traefik.http.routers.dockge-rtr.rule=Host(`dockge.sthome.org`)
      - traefik.http.routers.dockge-rtr.entrypoints=websecure
      - traefik.http.routers.dockge-rtr.tls.certresolver=sthomeresolver
      - traefik.http.routers.dockge-rtr.service=dockge-svc
      #
      - traefik.http.services.dockge-svc.loadbalancer.server.port=5001
    networks:
      - traefik-net1

# Add the following root entry:
networks:
  traefik-net1:
    external: true
# You can remove "version" line, i.e. the first line, to avoid docker complaining about version being obsolete on container restart
# Finished product should look as follows:
------------- snip -------------
services:
  dockge:
    image: louislam/dockge:1
    restart: unless-stopped
#    ports:
#      - 5001:5001
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data:/app/data
      # Stacks Directory
      # ⚠️ READ IT CAREFULLY. If you did it wrong, your data could end up writing into a WRONG PATH.
      # ⚠️ 1. FULL path only. No relative path (MUST)
      # ⚠️ 2. Left Stacks Path === Right Stacks Path (MUST)
      - /opt/stacks:/opt/stacks
    environment:
      # Tell Dockge where to find the stacks
      - DOCKGE_STACKS_DIR=/opt/stacks
    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik-net1
      - traefik.http.routers.dockge-rtr.rule=Host(`dockge.sthome.org`)
      - traefik.http.routers.dockge-rtr.entrypoints=websecure
      - traefik.http.routers.dockge-rtr.tls.certresolver=sthomeresolver
      - traefik.http.routers.dockge-rtr.service=dockge-svc
      #
      - traefik.http.services.dockge-svc.loadbalancer.server.port=5001
    networks:
      - traefik-net1
networks:
  traefik-net1:
    external: true
------------- snip -------------
# Save and exit
docker compose restart
# Test access to dockge using https://dockge.sthome.org (no port should be specified)

Replacing dockge docker.sock with a socket proxy
================================================
# refer: https://github.com/louislam/dockge/discussions/369
# The following instruction:
#   1. Replaces dockge service's /var/run/docker.sock volume with a DOCKER_HOST environment variable
#   2. Adds socket-proxy service to project 
# in Truenas shell, edit dockge compose.yml
nano /mnt/SSD1/jailmaker/jails/docker/rootfs/opt/dockge/compose.yaml
# under services.dockge.volumes comment out "/var/run/docker.sock:/var/run/docker.sock"
# under services.dockge.environment add "DOCKER_HOST=tcp://socket-proxy-dockge:2375"
# under services.dockge.networks add "socket_proxy"
# Add the socket_proxy under top level "networks", e.g.:
# ---- snip ----
networks:
  traefik-net1:
    external: true
  socket_proxy:
    driver: bridge
# ---- snip ----

# Amend services.dockge.networks by adding socket-proxy
# ---- snip ----
    networks:
      - traefik-net1
      - socket_proxy
# ---- snip ----

# Add the following under services
# --- snip ---
  socket-proxy:
    container_name: socket-proxy-dockge
    image: tecnativa/docker-socket-proxy
    restart: always
    networks:
      socket_proxy:
    #     ipv4_address: xyz
    privileged: true
    # ports:
    #   - "127.0.0.1:2375:2375"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    environment:
      - LOG_LEVEL=debug # debug,info,notice,warning,err,crit,alert,emerg
      ## Variables match the URL prefix (i.e. AUTH blocks access to /auth/* parts of the API, etc.).
      # 0 to revoke access.
      # 1 to grant access.
      ## Granted by Default
      # - EVENTS=1
      # - PING=1
      # - VERSION=1
      # # Security critical
      - AUTH=0
      - SECRETS=0
      - POST=1          # dockge 
      # # Not always needed      
      - BUILD=0
      - COMMIT=0
      - CONFIGS=0
      - CONTAINERS=1    # dockge, Portainer, Traefik
      - DISTRIBUTION=0
      - EXEC=1          # dockge (run Container Shell)
      - IMAGES=1        # dockge, Portainer
      - INFO=1          # dockge, Portainer
      - NETWORKS=1      # dockge, Portainer
      - NODES=0
      - PLUGINS=0
      - SERVICES=1      # dockge, Portainer
      - SESSION=0
      - SWARM=0
      - SYSTEM=1
      - TASKS=1
      - VOLUMES=1
# --- snip ---
# See dockge\compose.yml for file with traefik reverse proxy and socket-proxy added
# This is the preferred config for dockge

Troubleshooting
---------------
If you having trouble accessing dockge after applying the above:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Check if traefik is running (note: if traefik is not running, Dockge will be unavailable)
# Restart traefik inside the /opt/stacks/traefik folder with docker compose up -d or 
# if traefik cannot be restarted, undo the secure dockge login by stepping through the following:
 # uncomment the ports stanza in /opt/dockge/compose.yaml
 cd /opt/dockge
 docker compose stop
 docker compose build
 docker compose up -d
 # use dockge.sthome.org:5001 to access Dockge
# After sorting out the issue with traefik, it is important to comment out the ports stanza in the dockge compose file and rebuild & restart as explained above.
# Otherwise insecure access to dockge will remain available.

If containers use the wrong default route
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# in docker shell, list ip routes
ip r
# note the default gateway entry(ies), i.e. lines starting with "default via" followed by a gateway ip and an network interface name. See if there are more than one of these lines
# if more than one, you need to remove the default route(s) that are irrelevant / incorrect
# to remove extraneous default route(s) use "ip route del" followed by the text displayed with "ip r" command of the extraneous default gateway, e.g.:
ip route del default via 192.168.2.1 dev vee-eth1 proto static
# double check the network configs to see if there are more than one gateway in total, e.g.:
nano /etc/systemd/network/vee-dhcp.network
nano /etc/systemd/network/80-container-host0.network
# comment out extraneous gateway(s)