¶ Mise en place Cluster Swarm
¶ Réferences
¶ Architecture
¶ Installation Swarm
Déclaration DNS interne des 4 VM
Base Debian 12.4 + ajout clé ssh ( dans le template )
¶ Installation Docker
Sur les 4 hosts
# Update et setting repo
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the repository to Apt sources:
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker $USER
¶ Initialisation Swarm
Sur swarm-node01
docker swarm init --advertise-addr 192.168.88.51 --listen-addr 192.168.88.51
Swarm initialized: current node (vhj7qbdz8sefl1g4nebwx8piq) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-66w5tl9s3ykb6haddzn01442wzw3jqi9lgy6zsosg7eogjl9yt-0fw2v4mvn6tlo3fn1vrhvy0iw 192.168.88.51:2377
Sur swarm-node02
docker swarm join --token SWMTKN-1-66w5tl9s3ykb6haddzn01442wzw3jqi9lgy6zsosg7eogjl9yt-0fw2v4mvn6tlo3fn1vrhvy0iw \
--listen-addr 192.168.88.52 \
--advertise-addr 192.168.88.52 \
192.168.88.51:2377
Sur swarm-node03
docker swarm join --token SWMTKN-1-66w5tl9s3ykb6haddzn01442wzw3jqi9lgy6zsosg7eogjl9yt-0fw2v4mvn6tlo3fn1vrhvy0iw \
--listen-addr 192.168.88.53 \
--advertise-addr 192.168.88.53 \
192.168.88.51:2377
Sur swarm-node04
docker swarm join --token SWMTKN-1-66w5tl9s3ykb6haddzn01442wzw3jqi9lgy6zsosg7eogjl9yt-0fw2v4mvn6tlo3fn1vrhvy0iw \
--listen-addr 192.168.88.54 \
--advertise-addr 192.168.88.54 \
192.168.88.51:2377
¶ Installation Traefik
Ajout reseau type overlay
docker network create \
--driver overlay \
traefik-public
mkdir traefik
docker-compose.yml
version: '3'
services:
traefik:
image: "traefik:v2.10"
networks:
- traefik-public
command:
- --log.level=DEBUG
- --accesslog=true
- --api=true
- --api.dashboard=true
- --api.insecure=true
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --providers.docker.network=traefik-public
- --serverstransport.insecureskipverify=true
- --providers.docker.swarmmode
- --providers.file.directory=/etc/traefik/conf
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- ./conf:/etc/traefik/conf
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
- "traefik.http.routers.api-public.rule=Host(`dashboardswarm.msi.stef.lan`)"
- "traefik.http.routers.api-public.entrypoints=websecure"
- "traefik.http.routers.api-public.tls=true"
- "traefik.http.services.api-public.loadbalancer.server.port=8080"
deploy:
placement:
constraints:
- node.role == manager
networks:
traefik-public:
external: true
Création répertoire conf et config
Mes certificat SSL internes sont gérér par pfsense
Je dois déposer ces fichiers de certificats dans treafik
Tout les certificats ne sont pas utiles, mais vu que j'ai une archive toute faite ...
~/traefik:
.
├── conf
│ ├── ssl
│ │ ├── domain-stef-lan.crt.pem
│ │ ├── domain-stef-lan.key.pem
│ │ ├── ssl.toml
│ │ ├── wildcard.ca01.stef.local-cert.pem
│ │ ├── wildcard.ca01.stef.local-key.pem
│ │ ├── wildcard.stef.local-cert.pem
│ │ └── wildcard.stef.local-key.pem
│ └── traefik.toml
└── docker-compose.yml
fichier: conf/traefik.toml
[providers.file]
filename = "/etc/traefik/conf/ssl/ssl.toml"
Fichier conf/ssl/ssl.toml
[[tls.certificates]]
certFile = "/etc/traefik/conf/ssl/domain-stef-lan.crt.pem"
keyFile = "/etc/traefik/conf/ssl/domain-stef-lan.key.pem"
[tls.stores]
[tls.stores.default]
[tls.stores.default.defaultCertificate]
certFile = "/etc/traefik/conf/ssl/domain-stef-lan.crt.pem"
keyFile = "/etc/traefik/conf/ssl/domain-stef-lan.key.pem"
Tout les appels en *.msi.stef.lan & *.stef.lan sont pris en compte par ma chaine de certification.
Etat Architecture
¶ Demarrage du stack
Toute les commande docker deploy sont a faire depuis le manager
stef@swarm-node01:~$ docker stack deploy --compose-file traefik/docker-compose.yml traefik
stef@swarm-node01:~$ docker stack ls
NAME SERVICES
traefik 1
Remarque: la configuration est a corriger le dashboard n'est pas accessible via fqdn mais en http://192.168.88.51:8080 ( non prio pour le projet )
¶ Test avec stack whoami
Whoami est un container qui retourne les informations de connexion web
stef@swarm-node01:~$ mkdir whoami
stef@swarm-node01:~$ cd whoami/
stef@swarm-node01:~/whoami$
¶ Creation du stack
Fichier docker-compose.yml
version: "3.3"
services:
whoami:
image: containous/whoami:v1.3.0
networks:
- traefik-public
command:
- --port=8082 # Our service listens on 8082
deploy:
replicas: 3
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
- "traefik.http.routers.whoami.rule=Host(`whoami.msi.stef.lan`)"
- "traefik.http.services.whoami.loadbalancer.server.port=8082"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls=true"
networks:
traefik-public:
external: true
¶ Demarrage du stack
stef@swarm-node01:~/whoami$ cd
stef@swarm-node01:~$
stef@swarm-node01:~$ docker stack deploy --compose-file whoami/docker-compose.yml whoami
Creating service whoami_whoami
stef@swarm-node01:~$ docker stack ls
NAME SERVICES
javaapp 1
whoami 1
Le stack est up
stef@swarm-node01:~$ docker stack ps whoami
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
uxtrgyzz0rta whoami_whoami.1 containous/whoami:v1.3.0 swarm-node04 Running Running about a minute ago
u6gf02q839p8 whoami_whoami.2 containous/whoami:v1.3.0 swarm-node02 Running Running about a minute ago
y5z7bztxspmp whoami_whoami.3 containous/whoami:v1.3.0 swarm-node03 Running Running about a minute ago
Il y a bien 3 replicas
Vu Treafik
Test sur https://whoami.msi.stef.lan
Hostname: 0d855a32e4f4
IP: 127.0.0.1
IP: 10.0.6.152
IP: 172.18.0.6
RemoteAddr: 10.0.6.4:49138
GET / HTTP/1.1
Host: whoami.msi.stef.lan
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br
Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: _ga=GA1.1.577453403.1690623939; _ga_0C4M1PWYZ7=GS1.1.1691738490.12.1.1691738494.0.0.0; _ga_T11SF3WXX2=GS1.1.1691738490.12.1.1691738494.56.0.0; _ga_K2SPJK2C73=GS1.1.1691738490.12.1.1691738494.56.0.0
Purpose: prefetch
Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Sec-Purpose: prefetch;prerender
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.0.0.2
X-Forwarded-Host: whoami.msi.stef.lan
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: cfe0913a256a
X-Real-Ip: 10.0.0.2
Second test:
Hostname: c23769b2f3ce
IP: 127.0.0.1
IP: 10.0.6.153
IP: 172.18.0.4
RemoteAddr: 10.0.6.4:48034
GET / HTTP/1.1
Host: whoami.msi.stef.lan
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br
Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Cookie: _ga=GA1.1.577453403.1690623939; _ga_0C4M1PWYZ7=GS1.1.1691738490.12.1.1691738494.0.0.0; _ga_T11SF3WXX2=GS1.1.1691738490.12.1.1691738494.56.0.0; _ga_K2SPJK2C73=GS1.1.1691738490.12.1.1691738494.56.0.0
Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.0.0.2
X-Forwarded-Host: whoami.msi.stef.lan
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: cfe0913a256a
X-Real-Ip: 10.0.0.2
Le loadbalancing est ok
Fixons le suivie de session
Fichier docker-compose-sticky.yml
version: "3.3"
services:
whoami-sticky:
image: containous/whoami:v1.3.0
networks:
- traefik-public
command:
- --port=8082 # Our service listens on 8082
deploy:
replicas: 3
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
- "traefik.http.routers.whoami.rule=Host(`whoami.msi.stef.lan`)"
- "traefik.http.services.whoami.loadbalancer.server.port=8082"
- "traefik.http.routers.whoami.entrypoints=websecure"
- "traefik.http.routers.whoami.tls=true"
- "traefik.http.services.whoami.loadbalancer.sticky=true"
- "traefik.http.services.whoami.loadbalancer.sticky.cookie.name=StickyCookie"
networks:
traefik-public:
external: true
Arret du précedant stack
stef@swarm-node01:~$ docker stack rm whoami
Removing service whoami_whoami
Demmarage du nouveau
stef@swarm-node01:~$ docker stack deploy --compose-file whoami/docker-compose-sticky.yml whoami
Creating service whoami_whoami-sticky
Verifications
stef@swarm-node01:~$ docker stack ps whoami
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
nptrdlqzkljb whoami_whoami-sticky.1 containous/whoami:v1.3.0 swarm-node03 Running Running 3 minutes ago
lgodfgrut7jj whoami_whoami-sticky.2 containous/whoami:v1.3.0 swarm-node04 Running Running 3 minutes ago
8l1duslov1bf whoami_whoami-sticky.3 containous/whoami:v1.3.0 swarm-node02 Running Running 3 minutes ago
Cette fois-ci les requetes restent sur le meme endpoint
Noter l'apparition du StickyCookie en ligne 13
Hostname: 48f7cb2a996e
IP: 127.0.0.1
IP: 10.0.6.156
IP: 172.18.0.6
RemoteAddr: 10.0.6.4:49178
GET / HTTP/1.1
Host: whoami.msi.stef.lan
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Accept-Encoding: gzip, deflate, br
Accept-Language: fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Cookie: _ga=GA1.1.577453403.1690623939; _ga_0C4M1PWYZ7=GS1.1.1691738490.12.1.1691738494.0.0.0; _ga_T11SF3WXX2=GS1.1.1691738490.12.1.1691738494.56.0.0; _ga_K2SPJK2C73=GS1.1.1691738490.12.1.1691738494.56.0.0; StickyCookie=6de61de14f9ca39a
Sec-Ch-Ua: "Not_A Brand";v="8", "Chromium";v="120", "Google Chrome";v="120"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
X-Forwarded-For: 10.0.0.2
X-Forwarded-Host: whoami.msi.stef.lan
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: cfe0913a256a
X-Real-Ip: 10.0.0.2
¶ Avancement
¶ Installation Portainer
mkdir ~/portainer
cd portainer
Nous allons deploiyer un stack de deux services
- Service agent: Ce service est deployé sur tout les nodes
- Service portainer: Ces le GUI qui est deployé sur les nodes manager
J'ai modifié le port par défault des agents 9000 a 9002 pour de nécessités futur ( port aussi occupé par minio )
Fichier ~/portainer/portainer-agent-stack.yml
version: '3.2'
services:
agent:
environment:
- AGENT_PORT=9002
image: portainer/agent:2.19.4
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/volumes:/var/lib/docker/volumes
networks:
- agent_network
deploy:
mode: global
placement:
constraints: [node.platform.os == linux]
portainer:
environment:
- AGENT_PORT=9002
image: portainer/portainer-ce:2.19.4
command: -H tcp://tasks.agent:9002 --tlsskipverify
ports:
- "9443:9443"
- "9002:9002"
- "8000:8000"
volumes:
- portainer_data:/data
networks:
- traefik-public
- agent_network
deploy:
mode: replicated
replicas: 1
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
- "traefik.http.routers.portainer.rule=Host(`portainer-swarm.msi.stef.lan`)"
- "traefik.http.services.portainer.loadbalancer.server.port=9000"
- "traefik.http.routers.portainer.entrypoints=websecure"
- "traefik.http.routers.portainer.tls=true"
placement:
constraints: [node.role == manager]
networks:
agent_network:
driver: overlay
attachable: true
traefik-public:
external: true
volumes:
portainer_data:
¶ Deploiement du stack
stef@swarm-node01:~$ docker stack deploy --compose-file portainer/portainer-agent-stack.yml portainer
Creating network portainer_agent_network
Creating service portainer_agent
Creating service portainer_portainer
URL: https://portainer-swarm.msi.stef.lan
Vue Swarm
Vue du stack portainer
¶ Installation Minio
Minio est une solution de stockage S3, Nous allons mettre en place un cluster.
Il faut minimum 4 noeuds ( pour un fonctionnement replicats )
On deplois 4 services ( 1/ noeud)
Pour chaque services
Deux url traefik
- minio.msi.stef.lan : Frontal ( port minio 9000 )
- s3.msi.stef.lan: Interface S3 ( port minio 90001 )
Chaque service est limité a sont propre host
services:
minio1:
...
placement:
constraints:
- node.hostname == swarm-node01
...
minio2:
...
placement:
constraints:
- node.hostname == swarm-node02
...
Fichier ~/minio/docker-composer-stack.yml
version: '3.9'
services:
minio1:
image: 'bitnami/minio:latest'
hostname: minio1
volumes:
- minio1-data1:/bitnami/minio/data
networks:
- traefik-public
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
- MINIO_DISTRIBUTED_MODE_ENABLED=yes
- MINIO_DISTRIBUTED_NODES=minio1,minio2,minio3,minio4
- MINIO_SKIP_CLIENT=yes
- MINIO_BROWSER_REDIRECT_URL=https://minio.msi.stef.lan
deploy:
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
# MINIO console
- "traefik.http.routers.minio.tls=true"
- "traefik.http.routers.minio.rule=Host(`minio.msi.stef.lan`)"
- "traefik.http.routers.minio.service=minio-s"
- "traefik.http.services.minio-s.loadbalancer.server.port=9001"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.secure=true"
# MINIO api
- "traefik.http.routers.cdn.tls=true"
- "traefik.http.routers.cdn.rule=Host(`s3.msi.stef.lan`)"
- "traefik.http.routers.cdn.service=cdn-s"
- "traefik.http.services.cdn-s.loadbalancer.server.port=9000"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.secure=true"
restart_policy:
delay: 10s
max_attempts: 5
window: 60s
placement:
constraints:
- node.hostname == swarm-node01
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
minio2:
image: 'bitnami/minio:latest'
hostname: minio2
volumes:
- minio2-data1:/bitnami/minio/data
networks:
- traefik-public
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
- MINIO_DISTRIBUTED_MODE_ENABLED=yes
- MINIO_DISTRIBUTED_NODES=minio1,minio2,minio3,minio4
- MINIO_SKIP_CLIENT=yes
- MINIO_BROWSER_REDIRECT_URL=https://minio.msi.stef.lan
deploy:
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
# MINIO console
- "traefik.http.routers.minio.tls=true"
- "traefik.http.routers.minio.rule=Host(`minio.msi.stef.lan`)"
- "traefik.http.routers.minio.service=minio-s"
- "traefik.http.services.minio-s.loadbalancer.server.port=9001"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.secure=true"
# MINIO api
- "traefik.http.routers.cdn.tls=true"
- "traefik.http.routers.cdn.rule=Host(`s3.msi.stef.lan`)"
- "traefik.http.routers.cdn.service=cdn-s"
- "traefik.http.services.cdn-s.loadbalancer.server.port=9000"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.secure=true"
restart_policy:
delay: 10s
max_attempts: 5
window: 60s
placement:
constraints:
- node.hostname == swarm-node02
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
minio3:
image: 'bitnami/minio:latest'
hostname: minio3
volumes:
- minio3-data1:/bitnami/minio/data
networks:
- traefik-public
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
- MINIO_DISTRIBUTED_MODE_ENABLED=yes
- MINIO_DISTRIBUTED_NODES=minio1,minio2,minio3,minio4
- MINIO_SKIP_CLIENT=yes
- MINIO_BROWSER_REDIRECT_URL=https://minio.msi.stef.lan
deploy:
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
# MINIO console
- "traefik.http.routers.minio.tls=true"
- "traefik.http.routers.minio.rule=Host(`minio.msi.stef.lan`)"
- "traefik.http.routers.minio.service=minio-s"
- "traefik.http.services.minio-s.loadbalancer.server.port=9001"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.secure=true"
# MINIO api
- "traefik.http.routers.cdn.tls=true"
- "traefik.http.routers.cdn.rule=Host(`s3.msi.stef.lan`)"
- "traefik.http.routers.cdn.service=cdn-s"
- "traefik.http.services.cdn-s.loadbalancer.server.port=9000"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.secure=true"
restart_policy:
delay: 10s
max_attempts: 5
window: 60s
placement:
constraints:
- node.hostname == swarm-node03
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
minio4:
image: 'bitnami/minio:latest'
hostname: minio4
volumes:
- minio4-data1:/bitnami/minio/data
networks:
- traefik-public
environment:
- MINIO_ROOT_USER=minioadmin
- MINIO_ROOT_PASSWORD=minioadmin
- MINIO_DISTRIBUTED_MODE_ENABLED=yes
- MINIO_DISTRIBUTED_NODES=minio1,minio2,minio3,minio4
- MINIO_SKIP_CLIENT=yes
- MINIO_BROWSER_REDIRECT_URL=https://minio.msi.stef.lan
deploy:
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
# MINIO console
- "traefik.http.routers.minio.tls=true"
- "traefik.http.routers.minio.rule=Host(`minio.msi.stef.lan`)"
- "traefik.http.routers.minio.service=minio-s"
- "traefik.http.services.minio-s.loadbalancer.server.port=9001"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.minio-s.loadbalancer.sticky.cookie.secure=true"
# MINIO api
- "traefik.http.routers.cdn.tls=true"
- "traefik.http.routers.cdn.rule=Host(`s3.msi.stef.lan`)"
- "traefik.http.routers.cdn.service=cdn-s"
- "traefik.http.services.cdn-s.loadbalancer.server.port=9000"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.name=stickycookie"
- "traefik.http.services.cdn-s.loadbalancer.sticky.cookie.secure=true"
restart_policy:
delay: 10s
max_attempts: 5
window: 60s
placement:
constraints:
- node.hostname == swarm-node04
healthcheck:
test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
interval: 30s
timeout: 20s
retries: 3
networks:
traefik-public:
external: true
volumes:
minio1-data1:
minio2-data1:
minio3-data1:
minio4-data1:
Deploiement
stef@swarm-node01:~$ docker stack deploy --compose-file minio/docker-compose-stack.yml minio
Updating service minio_minio3 (id: wmgffz0yqxe8f0z0s4hirikl3)
Updating service minio_minio4 (id: gaz0lenl9tvs3yq6nipx24hnx)
Updating service minio_minio1 (id: eslpmn9umr6ne17ucukc13ola)
Updating service minio_minio2 (id: hagmzansn0ppztwr3nrz4e3pv)
Interface Minio
Je ne detail pas ici la configuration minio & s3
Création d'un bucket nginx-data
Contenant :
.
├── conf.d <= Conf nginx
│ └── default.conf
├── content <= Contenu site nginx
│ ├── 50x.html
│ ├── images
│ │ ├── 5c9387e5240000b600c80643 (2).jpeg
│ │ ├── e8987c7bb734461acd0158daad7b0fa9.jpg
│ │ ├── happyb.jpg
│ │ └── motarde.gif
│ └── index.html
└── nginx.conf <= Conf nginx
Le stockage distribué est maintenant disponible
Pour utilisé ce stockage comme un acces fichier unix classique, il faut ajouter le composant fuse s3
¶ Installation s3fs
Sur l'ensemble des noeuds
sudo apt install s3fs
Créer le fichier des credential en /etc/.passwd-s3fs
( note on peut en creer plusieurs en fonction des buckets a monter )
sudo su -
Exemple si votre compte pour l'acces au bucket est public et le mot de passe est publicpublic
cat > /etc/.passwd-s3fs << _EOF_
public:publicpublic
_EOF_
chmod 600 /etc/.passwd-s3fs
Ajouter dans /etc/fstab
nginx-data:/ /mnt/bucket/nginx-data fuse.s3fs _netdev,allow_other,passwd_file=/etc/.passwd-s3fs,url=https://s3.msi.stef.lan,use_path_request_style 0 0
Créer le/les points de montages
mkdir -p /mnt/bucket/{test,nginx-data}
systemctl daemon-reload
mount -a
Vérification
stef@swarm-node01:~$ df
Sys. de fichiers blocs de 1K Utilisé Disponible Uti% Monté sur
udev 965476 0 965476 0% /dev
tmpfs 197876 1080 196796 1% /run
/dev/mapper/debian--12--model--vg-root 8714684 5197816 3052588 64% /
...
s3fs 18014398509465600 0 18014398509465600 0% /mnt/bucket/nginx-data
Les fichiers sont maintenant accéssible par tout les noeuds
stef@swarm-node01:~$ echo "Ecriture de swarm-node01" > /mnt/bucket/nginx-data/filetest.txt
stef@swarm-node02:~$ echo "Ecriture de swarm-node02" >> /mnt/bucket/nginx-data/filetest.txt
stef@swarm-node03:~$ echo "Ecriture de swarm-node03" >> /mnt/bucket/nginx-data/filetest.txt
stef@swarm-node04:~$ echo "Ecriture de swarm-node04" >> /mnt/bucket/nginx-data/filetest.txt
stef@swarm-node01:~$ cat /mnt/bucket/nginx-data/filetest.txt
Ecriture de swarm-node01
Ecriture de swarm-node02
Ecriture de swarm-node03
Ecriture de swarm-node04
Et dans Minio
¶ Installation application volume partagé (Nginx )
Limitation
Ce modèle d'infra est juste un POC, car au redemmarage des systèmes, minio n'etant pas encore up, les montages s3 ne peuvent se faire.
Il est donc nécessaire pas passer sur tout les nodes pour faire un mount -a apres demarrage du stack minio.
Idéalement, l'infra de stockage doit être deployé sur un cluter dédié.
Rien de spécifique à faire, les montage sur les containeurs étant des share S3 monté sur les quatres hosts:
Fichier docker-compose.yml
version: "3.3"
services:
nginx:
image: nginx
networks:
- traefik-public
environment:
- NGINX_PORT:80
volumes:
- /mnt/bucket/nginx-data/nginx.conf:/etc/nginx/nginx.conf:ro
- /mnt/bucket/nginx-data/conf.d:/etc/nginx/conf.d
- /mnt/bucket/nginx-data/content:/usr/share/nginx/html
deploy:
labels:
- "traefik.enable=true"
- "traefik.docker.network=traefik-public"
- "traefik.http.routers.nginx.rule=Host(`nginx.msi.stef.lan`)"
- "traefik.http.services.nginx.loadbalancer.server.port=80"
- "traefik.http.routers.nginx.entrypoints=websecure"
- "traefik.http.routers.nginx.tls=true"
networks:
traefik-public:
external: true
¶ Installation App Java Springboot
Voir https://gitea.zen6.info/stef/springboot_demo
Déployer le stack comme indiqué dans le README.md
On note que le suivie de session ne suffira pas a maintenir un fonctionnement cohérent car
la valeur incrementé est interne à chaque instance.
Pour corriger ceci, externaliser les datas ( base classique , redis ou persistance via cookie , etc ..)