HA docker-compose stack & todo's


#1

Hey ya’all,

here is my docker-compose wich creates a mongo-replica set of 3 containers and 2 rocketchat containers on port 3000 & 3001.
atm this is only for testing scenarios but maybe in near future it can be used in a production environment too.

version: ‘2’

services:

rocketchat:
container_name: rc1
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=http://localhost:3000
- MONGO_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
depends_on:
- mongo1
- mongo2
- mongo3
ports:
- 3000:3000

rocketchat2:
container_name: rc2
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3001
- ROOT_URL=http://localhost:3001
- MONGO_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
depends_on:
- mongo1
- mongo2
- mongo3
ports:
- 3001:3001

mongo1:
container_name: db1
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db1:/data/db1

mongo2:
container_name: db2
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db2:/data/db2

mongo3:
container_name: db3
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db3:/data/db3

mongo-init-replica:
image: mongo:3.2
command: ‘mongo mongo1/rocketchat --eval “rs.initiate({ _id: ‘‘rs0’’, members: [ { _id: 0, host: ‘‘mongo1:27017’’ },{ _id: 1, host: ‘‘mongo2:27017’’ },{ _id: 2, host: ‘‘mongo3:27017’’ } ]})”’
depends_on:
- mongo1
- mongo2
- mongo3

maybe someone’s here who will help me with an nginx-part who will be responsible for reverse-proxy’ing. so he should listen on 80 and redirect all traffic to 443.
for this it’ll be great if we can implement letsencrypt.

I’d be happy if there’s someone here who wants to help out. :slight_smile:


#2

I moved this over to community creations since this is more about sharing.

Thanks for sharing btw :slight_smile:


#3

update to newer version with working hubot.
if you have your cert you only have to put it into a local “cert” folder wich is mapped into nginx container:

version: ‘2’
services:

nginx-proxy:
image: jwilder/nginx-proxy
container_name: nginx-proxy
restart: unless-stopped
ports:
- ‘80:80’
- ‘443:443’
volumes:
- ‘/etc/nginx/vhost.d’
- ‘/usr/share/nginx/html’
- ‘./certs:/etc/nginx/certs:ro’
- ‘/var/run/docker.sock:/tmp/docker.sock:ro’

db1:
container_name: db1
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db1:/data/db

db2:
container_name: db2
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db2:/data/db

db3:
container_name: db3
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db3:/data/db

mongo-init-replica:
image: mongo:3.2
command: ‘mongo db1/rocketchat --eval “rs.initiate({ _id: ‘‘rs0’’, members: [ { _id: 0, host: ‘‘db1:27017’’ },{ _id: 1, host: ‘‘db2:27017’’ },{ _id: 2, host: ‘‘db3:27017’’ } ]})”’
depends_on:
- db1
- db2
- db3

rc1:
container_name: rc1
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=http://127.0.0.1:3000
- MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
- VIRTUAL_HOST=rctest.yourdomain.co
depends_on:
- db1
- db2
- db3
- mongo-init-replica

rc2:
container_name: rc2
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=http://127.0.0.1:3000
- MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
- VIRTUAL_HOST=rctest.yourdomain.co
depends_on:
- db1
- db2
- db3
- mongo-init-replica

hubot:
container_name: hubot
restart: unless-stopped
image: rocketchat/hubot-rocketchat:latest
environment:
- ROCKETCHAT_URL=rc1:3000
- LISTEN_ON_ALL_PUBLIC=true
- ROCKETCHAT_USER=tutor
- ROCKETCHAT_PASSWORD=tutor
- BOT_NAME=tutor
- EXTERNAL_SCRIPTS=hubot-help,hubot-seen,hubot-links,hubot-diagnostics,hubot-google,hubot-reddit,hubot-bofh,hubot-bookmark,hubot-shipit,hubot-maps

networks:
default:
external:
name: nginx-net


#4

ATM i’m searching for a cool & valid backup strategy for all this.
maybe someone has a cool idea.
very important for me right now is to backing up the mongo db replica-set.


#5

Usually people just do mongodump with some clever way of scheduling it


#6

mmh i see.
for testing purposes i quck & dirty wrote this one:

d=$(date +%Y-%m-%d)
docker-compose exec db1 mongodump --oplog
docker cp db1:/dump /opt/container/rocketchat/mongodump
docker exec db1 rm -rf /dump
mkdir -p /opt/container/rocketchat/tmp
tar -zcvf /opt/container/rocketchat/tmp/"$d"_mongodump_rocketchat.tar.gz /opt/container/rocketchat/mongodump/rocketchat

upload to s3

/usr/local/bin/aws s3 cp /opt/container/rocketchat/tmp/*.tar.gz s3://mongobucket/"$d"_mongodump_rocketchat.tar.gz
rm -rf /opt/container/rocketchat/tmp
rm -rf /opt/container/rocketchat/mongodump

crontab -l
–>

0 3 * * * /opt/container/rocketchat/moo > /opt/container/rocketchat/output

i’m sure it’s not the best but right now it does the trick.

for downloading the latest dump from s3 back to local i wrote this:

BUCKET="the_mongo_backup_bucket"
FILE=`aws s3 ls $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}'`
 
 
aws s3 cp s3://$BUCKET/$FILE .

next will be rolling back the latest backup into mongo replica set.
will write about that later.

cheers!


#7

to bad that there’s an issue which is relatet to this construct i’m building here:

you cant use s3 to storage your files - because of this issue!


#8

Sticky sessions is one way some people work around this.


#9

How do you implement stickiness in this context - and how to do do thin on open.rocket.chat?


#10

That’s the thing. We use object store and no stickiness. Definitely multi-instance. No issues.


#11

Is there any documentation out there about this?
tryed to get it working with an jwilder-fork but had no luck yet.

i’m always getting the error no host in upstream using tpcwang/nginx-proxy:latest with this docker-compose.yml:

version: '3'
services:

  nginx-proxy:
    image: tpcwang/nginx-proxy:latest
    container_name: nginx-proxy
    restart: unless-stopped
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - '/etc/nginx/vhost.d'
      - '/usr/share/nginx/html'
      - './certs:/etc/nginx/certs:ro'
      - './conf.d:/etc/nginx/conf.d'
      - '/var/run/docker.sock:/tmp/docker.sock:ro'
    environment:
      - USE_IP_HASH=1
    depends_on:
      - rc

  db1:
    container_name: db1
    image: mongo:3.2
    restart: unless-stopped
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    volumes:
      - ./data/db1:/data/db

  db2:
    container_name: db2
    image: mongo:3.2
    restart: unless-stopped
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    volumes:
      - ./data/db2:/data/db

  db3:
    container_name: db3
    image: mongo:3.2
    restart: unless-stopped
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    volumes:
      - ./data/db3:/data/db

  mongo-init-replica:
    image: mongo:3.2
    command: 'mongo db1/rocketchat --eval "rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''db1:27017'' },{ _id: 1, host: ''db2:27017'' },{ _id: 2, host: ''db3:27017'' } ]})"'
    depends_on:
      - db1
      - db2
      - db3

  rc:
    image: rocketchat/rocket.chat:latest
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=http://127.0.0.1:3000
      - MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
      - MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
      - MAIL_URL=smtp://smtp.email
      - VIRTUAL_HOST=rctest.foo.bar
      - VIRTUAL_NETWORK=1
    depends_on:
      - db1
      - db2
      - db3
      - mongo-init-replica

ideas?


#12

i’ve just solved the stickyness problem with traefik!
—>

version: '3'
services:

 traefik:
   image: traefik
   command: --web --docker --docker.domain=docker.localhost --logLevel=DEBUG
   ports:
     - "80:80"
     - "8080:8080"
     - "443:443"
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /dev/null:/traefik.toml

 db1:
   container_name: db1
   image: mongo:3.2
   restart: unless-stopped
   command: mongod --smallfiles --oplogSize 128 --replSet rs0
   volumes:
     - ./data/db1:/data/db

 db2:
   container_name: db2
   image: mongo:3.2
   restart: unless-stopped
   command: mongod --smallfiles --oplogSize 128 --replSet rs0
   volumes:
     - ./data/db2:/data/db

 db3:
   container_name: db3
   image: mongo:3.2
   restart: unless-stopped
   command: mongod --smallfiles --oplogSize 128 --replSet rs0
   volumes:
     - ./data/db3:/data/db

 mongo-init-replica:
   image: mongo:3.2
   command: 'mongo db1/rocketchat --eval "rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''db1:27017'' },{ _id: 1, host: ''db2:27017'' },{ _id: 2, host: ''db3:27017'' } ]})"'
   depends_on:
     - db1
     - db2
     - db3

 rc:
   image: rocketchat/rocket.chat
   restart: unless-stopped
   volumes:
     - ./uploads:/app/uploads
   environment:
     - PORT=3000
     - ROOT_URL=http://127.0.0.1:3000
     - MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
     - MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
     - MAIL_URL=smtp://smtp.email
   depends_on:
     - db1
     - db2
     - db3
     - mongo-init-replica
   labels:
     - traefik.backend=echo
     - traefik.frontend.rule=Host:rctest.bla.foo
     - traefik.backend.loadbalancer.stickiness=true

todo: get tls running and for now this will be my setup for HA!


#13

i’m not be able to get my wildcard TLS-Certificate to work within tis setup…
anyone has experience working with traefik and SSL in docker?


#14

traefik is definitely my choice. A wildcard you should be able to include in a volume mount to bring the wildcard cert into the traefik container.


#15

tryed many times with different errors but no lucky…


#16

So from what I know you need to do a couple of things.

  1. Get wildcard cert and a traefik.toml inside the container
  2. Update the traefik.toml with details
  3. In traefik.toml set it up for docker so it can detect labels and such and add routes

Take a look at: https://github.com/RocketChat/Rocket.Chat/blob/develop/docker-compose.yml#L68

and https://github.com/RocketChat/Rocket.Chat/blob/develop/docker-compose.yml#L22