HA docker-compose stack & todo's

DeployBoy · April 25, 2018, 8:52am

Hey ya’all,

here is my docker-compose wich creates a mongo-replica set of 3 containers and 2 rocketchat containers on port 3000 & 3001.
atm this is only for testing scenarios but maybe in near future it can be used in a production environment too.

version: ‘2’

services:

rocketchat:
container_name: rc1
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=http://localhost:3000
- MONGO_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
depends_on:
- mongo1
- mongo2
- mongo3
ports:
- 3000:3000

rocketchat2:
container_name: rc2
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3001
- ROOT_URL=http://localhost:3001
- MONGO_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://mongo1:27017,mongo2:27017,mongo3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
depends_on:
- mongo1
- mongo2
- mongo3
ports:
- 3001:3001

mongo1:
container_name: db1
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db1:/data/db1

mongo2:
container_name: db2
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db2:/data/db2

mongo3:
container_name: db3
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db3:/data/db3

mongo-init-replica:
image: mongo:3.2
command: ‘mongo mongo1/rocketchat --eval “rs.initiate({ _id: ‘‘rs0’’, members: [ { _id: 0, host: ‘‘mongo1:27017’’ },{ _id: 1, host: ‘‘mongo2:27017’’ },{ _id: 2, host: ‘‘mongo3:27017’’ } ]})”’
depends_on:
- mongo1
- mongo2
- mongo3

maybe someone’s here who will help me with an nginx-part who will be responsible for reverse-proxy’ing. so he should listen on 80 and redirect all traffic to 443.
for this it’ll be great if we can implement letsencrypt.

I’d be happy if there’s someone here who wants to help out.

aaron.ogle · May 31, 2018, 8:17pm

I moved this over to community creations since this is more about sharing.

Thanks for sharing btw

DeployBoy · June 6, 2018, 9:09am

update to newer version with working hubot.
if you have your cert you only have to put it into a local “cert” folder wich is mapped into nginx container:

version: ‘2’
services:

nginx-proxy:
image: jwilder/nginx-proxy
container_name: nginx-proxy
restart: unless-stopped
ports:
- ‘80:80’
- ‘443:443’
volumes:
- ‘/etc/nginx/vhost.d’
- ‘/usr/share/nginx/html’
- ‘./certs:/etc/nginx/certs:ro’
- ‘/var/run/docker.sock:/tmp/docker.sock:ro’

db1:
container_name: db1
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db1:/data/db

db2:
container_name: db2
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db2:/data/db

db3:
container_name: db3
image: mongo:3.2
restart: unless-stopped
command: mongod --smallfiles --oplogSize 128 --replSet rs0
volumes:
- ./data/db3:/data/db

mongo-init-replica:
image: mongo:3.2
command: ‘mongo db1/rocketchat --eval “rs.initiate({ _id: ‘‘rs0’’, members: [ { _id: 0, host: ‘‘db1:27017’’ },{ _id: 1, host: ‘‘db2:27017’’ },{ _id: 2, host: ‘‘db3:27017’’ } ]})”’
depends_on:
- db1
- db2
- db3

rc1:
container_name: rc1
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=http://127.0.0.1:3000
- MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
- VIRTUAL_HOST=rctest.yourdomain.co
depends_on:
- db1
- db2
- db3
- mongo-init-replica

rc2:
container_name: rc2
image: rocketchat/rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=http://127.0.0.1:3000
- MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
- MAIL_URL=smtp://smtp.email
- VIRTUAL_HOST=rctest.yourdomain.co
depends_on:
- db1
- db2
- db3
- mongo-init-replica

hubot:
container_name: hubot
restart: unless-stopped
image: rocketchat/hubot-rocketchat:latest
environment:
- ROCKETCHAT_URL=rc1:3000
- LISTEN_ON_ALL_PUBLIC=true
- ROCKETCHAT_USER=tutor
- ROCKETCHAT_PASSWORD=tutor
- BOT_NAME=tutor
- EXTERNAL_SCRIPTS=hubot-help,hubot-seen,hubot-links,hubot-diagnostics,hubot-google,hubot-reddit,hubot-bofh,hubot-bookmark,hubot-shipit,hubot-maps

networks:
default:
external:
name: nginx-net

DeployBoy · June 12, 2018, 11:06am

ATM i’m searching for a cool & valid backup strategy for all this.
maybe someone has a cool idea.
very important for me right now is to backing up the mongo db replica-set.

aaron.ogle · June 13, 2018, 5:02am

Usually people just do mongodump with some clever way of scheduling it

DeployBoy · June 13, 2018, 6:54am

mmh i see.
for testing purposes i quck & dirty wrote this one:

d=$(date +%Y-%m-%d)
docker-compose exec db1 mongodump --oplog
docker cp db1:/dump /opt/container/rocketchat/mongodump
docker exec db1 rm -rf /dump
mkdir -p /opt/container/rocketchat/tmp
tar -zcvf /opt/container/rocketchat/tmp/“$d”_mongodump_rocketchat.tar.gz /opt/container/rocketchat/mongodump/rocketchat

upload to s3

/usr/local/bin/aws s3 cp /opt/container/rocketchat/tmp/*.tar.gz s3://mongobucket/“$d”_mongodump_rocketchat.tar.gz
rm -rf /opt/container/rocketchat/tmp
rm -rf /opt/container/rocketchat/mongodump

crontab -l
→

0 3 * * * /opt/container/rocketchat/moo > /opt/container/rocketchat/output

i’m sure it’s not the best but right now it does the trick.

for downloading the latest dump from s3 back to local i wrote this:

BUCKET="the_mongo_backup_bucket"
FILE=`aws s3 ls $BUCKET --recursive | sort | tail -n 1 | awk '{print $4}'`
 
 
aws s3 cp s3://$BUCKET/$FILE .

next will be rolling back the latest backup into mongo replica set.
will write about that later.

cheers!

DeployBoy · June 13, 2018, 1:44pm

to bad that there’s an issue which is relatet to this construct i’m building here:

you cant use s3 to storage your files - because of this issue!

aaron.ogle · June 13, 2018, 9:53pm

Sticky sessions is one way some people work around this.

ojaegle · June 14, 2018, 3:44am

How do you implement stickiness in this context - and how to do do thin on open.rocket.chat?

aaron.ogle · June 14, 2018, 4:10am

That’s the thing. We use object store and no stickiness. Definitely multi-instance. No issues.

DeployBoy · June 14, 2018, 8:51am

Is there any documentation out there about this?
tryed to get it working with an jwilder-fork but had no luck yet.

i’m always getting the error no host in upstream using tpcwang/nginx-proxy:latest with this docker-compose.yml:

version: '3'
services:

  nginx-proxy:
    image: tpcwang/nginx-proxy:latest
    container_name: nginx-proxy
    restart: unless-stopped
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - '/etc/nginx/vhost.d'
      - '/usr/share/nginx/html'
      - './certs:/etc/nginx/certs:ro'
      - './conf.d:/etc/nginx/conf.d'
      - '/var/run/docker.sock:/tmp/docker.sock:ro'
    environment:
      - USE_IP_HASH=1
    depends_on:
      - rc

  db1:
    container_name: db1
    image: mongo:3.2
    restart: unless-stopped
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    volumes:
      - ./data/db1:/data/db

  db2:
    container_name: db2
    image: mongo:3.2
    restart: unless-stopped
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    volumes:
      - ./data/db2:/data/db

  db3:
    container_name: db3
    image: mongo:3.2
    restart: unless-stopped
    command: mongod --smallfiles --oplogSize 128 --replSet rs0
    volumes:
      - ./data/db3:/data/db

  mongo-init-replica:
    image: mongo:3.2
    command: 'mongo db1/rocketchat --eval "rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''db1:27017'' },{ _id: 1, host: ''db2:27017'' },{ _id: 2, host: ''db3:27017'' } ]})"'
    depends_on:
      - db1
      - db2
      - db3

  rc:
    image: rocketchat/rocket.chat:latest
    restart: unless-stopped
    volumes:
      - ./uploads:/app/uploads
    environment:
      - PORT=3000
      - ROOT_URL=http://127.0.0.1:3000
      - MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
      - MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
      - MAIL_URL=smtp://smtp.email
      - VIRTUAL_HOST=rctest.foo.bar
      - VIRTUAL_NETWORK=1
    depends_on:
      - db1
      - db2
      - db3
      - mongo-init-replica

ideas?

DeployBoy · June 14, 2018, 10:49am

i’ve just solved the stickyness problem with traefik!
—>

version: '3'
services:

 traefik:
   image: traefik
   command: --web --docker --docker.domain=docker.localhost --logLevel=DEBUG
   ports:
     - "80:80"
     - "8080:8080"
     - "443:443"
   volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /dev/null:/traefik.toml

 db1:
   container_name: db1
   image: mongo:3.2
   restart: unless-stopped
   command: mongod --smallfiles --oplogSize 128 --replSet rs0
   volumes:
     - ./data/db1:/data/db

 db2:
   container_name: db2
   image: mongo:3.2
   restart: unless-stopped
   command: mongod --smallfiles --oplogSize 128 --replSet rs0
   volumes:
     - ./data/db2:/data/db

 db3:
   container_name: db3
   image: mongo:3.2
   restart: unless-stopped
   command: mongod --smallfiles --oplogSize 128 --replSet rs0
   volumes:
     - ./data/db3:/data/db

 mongo-init-replica:
   image: mongo:3.2
   command: 'mongo db1/rocketchat --eval "rs.initiate({ _id: ''rs0'', members: [ { _id: 0, host: ''db1:27017'' },{ _id: 1, host: ''db2:27017'' },{ _id: 2, host: ''db3:27017'' } ]})"'
   depends_on:
     - db1
     - db2
     - db3

 rc:
   image: rocketchat/rocket.chat
   restart: unless-stopped
   volumes:
     - ./uploads:/app/uploads
   environment:
     - PORT=3000
     - ROOT_URL=http://127.0.0.1:3000
     - MONGO_URL=mongodb://db1:27017,db2:27017,db3:27017/rocketchat?replicaSet=rs0
     - MONGO_OPLOG_URL=mongodb://db1:27017,db2:27017,db3:27017/local?replicaSet=rs0
     - MAIL_URL=smtp://smtp.email
   depends_on:
     - db1
     - db2
     - db3
     - mongo-init-replica
   labels:
     - traefik.backend=echo
     - traefik.frontend.rule=Host:rctest.bla.foo
     - traefik.backend.loadbalancer.stickiness=true

todo: get tls running and for now this will be my setup for HA!

DeployBoy · June 14, 2018, 12:56pm

i’m not be able to get my wildcard TLS-Certificate to work within tis setup…
anyone has experience working with traefik and SSL in docker?

aaron.ogle · June 15, 2018, 9:20pm

traefik is definitely my choice. A wildcard you should be able to include in a volume mount to bring the wildcard cert into the traefik container.

DeployBoy · June 18, 2018, 8:05am

tryed many times with different errors but no lucky…

aaron.ogle · June 18, 2018, 5:21pm

So from what I know you need to do a couple of things.

Get wildcard cert and a traefik.toml inside the container
Update the traefik.toml with details
In traefik.toml set it up for docker so it can detect labels and such and add routes

Take a look at: https://github.com/RocketChat/Rocket.Chat/blob/develop/docker-compose.yml#L68

and https://github.com/RocketChat/Rocket.Chat/blob/develop/docker-compose.yml#L22

Adriano · January 26, 2021, 2:49pm

@aaron.ogle

Do you currently have a manual for installation in HA, from rocket.chat using the installation standard via docker compose?

I’m using this material from your colleague frdmn, I can scale the apps but I’m having problems with mongo.

For the mongo would it be like the example of the yml above? Or do you currently have another way?

Thanks!

aaron.ogle · February 24, 2021, 11:10pm

if you’re looking for an HA deployment I wouldn’t probably use docker-compose. To me thats a let me do this super quick and easy. An HA deployment i’d do one of our manual installs. Now you could if you want remove mongo from the docker-compose put it external in a normal mongo cluster and then scale just Rocket.Chat with docker-compose. That might work.

But honestly even then… its HA on a single machine with docker-compose. So would be better off using kubernetes with helm chart or just a regular non-docker Rocket.Chat install. That way you have Rocket.Chat running on multiple machines and actually pull off an HA setup.

Topic		Replies	Views
Docker compose backup mongo Community Support rocketchat-apps , docker	2	358	April 14, 2023
Multitple RC instances with docker-compose Community Support	1	756	March 31, 2021
RocketChat: Multiple setup and mongo db restore issue Community Support	0	788	November 3, 2020
Using Docker and a local mongodb, outside of the container Community Support	3	810	October 17, 2022
Restored Mongo in docker compose not working Community Support rocketchat-apps , docker	1	1034	April 18, 2023

Join our Community Open Call tomorrow where we'll share more details regarding recent changes and answer questions about updating your workspace to the latest Rocket.Chat

HA docker-compose stack & todo's

upload to s3

Related Topics