Snap 2.1 Update Issues

As these seem to be the only channels available, I seem to be stuck:

channels:
latest/stable: 2.4.11 2020-02-28 (1427) 267MB -
latest/candidate: 2.4.11 2020-02-27 (1427) 267MB -
latest/beta: 2.1.1 2019-10-21 (1414) 251MB -
latest/edge: 2.4.11 2020-02-27 (1427) 267MB -
3.x/stable: 3.0.10 2020-04-02 (1430) 272MB -
3.x/candidate: 3.0.10 2020-04-02 (1430) 272MB -
3.x/beta: ↑
3.x/edge: ↑
2.x/stable: 2.4.11 2020-02-28 (1427) 267MB -
2.x/candidate: 2.4.11 2020-02-28 (1427) 267MB -
2.x/beta: ↑
2.x/edge: ↑
installed: 0.74.3 (1367) 235MB -

When you goto mongo 3.4 what do you have in your mongo logs?

When I use the flags mentioned above, it always attempts to load v3.6.14 - it does not seem to attempt to move to 3.4 at all.

[initandlisten] db version v3.6.14
[initandlisten] git version: cbef87692475857c7ee6e764c8f5104b39c342a1
[initandlisten] allocator: tcmalloc
[initandlisten] modules: none
[initandlisten] build environment:
[initandlisten] distarch: x86_64
[initandlisten] target_arch: x86_64

IMPORTANT: UPGRADE PROBLEM: The data files need to be fully upgraded to version 3.4 before attempting an upgrade to 3.6; see http://dochub.mongodb.org/core/3.6-upgrade-fcv for more details.

rocketchat-server.mongo --eval also outputs:
MongoDB shell version v3.6.14

After revert, it shows: MongoDB shell version: 3.2.7

So you do this and nothing else and it’s running mongo 3.6?

Yes, that is correct.

What do you get on snap list ? Something don’t seem right there…

snap list
    Name               Version    Rev   Tracking         Publisher    Notes
    core               16-2.44.1  8935  latest/stable    canonical✓   core
    rocketchat-server  0.74.3     1367  latest/stable/…  rocketchat✓  -

In snap info rocketchat-server it shows:

tracking: latest/stable/migrate-mongo-3.4

Here is the output:

$ snap refresh rocketchat-server --channel=stable/migrate-mongo-3.4
rocketchat-server 2.4.11 from Rocket.Chat (rocketchat✓) refreshed

$ snap list
Name Version Rev Tracking Publisher Notes
core 16-2.44.1 8935 latest/stable canonical✓ core
rocketchat-server 2.4.11 1427 latest/stable/… rocketchat✓ -

$ rocketchat-server.mongo --eval “printjson(db.adminCommand ({getParameter: 1, featureCompatibilityVersion: 1}))”
MongoDB shell version v3.6.14
connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
2020-04-14T22:01:31.038-0400 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2020-04-14T22:01:31.039-0400 E QUERY [thread1] Error: couldn’t connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:263:13
@(connect):1:6
exception: connect failed

Having the exact same issues and error messages as well.

Any suggestions? Or people with snaps running Mongodb v3.2 are just stuck now?

I have similar issues over here, RocketChat just stopped working.

I am not able to use the migration channels:

# snap refresh rocketchat-server --channel=stable/migrate-mongo-3.4
error: requested a non-existing branch on 2.x/stable for snap "rocketchat-server":
       migrate-mongo-3.4

Oddly it seems that these custom tags we created to aid expired and have just been pruned by the snap store.

This is why disabling the snap refresh is not good and using our tracks is a better idea.

If it stopped working specifically today your issue might be related to a bad revision that went out

a refresh should do the trick in that case

Sudden 502 error please try this solution

It did not work for unfortunately. I also tried snap refresh ... as suggested by @aaron.ogle but it did not help either. Notice that it’s saying it’s 3.2.2. I am a bit confused about the auto updates of snap.

root@pi1145:~# snap refresh rocketchat-server
snap "rocketchat-server" has no updates available
root@pi1145:~# snap revert rocketchat-server
rocketchat-server reverted to 3.2.2
root@pi1145:~# snap run --shell rocketchat-server
root@pi1145:/root# snapctl get snap-refreshing
true
root@pi1145:/root# snapctl set snap-refreshing=false
root@pi1145:/root# exit
exit
root@pi1145:~# systemctl restart snap.rocketchat-server.rocketchat-server

I should probably open new thread since I am not sure anymore if this is really the same issue. It seems that it has problems connecting to the mongoDB server:

root@pi1145:~# service snap.rocketchat-server.rocketchat-server status
* snap.rocketchat-server.rocketchat-server.service - Service for snap application rocketchat-server.rocketchat-server
   Loaded: loaded (/etc/systemd/system/snap.rocketchat-server.rocketchat-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-05-13 01:26:03 CEST; 13s ago
 Main PID: 4090 (startRocketChat)
    Tasks: 12 (limit: 4915)
   CGroup: /system.slice/snap.rocketchat-server.rocketchat-server.service
           |-4090 /bin/bash /snap/rocketchat-server/1433/bin/startRocketChat
           `-4276 node /snap/rocketchat-server/1433/main.js

May 13 01:26:13 pi1145 rocketchat-server.rocketchat-server[4090]: MongoDB shell version v3.6.14
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: 2020-05-13T01:26:14.049+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: 2020-05-13T01:26:14.050+0200 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: connect@src/mongo/shell/mongo.js:263:13
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: @(connect):1:6
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: exception: connect failed
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: Checking if mongo featureCompatibilityVersion is correct, changing if not
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: exception: connect failed
May 13 01:26:14 pi1145 rocketchat-server.rocketchat-server[4090]: exception: connect failed

Well… our outdated server was running along just fine until yesterday:

$ snap changes
ID   Status  Spawn                   Ready                   Summary
147  Done    yesterday at 13:34 EDT  yesterday at 13:35 EDT  Auto-refresh snap "rocketchat-server"
148  Done    yesterday at 19:59 EDT  yesterday at 20:00 EDT  Auto-refresh snap "rocketchat-server"

Which has now broken our rocketchat server beyond recovery. We can no longer revert to the working version, and mongodb is permanently stuck in a loop.

I honestly don’t get how this is considered a reasonable approach for deployment. Random snap updates at random times impacting services that people are using in the middle of the day, and if you don’t let it randomly update, you’ll be stuck without the ability to update, then subsequent “automatic” updates will brick your deployment… :confused:

Alright retagged the ones that apparently expired on the snap store.

For the record in case need to retag again. :slight_smile:

stable/migrate-mongo3.2  0.74.3     1367        2020-06-12T18:55:45.872970
stable/migrate-mongo3.4  1.3.2      1416        2020-06-12T18:55:27.337586
stable/migrate-mongo3.6  2.1.1      1414        2020-06-12T18:56:54.363648

Now the previous instructions should work.

Depending on what version you were on you should be able to fix your deployment. If you were on pre 1.x you can go back and set your compatibility version as indicated in the first post.

If your db jumped from 3.2 to 3.6 you will need to take it back down to mongo 3.4 so that mongo can migrate the data format.

If you were on rocket.chat 2.4.11 and it broke when going to 2.4.12… just refresh and will be back to 2.4.11 and should work

So what is the preferred solution now? I am a bit confused.

The status on our server with 300 users is now (in short: connection to MongDB failed):

The overall message from the Web server is

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Apache/2.4.25 (Debian) Server at chat.km3net.de Port 443

Snap status reports it’s running rocketchat-server 3.2.2:

root@pi1145:~# snap list
Name               Version    Rev   Tracking       Publisher    Notes
core               16-2.44.3  9066  latest/stable  canonical*   core
rocketchat-server  3.2.2      1433  3.x/stable     rocketchat*  -

Caddy:

root@pi1145:~# service snap.rocketchat-server.rocketchat-caddy status
* snap.rocketchat-server.rocketchat-caddy.service - Service for snap application rocketchat-server.rocketchat-caddy
   Loaded: loaded (/etc/systemd/system/snap.rocketchat-server.rocketchat-caddy.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2020-05-13 01:54:16 CEST; 20h ago
  Process: 4037 ExecStart=/usr/bin/snap run rocketchat-server.rocketchat-caddy (code=exited, status=1/FAILURE)
 Main PID: 4037 (code=exited, status=1/FAILURE)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Server

root@pi1145:~# service snap.rocketchat-server.rocketchat-server status
* snap.rocketchat-server.rocketchat-server.service - Service for snap application rocketchat-server.rocketchat-server
   Loaded: loaded (/etc/systemd/system/snap.rocketchat-server.rocketchat-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-05-13 22:30:47 CEST; 2s ago
 Main PID: 20632 (startRocketChat)
    Tasks: 8 (limit: 4915)
   CGroup: /system.slice/snap.rocketchat-server.rocketchat-server.service
           |-20632 /bin/bash /snap/rocketchat-server/1433/bin/startRocketChat
           `-20693 node /snap/rocketchat-server/1433/main.js

May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: MongoDB shell version v3.6.14
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: connecting to: mongodb://127.0.0.1:27017/?gssapiServiceName=mongodb
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: 2020-05-13T22:30:47.559+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: 2020-05-13T22:30:47.560+0200 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: connect@src/mongo/shell/mongo.js:263:13
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: @(connect):1:6
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: exception: connect failed
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: Checking if mongo featureCompatibilityVersion is correct, changing if not
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: exception: connect failed
May 13 22:30:47 pi1145 rocketchat-server.rocketchat-server[20632]: exception: connect failed

MongoDB

* snap.rocketchat-server.rocketchat-mongo.service - Service for snap application rocketchat-server.rocketchat-mongo
   Loaded: loaded (/etc/systemd/system/snap.rocketchat-server.rocketchat-mongo.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-05-13 22:31:06 CEST; 835ms ago
 Main PID: 21109 (sh)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/snap.rocketchat-server.rocketchat-mongo.service
           |-21109 /bin/sh /snap/rocketchat-server/1433/bin/startmongo
           `-21138 mongod --bind_ip 127.0.0.1 --pidfilepath /var/snap/rocketchat-server/common/mongod.pid --smallfiles --journal --dbpath=/var/snap/rocketchat-server/common --replSet rs0

May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.201+0200 I CONTROL  [initandlisten] modules: none
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.202+0200 I CONTROL  [initandlisten] build environment:
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.202+0200 I CONTROL  [initandlisten]     distarch: x86_64
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.202+0200 I CONTROL  [initandlisten]     target_arch: x86_64
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.202+0200 I CONTROL  [initandlisten] options: { net: { bindIp: "127.0.0.1" }, processManagement: { pidFilePath: "/var/snap/rocketchat-server/common/mongod.pid" }, replicat
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.202+0200 I -        [initandlisten] Detected data files in /var/snap/rocketchat-server/common created by the 'wiredTiger' storage engine, so setting the active storage en
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.202+0200 I STORAGE  [initandlisten]
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.203+0200 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.203+0200 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
May 13 22:31:06 pi1145 rocketchat-server.rocketchat-mongo[21109]: 2020-05-13T22:31:06.204+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=2402M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=

None of the migrations work:

root@pi1145:~# snap refresh rocketchat-server --channel=stable/migrate-mongo-3.2
error: requested a non-existing branch on 3.x/stable for snap "rocketchat-server":
       migrate-mongo-3.2
root@pi1145:~# snap refresh rocketchat-server --channel=stable/migrate-mongo-3.4
error: requested a non-existing branch on 3.x/stable for snap "rocketchat-server":
       migrate-mongo-3.4

What version did you upgrade from? My guess is this thread doesn’t apply to you at all. Unless you are coming from 1.x to 3.x or 1.x to 2.x then this is likely a different problem.

I see pi in the hostname is this on a raspberry pi?

Honestly the server was just running fine and I it was on 2.x before. As far as I can tell the problem occurred at the first refresh, then I tried to revert and after a while I also removed and reinstalled it and restored the last backup:

root@pi1145:~# snap changes --abs-time rocketchat-server
ID   Status  Spawn                      Ready                      Summary
50   Done    2020-05-12T23:44:23+02:00  2020-05-12T23:44:59+02:00  Refresh "rocketchat-server" snap from "2.x/stable" channel
51   Done    2020-05-13T01:23:21+02:00  2020-05-13T01:23:30+02:00  Revert "rocketchat-server" snap
52   Done    2020-05-13T01:29:53+02:00  2020-05-13T01:29:53+02:00  Change configuration of "rocketchat-server" snap
54   Done    2020-05-13T01:42:22+02:00  2020-05-13T01:46:18+02:00  Remove "rocketchat-server" snap
55   Done    2020-05-13T01:46:41+02:00  2020-05-13T01:46:56+02:00  Install "rocketchat-server" snap

No it’s not a raspberry, it’s a Xeon :wink: (pi stands for Physikalisches Institut).