Upgrade from 4.5.4 to 4.8.1 failing

Community Support
1

Description

I am attempting to update from 4.5.4 to 4.8.1. The upgrade finishes without errors (a few warnings regarding deprecated stuff) and the versions of the supporting software line up but starting the application after the upgrade results in the 11/SEGV and a failed service.

Server Setup Information

  • Version of Rocket.Chat Server: 4.5.4 → 4.8.1
  • Operating System: Centos 7
  • Deployment Method: tar
  • Number of Running Instances: 1
  • DB Replicaset Oplog: Enabled
  • NodeJS Version: v14.18.3
  • MongoDB Version: 4.2.20
  • Proxy: HAProxy
  • Firewalls involved: No
2

Hi!

Can you provide any other outstanding log?

Have you seen this issue? It may be related:

3

Hi, sorry it took me so long to get back to this. I got another window last night and tried again, this tome to the new 5.0.0 version. Here are some logs and more from the attempt.

Current Server:

[root@wprocket ~]# systemctl status rocketchat
● rocketchat.service - The Rocket.Chat server
Loaded: loaded (/usr/lib/systemd/system/rocketchat.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2022-07-27 23:04:28 EDT; 19s ago
Main PID: 1682 (node)
CGroup: /system.slice/rocketchat.service
└─1682 /usr/local/bin/node /opt/Rocket.Chat/main.js

Jul 27 23:04:48 wprocket rocketchat[1682]: | MongoDB Version: 4.2.20 |
Jul 27 23:04:48 wprocket rocketchat[1682]: | MongoDB Engine: wiredTiger |
Jul 27 23:04:48 wprocket rocketchat[1682]: | Platform: linux |
Jul 27 23:04:48 wprocket rocketchat[1682]: | Process Port: 3000 |
Jul 27 23:04:48 wprocket rocketchat[1682]: | Site URL: https:// |
Jul 27 23:04:48 wprocket rocketchat[1682]: | ReplicaSet OpLog: Enabled |
Jul 27 23:04:48 wprocket rocketchat[1682]: | Commit Hash: c4066e452b |
Jul 27 23:04:48 wprocket rocketchat[1682]: | Commit Branch: HEAD |
Jul 27 23:04:48 wprocket rocketchat[1682]: | |
Jul 27 23:04:48 wprocket rocketchat[1682]: ±------------------------------------------------------+

Update Node and NPM:

[root@wprocket ~]# n install 14.19.3

installing : node-v14.19.3
mkdir : /usr/local/n/versions/node/14.19.3
fetch : https://nodejs.org/dist/v14.19.3/node-v14.19.3-linux-x64.tar.xz
installed : v14.19.3 (with npm 6.14.17)

[root@wprocket ~]# npm install -g npm@6.14.17
/usr/local/bin/npm → /usr/local/lib/node_modules/npm/bin/npm-cli.js
/usr/local/bin/npx → /usr/local/lib/node_modules/npm/bin/npx-cli.js

  • npm@6.14.17
    updated 1 package in 23.534s

    ╭───────────────────────────────────────────────────────────────╮
    │ │
    │ New major version of npm available! 6.14.15 → 8.5.5 │
    │ Changelog: Release v8.5.5 · npm/cli · GitHub │
    │ Run npm install -g npm to update! │
    │ │
    ╰───────────────────────────────────────────────────────────────╯

Pre-Update:

[root@wprocket /]# mongo --version
MongoDB shell version v4.2.20
git version: 15c0712952c356cb711c13a42cb3bca8617d4ebc
OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
allocator: tcmalloc
modules: none
build environment:
distmod: rhel70
distarch: x86_64
target_arch: x86_64
[root@wprocket /]# node -v
v14.19.3
[root@wprocket /]# npm -v
6.14.17

Update stdio:

[root@wprocket ~]# cd /tmp/bundle/programs/server && npm install
npm WARN deprecated node-pre-gyp@0.15.0: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future

fibers@5.0.1 install /tmp/bundle/programs/server/node_modules/fibers
node build.js || nodejs build.js

linux-x64-83-glibc exists; testing
Problem with the binary; manual build incoming
make: Entering directory /tmp/bundle/programs/server/node_modules/fibers/build' CXX(target) Release/obj.target/fibers/src/fibers.o In file included from /root/.cache/node-gyp/14.19.3/include/node/node.h:67:0, from ../src/coroutine.h:1, from ../src/fibers.cc:1: /root/.cache/node-gyp/14.19.3/include/node/v8.h:1585:58: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<Value> export_value); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:3359:23: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<Value> Name() const { return Description(); } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5178:76: warning: ‘deprecated’ attribute directive ignored [-Wattributes] ArrayBufferCreationMode mode = ArrayBufferCreationMode::kExternalized); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5225:21: warning: ‘deprecated’ attribute directive ignored [-Wattributes] bool IsExternal() const; ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5252:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents Externalize(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5262:70: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void Externalize(const std::shared_ptr<BackingStore>& backing_store); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5273:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents GetContents(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5660:76: warning: ‘deprecated’ attribute directive ignored [-Wattributes] ArrayBufferCreationMode mode = ArrayBufferCreationMode::kExternalized); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5709:76: warning: ‘deprecated’ attribute directive ignored [-Wattributes] ArrayBufferCreationMode mode = ArrayBufferCreationMode::kExternalized); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5718:21: warning: ‘deprecated’ attribute directive ignored [-Wattributes] bool IsExternal() const; ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5734:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents Externalize(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5744:70: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void Externalize(const std::shared_ptr<BackingStore>& backing_store); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5759:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents GetContents(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5984:50: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<FinalizationGroup> finalization_group); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7135:28: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t code_range_size() const { return code_range_size_ / kMB; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7137:46: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_code_range_size(size_t limit_in_mb) { ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7141:38: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t max_semi_space_size_in_kb() const; ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7143:56: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_max_semi_space_size_in_kb(size_t limit_in_kb); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7145:31: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t max_old_space_size() const { return max_old_generation_size_ / kMB; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7147:49: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_max_old_space_size(size_t limit_in_mb) { ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7151:31: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t max_zone_pool_size() const { return max_zone_pool_size_; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7153:43: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_max_zone_pool_size(size_t bytes) { max_zone_pool_size_ = bytes; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:8601:52: warning: ‘deprecated’ attribute directive ignored [-Wattributes] HostCleanupFinalizationGroupCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:8765:59: warning: ‘deprecated’ attribute directive ignored [-Wattributes] MeasureMemoryMode mode); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:8829:36: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<Context> GetEnteredContext(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9106:22: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void RunMicrotasks() { PerformMicrotaskCheckpoint(); } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9150:75: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void AddMicrotasksCompletedCallback(MicrotasksCompletedCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9158:78: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void RemoveMicrotasksCompletedCallback(MicrotasksCompletedCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9330:30: warning: ‘deprecated’ attribute directive ignored [-Wattributes] UnwindState GetUnwindState(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9393:54: warning: ‘deprecated’ attribute directive ignored [-Wattributes] AllowCodeGenerationFromStringsCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9750:75: warning: ‘deprecated’ attribute directive ignored [-Wattributes] static bool TryHandleSignal(int signal_number, void* info, void* context); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:10723:55: warning: ‘deprecated’ attribute directive ignored [-Wattributes] const void* stack_base); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:10749:65: warning: ‘deprecated’ attribute directive ignored [-Wattributes] static bool PCIsInV8(const UnwindState& unwind_state, void* pc); ^ CXX(target) Release/obj.target/fibers/src/coroutine.o In file included from /root/.cache/node-gyp/14.19.3/include/node/node.h:67:0, from ../src/coroutine.h:1, from ../src/coroutine.cc:1: /root/.cache/node-gyp/14.19.3/include/node/v8.h:1585:58: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<Value> export_value); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:3359:23: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<Value> Name() const { return Description(); } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5178:76: warning: ‘deprecated’ attribute directive ignored [-Wattributes] ArrayBufferCreationMode mode = ArrayBufferCreationMode::kExternalized); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5225:21: warning: ‘deprecated’ attribute directive ignored [-Wattributes] bool IsExternal() const; ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5252:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents Externalize(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5262:70: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void Externalize(const std::shared_ptr<BackingStore>& backing_store); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5273:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents GetContents(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5660:76: warning: ‘deprecated’ attribute directive ignored [-Wattributes] ArrayBufferCreationMode mode = ArrayBufferCreationMode::kExternalized); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5709:76: warning: ‘deprecated’ attribute directive ignored [-Wattributes] ArrayBufferCreationMode mode = ArrayBufferCreationMode::kExternalized); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5718:21: warning: ‘deprecated’ attribute directive ignored [-Wattributes] bool IsExternal() const; ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5734:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents Externalize(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5744:70: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void Externalize(const std::shared_ptr<BackingStore>& backing_store); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5759:24: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Contents GetContents(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:5984:50: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<FinalizationGroup> finalization_group); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7135:28: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t code_range_size() const { return code_range_size_ / kMB; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7137:46: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_code_range_size(size_t limit_in_mb) { ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7141:38: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t max_semi_space_size_in_kb() const; ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7143:56: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_max_semi_space_size_in_kb(size_t limit_in_kb); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7145:31: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t max_old_space_size() const { return max_old_generation_size_ / kMB; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7147:49: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_max_old_space_size(size_t limit_in_mb) { ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7151:31: warning: ‘deprecated’ attribute directive ignored [-Wattributes] size_t max_zone_pool_size() const { return max_zone_pool_size_; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:7153:43: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void set_max_zone_pool_size(size_t bytes) { max_zone_pool_size_ = bytes; } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:8601:52: warning: ‘deprecated’ attribute directive ignored [-Wattributes] HostCleanupFinalizationGroupCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:8765:59: warning: ‘deprecated’ attribute directive ignored [-Wattributes] MeasureMemoryMode mode); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:8829:36: warning: ‘deprecated’ attribute directive ignored [-Wattributes] Local<Context> GetEnteredContext(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9106:22: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void RunMicrotasks() { PerformMicrotaskCheckpoint(); } ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9150:75: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void AddMicrotasksCompletedCallback(MicrotasksCompletedCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9158:78: warning: ‘deprecated’ attribute directive ignored [-Wattributes] void RemoveMicrotasksCompletedCallback(MicrotasksCompletedCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9330:30: warning: ‘deprecated’ attribute directive ignored [-Wattributes] UnwindState GetUnwindState(); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9393:54: warning: ‘deprecated’ attribute directive ignored [-Wattributes] AllowCodeGenerationFromStringsCallback callback); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:9750:75: warning: ‘deprecated’ attribute directive ignored [-Wattributes] static bool TryHandleSignal(int signal_number, void* info, void* context); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:10723:55: warning: ‘deprecated’ attribute directive ignored [-Wattributes] const void* stack_base); ^ /root/.cache/node-gyp/14.19.3/include/node/v8.h:10749:65: warning: ‘deprecated’ attribute directive ignored [-Wattributes] static bool PCIsInV8(const UnwindState& unwind_state, void* pc); ^ CC(target) Release/obj.target/fibers/src/libcoro/coro.o SOLINK_MODULE(target) Release/obj.target/fibers.node COPY Release/fibers.node make: Leaving directory /tmp/bundle/programs/server/node_modules/fibers/build’
Installed in /tmp/bundle/programs/server/node_modules/fibers/bin/linux-x64-83-glibc/fibers.node
npm WARN lifecycle meteor-dev-bundle@~install: cannot run in wd meteor-dev-bundle@ node npm-rebuild.js (wd=/tmp/bundle/programs/server)
added 157 packages from 85 contributors and audited 157 packages in 17.584s

4 packages are looking for funding
run npm fund for details

found 0 vulnerabilities

╭────────────────────────────────────────────────────────────────╮
│ │
│ New major version of npm available! 6.14.17 → 8.15.1 │
│ Changelog: Release v8.15.1 · npm/cli · GitHub │
│ Run npm install -g npm to update! │
│ │
╰────────────────────────────────────────────────────────────────╯

[root@wprocket server]# mv /tmp/bundle /opt/Rocket.Chat
[root@wprocket server]# chown -R rocketchat:rocketchat /opt/Rocket.Chat
[root@wprocket server]# systemctl start rocketchat

Result:

[root@wprocket tmp]# systemctl status rocketchat
● rocketchat.service - The Rocket.Chat server
Loaded: loaded (/usr/lib/systemd/system/rocketchat.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Wed 2022-07-27 22:50:30 EDT; 1s ago
Process: 11860 ExecStart=/usr/local/bin/node /opt/Rocket.Chat/main.js (code=killed, signal=SEGV)
Main PID: 11860 (code=killed, signal=SEGV)

Jul 27 22:50:21 wprocket systemd[1]: Started The Rocket.Chat server.
Jul 27 22:50:30 wprocket systemd[1]: rocketchat.service: main process exited, code=killed, status=11/SEGV
Jul 27 22:50:30 wprocket systemd[1]: Unit rocketchat.service entered failed state.
Jul 27 22:50:30 wprocket systemd[1]: rocketchat.service failed.

I then roll back with a snapshot and things are good again. I am not sure what the newer versions don’t like about the environment as there isn’t much in the logs to point me in any particular direction. Any pointers would be welcome!

Thanks!

4

Are there any other logs that might be helpful? Should I move everything to a new server? Thanks

5

Hi!

This probably some build problems somewhere.

Can you move this installation to a docker one? I believe all those problems can be fixed with fixed on this kind of deployment, as manual installs can have some problems while upgrading to newer versions.

6

Thanks for the suggestion! Is there a documented way to make that move?

Thanks!

7

Sure thing!

Here you go:

8

I found the exact error I am having in the Github forum under this issue number #26046. It seems I am not alone with this. Are there any other suggestions for a fix before I have to change the architecture?
Thanks!

9

Not sure.

This is kind of out of our hands as the issue comes from node versions, apparently.

10

Oh, would trying different node versions do anything or do I have to go to docker. Will docker work for sure or am I just stuck at this version. I would hate to put in the work to get migrated and run into the same issues.

11

Docker may solve it here, as it will come built with all dependencies.

But I am not completely sure.

In any case, Docker will bring you a lot of improvements, and you can test it pretty easily in your server already.

Let me know if this helps :slight_smile:

12

I am more worried about moving all the existing data over to the new instance than anything. Ok, I guess I know what to try from here. Thanks

1 Like
13

It appears that converting to a docker instance was the answer. Thanks!