For a little while now the armhf snap builds primarily used on the raspberry pi… have become unusable. I’m opening this thread to try and bring information related to it together so we can get this resolved.
The primary issue is the sharp package which was introduced a few versions back.
Basically seems like its shipping 64 bit arm binaries. We use a build service from Canonical on launchpad. They use arm64 workers and use lxd to emulate armhf. My suspicion is some how the sharp installation is getting information leaked through and is including the arm64 binaries instead.
If you have logs or information please add to this thread so we can try to work together to get this resolved.
1 Like
I have written about this over at open.rocket.chat but might as well post it here since I saw one other having the same problem.
I have a rpi3 running ubuntu core and rocket.chat installed as a snap. Regular standard install, nothing custom, just have caddy enabled.
A few days ago, might be a week, cant remember, it updated automatically to version 0.62.1, but since then I have been unable to access the site. It just says “502 bad gateway”. After digging around this is the error that I found after using this command “sudo journalctl -u snap.rocketchat-server.rocketchat-server”:
Any suggestions/help/ideas would be helpful, as I kind find any other piece of information that might point to where the problem is. Every service is up and running, I just can’t access it after the update.
Just to follow up, got some clarification over on open.rocket.chat from “sing.li” regarding the exception. According to him it seems that the module “sharp” is trying to load a non ARM binary, resulting in the error.
Is it possible, and safe, to roll back to the previous version?
Should I perhaps make this an issue on github? Others on open.rocket.chat have mentioned the same issue.
I have tried starting over, and rolling back but still same issue. I have created an issue over on github. It would be great if this issue was dealt with soon as the raspberry pi version is broken.
https://github.com/RocketChat/Rocket.Chat.RaspberryPi/issues/31
Thanks for spending time on this issue. Is there anything else I can do to help? I have some more information on open.rocket.chat, but I don’t know if it will help in posting it here. Anyway, just give a heads up if there are any builds I should try. Or, anything I can help with.
Basically just looking to consolidate the information. If I can see it all together I can troubleshoot it a bit better.
If you do want to try out building and running…
If you are on ubuntu core install the classic snap.
sudo snap install --beta --devmode classic
Which is basically a chroot environment that you can hop into by doing:
sudo classic
Inside then install snapcraft from apt.
From there you can clone the repo… hop into the .snapcraft folder… rm README.md (annoying step I should really solve) and then run: snapcraft That will do a full build.
Once thats finished you can exit the classic snap. Goto the same folder where you cloned and .snapcraft and then run: sudo snap try prime This will install the snap you just build to “try” it.
If not ubuntu core can swap classic snap for installing snapcraft. I like doing: sudo snap install snapcraft --classic 
Thanks Aaron.
It would be great if you or someone else can explain these steps in more details for a total noob like me. I have tried to do what I think you re explaining but it did not work for me.
Any updates on this issue?
Unfortunately no. I’m a bit spread thin and haven’t had time to really get to it yet 
On up note… We are looking for someone to join team to make things like this progress faster, so if you are interested or know someone:
https://rocketchat.recruitee.com/o/deployments-specialist
I build the snap according to your instructions on a Raspberry 3.
After installing and starting the snap i got the exact same error as frederik.t
so this might not be a problem with the builds on launchpad.
It would be great if we cloud make a little progresss here. I think lots of people are trying to install the rocketchart server on a raspberrypi
Got the 'ol ssh up and have been doing a bit of hacking on it today.
CXX(target) Release/obj.target/sharp/src/common.o
In file included from ../vendor/lib/glib-2.0/include/glibconfig.h:9:0,
from ../vendor/include/glib-2.0/glib/gtypes.h:32,
from ../vendor/include/glib-2.0/glib/galloca.h:32,
from ../vendor/include/glib-2.0/glib.h:30,
from ../vendor/include/glib-2.0/gobject/gbinding.h:28,
from ../vendor/include/glib-2.0/glib-object.h:23,
from ../vendor/include/vips/vips8:35,
from ../src/common.cc:25:
../vendor/include/glib-2.0/glib/gtypes.h: In function ‘gboolean _GLIB_CHECKED_ADD_U64(guint64*, guint64, guint64)’:
../vendor/include/glib-2.0/glib/gmacros.h:145:29: error: size of array ‘_GStaticAssertCompileTimeAssertion_0’ is negative
__attribute__((__unused__))
^
../vendor/include/glib-2.0/glib/gmacros.h:241:120: note: in expansion of macro ‘G_GNUC_UNUSED’
#define G_STATIC_ASSERT(expr) typedef char G_PASTE (_GStaticAssertCompileTimeAssertion_, __COUNTER__)[(expr) ? 1 : -1] G_GNUC_UNUSED
^
../vendor/include/glib-2.0/glib/gtypes.h:423:3: note: in expansion of macro ‘G_STATIC_ASSERT’
G_STATIC_ASSERT(sizeof (unsigned long long) == sizeof (guint64));
^
sharp.target.mk:118: recipe for target 'Release/obj.target/sharp/src/common.o' failed
make: *** [Release/obj.target/sharp/src/common.o] Error 1
Is unfortunately where I error out. Which of course is from the sharp package that’s giving people issues
I think one issue might be that the sharp (as well as the grpc) package in the downloaded RocketChat Release
(“curl -SLf “https://releases.rocket.chat/#{RC_VERSION}/download/” -o rocket.chat.tgz”) are only build for x64?
swap #{RC_VERSION} for 0.65.2
I’d recommend trying from this branch: https://github.com/rocketchat/rocket.chat/tree/snap-node-prepare-script
This is where i’m currently trying from. Of course still need to swap #{RC_VERSION} for 0.65.2 so it’ll download the right bundle.
But the bundles aren’t for amd64 only. All of the arch specific stuff is installed with npm install after you download the bundle. One reason we don’t bundle with all dependencies installed
Yeah i swaped the version in the url already.
What i meant is that e.g. in “\programs\server\npm\node_modules\grpc\src\node\extension_binary” only x64 stuff is included and by the npm install later on i will not be installed for arm (same for sharp)?
Ok just irgnore my previous post.
The npm install failed because of a missing phantomjs (what would be the right way so resolve this dependency here? I just downloaded a precompiled one and placed it somewhere in the path?).
Right now i think you’ll have to replace the libs in \programs\server\npm\node_modules\sharp\vendor\lib (which are included in the bundle) with libs compiled for arm because they are used to compile sharp (removing the vendor dir might work as well?)
My hack at the moment is to actually remove the package that needs phantomjs. Because phantomjs is for sure not needed for the execution of Rocket.Chat.
#!/bin/bash
curl -SLf "https://releases.rocket.chat/0.65.2/download/" -o rocket.chat.tgz
tar xvf rocket.chat.tgz --strip 1
cd programs/server
rm -rf npm/node_modules/meteor/emojione_emojione/node_modules/grunt-contrib-qunit
Added this last line. Basically the emojione_emojione package depends on grunt-contrib-qunit in its dev dependencies, which is actually shipped in bundle. Then grunt-contrib-qunit depends on phantomjs in its direct dependencies.
I think this may be an indicator that upstream we need to be building and omitting dev Dependencies.
Looks like meteor build is blindly including dev dependencies in the build bundle when it shouldn’t be.
AH! Success building locally! Finally!
Now pushing changes and sending to launchpad and seeing if it will produce workable snap for arm also
Well… launchpad is still failing… But I went ahead and pushed up my locally build snap.
Revision 1278 is in the stable channel.
Please let me know how it goes.
I build buid a arm version locally too that worked. But the one you uploaded to the stable channe works fine too.
What was your solution? Deleting the vendor dir in the sharp dir as suggested?
Though still hacking away…
Launchpad uses arm64 and convincing node/npm to install the emulated armhf seems to be harder then one would expect.
