The good, the bad and the ugly - My personal assessment of the Rocket.Chat (Github) state


#1

Abstract

First off: this is a follow up to Possibility of Github moderation for community members.

In lights of the breathtakingly high number of open issues I asked, whether it would make sense to give github moderation privileges to assorted community members. Together with with Theo Renck and @sing.li we got a prototype of a Github bot off the ground that I used to delve into the depths of the Rocket.Chat Github repository, more precisely their open bugs. At the time of writing this, I have closed 106 issues with the help of this bot and skimmed the >90 pages of open issues in the process. I won’t say that I have looked at every single issue but I combed through every page to find issues that are not relevant anymore, don’t provide enough information or in some other way justify to close the respective issue.

This post tries to capture this experience and derive actionable information from it.

Disclaimer

Quite obviously, I have an interest in Rocket.Chat flourishing. Otherwise I wouldn’t have invested such a huge amount of my spare time (this post alone took me 3h). That being said: I am going to criticize things and I will most probably be using strong language. If you can’t handle that, please move on. I will try to restrain myself, but you know how it is, when you are involved with something…

Also, my critique might, in some places, not be fully justified because I don’t see the inner workings of the Rocket.Chat Company. I can only interpolate from the things I see on Github / other Rocket.Chat means of public communication.

Also I most certainly made mistakes or have thoughts that might be wrong. If you see those, point them out to me, correct them and put things right. I’m thankful in advance already :slight_smile:

What I did and what I didn’t do

  • I did close issues that were missing information or were filed against some ancient version of Rocket.Chat
  • I did close issues that were, quite obviously support-only question where there was no bug involved (and where it was not likely that one was to be found, even when digging deeper)
  • I did close issues that were duplicates of other issues. I tried to leave open the bug that had the most valuable information in it in order to, at some point, do the thing that was requested
  • I did not close issues that were feature requests (only in very very rare cases, where it was extremely unlikely that this will ever be implemented) - although sometimes you take a step back and think for yourself: has this person EVER considered whether this feature request is of importance to anyone but him and his mum. I could have closed a lot of those, let’s say at least 50. Quite possibly more.
  • I did not close issues that needed some kind of decision of the Rocket.Chat company (I will come back to that topic later in this post)
  • I did not close issues that had recent activity by Rocket.Chat core members

Furthermore, in the process of interacting with those issues and the community members behind them, I deemed it necessary to justify actions and to do so in a positive and generic manner that, on the one hand doesn’t burn my time by individually replying to each case, and on the other hand, do so in a friendly and determined way. For that I wrote a (non-exhaustive) set of templates that I used.

Of those 106 issues, there was maybe a handful of issues that got a reply after I closed them. Some of them thanked me, some of them made me rethink and reopen the issue, as their argument to not close the issue was good and valid. Overall, I would estimate, that the majority (maybe 60%) of issues are actually dead for one or more of the following reasons:

  • the person opening them lost their interest
  • the issue they were reporting has already been fixed
  • the issue is not relevant anymore because the feature has been superseded / replaced by some other feature
  • the person opening them is not willing to provide meaningful information to enable community members to first narrow down and reproduce the issue and then fix it

Common themes, maybe priorities for the future?

In the process of hunting, there were some common themes amongst issues that not always justified closing them as duplicate but could be summarized into “the same category” after all. As a byproduct of this venture, I collected them. This list is not exhaustive but every topic went across my screen at least 3 or (sometimes far) more times:

  • Make it possible to delete or limit (in a temporal dimension) chat history
  • Support full Markdown speccs, also see: https://github.com/RocketChat/Rocket.Chat/pull/7454
  • Support HTTP-Proxy Configuration for every component that fetches HTTP-stuff
  • Backing up Rocket.Chat and restoring it
  • Show various information in the user profile from various sources (sometimes from LDAP, sometimes stuff the people can enter themselves… you get the gist…)
  • Theming Rocket.Chat (no, not only colors, fonts or CSS separately. Real theming, like in "download this theme zip and boom, completely different style, preferrably one default theme and then as a user preference, so every user can choose their theme (provided only by admins of course))
  • Labelling or tagging messages (this is something that deserves a discussion I think, because some proper use case is not inherent in my opinion)
  • Possibility to see all users on a server / in a channel / in a group … so… visibility of users as a general topic
  • Reporting feature for abusive / malicious messages (maybe including moderation features of some kind)
  • Big topic: more fine-grained control over notifications. On a multi-dimensional level, read as: on channel basis, on broadcast-level (@here and friends), on device level, on user-status basis
  • Running Rocket.Chat from a subfolder (maybe not subfolder, but rather from something like https://my.host.name/Rocket1 and https://my.host.name/Rocket2)

You know what… why not do a community voting on this? To get some priorities. Of course, the Rocket.Chat team has their own priorities, but getting the community opinion? Why not!

Mistakes by Rocket.Chat in their handling of Github

Quite obviosly, there have been numerous attempts to somehow deal with the huge influx of new issues. Most of them were, if I may say so, implemented half-heartedly:

  • labelling issues (there are currently ~100 labels) and never doing anything else with them except for labelling
  • assigning issues to core developers - some of them are assigned until today and either dead or, as it seems, of very low priority
  • request more information in issues that didn’t provide enough information (labels like “stat: need more info”) but then not closing those issues after a certain time or when there was no feedback
  • accepting every feature request that’s relevant to maybe 2 but not more than 10 people in the world
  • introducing an issue template far too late (and enforcing it by closing issues that do not adhere)

General stuff that goes wrong

  • I still have the feeling there is barely any testing. Sometimes there are issues that have two or even three incarnations until they are truly fixed. Don’t believe me? Well there you go!
  • no definition of supported platforms or versions. The practice seems to be to only support current stable. Supported (whatever that means by the way) platforms seem to be like: “well… you know, you try to build it on your Temple OS and we’ll even talk to you and try help you out.” That is surely very nice of the developers, but it’s an utter waste of time that could be invested more wisely!
  • Think through issues from the start to the end. I have had multiple occasions where the implication of one new feature or a change in a feature has not been considered for other parts of the application. For example the rendering of a room name (stripping or converting specials chars, putting a hash sign in front of it, take case into account) … it’s been implemented differently in at least two areas of the application currently? This is madness! For details follow up the traces of this issue & this issue and you will have some proper facepalm moments.
  • Documentation: I don’t even know where to start here. The documentation of Rocket.Chat is a mess currently. Point in case: it’s gotten better already, true, but try to get a list of every admin option and the way it works. Or how it may or may not intersect with other options. Good luck. This leads to soooo much support overhead and also to errors when developing new features (see bullet point before). Want a good example of how to do it? Sure! Check out the mattermost docs. While not being perfect, this is lightyears ahead of Rocket.Chat docs. EDIT: I didn’t bother to look into the documentation again before writing this. In fact, some things have changed and improved and it seems to be getting better, indeed. The direction is the right one :slight_smile:

Actions to be taken

This is a list of actions I think would help solve lots of problems. You will most likely disagree with me, but let’s just have a proper discussion.

Organizational

  • implement regular issue triaging. Don’t say you already have because the last two or three weeks almost nobody else than me and @JSzaszvari have replied to new issues on Github. Rocket.Chat team has to manage the influx of new tickets. One realization of this whole Github Bot for community members is: it is a nice addition to enable community members, but by no means can they (nor should they) manage the influx of new issues
  • MAKE DECISIONS! (yes, I find this to be paramount, that’s why I’m screaming!) When there’s a new feature request, decide whether you implement it or not and then
    • either close the issue (with a template) telling the reporter that this will not be implemented (and preferrably even why)
    • or assign it to a realistic milestone (might also be meta-milestones like long-term or whatever) but do so only, if you have thought it through. There’s far more “nice-to-have’s” than you will ever be able to implement!
    • then, do regular triaging of your milestones and weed them out
  • do not allow new features without them being documented exhaustively. Yes. Sometimes writing documention may even take more time than actually implementing the feature. That’s OK. Make this a part of your culture
  • decide for a release schedule. For that, see my issue. You currently have no resources to support more than current stable? It’s OK. Just state it somewhere, it gives you leverage when combatting issues that get reported against old versions.
  • do PR reviews fast. Nothing is more disappointing for someone wanting to help, than having to wait for some feedback. One PR that’s taking weeks to review may put off that person forever!
  • DO NOT TRY TO SUPPORT A NON-TECH AUDIENCE. This might collide with your business goals of spreading the word about Rocket.Chat but somebody running a server application should know the basics. Somebody not even able to fetch the logs of their instance is just a wast of time. Not as a human being but as someone to tackle issues with on eye level. The community can help those people, but you shouldn’t. Unless they pay you, of course. But even then it might be worth thinking about it…

Technical

  • implement UI smoke testing. So much issues get reported just because of wrong symbols, wrong place of buttons, disappearance of buttons, non-scrollable pages and the likes
  • implement general (automated) testing. Yes, more of it than currently. Write the test before the feature.
  • decide for supported deployment methods and document them extensively
  • remove all documentation that belongs to non-supported deployment methods (or introduce something like “Community sourced documentation” in a separate place). Make it clear there, that there will be NO official support for those deployment methods
  • Consider dropping support for the Electron Client. There are better alternatives currently and the time saved could be invested somewhere else.

Considering the title of this post, you might ask yourselves: Where is the good? The good is, I think, that the Rocket.Chat developers and deciders are open to critique and suggestions and do communicate with the community very much. That’s why I think this is gonna be a nice discussion. I’m looking forward to your feedback!

Cheers
Thomas

:slight_smile:


What to do with feature requests?
#2

Thank you @TwizzyDizzy ! Very thoughtful comments such as yours are what moves us forward, we really appreciate your comments and your help with the community.

Hopefully during your upcoming meeting with Gabriel Engel, you can bring up and we can help address as many of your suggestions as possible.

Again, thank you very much and I’m personally looking forward to start acting on changes for the better on Rocket.Chat :wink:


#3

Thank you! Let’s see where it leads us (or more specific: Rocket.Chat). :slight_smile: I’d be happy to hear any comments!

Cheers
Thomas


#4

Out of curiosity, Why?

The Electron App requires minimal maintenance and works well. In regards to the link you posted, That’s pretty cool - but some people want a dedicated app per tool, not a single app that has everything in it.

For such a small amount of work, it gives Rocket.Chat a ‘cross platform’ app that requires minimal upkeep.

The current amount of work to upkeep the electron client is ~5-10 mins per month.


#5

To free resources.

Well, I can’t argue with that, as it’s a matter of personal preference.

Yet the 5-10 mins per month seem to be estimated very optimistic, don’t you think? I mean, keeping all those dependencies up to date and tackling Github issues for the Electron Client. Still I have to take your word for it and if it’s really that small amount of time (or at least in that direction), then I agree with you: the benefit surely outweighs the “cost” in that case.

Cheers
Thomas


#6

We all just got back from our company summit. During which this was definitely a hot topic. We are going to be addressing much of this very soon.

Might be worth creating a topic to break down the things we are going to be doing to change and hopefully bring about this change.


#7

Going to go ahead and follow up. @TwizzyDizzy I know we’ve had some lengthy discussions last week but think its worth trying to summarize a bit here. :grin:

Community is super important to us. Its one of the things that makes us what we are. So for the community sake going to summarize a bit here. Even if we end up posting a bit more about this in another topic.

Github has this issue voting thing… but it sucks. Sure you can sort by number of thumbs up… but visually it just isn’t that helpful.

So the idea here is we’d make a bit better use of these wonderful forums. Feature requests can be made and drafted out. Ideally in as much detail as possible by the requester. People can vote and discuss on it.

When we decide to actually put this feature on the roadmap, we would communicate that on the topic, as well as create the issue on github and label it with feature planned.

That way our github feature requests only represent what we truly plan to implement.

This basically continuing on with what I said above. Feature requests that are planned only. If its “hey I need help” we will be referring to this forum. The community support category. Or over to #support on open.rocket.chat

Not in any way trying to silence things. But we need to preserve github for real bugs that we can fix, as well as feature requests we plan to implement.

So being more decisive. :slight_smile:

Yeah… this one is on us. We have unit tests and end to end tests. Obviously we need to enforce creating of these along with new features, because things like that still happen.

We are acting on this one now. We will have Officially supported installation methods. These we will curate and make rock solid docs for. We will also add automated testing to make sure the installation on these platforms works with every release.

For those that don’t fall in the officially supported category. They will be moved here to the forum. We have a category called #community-guides . Topics created here are wiki style and community supported methods will live here.

This will make your life easier as well as ours. An end user will only see docs on our website for things that are tested and we guarantee will work. While also allowing someone that is passionate about freebsd and wants to use it a community supported installation method. They can work with other enthusiast to get it working.

We think turning these over to you guys, the community is very beneficial to everyone.

Consistency and stability. This is going to be a major focus of ours. Spending more time on stability then on new features. Making sure that the code base is consistent. We still have some planning to do here, but things should be getting better.

This is constantly evolving. We are working to simplify this even further. We want to make it very easy to get to the information you as an end user want and need. As you said this is a process, and glad to see you say we are on the right track :slight_smile:

:blush: Thanks for the prod. We definitely agree.

This one was hard for us to realize. But yes we totally agree. As mentioned above we are going to be implementing regular processes to evaluate proposed features and make it clear the ones we actually plan to do.

We’ve taken an initial step on this. All new rest api endpoints have to be documented before we merge. I don’t see why we couldn’t enforce this for all new features.

This one we just kind of started doing. I didn’t realize that not everyone picked up on it. :grin:

New releases happen the 27th of every month. Feature freeze happens on the 20th of the month and accompanied by a release candidate. After that freeze only fixes will be allowed.

We also have a checklist that is posted for every release so its clear where we are on the process. Example of that here:

We will be documenting this in our docs as well.

Agreed. We have some PR’s that have maybe been open a bit too long. We must do a better job at this.

I think we should break this off into its own topic. Would love to discuss a bit more of why this feeling. :slight_smile:

I didn’t answer all, but hopefully you can see we are working to improve.

Thanks for the feedback. It may be rough and you may feel bad for being that guy… But don’t! We really do appreciate it.


#8

Hey Aaron!

thanks for making transparent the steps we talked about at the summit (my internet is not really reliable as I’m still on the road in brazil). Now that I’ve experienced everybody first-hand I’m pretty optimistic about where things are heading. I don’t believe everything will be perfect in an instant, yet I truly believe the right steps will be taken and things will slowly but surely improve :slight_smile:

High five, team Rocket.Chat! <3

Cheers
Thomas