The good, the bad and the ugly - My personal assessment of the Rocket.Chat (Github) state

Feedback
1

Abstract

First off: this is a follow up to Possibility of Github moderation for community members - #36 by sing.li.

In lights of the breathtakingly high number of open issues I asked, whether it would make sense to give github moderation privileges to assorted community members. Together with with Theo Renck and @sing.li we got a prototype of a Github bot off the ground that I used to delve into the depths of the Rocket.Chat Github repository, more precisely their open bugs. At the time of writing this, I have closed 106 issues with the help of this bot and skimmed the >90 pages of open issues in the process. I wonā€™t say that I have looked at every single issue but I combed through every page to find issues that are not relevant anymore, donā€™t provide enough information or in some other way justify to close the respective issue.

This post tries to capture this experience and derive actionable information from it.

Disclaimer

Quite obviously, I have an interest in Rocket.Chat flourishing. Otherwise I wouldnā€™t have invested such a huge amount of my spare time (this post alone took me 3h). That being said: I am going to criticize things and I will most probably be using strong language. If you canā€™t handle that, please move on. I will try to restrain myself, but you know how it is, when you are involved with somethingā€¦

Also, my critique might, in some places, not be fully justified because I donā€™t see the inner workings of the Rocket.Chat Company. I can only interpolate from the things I see on Github / other Rocket.Chat means of public communication.

Also I most certainly made mistakes or have thoughts that might be wrong. If you see those, point them out to me, correct them and put things right. Iā€™m thankful in advance already :slight_smile:

What I did and what I didnā€™t do

  • I did close issues that were missing information or were filed against some ancient version of Rocket.Chat
  • I did close issues that were, quite obviously support-only question where there was no bug involved (and where it was not likely that one was to be found, even when digging deeper)
  • I did close issues that were duplicates of other issues. I tried to leave open the bug that had the most valuable information in it in order to, at some point, do the thing that was requested
  • I did not close issues that were feature requests (only in very very rare cases, where it was extremely unlikely that this will ever be implemented) - although sometimes you take a step back and think for yourself: has this person EVER considered whether this feature request is of importance to anyone but him and his mum. I could have closed a lot of those, letā€™s say at least 50. Quite possibly more.
  • I did not close issues that needed some kind of decision of the Rocket.Chat company (I will come back to that topic later in this post)
  • I did not close issues that had recent activity by Rocket.Chat core members

Furthermore, in the process of interacting with those issues and the community members behind them, I deemed it necessary to justify actions and to do so in a positive and generic manner that, on the one hand doesnā€™t burn my time by individually replying to each case, and on the other hand, do so in a friendly and determined way. For that I wrote a (non-exhaustive) set of templates that I used.

Of those 106 issues, there was maybe a handful of issues that got a reply after I closed them. Some of them thanked me, some of them made me rethink and reopen the issue, as their argument to not close the issue was good and valid. Overall, I would estimate, that the majority (maybe 60%) of issues are actually dead for one or more of the following reasons:

  • the person opening them lost their interest
  • the issue they were reporting has already been fixed
  • the issue is not relevant anymore because the feature has been superseded / replaced by some other feature
  • the person opening them is not willing to provide meaningful information to enable community members to first narrow down and reproduce the issue and then fix it

Common themes, maybe priorities for the future?

In the process of hunting, there were some common themes amongst issues that not always justified closing them as duplicate but could be summarized into ā€œthe same categoryā€ after all. As a byproduct of this venture, I collected them. This list is not exhaustive but every topic went across my screen at least 3 or (sometimes far) more times:

  • Make it possible to delete or limit (in a temporal dimension) chat history
  • Support full Markdown speccs, also see: [FIX] Support complete markdown specification by gromain Ā· Pull Request #7454 Ā· RocketChat/Rocket.Chat Ā· GitHub
  • Support HTTP-Proxy Configuration for every component that fetches HTTP-stuff
  • Backing up Rocket.Chat and restoring it
  • Show various information in the user profile from various sources (sometimes from LDAP, sometimes stuff the people can enter themselvesā€¦ you get the gistā€¦)
  • Theming Rocket.Chat (no, not only colors, fonts or CSS separately. Real theming, like in "download this theme zip and boom, completely different style, preferrably one default theme and then as a user preference, so every user can choose their theme (provided only by admins of course))
  • Labelling or tagging messages (this is something that deserves a discussion I think, because some proper use case is not inherent in my opinion)
  • Possibility to see all users on a server / in a channel / in a group ā€¦ soā€¦ visibility of users as a general topic
  • Reporting feature for abusive / malicious messages (maybe including moderation features of some kind)
  • Big topic: more fine-grained control over notifications. On a multi-dimensional level, read as: on channel basis, on broadcast-level (@here and friends), on device level, on user-status basis
  • Running Rocket.Chat from a subfolder (maybe not subfolder, but rather from something like https://my.host.name/Rocket1 and https://my.host.name/Rocket2)

You know whatā€¦ why not do a community voting on this? To get some priorities. Of course, the Rocket.Chat team has their own priorities, but getting the community opinion? Why not!

Mistakes by Rocket.Chat in their handling of Github

Quite obviosly, there have been numerous attempts to somehow deal with the huge influx of new issues. Most of them were, if I may say so, implemented half-heartedly:

  • labelling issues (there are currently ~100 labels) and never doing anything else with them except for labelling
  • assigning issues to core developers - some of them are assigned until today and either dead or, as it seems, of very low priority
  • request more information in issues that didnā€™t provide enough information (labels like ā€œstat: need more infoā€) but then not closing those issues after a certain time or when there was no feedback
  • accepting every feature request thatā€™s relevant to maybe 2 but not more than 10 people in the world
  • introducing an issue template far too late (and enforcing it by closing issues that do not adhere)

General stuff that goes wrong

  • I still have the feeling there is barely any testing. Sometimes there are issues that have two or even three incarnations until they are truly fixed. Donā€™t believe me? Well there you go!
  • no definition of supported platforms or versions. The practice seems to be to only support current stable. Supported (whatever that means by the way) platforms seem to be like: ā€œwellā€¦ you know, you try to build it on your Temple OS and weā€™ll even talk to you and try help you out.ā€ That is surely very nice of the developers, but itā€™s an utter waste of time that could be invested more wisely!
  • Think through issues from the start to the end. I have had multiple occasions where the implication of one new feature or a change in a feature has not been considered for other parts of the application. For example the rendering of a room name (stripping or converting specials chars, putting a hash sign in front of it, take case into account) ā€¦ itā€™s been implemented differently in at least two areas of the application currently? This is madness! For details follow up the traces of this issue & this issue and you will have some proper facepalm moments.
  • Documentation: I donā€™t even know where to start here. The documentation of Rocket.Chat is a mess currently. Point in case: itā€™s gotten better already, true, but try to get a list of every admin option and the way it works. Or how it may or may not intersect with other options. Good luck. This leads to soooo much support overhead and also to errors when developing new features (see bullet point before). Want a good example of how to do it? Sure! Check out the mattermost docs. While not being perfect, this is lightyears ahead of Rocket.Chat docs. EDIT: I didnā€™t bother to look into the documentation again before writing this. In fact, some things have changed and improved and it seems to be getting better, indeed. The direction is the right one :slight_smile:

Actions to be taken

This is a list of actions I think would help solve lots of problems. You will most likely disagree with me, but letā€™s just have a proper discussion.

Organizational

  • implement regular issue triaging. Donā€™t say you already have because the last two or three weeks almost nobody else than me and @JSzaszvari have replied to new issues on Github. Rocket.Chat team has to manage the influx of new tickets. One realization of this whole Github Bot for community members is: it is a nice addition to enable community members, but by no means can they (nor should they) manage the influx of new issues
  • MAKE DECISIONS! (yes, I find this to be paramount, thatā€™s why Iā€™m screaming!) When thereā€™s a new feature request, decide whether you implement it or not and then
    • either close the issue (with a template) telling the reporter that this will not be implemented (and preferrably even why)
    • or assign it to a realistic milestone (might also be meta-milestones like long-term or whatever) but do so only, if you have thought it through. Thereā€™s far more ā€œnice-to-haveā€™sā€ than you will ever be able to implement!
    • then, do regular triaging of your milestones and weed them out
  • do not allow new features without them being documented exhaustively. Yes. Sometimes writing documention may even take more time than actually implementing the feature. Thatā€™s OK. Make this a part of your culture
  • decide for a release schedule. For that, see my issue. You currently have no resources to support more than current stable? Itā€™s OK. Just state it somewhere, it gives you leverage when combatting issues that get reported against old versions.
  • do PR reviews fast. Nothing is more disappointing for someone wanting to help, than having to wait for some feedback. One PR thatā€™s taking weeks to review may put off that person forever!
  • DO NOT TRY TO SUPPORT A NON-TECH AUDIENCE. This might collide with your business goals of spreading the word about Rocket.Chat but somebody running a server application should know the basics. Somebody not even able to fetch the logs of their instance is just a wast of time. Not as a human being but as someone to tackle issues with on eye level. The community can help those people, but you shouldnā€™t. Unless they pay you, of course. But even then it might be worth thinking about itā€¦

Technical

  • implement UI smoke testing. So much issues get reported just because of wrong symbols, wrong place of buttons, disappearance of buttons, non-scrollable pages and the likes
  • implement general (automated) testing. Yes, more of it than currently. Write the test before the feature.
  • decide for supported deployment methods and document them extensively
  • remove all documentation that belongs to non-supported deployment methods (or introduce something like ā€œCommunity sourced documentationā€ in a separate place). Make it clear there, that there will be NO official support for those deployment methods
  • Consider dropping support for the Electron Client. There are better alternatives currently and the time saved could be invested somewhere else.

Considering the title of this post, you might ask yourselves: Where is the good? The good is, I think, that the Rocket.Chat developers and deciders are open to critique and suggestions and do communicate with the community very much. Thatā€™s why I think this is gonna be a nice discussion. Iā€™m looking forward to your feedback!

Cheers
Thomas

:slight_smile:

12 Likes
2

Thank you @TwizzyDizzy ! Very thoughtful comments such as yours are what moves us forward, we really appreciate your comments and your help with the community.

Hopefully during your upcoming meeting with Gabriel Engel, you can bring up and we can help address as many of your suggestions as possible.

Again, thank you very much and Iā€™m personally looking forward to start acting on changes for the better on Rocket.Chat :wink:

1 Like
3

Thank you! Letā€™s see where it leads us (or more specific: Rocket.Chat). :slight_smile: Iā€™d be happy to hear any comments!

Cheers
Thomas

4

Out of curiosity, Why?

The Electron App requires minimal maintenance and works well. In regards to the link you posted, Thatā€™s pretty cool - but some people want a dedicated app per tool, not a single app that has everything in it.

For such a small amount of work, it gives Rocket.Chat a ā€˜cross platformā€™ app that requires minimal upkeep.

The current amount of work to upkeep the electron client is ~5-10 mins per month.

5

To free resources.

Well, I canā€™t argue with that, as itā€™s a matter of personal preference.

Yet the 5-10 mins per month seem to be estimated very optimistic, donā€™t you think? I mean, keeping all those dependencies up to date and tackling Github issues for the Electron Client. Still I have to take your word for it and if itā€™s really that small amount of time (or at least in that direction), then I agree with you: the benefit surely outweighs the ā€œcostā€ in that case.

Cheers
Thomas

6

We all just got back from our company summit. During which this was definitely a hot topic. We are going to be addressing much of this very soon.

Might be worth creating a topic to break down the things we are going to be doing to change and hopefully bring about this change.

2 Likes
7

Going to go ahead and follow up. @TwizzyDizzy I know weā€™ve had some lengthy discussions last week but think its worth trying to summarize a bit here. :grin:

Community is super important to us. Its one of the things that makes us what we are. So for the community sake going to summarize a bit here. Even if we end up posting a bit more about this in another topic.

Github has this issue voting thingā€¦ but it sucks. Sure you can sort by number of thumbs upā€¦ but visually it just isnā€™t that helpful.

So the idea here is weā€™d make a bit better use of these wonderful forums. Feature requests can be made and drafted out. Ideally in as much detail as possible by the requester. People can vote and discuss on it.

When we decide to actually put this feature on the roadmap, we would communicate that on the topic, as well as create the issue on github and label it with feature planned.

That way our github feature requests only represent what we truly plan to implement.

This basically continuing on with what I said above. Feature requests that are planned only. If its ā€œhey I need helpā€ we will be referring to this forum. The community support category. Or over to #support on open.rocket.chat

Not in any way trying to silence things. But we need to preserve github for real bugs that we can fix, as well as feature requests we plan to implement.

So being more decisive. :slight_smile:

Yeahā€¦ this one is on us. We have unit tests and end to end tests. Obviously we need to enforce creating of these along with new features, because things like that still happen.

We are acting on this one now. We will have Officially supported installation methods. These we will curate and make rock solid docs for. We will also add automated testing to make sure the installation on these platforms works with every release.

For those that donā€™t fall in the officially supported category. They will be moved here to the forum. We have a category called Community Guides . Topics created here are wiki style and community supported methods will live here.

This will make your life easier as well as ours. An end user will only see docs on our website for things that are tested and we guarantee will work. While also allowing someone that is passionate about freebsd and wants to use it a community supported installation method. They can work with other enthusiast to get it working.

We think turning these over to you guys, the community is very beneficial to everyone.

Consistency and stability. This is going to be a major focus of ours. Spending more time on stability then on new features. Making sure that the code base is consistent. We still have some planning to do here, but things should be getting better.

This is constantly evolving. We are working to simplify this even further. We want to make it very easy to get to the information you as an end user want and need. As you said this is a process, and glad to see you say we are on the right track :slight_smile:

:blush: Thanks for the prod. We definitely agree.

This one was hard for us to realize. But yes we totally agree. As mentioned above we are going to be implementing regular processes to evaluate proposed features and make it clear the ones we actually plan to do.

Weā€™ve taken an initial step on this. All new rest api endpoints have to be documented before we merge. I donā€™t see why we couldnā€™t enforce this for all new features.

This one we just kind of started doing. I didnā€™t realize that not everyone picked up on it. :grin:

New releases happen the 27th of every month. Feature freeze happens on the 20th of the month and accompanied by a release candidate. After that freeze only fixes will be allowed.

We also have a checklist that is posted for every release so its clear where we are on the process. Example of that here:

We will be documenting this in our docs as well.

Agreed. We have some PRā€™s that have maybe been open a bit too long. We must do a better job at this.

I think we should break this off into its own topic. Would love to discuss a bit more of why this feeling. :slight_smile:

I didnā€™t answer all, but hopefully you can see we are working to improve.

Thanks for the feedback. It may be rough and you may feel bad for being that guyā€¦ But donā€™t! We really do appreciate it.

5 Likes
8

Hey Aaron!

thanks for making transparent the steps we talked about at the summit (my internet is not really reliable as Iā€™m still on the road in brazil). Now that Iā€™ve experienced everybody first-hand Iā€™m pretty optimistic about where things are heading. I donā€™t believe everything will be perfect in an instant, yet I truly believe the right steps will be taken and things will slowly but surely improve :slight_smile:

High five, team Rocket.Chat! <3

Cheers
Thomas

3 Likes
10

So Iā€™m gonna necro this because itā€™s been 3 years and it feels like not much has changed.
Thereā€™s still an overwhelming number of open issues, many seeming to be for versions that arenā€™t being developed for anymore, does triage happen? it feels like very little testing goes into builds, new versions shouldnā€™t ship with UI bugs. I could rant and rant but do some QA please

3 Likes
11

Hi,

Some things have changed, but it takes time to filter through.

We now have a community team, and we care :slight_smile:

I think you can see that we are now making sure enquiries in open and forums are answered.

Weā€™d love to see more community members step up and put a bit back into the product - helping on open/forums/bugs.

We are triaging bugs - but there are a lot and it will take time. Due to the internal tracking system it doesnā€™t look like some of them get attention, but they do. I find this frustrating that the community canā€™t see issues being worked on, and we intend to try and resolve this in due course. But it takes time.

We got rid of the hated ā€˜close botā€™, but as you rightly say, some bugs are old and we need to go back and kill a lot off. I have a daily with my team where we are starting to address this.

We have already taken community input on the fixing some of the most annoying bugs - 3.15.x contained a number of them - and the product team are focusing their work on stability fixes currently.

As Rocket.Chat grows we are also building a QA team and will be working hard to up our game there as well.

No, none of it will be fixed in 5 minutes, but to quote a well known song ā€˜The Times They Are A-Changinā€™ !!

If you want to influence change then please, donā€™t sit on the sidelines and snipe. Get involved!! We are listening.

3 Likes
12

Still an issue imho. Year old issues are left open for no apparent reason with no action in years (example I randomly picked: App is stuck and the routing is triggered after a very long time Ā· Issue #9014 Ā· RocketChat/Rocket.Chat Ā· GitHub ).

Please do yourself a favor and be more agressive in closing irrelevant issues. It makes it easier for everyone. If it takes 10 seconds to check if an issue is blatantly outdated or irrelevant (just catching the obvious ones: Pareto principle) you could work through them all in a workday.

13

hi @stanner !

Thanks for your input. We really appreciate it.

As said previously, things are changing. We are doing some house keeping tasks before attacking the long time opened issues.

The one you picked, for example, is probably already closed, considering it is possible related to Trying to access chatMessages before data present Ā· Issue #9038 Ā· RocketChat/Rocket.Chat Ā· GitHub

Please, consider participating in our Community Open Call so we can discuss this issues: Rocket.Chat's Community Open Call | Rocket.Chat

Thanks!