Introducing a new utility to manage custom clients from the server


#1

I’d like to introduce a new utility called Client Commands, a solution created to allow the Rocket.Chat server to trigger actions in subscriber clients (bots and possibly other websocket clients). This is handled at the adapter and/or SDK level, not by final users (e.g. normal bot developers).

The problem

Bots subscribe to a message stream and respond to message events, but there’s no way for the server to prompt them to do anything other than that.

In order to provide a range of new management features for administrating bot clients, getting data or triggering any non message response action, we need to send data to be interpreted by the client as a command. Such data, identified here by ClientCommands, could not be sent through the normal message stream, because:

  • a) it would need hack workarounds to filter commands from normal message data
  • b) it would be kept forever, bloating message collection storage

The solution

While some features could be added with specific implementations, ClientCommands provides flexibility, making it useful for multiple cases. It keeps code DRY by providng a core utility to developers so they’re not re-inventing a range of inconsistent solutions. It should be a fairly obvious architecture to learn as well, accelerating the pace of development for new client management features.

Some management features that can be implemented with the ClientCommands are:

  1. Check bot’s aliveness - As commented, it could be a simple endpoint, probably HTTP. But we could send instead a simple 'heartbeat' ClientCommand and the client will send a response indicating that it is alive.
  2. Pause/resume bots - Providing a single interface for an admin to pause the operation of bots can be done by sending a 'pauseMessageStream' command, it will be handled in the SDK that will stop receiving any messages from the server. This can be useful for admins that don’t have access to where the bot is being hosted and need to stop it.
  3. Clear cache / reset memory - This is theoretical, but could be something useful for adapters that have cache that might not be working correctly or contain data that needs to be deleted (e.g. user was removed from server, for GDPR compliance), so an admin can send a command to the adapter, that will then clear its cache and reply indicating success or failure. While this particular management option might not be useful to all, it shows that you can manage anything implemented by the SDK/Adapter, as long as it listens to the specified command.

Useful features that improve UX and can be added with ClientCommands:

  1. Autocomplete bots commands - Since each bot framework behaves differently when asking for the available commands, even if they all use the 'help' message, the response format is different. Each adapter can listen to the 'availableCommands' ClientCommand, and knowing its architecture, get the commands in an array and reply. On server-side, by sending the ClientCommand 'availableCommands', we can store the response in the bot User model with a common syntax for all adapters, making it possible for the server to know which commands the bot can respond to.
  2. Callbacks - An adapter might provide an interface, on top of the ClientCommands, to add callbacks to be called upon conversational context. As approached by @tim.kinnane here.

It is also important to note that bots that don’t support ClientCommands will keep operating as normal, since all features that depend on ClientCommands will check if the client is listening to the specific commands and will not appear for the bots that don’t.

The implementation

While a simple publication and collection would be enough to send ClientCommands, there is an overhead when dealing with collections that might become a problem in larger systems.

Therefore, after discussing with members of the Rocket.Chat team, meteor-streamer was picked to handle the communication from server to clients, it does not use a collection on server-side so it reduces a lot of overhead.

The package rocketchat-client-commands has a function to send ClientCommands, a method to reply to ClientCommands (called by the client) and a startup file to set up the stream.

Sending a command (in the server)

To send a ClientCommand all you need to do is to call the function via RocketChat.sendClientCommand(user, command [, timeout]), where:

  • user: Object of the target user, containing the _id and username properties
  • command: Object of the command, where it must have at least the key property, a unique string
  • timeout: Optional parameter of the timeout for the client to reply the command, defaults to 5 seconds.
  • It returns a promise that resolves with a reply or rejects with a timeout error

This function adds a listener for a 'client-command-response-<_id of the command>' event in the internal EventEmitter called RocketChat and initiates a timeout.

If the listener is called, the timeout is cleared and the function resolves with the event’s first parameter, the ClientCommand’s response object.

If the listener is not called within the timeout period, the timeout removes the listener and rejects the promise.

Example - Sending a pauseMessageStream command:

const bot = {
  _id: 'e9e89dqw823d81',
  username: 'bot'
};

RocketChat.sendClientCommand(bot, { key: 'pauseMessageStream' })
  .then((response) => {
    // client replied the command
    console.log(response);
  })
  .catch((err) => {
    // client did not reply within 5 seconds
    console.log(err);
  });

Replying to commands (in the client)

Once the client receives the command and acts according to it, it should call the replyClientCommand method (on the server, via SDK driver) with two parameters, the _id of the command and the response object.

The method will then emit a 'client-command-response-<_id of the command>' event via RocketChat.


#2

:heart: :tada: :100: :fire:

Just commenting to say I’m really happy with how this solution has evolved. There was some early resistance to architecture choices and some confusion about the use cases, but I think Mikael has done an excellent job of evolving it to overcome those issues and explaining the approach. I hope the community can see as much potential in this as I do.


#3

Another feature this enables, would be requesting data about a bot’s usage statistics, e.g. for a custom dashboard view within admin screens, showing how many unique users it’s interacted with, errors it’s logged, messages received etc. These stats can’t be requested form Rocket.Chat’s data alone because in a public room, there’s no data point to track when a user was addressing a bot, or being specifically responded to by the bot. They could be found in the bot’s own data or logs, but that would distribute admin oversight to multiple systems. It would be so great to have this high level data available at a glance from within Rocket.Chat - and ClientCommands enables that fundamental communication layer.


#4

This approach looks very powerful and useful.

I’m very keen on the concept that admins can use single point of management for the common scenarios that @mikaelmello has mentioned - I really like the proposed solution for my previous request about simplified bot heartbeat monitoring.

Also very pleased that this approach drives standardisation to access bot functionality and avoids the requirement that otherwise exists to try and use custom syntaxes and filters to deliver commands to bots inside messages while protecting humans from reading such commands.

I also agree with @tim.kinnane that this approach promises great ability to gather better intelligence from bots. One of the bot-related issues my team is grappling with is how best to observe and understand how human users interact with the bot services that we provide, so we can iterate and refine our bot scripts and interactions.

Great work @mikaelmello, I am very much looking forward to seeing this new capability in Rocket.chat


#5

Very well wrote up! I’m glad to see the use of streams instead of collections.

Collections are definitely very heavy and put strain on DB, Rocket.Chat and client. Over all way more information being synced everywhere then is good for non-persistent events :slight_smile:

One note of caution or advice. Make sure you test with multiple instances. Single instance this is an absolute piece of cake. Event happens on instance 1 and bot would be connected to instance 1.

It does get a bit more challenging when you have multiple instances. Because at that point event might happen on instance 2 but the bot is connected to instance 1.

In this case you need to make sure that the event is fired over what I call “The event matrix”. (I don’t think we have an official term for it)

Basically each server maintains a DDP connection to the other instances running. The streams are then relayed to the other servers via the matrix

Think in the case of typing indicator. User is typing that event is sent to the instance they are connected, then its broadcast over the event matrix and the other instances let subscribers to that stream know about the typing.

You may have gone down that rabbit hole already, but just incase figured i’d make sure to point out.


#6

@aaron.ogle How do we test this? I thought that would be handled at a lower level, by the streamer package or meteor itself. Are there developer docs for this event matrix of which you speak?

Just to be clear, we’re talking about multiple instances like a cloud server cluster, not the other kind of multiple instances referred to sometimes as federation, where on Rocket.Chat server runs multiple instances of Rocket.Chat? I thought the latter was still hypothetical but if it’s not, I’ve got lots of catching up to do.


#7

Some documentation: https://github.com/RocketChat/meteor-streamer/blob/master/packages%2Frocketchat-streamer%2FREADME.md

I think by default it should work okay depending on configuration. Some configurations will isolate to the same instance.

A good example of the streamer being used is this file: https://github.com/RocketChat/Rocket.Chat/blob/develop/packages/rocketchat-lib/client/Notifications.js#L12

Right. The distinction I try to make is:

  • Instance - a copy of Rocket.Chat
  • Server - one or more copies of Rocket.Chat basically a Rocket.Chat install.