Supporting rich chatbot messages/menus

Feature Requests (historic archive)
1

Issue #6786 in Rocket.chat repo is an issue covering this general topic. We have an urgent need to add FB Messenger style rich chatbot messages and plan to do it in our fork but want to make sure our approach is at least somewhat aligned with the overall team vision for this. Our immediate need is Android support but of course we’d like to see the features implemented in web and IOS as well. The issue for tracking Android support is #1027 in Rocket.chat.android repo.

Our requirement is to support a subset (not clear which we need yet) of the following rich messages (https://developers.facebook.com/docs/messenger-platform/send-messages) as well as persistent menus, https://developers.facebook.com/docs/messenger-platform/send-messages/persistent-menu.

I’d like to discuss here some aspects of the design to see if we can implement an approach that will align with the upstream Rocket.chat team thinking/approach.

First, we are using a modified version of hubot-apiai adapter to direct message a bot that is implemented in dialogflow/api.ai. This bot will send back rich messages to the hubot adapter which will send them to the Rocket.chat server, which will send them to the Rocket.chat client. Our initial version will be Android (IOS and Web to follow).

Design question #1: Can we simply add a new attribute (“customResponse” or something) to the message object that is passed through the Rocket.chat server and sent to the Android client? Would this require a modification to the Rocket.chat server or would it simply pass the attribute through to the client? And, would this break IOS and web clients?

Note: if a server change is required we understand that it should probably be implemented as a Rocket.chat.app but that feature is not yet ready so we’re guessing we’ll need to fork the server code and modify it.

Design question #2: For persistent menu support, should this be an attribute associated with the user that the bot is associated with? Is it possible to extend the user object with attributes that would include the data needed to describe these persistent menus?

2

@aaron.ogle @bradley.hilton Not sure if either of you would know, but if you could point us in the right direction - But what sort of consequences (if any) would there be to adding a new attribute to the message object

I fear that this could break a lot of things, but I am honestly not too sure. As mentioned above, the current thinking is around adding a new item there that we can use for rich messaging metadata/info.

Is it possible to extend the user object with attributes that would include the data needed to describe these persistent menus

@bizzbyster Sorry i’m not following? Do you mean something like this - Legacy: Attaching interactive message menus | Slack ?

associated with the user that the bot is associated with?

Got to think more than bots as well, as this would need to be supported by the REST API as well for incoming messages - https://rocket.chat/docs/developer-guides/rest-api/chat/postmessage/
If so we can use the same thinking of storing that data in the custom field as a nested JSON array right?

A complete message could look something like this

{
  "alias": "Gruggy",
  "avatar": "http://res.guggy.com/logo_128.png",
  "channel": "#general",
  "emoji": ":smirk:",
  "roomId": "Xnb2kLD2Pnhdwe3RH",
  "text": "Sample message",
  "attachments": [
    {
      "audio_url": "http://www.w3schools.com/tags/horse.mp3",
      "author_icon": "https://avatars.githubusercontent.com/u/850391?v=3",
      "author_link": "https://rocket.chat/",
      "author_name": "Bradley Hilton",
      "collapsed": false,
      "color": "#ff0000",
      "fields": [
        {
          "short": true,
          "title": "Test",
          "value": "Testing out something or other"
        },
        {
          "short": true,
          "title": "Another Test",
          "value": "[Link](https://google.com/) something and this and that."
        }
      ],
      "image_url": "http://res.guggy.com/logo_128.png",
      "message_link": "https://google.com",
      "text": "Yay for gruggy!",
      "thumb_url": "http://res.guggy.com/logo_128.png",
      "title": "Attachment Example",
      "title_link": "https://youtube.com",
      "title_link_download": true,
      "ts": "2016-12-09T16:53:06.761Z",
      "video_url": "http://www.w3schools.com/tags/movie.mp4"
    }
  ],
  "richAttachments": [
    {
      "text": "Select a button",
      "fallback": "You are unable to select a button",
      "callback_id": "some_function",
      "color": "#3AA3E3",
      "attachment_type": "buttons",
      "actions": [
        {
          "name": "button1",
          "text": "This is a button",
          "type": "button",
          "value": "button1"
        },
        {
          "name": "button1",
          "text": "This is a button",
          "type": "button",
          "value": "button1"
        },
        {
          "name": "button1",
          "text": "This is a button",
          "type": "button",
          "value": "button1",
          "confirm": {
            "title": "Are you sure?",
            "text": "Please confirm you want to push this button",
            "ok_text": "Yes",
            "dismiss_text": "No"
          }
        }
      ]
    },
    {
      "name": "my_dropdown_list",
      "text": "Pick a game...",
      "type": "dropdown",
      "options": [
        {
          "text": "Selection 3",
          "value": "option1"
        },
        {
          "text": "Selection 3",
          "value": "option2"
        },
        {
          "text": "Selection 3",
          "value": "option3"
        }
      ]
    }
  ]
}

See the richAttachments field at the end

Note: if a server change is required we understand that it should probably be implemented as a Rocket.chat.app but that feature is not yet ready so we’re guessing we’ll need to fork the server code and modify it.

Yep more than likely - Although integrating this into the core of the product might be worthwhile as well. I dont think this should be an optional app that people install, rather something that comes with Rocket.Chat.

Rocket.Chat is pretty much the only chat platform out there (Slack style ones) that does not support interactive messages at this time.

3

@bizzbyster

See here for some ideas on how it could be structured

https://docs.mattermost.com/developer/interactive-message-buttons.html

From my perspective, it would be good to start the implementation replicating how slack or something similar does it and then build it out from there.

4

Yes, I want Rocket.Chat to support this however I want to take some time and investigate how the various platforms out there accomplish it via not only code but also design. This is one feature that if we get wrong, it won’t be easy to fix. Thus I am not promising we will get something in by the next release or two. I know that’s not what we want to hear, but I’m just being real with you.

5

That’s to be expected. I have no time frame attached to this. Not sure about @bizzbyster

I agree, That’s why I’m looking to transform what I’ve done into something more in line with the standards that Slack + Other platforms have designed around this.

Then I’ll either submit a PR, or release it as a separate meteor/rocket.chat module and then people can do with what they wish. Honestly I could not care less if it gets merged / accepted - I’m merely going to contribute back what I’ve done internally for the company I work with and then if you want to throw it away, modify it, whatever - I don’t really care.

I’m asking about modifying the message object not to make a decision on this but to help my project that I am working on, for my self.

6

@bradley.hilton Also, heres the complete spec of slacks implementation if you ever want to have a read. Pretty sure every one else who has implemented that has followed it.

Looked at MatterMost’s documentation and its pretty much a copy and paste

For my project I might just continue down the path of implementing it exactly how slack has done it. The RC and Slack Message object are the same almost, and the concepts are the same. Just going to clone it 1:1 . Considering Rocket.Chat is sold as a “Slack Alternative” and the webhook message format is the same - Then i’m just going to assume that its a good way to start it.

Also @bradley.hilton regarding about getting something into the next release, we’re not asking you guys to work on it - we are going to work on it for our selves as we both have custom deployments.

If you guys want to use it, then up too you. But we’re more asking here for our own project. Not to force it into a public release.

7

From the way he was talking and he said they urgently needed this, it sounds like he does so that’s why I said it.

I doubt it will break anything. Only places it would break are the typed class environments which represent the message object as typed classes.

Honestly, probably the safest route to go for now. I’ve read it in the past and liked it.

I understand. I was stating that to ensure all ideas of it making it into the official codebase are not entertained and also on our mobile applications either, as they have started to plan out the next quarter from what I understand.

8

Thank you @bradley.hilton I appreciate it.

9

@JSzaszvari Facebook messenger “persistent menus” sit at the bottom of the chat view and are not tied to individual messages. Like this:

image
image1080×2220 422 KB

Since they are not tied to individual messages, they need to either be associated with a “room” or with a “user”. I was proposing that they be properties associated with the chatbot user. One way to think of these menu items is that they map to commands that the bot supports. So, if I click “Features Menu” in the above screenshot, then I send the command “list features” to the bot. If its more clear, I can start a new thread on persistent menus since they are architecturally very different from rich messages.

10

Yes I need this soon. We have a developer who has expressed interest in doing the Android side work. I plan to put a bounty on it once the design is specified well enough.

We will test it out and see but this would be great.

About following the Slack spec, there are multiple different aspects to this design. If we are just referring to the json associated with each type of rich message, then ideally I think we’d operate the way that DialogFlow does, and that is to support multiple different types. DF supports Facebook Messenger, Slack, and other different custom message types.

Our plan is to modify the api.ai hubot coffee script to support Facebook Messenger style rich messages but we could potentially also add support for Slack style messages, especially if that means we could leverage some of the work that @JSzaszvari plans to do for his Slack-oriented project. The style (facebook or slack or other) would be indicated in the custom message and the api.ai coffee script would know how to interpret these different types.

The only thing we need to borrow from Slack is the notion of the context of the user action. In Slack this is handled by the “callback_id”. In Facebook, this is called a “payload” of the response. Either approach is fine with me though the Facebook approach seems more extensible since it is a dictionary so for instance the payload can also include the user’s location coordinates see Quick Replies - Messenger Platform - Documentation - Meta for Developers. To be clear, I don’t need location coordinates for my immediate project but I like the extensibility of the dictionary as opposed to just the callback_id used by Slack.

@bradley.hilton
I understand that our working on this now may not fit with the overall project timeline. At the same time I want the project team to be aware that we are working this if they would like to influence our design. We are more than happy to accommodate if it means our work might be merged into the upstream repos / core modules of Rocket.chat. Now’s your chance to direct our work in order to hopefully save others having to do this work later on. Thanks!

11

I’ve just posted a draft spec for the main issue to GH - linked at bottom.

I got some feedback from the team that these more immediate concerns can be better handled on the issues, because it simply gets more attention than the forum. I don’t mean to wipe over your contributions here though, so please re-iterate any important points there if you have time.

@bizzbyster - I think there’s two issues in discussion here and trying to capture the idea of persistent menus for bots might be a bit complex and possibly confuse the simpler task of just getting some parity with basic buttons (and possibly cards). My main concern is that this would require a considerable front-end overhaul, as opposed to just a floating button over the message stream. It would take a lot of effort to get the mobile teams in sync with any solution too.

So I think we should split that issue off into a new forum topic for the longer term roadmap. You can push it independently, with your own contributions or bounties etc, but let’s not expand the scope from what could be a fairly quick win for the basic rich message support.

https://github.com/RocketChat/Rocket.Chat/issues/6786#issuecomment-381461138

12

All,

As per the plan on this:

  1. Survey of rich message types across popular chat platforms so we get a sense of what is implemented and how those things overlap. Also we want to put a name next to each category of thing – quick replies versus attachments with buttons, etc. – with screenshots before we specify what we’d like to see added to Rocket.chat.
  2. Write a short spec of what we’d like to see implemented that refers to that survey document and post here for comments. The spec will also include the payload structure which will be in keeping with your “Support payload structure from Slack and Messenger with minimal migrations”.
  3. Integrate feedback from @slyk and others into the spec and then post a bounty on this issue for the work required to implement it in the Web client.
  4. Create seperate issues in Android and IOS repos pointing to this issue and post bounties for each of those implementations as well.
  5. Work with implementors to get this feature in across the three platforms.

@jaytat0 has completed this draft of the survey of rich message types: https://github.com/WideChat/Rocket.Chat.Android/wiki/A-Survey-of-Rich-Messaging-in-Chatbots. Please comment to inform the spec we’ll be writing as step #2 in the above.

13

@tim.kinnane @bradley.hilton @JSzaszvari @gabriel.engel @diego.dorgam

It would be great to support all of the rich message types in the survey document in Rocket.chat, including the Webview type which I think is especially powerful b/c it is so extensible.

Two things need to happen next: 1) graphic design of UI elements and 2) JSON schema format changes to https://rocket.chat/docs/developer-guides/realtime-api/the-message-object/ to support these rich messages. Is there a design person available to work on #1? Or should our team take the initiative and move #2 forward? I can also ask the summer interns if they are interested in #2. Thanks!

14

All,

We need feedback on this design proposal for updates to the Message object to support rich messaging: https://github.com/RocketChat/docs/pull/740.

We would like to start implementing soon so please weigh in.

15

I made some comments on a docs PR review. I’m not sure a PR to docs is the best first step though. It’s good to surface schema changes, but the PR will have to stay open for quite some time until changes are implemented in a core release.

I like the wiki approach you did with the survey though. Perhaps a list of proposed schema changes there instead? At least for the spec discussion, then I can setup a branch to start testing the schema in operation. We’d need a companion branch on the SDK to build and test bots that are capable of sending the payload attributes.

16

@tim.kinnane We definitely want to do this the recommended way but I’m struggling to understand how a wiki is different from this: https://github.com/WideChat/docs/tree/ear_richMessages/developer-guides/realtime-api/the-message-object.

Perhaps you are just saying we should close the PR? Can you explain a bit more how you want this done and why?

17

Right. Our current plan is to implement these in our fork of the Android client. We will use our Hubot api.ai coffee file with modifications to pass payload and type through to the Android client for testing. If the SDK in a companion branch is ready at any point we will switch to that for testing. Hope that makes sense to you.

18

I wasn’t suggesting anything that prescriptive. Just pointing out that the realtime API docs is probably the last place that a PR will be accepted, so it might go stale or become a challenge to maintain the PR with merge conflicts by the time it’s an actual reflection of the state of the code.

However, I don’t know if a PR to the actual schema in code is the best way to surface and communicate the proposed changes either. So maybe the wiki is a middle ground. There’s no right answers, we’re discovering the best approach as we go I think.

19

Another model to consider, fresh out of slack…

This isn’t so much limited to bots, actions could be added by 3rd party packages as options on all messages, or if specific conditions are met. But either way, they could potentially be added using the same payload schema, with a new attribute.

20

We’ve converted our original docs PR into a wiki on our fork here: https://github.com/WideChat/Rocket.Chat.Android/wiki/Proposed-rich-message-implementation

It describes the new payload field in the message object. We made some adjustments from the original proposal that we think should help clarify and simplify.

We also closed the original PR, but put a link in the wiki to our fork of the developer-guide for reference.