The Great Open Source Debate with Elastic, Gluu and GitHub 🎤 Oct 26

Moving away from Meteor and beyond

This idea is old, but we really need to go to this path.

We already started this movement with the merging of the Globals Removal and the eminent merge of the Depackaging Pull Request, but it isn’t enough for our goal of having the server side running as a micro-service architecture.

Globals Removal
It was the first step, remove everything that was shared among packages and the main code and convert them to imports, only then we could start having the benefits of the import, start structuring our code better and start thinking about use TypeScript.

Depackaging
This consist in move all files out from the packages folder and let them start importing themselves instead of import the entire package what makes hard to prevent cyclical dependencies, it allow the code editors to handle the autocomplete what is impossible using the Meteor’s package way of import, and it makes the code close to what is a pure NodeJS application and easier to achieve the Demeteorize goal.

Models restrutoring
The code of everything is the database, and the core of our server code is our Model system, everything based on Meteor’s collection allowing us to execute finds in a synchronous way (using Fibers behind) and having the benefits of observers (when needed).
This needs to be our first step to move away from meteor, start using the MongoDB driver directly and return promises instead of use Fibers, that will allow us to start changing the rest of the code to stop using Meteor.
If we change other parts of the application first, the database finds will fail since it expects to run in a Fiber, Meteor provide it for us transparently when we receive external calls via DDP, but if we try to call the models out of a Fiber it will rise an exception, that’s why the models needs to be the first step.

TypeScript
Since the Models needs to go first, we may start this change using TypeScript, it would allow us to define better the structure of our data and have a well documentation for the types and methods, helping us on the development and the community when extending us.
If we decide to go this way we should think this in a way we could share with other projects like our JS SDK or the Apps Engine.

Micro Services
The main idea here is to start using a library that allow us to register the services and call other services, we already have some work done in this way, allowing us to run multiple services in the same process or different ones, it doesn’t matter to the framework, it will allow us to scale parts of the system like the notification part.
We can start this with the models as well, since it may run in a separated process or even inside the big one, but already start changing the code to call the micro service API, allowing us to split it in the future without any new change in the way we request and save data.
We may need to think about a way to abstract the micro service inside an SDK allowing us to take advantage of the autocomplete and lint (check) from TypeScript.

6 Likes

:clap: Thanks for going through the effort to put this together!

This sounds awesome! Part of what makes our code base so hard some times is all of the magic meteor does when it bundles. Dependencies kind of randomly compiled and bundled vs others installed at deploy time.

I imagine because of the need to bundle packages going away our builds should also get easier? Less to actually build and more of just executing the app?

Are we going to loose anything significant by moving away from fibers? Beyond being easier to understand… are promises going to improve things or will we loose some performance potentially?

So they would be separate packages that can be consumed by importing, or by standing up I’m guessing with a light wrapper to run as microservice? What would they use to communicate between each other?

Thanks again for sharing! :pray:

2 Likes

:heart: this! Great job and thanks for sharing @rodrigo.nascimento! (I may provide more thoughts later, I just wanted to say thanks!)

3 Likes

Not really, the build is taking the same time, the most of it is doing the minification of all JS files which remains almost the same, but may change if we change the build framework.

I don’t have this answer, but I bet we will gain performance since we remove the fibers overload and start using a native solution (promises).

We have being testing the moleculer framework for microservices, but I think we should have a NPM module in TypeScript that we import and call the functions directly and it will translate to the microservice framework, that is just an idea to keep the typing from typescript and be able to check the parameters and results of methods.

3 Likes

@rodrigo.nascimento Hi,

Do you have any updates regarding your idea of running the Rocket.Chat as a set of microservices instead of the one still application?

Hi @markb-trustifi,

We are still working on it, it will take a long way to land since we still need to rewrite a lot of code to be able to run some parts outside meteor.

I hope this semester we will have something to show.

1 Like

Hey @rodrigo.nascimento amazing description of the situation and your plans of moving away from meteor, is by any means the code you are writting open source or you will make it once you have something more stable that you feel confident about?
Just one last question, did you move away from meteor accounts? if so, into what?
Thanks for your input

Hi @tiagocorreiaitcenter, yes, everything will be open source, this not mean it will be free, we are merging our enterprise solution into our community repo inside a folder under a proprietary license.

We didn’t move away from the accounts system yet, but we will do, I’m not sure how yet.

Thanks

@rodrigo.nascimento Makes complete sense, I was just really interested in how you handled it so it’s mostly a matter of curiosity over anything else
If I may ask, and if its not a bit of repetition, what was the main reason to move away from meteor? and to what are you moving now? rest? graphql?
Once again thanks for your input, and good work :slight_smile:

@tiagocorreiaitcenter no problem. We are not using subscriptions anymore, moved some methods from DDP methods to REST endpoints, trying to be away from fibers and use promises as much as possible and stop using the Meteor Collections to use Raw Collections for performance reasons. So we are already stop using Meteor gradationally. The other reason is we are splitting things in microservices and since we don’t want to use the technologies mentioned before we are using pure nodejs for them, so the natural path will be move away from Meteor.

Right now we use only REST for the calls to the server on our mobile apps and DDP only for Streams (our solution on top of subscriptions to receive notifications from the server), but we are still investigating other technologies to replace the DDP, may be MQTT or something else with the objective to reduce the payload size as well.

Since it’s just in the beginning we are doing a lot of tests and trying to define the best approach, we will make things public as soon as we have some initial definitions.

Right now our main priority is to implement our system of chained messages in order to ensure consistency and improve data sync, so we will go back to the microservices development soon.

3 Likes

@tiagocorreiaitcenter hi.
First of all, thank you guys for all your hard work.
Something tells me that this topic deserves to be updated already :innocent:
We’re excited to see how things are going in moving away from Meteor.

Also, does that mean that you will run :rocket:-chat solely as microservices. Or it will have the option to run as a monolith?

2 Likes