Moving away from Meteor and beyond

#1

This idea is old, but we really need to go to this path.

We already started this movement with the merging of the Globals Removal and the eminent merge of the Depackaging Pull Request, but it isn’t enough for our goal of having the server side running as a micro-service architecture.

Globals Removal
It was the first step, remove everything that was shared among packages and the main code and convert them to imports, only then we could start having the benefits of the import, start structuring our code better and start thinking about use TypeScript.

Depackaging
This consist in move all files out from the packages folder and let them start importing themselves instead of import the entire package what makes hard to prevent cyclical dependencies, it allow the code editors to handle the autocomplete what is impossible using the Meteor’s package way of import, and it makes the code close to what is a pure NodeJS application and easier to achieve the Demeteorize goal.

Models restrutoring
The code of everything is the database, and the core of our server code is our Model system, everything based on Meteor’s collection allowing us to execute finds in a synchronous way (using Fibers behind) and having the benefits of observers (when needed).
This needs to be our first step to move away from meteor, start using the MongoDB driver directly and return promises instead of use Fibers, that will allow us to start changing the rest of the code to stop using Meteor.
If we change other parts of the application first, the database finds will fail since it expects to run in a Fiber, Meteor provide it for us transparently when we receive external calls via DDP, but if we try to call the models out of a Fiber it will rise an exception, that’s why the models needs to be the first step.

TypeScript
Since the Models needs to go first, we may start this change using TypeScript, it would allow us to define better the structure of our data and have a well documentation for the types and methods, helping us on the development and the community when extending us.
If we decide to go this way we should think this in a way we could share with other projects like our JS SDK or the Apps Engine.

Micro Services
The main idea here is to start using a library that allow us to register the services and call other services, we already have some work done in this way, allowing us to run multiple services in the same process or different ones, it doesn’t matter to the framework, it will allow us to scale parts of the system like the notification part.
We can start this with the models as well, since it may run in a separated process or even inside the big one, but already start changing the code to call the micro service API, allowing us to split it in the future without any new change in the way we request and save data.
We may need to think about a way to abstract the micro service inside an SDK allowing us to take advantage of the autocomplete and lint (check) from TypeScript.

4 Likes

#2

:clap: Thanks for going through the effort to put this together!

This sounds awesome! Part of what makes our code base so hard some times is all of the magic meteor does when it bundles. Dependencies kind of randomly compiled and bundled vs others installed at deploy time.

I imagine because of the need to bundle packages going away our builds should also get easier? Less to actually build and more of just executing the app?

Are we going to loose anything significant by moving away from fibers? Beyond being easier to understand… are promises going to improve things or will we loose some performance potentially?

So they would be separate packages that can be consumed by importing, or by standing up I’m guessing with a light wrapper to run as microservice? What would they use to communicate between each other?

Thanks again for sharing! :pray:

1 Like

#3

:heart: this! Great job and thanks for sharing @rodrigo.nascimento! (I may provide more thoughts later, I just wanted to say thanks!)

3 Likes

#4

Not really, the build is taking the same time, the most of it is doing the minification of all JS files which remains almost the same, but may change if we change the build framework.

I don’t have this answer, but I bet we will gain performance since we remove the fibers overload and start using a native solution (promises).

We have being testing the moleculer framework for microservices, but I think we should have a NPM module in TypeScript that we import and call the functions directly and it will translate to the microservice framework, that is just an idea to keep the typing from typescript and be able to check the parameters and results of methods.

2 Likes