Developing graceful microservices by CorpJS
Introduction
A process that runs in a production environment has to start and stop gracefully, regardless of being monolothic service or a microservice. The reason of having such graceful operation is the following: usually, there are dependencies beetween the runtime resources of the process so to have an error-free start, these need to start in a particular order, waiting for each other. The same is true for stopping.
Graceful start is trivial in a way that we cannot force our app to start if things do not happen this way. Graceful stop however is missed by many, saying for example that “it will stop eventually”. If you have a monolithic app, this is not that much of a problem, while in a microservices architecture suddenly terminating a process can have nasty side-effects.
Having graceful operation is not a new thing. Its main concept is to separate the runtime resources as the components of the app (config, logger, server, db client, message queue client, etc.), implementing well-defined asynchronous start
and stop
methods. We start the components in their dependency order, so the whole app has an async start
and stop
method.
In this post, we’ll demonstrate graceful implementation by using the corpjs-system module, also introducing a few weaker, additional requirements and practicies which are connected to this topic.
System basics
corpjs-system is a graceful component-management module written in TypeScript. Its components must have the following structure:
1 2 3 4 5 |
{ async start(dependencies?, restart?, stop?) { ... }, async stop() { ... } } |
start
and stop
methods must return with Promise
. Definition of the former is mandatory, while the latter is not. The resolved return value of the start
method (which is a Promise
) will be resource delegated by the component.
It’s recommended to define the components as classes or factory function, to enable multiinstance usage, such asusing them as closure, but we can also give other parameters to component instance through the factory arguments.
Finally, we unite the components into a System
:
1 2 3 4 5 6 7 8 9 10 11 |
const system = new System() .add('config', Config()) .add('logger', Logger()).dependsOn('config') .add('mongodb', MongoDb()).dependsOn('config', 'logger') system.start().then(resources => console.log(resources)) ... system.stop().then(() => console.log("Good bye!")) |
When we start the system
, that will start the components in the dependency order, giving them their dependencies:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
function MongoDb(useUri) { let db return { async start({config, logger}) { const connectionConfig = useUri ? config.mongodb.connUri : config.mongodb.connConfig db = await MongoClient.connect(connectionConfig) logger.log("Connection details:", connectionConfig) return db }, async stop() { if (db) await db.close() } } } |
Dependencies
When we define the dependencies, we have the chance to overwrite their name and/or content for the dependent component:
1 2 3 4 5 |
const system = new System() .add('config', Config()) .add('logger', Logger()).dependsOn('config') .add('mongodb', MongoDb()).dependsOn({component: 'config', source: 'mongodb', as: 'mongodbConfig'}, 'logger') |
The MongoDB
component only gets the MongoDB
part from the config
, under the name mongodbConfig
.
Lifecycle methods
The start
function of the component will receive two additional lifecycle methods as arguments, besides the resources of its dependencies: systemRestart
, systemStop
. Components will have the chance to invoke the full restart and stop of the whole system
with these functions.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
function Config(path) { let watcher return { async start(_, systemRestart, systemStop) { watcher = watch(path, () => systemRestart()) watcher.on('error', systemStop) return readJson(path) }, async stop() { if (watcher) watcher.close() } } } |
In the example above, the component will enforce the system to restart upon the change of the config file. After such a systemRestart
, the whole system
is stopped and restarted gracefully, without exiting the process.
By using the systemStop
function received as an argument, we can force the system to stop and exit. If we call the systemStop
with an error, then we can propagate our exception to the system, which will make the exit of the process with error code 1.
We can disable the exiting in case of an exception with the:
1 2 |
new System({ exitOnError: false }) |
setting, which can be beneficial when we’re running tests.
Error handling
When we constructed the underlying principles of corpjs-system, we deemed the usage in production environments more imortant, so error handling was created by following this formula.
1) When the starting flow of the system
encounters and error (meaning, any of the components throws a start
exception), the system stops all started components gracefully and exits the process.
2) When any of the components calls systemStop
with an exception, we do the same.
3) When a stop
method of a component in the stopping flow throws an exception, we skip the component and continue the graceful stop, finally exiting the process.
4) When stopping reaches its timeout, it exits the process. We can set this via the terminationTimeout
setting on the System
instance.
5) In case of uncaughtException
or unhandledRejection
, the system is stopped gracefully and the process exits.
The stop the exit of the process in case of an error, we must set exitOnError: false
.
Exit of a process is preferred by corpjs-system because its fate is handled by the infrastructure, and not the process itself. This is the preferred design in a microservices architecture, since stopping of a given process can have impact on other parts of the architecture as well. This way, the infrastructure can decide whether the service must be restarted without further consideration, or perhaps it should do some investigation regarding the cause of the error – for example if a database, which is a strong dependency, is not available.
Ignorable components
A system has important (mandatory) and less important (ignorable) components. Ignorable components will have their errors ingored in the first two cases above. We can define an ignorable component the following way:
1 2 3 4 5 |
const system = new System() .add('config', Config()) .add('logger', Logger()).ignorable().dependsOn('config') .add('mongodb', MongoDb()).dependsOn('config', 'logger') |
In this example, if there’s an error when the logger
component is started, we skip it. If it asks for stop because of an exception, we ignore it.
Signal handling
system
watches SIGINT
and SIGTERM
. It is stopped in a graceful way in the cases of such interruption or termination. Most of the process managers terminate processes by using such signals, so their handling is mandatory.
Grouping
So you might ask whether it makes sense to organize these systems in a deeper hierarchy or not. For example, when creating a closed test system, organizing system
into smaller parts can be useful, but we can also have a code organization where a single module only publishes a half component:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
// adminApiSystem.js const adminApi = new System() .add('adminRouter', AdminApiRouter()) .add('auth', AdminAuth()).dependsOn('adminRouter') .add('users', Users()).dependsOn('adminRouter') .add('orders', Orders()).dependsOn('adminRouter') .group() // publicApiSystem.js const publicApi = new System() .add('publicRouter', PublicApiRouter()) .add('auth', Auth()).dependsOn('publicRouter') .add('cart', Cart()).dependsOn('publicRouter') .add('products', Products()).dependsOn('publicRouter') .group() // system.js const system = new System() .add('root', ExpressApp()) .add('adminRouter', adminApi).dependsOn('root') .add('publicRouter', publicApi).dependsOn('root') .add('server', Server()).dependsOn('root', 'adminRouter', 'publicRouter') |
Emitted events
system
is also an EventEmitter, emitting a couple of useful events. As we prefer to know about every event, the logAllEvents()
method watches and logs every event to the standard output:
1 2 3 4 |
const system = new System() ... .logAllEvents() |
Upcoming features
corpjs-system is a prototype, perfecting and revamping the feature sets of electrician and systemic.
Enhancements currently missing but already in the backlog are:
- Removal and override of components: these can also be useful when you are developing test systems, helping us to mock existing systems.
- Option to set start timeout with further configuration options.
- Usage of the
logAllEvents()
function with other loggers, not only stdout.
The corpjs
component set
We’re publishing a basic component set for corpjs-system, which:
– has all System
compatible,
– are created using the principles described in this post.
This is an incomplete list of components, it will grow in the future:
- corpjs-config: A config reader and watcher module, based on confabulous ,which fallbacks the structures of the configs in the given order. It restarts the system when the config is changed.
- corpjs-endpoints: Handles a special case of config: endpoints. When creating a microservices architecture, it is paramount for the differents parts of the architecture to know the network endpoints of their dependencies. If the infrastrcture can serve this information into a file (for which molinio is great!), then this module is a perfect choice to watch this file and testart the system when it changes. Extending the endpoints to other protocols is in the backlog.
- corpjs-logger: This module wraps winston as a
System
compatible component. The component delegates aLogger
instance. - corpjs-express: express
App
– andServer
-components. - corpjs-mongodb: delegating MongoDB instance.
- corpjs-amqp: RabbitMQ connection- and channel-components.
The CorpJS microservice concepts
The main purpose of the CorpJS product family is to publish standards which cover the Corporate Microservices requirements, by using the design patterns described above. To achieve this, we started to create yeoman generators which will ease the pain for developers to create sekeletons, plus these boilerplates will serve as guidelines to create microservices.
The Yeoman generators
- Rest Service: Simple express rest service. Developers only need to implement the router components.- Message Controlled Worker Service: Workers service which can be read from RabbitMQ. Developers only need to implement consumers. The boilerplate is prepared to handle dynamic configs and endpoints.
- React Application Host Service: A simple React app packed with Webpack, using
redux
andredux-thunk
. The boilerplate constructs the whole folder structure, data- and workflow configuration. Developers only need to implement reducers, actionCreators and components. The service hosting the app is created based on the rest-service mentioned before. - MongoDB: Docker-compose file using
mongodb
image. - RabbitMQ: Docker-compose file using
rabbitmq:3-management
.
Upcoming generators
- OData Rest Services: OData based rest services based on odata-v4-server and
odata-v4-
database connectors.
Every generator uses the generator-corpjs-ts base generator, which creates a simple TypeScript-based boilerplate.
We will soon make some vanilla JS-based generators available as well.