Developing graceful microservices by CorpJS

Author: Peter Hauszknecht | March 22nd, 2017

Introduction

A process that runs in a production environment has to start and stop gracefully, regardless of being monolothic service or a microservice. The reason of having such graceful operation is the following: usually, there are dependencies beetween the runtime resources of the process so to have an error-free start, these need to start in a particular order, waiting for each other. The same is true for stopping.

Graceful start is trivial in a way that we cannot force our app to start if things do not happen this way. Graceful stop however is missed by many, saying for example that “it will stop eventually”. If you have a monolithic app, this is not that much of a problem, while in a microservices architecture suddenly terminating a process can have nasty side-effects.

Having graceful operation is not a new thing. Its main concept is to separate the runtime resources as the components of the app (config, logger, server, db client, message queue client, etc.), implementing well-defined asynchronous start and stop methods. We start the components in their dependency order, so the whole app has an async start and stop method.

In this post, we’ll demonstrate graceful implementation by using the corpjs-system module, also introducing a few weaker, additional requirements and practicies which are connected to this topic.

System basics

corpjs-system is a graceful component-management module written in TypeScript. Its components must have the following structure:

{
  async start(dependencies?, restart?, stop?) { ... },
  async stop() { ... }
}

{

async start(dependencies?, restart?, stop?) { ... },

async stop() { ... }

}

start and stop methods must return with Promise. Definition of the former is mandatory, while the latter is not. The resolved return value of the start method (which is a Promise) will be resource delegated by the component.

It’s recommended to define the components as classes or factory function, to enable multiinstance usage, such asusing them as closure, but we can also give other parameters to component instance through the factory arguments.

Finally, we unite the components into a System:

const system = new System()
  .add('config', Config())
  .add('logger', Logger()).dependsOn('config')
  .add('mongodb', MongoDb()).dependsOn('config', 'logger')

system.start().then(resources => console.log(resources))

...

system.stop().then(() => console.log("Good bye!"))

const system = new System()

.add('config', Config())

.add('logger', Logger()).dependsOn('config')

.add('mongodb', MongoDb()).dependsOn('config', 'logger')

system.start().then(resources => console.log(resources))

...

system.stop().then(() => console.log("Good bye!"))

When we start the system, that will start the components in the dependency order, giving them their dependencies:

function MongoDb(useUri) {
  let db
  return {
    async start({config, logger}) {
      const connectionConfig = useUri ? config.mongodb.connUri : config.mongodb.connConfig
      db = await MongoClient.connect(connectionConfig)
      logger.log("Connection details:", connectionConfig)
      return db
    },
    async stop() {
      if (db) await db.close()
    }
  }
}

function MongoDb(useUri) {

let db

return {

async start({config, logger}) {

const connectionConfig = useUri ? config.mongodb.connUri : config.mongodb.connConfig

db = await MongoClient.connect(connectionConfig)

logger.log("Connection details:", connectionConfig)

return db

async stop() {

if (db) await db.close()

}

Dependencies

When we define the dependencies, we have the chance to overwrite their name and/or content for the dependent component:

const system = new System()
  .add('config', Config())
  .add('logger', Logger()).dependsOn('config')
  .add('mongodb', MongoDb()).dependsOn({component: 'config', source: 'mongodb', as: 'mongodbConfig'}, 'logger')

const system = new System()

.add('config', Config())

.add('logger', Logger()).dependsOn('config')

.add('mongodb', MongoDb()).dependsOn({component: 'config', source: 'mongodb', as: 'mongodbConfig'}, 'logger')

The MongoDB component only gets the MongoDB part from the config, under the name mongodbConfig.

Lifecycle methods

The start function of the component will receive two additional lifecycle methods as arguments, besides the resources of its dependencies: systemRestart, systemStop. Components will have the chance to invoke the full restart and stop of the whole system with these functions.

function Config(path) {
  let watcher
  return {
    async start(_, systemRestart, systemStop) {
      watcher = watch(path, () => systemRestart())
      watcher.on('error', systemStop)
      return readJson(path)
    },
    async stop() {
      if (watcher) watcher.close()
    }
  }
}

function Config(path) {

let watcher

return {

async start(_, systemRestart, systemStop) {

watcher = watch(path, () => systemRestart())

watcher.on('error', systemStop)

return readJson(path)

async stop() {

if (watcher) watcher.close()

}

In the example above, the component will enforce the system to restart upon the change of the config file. After such a systemRestart, the whole system is stopped and restarted gracefully, without exiting the process.

By using the systemStop function received as an argument, we can force the system to stop and exit. If we call the systemStop with an error, then we can propagate our exception to the system, which will make the exit of the process with error code 1.

We can disable the exiting in case of an exception with the:

new System({ exitOnError: false })

1 2	new System({ exitOnError: false })

setting, which can be beneficial when we’re running tests.

Error handling

When we constructed the underlying principles of corpjs-system, we deemed the usage in production environments more imortant, so error handling was created by following this formula.

1) When the starting flow of the system encounters and error (meaning, any of the components throws a start exception), the system stops all started components gracefully and exits the process.
2) When any of the components calls systemStop with an exception, we do the same.
3) When a stop method of a component in the stopping flow throws an exception, we skip the component and continue the graceful stop, finally exiting the process.
4) When stopping reaches its timeout, it exits the process. We can set this via the terminationTimeout setting on the System instance.
5) In case of uncaughtException or unhandledRejection, the system is stopped gracefully and the process exits.

The stop the exit of the process in case of an error, we must set exitOnError: false.

Exit of a process is preferred by corpjs-system because its fate is handled by the infrastructure, and not the process itself. This is the preferred design in a microservices architecture, since stopping of a given process can have impact on other parts of the architecture as well. This way, the infrastructure can decide whether the service must be restarted without further consideration, or perhaps it should do some investigation regarding the cause of the error – for example if a database, which is a strong dependency, is not available.

Ignorable components

A system has important (mandatory) and less important (ignorable) components. Ignorable components will have their errors ingored in the first two cases above. We can define an ignorable component the following way:

const system = new System()
  .add('config', Config())
  .add('logger', Logger()).ignorable().dependsOn('config')
  .add('mongodb', MongoDb()).dependsOn('config', 'logger')

const system = new System()

.add('config', Config())

.add('logger', Logger()).ignorable().dependsOn('config')

.add('mongodb', MongoDb()).dependsOn('config', 'logger')

In this example, if there’s an error when the logger component is started, we skip it. If it asks for stop because of an exception, we ignore it.

Signal handling

system watches SIGINT and SIGTERM. It is stopped in a graceful way in the cases of such interruption or termination. Most of the process managers terminate processes by using such signals, so their handling is mandatory.

Grouping

So you might ask whether it makes sense to organize these systems in a deeper hierarchy or not. For example, when creating a closed test system, organizing system into smaller parts can be useful, but we can also have a code organization where a single module only publishes a half component:

// adminApiSystem.js
const adminApi = new System()
  .add('adminRouter', AdminApiRouter())
  .add('auth', AdminAuth()).dependsOn('adminRouter')
  .add('users', Users()).dependsOn('adminRouter')
  .add('orders', Orders()).dependsOn('adminRouter')
  .group()

// publicApiSystem.js
const publicApi = new System()
  .add('publicRouter', PublicApiRouter())
  .add('auth', Auth()).dependsOn('publicRouter')
  .add('cart', Cart()).dependsOn('publicRouter')
  .add('products', Products()).dependsOn('publicRouter')
  .group()

// system.js
const system = new System()
  .add('root', ExpressApp())
  .add('adminRouter', adminApi).dependsOn('root')
  .add('publicRouter', publicApi).dependsOn('root')
  .add('server', Server()).dependsOn('root', 'adminRouter', 'publicRouter')

// adminApiSystem.js

const adminApi = new System()

.add('adminRouter', AdminApiRouter())

.add('auth', AdminAuth()).dependsOn('adminRouter')

.add('users', Users()).dependsOn('adminRouter')

.add('orders', Orders()).dependsOn('adminRouter')

.group()

// publicApiSystem.js

const publicApi = new System()

.add('publicRouter', PublicApiRouter())

.add('auth', Auth()).dependsOn('publicRouter')

.add('cart', Cart()).dependsOn('publicRouter')

.add('products', Products()).dependsOn('publicRouter')

.group()

// system.js

const system = new System()

.add('root', ExpressApp())

.add('adminRouter', adminApi).dependsOn('root')

.add('publicRouter', publicApi).dependsOn('root')

.add('server', Server()).dependsOn('root', 'adminRouter', 'publicRouter')

Emitted events

system is also an EventEmitter, emitting a couple of useful events. As we prefer to know about every event, the logAllEvents() method watches and logs every event to the standard output:

const system = new System()
  ...
  .logAllEvents()

const system = new System()

...

.logAllEvents()

Upcoming features

corpjs-system is a prototype, perfecting and revamping the feature sets of electrician and systemic.

Enhancements currently missing but already in the backlog are:

Removal and override of components: these can also be useful when you are developing test systems, helping us to mock existing systems.
Option to set start timeout with further configuration options.
Usage of the logAllEvents() function with other loggers, not only stdout.

The `corpjs` component set

We’re publishing a basic component set for corpjs-system, which:
– has all System compatible,
– are created using the principles described in this post.

This is an incomplete list of components, it will grow in the future:

corpjs-config: A config reader and watcher module, based on confabulous ,which fallbacks the structures of the configs in the given order. It restarts the system when the config is changed.
corpjs-endpoints: Handles a special case of config: endpoints. When creating a microservices architecture, it is paramount for the differents parts of the architecture to know the network endpoints of their dependencies. If the infrastrcture can serve this information into a file (for which molinio is great!), then this module is a perfect choice to watch this file and testart the system when it changes. Extending the endpoints to other protocols is in the backlog.
corpjs-logger: This module wraps winston as a System compatible component. The component delegates a Logger instance.
corpjs-express: express App– and Server-components.
corpjs-mongodb: delegating MongoDB instance.
corpjs-amqp: RabbitMQ connection- and channel-components.

The CorpJS microservice concepts

The main purpose of the CorpJS product family is to publish standards which cover the Corporate Microservices requirements, by using the design patterns described above. To achieve this, we started to create yeoman generators which will ease the pain for developers to create sekeletons, plus these boilerplates will serve as guidelines to create microservices.

The Yeoman generators

Rest Service: Simple express rest service. Developers only need to implement the router components.- Message Controlled Worker Service: Workers service which can be read from RabbitMQ. Developers only need to implement consumers. The boilerplate is prepared to handle dynamic configs and endpoints.
React Application Host Service: A simple React app packed with Webpack, using redux and redux-thunk. The boilerplate constructs the whole folder structure, data- and workflow configuration. Developers only need to implement reducers, actionCreators and components. The service hosting the app is created based on the rest-service mentioned before.
MongoDB: Docker-compose file using mongodb image.
RabbitMQ: Docker-compose file using rabbitmq:3-management.

Upcoming generators

OData Rest Services: OData based rest services based on odata-v4-server and odata-v4- database connectors.

Every generator uses the generator-corpjs-ts base generator, which creates a simple TypeScript-based boilerplate.

We will soon make some vanilla JS-based generators available as well.

Molinio