Microservices–Do it right!

In my earlier post, A Financially Frugal Architectural Pattern for the Cloud,  I advocated the use of microservices. Microservices are similar to REST, a concept or pattern or architectural standard, unlike  SOAP which is standards based. The modern IT industry trend towards “good enough”,  “lip-service” and “we’ll fix it in the next release”.  A contemporary application may use relational database software (SQL Server, Oracle, MySql) and thus the developers (and their management) would assert that their is a relational database system. If I move a magnetic tape based system into tables (one table for each type of tape) using relational database software – would that make it a relational database system? My opinion is no – never!!!

 

Then what makes it one? The data has been fully normalized in the logical model. Often the database has never been reviewed for normalization  despite such information being ancient (see William Kent, A Simple Guide to Five Normal Forms in Relational Database Theory, 1982), older than many developers. The implementation may be de-normalized in the physical model (if you have just a ‘database model’ and not separate physical and logical, then you are likely heading to trouble – in time (usually after the original developers have left!). For NoSql database, there is a lot of ancient literature out there dealing with both hierarchical databases and network databases which should also be used with MongoDB and Neo4j – but likely not.

 

My academic training is in mathematics and thus axioms and deriving theorems from them though rigorous logic.  The normalization of databases is immediately attractive to me. Knowing the literature (especially Christopher J.Date’s early writings from the 1970’s) is essential since “"Those who do not learn history are doomed to repeat it.”

 

Microservices Normalization Rules

Below are my attempt to define equivalent rules for microservices. They will likely be revised over time. They are very mathematical in definition by intent. Martin Fowler’s article is also a good read. Much of the discussion on the web is at a high level (the hand waving level), such as Microservices Architecture and Design Principles,  Microservices Design Principles, with some echoing some of the issues cited below Adopting Microservices at Netflix: Lessons for Architectural Design

 

A public REST API consumed over the internet is probably not a microservice. It may use many microservices and other composite APIs.

 

• Composite API: A service that consumes other composite APIs and/or microservices but do not qualify below
• Independent Microservice: A service that does not call any other microservices
• Dependent Microservice: A service that calls Independent Microservices in parallel

An Independent Microservice

An independent microservice is the exclusive owner of a data store.

• No other service or system may access the data store.
• A microservice may change the software used to create the datastore with no consequences on any other system.

• A microservice does not make calls to other services
  • An corollary of this is that microservices rarely use any libraries that are not generic across the industry
    • Exception: libraries of static functions that are explicit to a firm, for example, encryption of  keys (i.e. Identity Integers –> strings)
• A microservice may contain publishers
  • The datastore that it controls may need to be pushed to reporting and other systems
• A microservice may create log records that are directly consumed by other systems.
  • Non-blocking outputs from a microservice are fine
• A microservice may make periodic calls.
  • A microservice may pull things off a queue, or push things to a queue
    • The nature of this data is transient. The queue services interaction must match the model of an API call and response. The call comes from one queue and written to another queue.
      • Ideally there will be no references to queues inside the microservice.
      • A call to the microservice would start reading a queue at a regular interval (the queue is in the call parameters)
      • The data on the queue would specify where the results should be sent
  • A microservice should not pull data from (non-microservice) data store.
    • Exception: a configurationless implementation such as described for queues above is fine.
• A microservice configuration should never reference anything outside of it’s own world.
  • Configuration Injection is allowed. Microservice is deployed and loaded, then a call is done to supply it’s configuration.

A Dependent Microservice

• Has exclusive ownership of it’s data store just like an independent microservice.
• Calls dependent microservice to obtain data only (no update, delete or create)
• Create Update Delete calls must always go to the independent microservice that owns it, no relaying should occur.
• Calls are in parallel, never sequential
• Note: Some types of queries may be inefficient, those should be directed at a reporting microservice (which independent microservices may publish to) or a composite service.

 

Control of Microservices

The best model that I have seen is one that some groups at Amazon used.

• All calls to the microservice must have an identifier (cookie?) that identifies the caller.
  • The microservice determines if it is an authorized caller based on the identifier and possibly the IP address
• The consumer of the microservice must be authorized by the owner of the microservice based on: a contract containing at least
  • Daily load estimates
  • Peak load estimates
  • Availability
• The microservice may disable any consumer that exceeds the contracted load.
• The consumer should be given a number from 1-100 indicating business importance.
  • If the microservice is stressed, then those services with lower values will be disabled.
• There should always be SLA in place

Microservices should be executed on isolated VMs behind  a load distributor. The datastore should be also on a dedicated set of VMs, for example a set of VMs supporting a Casandra implementation.

 

More suggestions?

To quote Martin Fowler, “If the components do not compose cleanly, then all you are doing is shifting complexity from inside a component to the connections between components. Not just does this just move complexity around, it moves it to a place that's less explicit and harder to control.” I have seen this happen – with the appearance of microservices (because there are tons of REST APis on different servers) but behind this layer, there are shared DLL’s accessing multiple databases causing endless pains with keeping DLL current as features are added or bugs fixed. A bug in a single library may require the fix to be propagated to a dozen REST APIs’. If it must be propagated it is not a microservice.

 

My goal with this post is to define a set of objective check items that can be clearly determined by inspection. The ideal implementation would have all of them passing.

 

One of the side-effects is that the rules can often be inconvenient for quick designs. A rethinking of what you are trying to do often results – similar to what I have seen happen when you push for full normalization in a logical model. 

 

Do you have further suggestions?