One Migration Strategy to Microservices

The concepts of microservices is nice, but if you have a complex existing system the path is neither obvious or easy. I have seen Principal Architects throw up their hands and persuade the business that we need to build a new replacement system and that the old system is impossible to work with. This path tends to lead into overruns and often complete failures – I have recently seen that happen at a firm: “Just one year needed to deliver…” and three years later it was killed off because it had not been delivered.  The typical reported in industry literature statistics of 80—90% failure are very believable.

 

Over decades, I have seen many failrues (usually on the side lines).  On the other hand, for a planned phrase migration I have seen repeated success. Often success or failure seem to be determined by the agile-ness of the management and technical leads coupled with the depth of analysis before the project start. Unfortunately deep analysis ends up with a waterfall like specification that result in locked-step development and no agile-ness around issues. Similarly, agile often result in superficial analysis (the time horizon for analysis is often just the end of the next sprint)  with many components failing to fit together properly over time!

 

This post is looking at a heritage system and seeing how it can be converted to a microservices framework in an evolutionary manner. No total rewrite, just a phrased migration ending with a system that is close to a classic pro-forma microservice system.

 

I made several runs at this problem, and what I describe below “feels good” – which to me usually mean a high probability of success with demonstrable steps at regular intervals.

 

Example System

I am going to borrow a university system template from my days working for Blackboard.  We have teachers, non-teaching staff, students, classes, building, security access cards, payment cards, etc.  At one point, components were in Delphi, C#, Java, C++ etc with the databases in SQL Server and Oracle. Not only is data shared, but permissions often need to be consistent and appropriate.

 

I have tried a few running starts of microservicing  such a design, and at present, my best suggestion is this:

• Do NOT extend the microservicing  down to the database – there is a more elegant way to proceed
• Look at the scope of the microservices API very carefully – this is a narrow path that can explode into infinite microservices or a resurrection of legacy patterns

Elegant Microservice Database Access

Do not touch the database design at the start. You are just compounding the migration path needlessly at the start. Instead, for each microservice create a new database login that is named for the microservice and has (typically) CRUD permissions to:

• A table
• A subset of columns in a table
• An updateable view
• A subset of columns in an updateable view

We term this the Crud-Columns. There is a temptation to incorporate multiple Crud-columns into one microservice – the problem is simple, what is the objective criteria to stop incorporating more Crud-Columns into this single microservice? If you go to one microservice for each Crud-Columns, then by counting the tables you have an estimate of the number of microservices that you will likely end up with…  oh… that may be a lot! At this point of time, you may really want to consider automatic code generation of many microservices – similar to what you see with Entity-Frameworks, except this would be called Microservices-Framework.

 

This microservice may also have Read only permissions to other tables.  This other tables read only access  may be transitory for the migration. Regardless of final resolution, these tables must be directly related to the CRUD columns, and used to determine CUD decisions. At some future time, these rest calls to these read only tables may be redirected elsewhere (for example using a Moved to directive to a reporting microservices).

 

Oh, I have introduced a new term “reporting microservices”.  This is a microservice with one or read Read Api’s – multiple calls may be exposed depending on filtering, sorting or user permissions.

 

Microservices are not domain level APIs but at sub-domains or even sub-sub-domains. You should not be making small steps, instead, put on your seven-league boots!

 

Tracking microservices

Consider creating a table where every database column is enumerated out and the microservice having CRUD over it is listed.

i.e.

• Server.Database.Table.Schema.Column –> CRUD – >Microservice Name

 

The ideal (but likely impractical goal) is to have just one Microservice per specified column. That is a microservices may have many CUD columns, but a column will have only one CUD microservice ( N columns :: 1 Microservice).

 

Similarly, a table with

• Server.Database.Table.Schema.Column –> R– >Microservice

can be used as a heat map to refactor as the migration occurs. We want to reduce hot spots (i.e. the number of Read microservices per column).

 

Building Microservices from Database Logins

Defining the actions that a microservice login can do cascades into a finite set of possible APIs. We are avoiding trying to define a microservice and then get the database access to support it. We are effectively changing the usual process upside down.

 

Instead of the typical path of asking the client what it needs for an API (to keep it’s life simple), we are insuring that there is a collection of APIs that satisfies its needs – although these may be complicated to call. What we need to return to the classical simplicity is intermediate APIs.

 

Intermediate APIs

Intermediate APIs are APIs are do not have explicit  database CUD rights. They are intended to be helper APIs that talk to the database microservices above and present a simpler API to clients. They will call the above APIs to change the database. They may also be caching APIs and database reporting APIs.

 

A Walk Thru

Using the university model cited above, the first naïve step could be to create a

• Teacher API
• Student API
• Class API

If you bring in column permissions you find that these can be decomposed further. The reason that there may be a single row in the database for each of the above comes from Relational Database Design Normalization theory.  Instead, we should try to decompose according to user permission sets. For example:

• Teacher API
  • Teacher MetaData API i.e. name,
  • Teacher Address Info API
  • Teacher Salary Info API
  • Teacher HR API
  • Teacher Card Access API
• Student API
  • Student MetaData API, i.e. name,
  • Student Address Info API
  • Student Tuition API
  • Student Awards API
  • Student Card Access API

Our wishful state is that if you are authorized for an API, there is no need to check for further permissions. As I said, wishful. If you apply this concept strictly then you will likely end up with an unmanageable number of APIs that would be counter productive. This would be the case for an enterprise class system. For less complex systems, like customer retail systems, the number of permissions sets may be greatly reduced.

 

With the Blackboard system (when I was working on it), we were enabling support for hundred of thousands permission sets that often contains hundred of permission each (i.e. each person had their own set, each set contains permissions to access building, Uris, copying machines, etc).

 

An Intermediate API may be ClassAssignmentViewer. In this API, information from Student Metadata API, Teacher Metadata API and other APIs. Alternatively, it may be directly read only from the database.

 

Next Step

Once you have the microservices defined, you can start looking at segmenting the data store to match the microservices. When you leave a classic relational database, you may need to deal with issues such as referential integrity and foreign keys between microservices. If you have the microservice and the database login permissions pre-defined, then these issues are a magnitude simpler.

Bottom Line

The above is a sketch of what I discovered about migration process by trying several different approaches and seeing ongoing headaches, or, massive and risky refactoring.

 

With the above, you can start with a small scope and implement it. The existing system keeps functioning and you have created a parallel access point to the data. As functioning sets are completed, you can cut over to some microservices while the rest is running on the classic big api approach.  You can eventually have the entire system up in parallel and then do a cut over to these microservices stubs. Over time, you may wish to decouple the data stores but that can be done later. You need to isolate the CUD first into microservice to be above to do that step.