Monday, June 6, 2016

One Migration Strategy to Microservices

The concepts of microservices is nice, but if you have a complex existing system the path is neither obvious or easy. I have seen Principal Architects throw up their hands and persuade the business that we need to build a new replacement system and that the old system is impossible to work with. This path tends to lead into overruns and often complete failures – I have recently seen that happen at a firm: “Just one year needed to deliver…” and three years later it was killed off because it had not been delivered.  The typical reported in industry literature statistics of 80—90% failure are very believable.

 

Over decades, I have seen many failrues (usually on the side lines).  On the other hand, for a planned phrase migration I have seen repeated success. Often success or failure seem to be determined by the agile-ness of the management and technical leads coupled with the depth of analysis before the project start. Unfortunately deep analysis ends up with a waterfall like specification that result in locked-step development and no agile-ness around issues. Similarly, agile often result in superficial analysis (the time horizon for analysis is often just the end of the next sprint)  with many components failing to fit together properly over time!

 

This post is looking at a heritage system and seeing how it can be converted to a microservices framework in an evolutionary manner. No total rewrite, just a phrased migration ending with a system that is close to a classic pro-forma microservice system.

 

I made several runs at this problem, and what I describe below “feels good” – which to me usually mean a high probability of success with demonstrable steps at regular intervals.

 

Example System

I am going to borrow a university system template from my days working for Blackboard.  We have teachers, non-teaching staff, students, classes, building, security access cards, payment cards, etc.  At one point, components were in Delphi, C#, Java, C++ etc with the databases in SQL Server and Oracle. Not only is data shared, but permissions often need to be consistent and appropriate.

 

I have tried a few running starts of microservicing  such a design, and at present, my best suggestion is this:

  • Do NOT extend the microservicing  down to the database – there is a more elegant way to proceed
  • Look at the scope of the microservices API very carefully – this is a narrow path that can explode into infinite microservices or a resurrection of legacy patterns

Elegant Microservice Database Access

Do not touch the database design at the start. You are just compounding the migration path needlessly at the start. Instead, for each microservice create a new database login that is named for the microservice and has (typically) CRUD permissions to:

  • A table
  • A subset of columns in a table
  • An updateable view
  • A subset of columns in an updateable view

We term this the Crud-Columns. There is a temptation to incorporate multiple Crud-columns into one microservice – the problem is simple, what is the objective criteria to stop incorporating more Crud-Columns into this single microservice? If you go to one microservice for each Crud-Columns, then by counting the tables you have an estimate of the number of microservices that you will likely end up with…  oh… that may be a lot! At this point of time, you may really want to consider automatic code generation of many microservices – similar to what you see with Entity-Frameworks, except this would be called Microservices-Framework.

 

This microservice may also have Read only permissions to other tables.  This other tables read only access  may be transitory for the migration. Regardless of final resolution, these tables must be directly related to the CRUD columns, and used to determine CUD decisions. At some future time, these rest calls to these read only tables may be redirected elsewhere (for example using a Moved to directive to a reporting microservices).

 

Oh, I have introduced a new term “reporting microservices”.  This is a microservice with one or read Read Api’s – multiple calls may be exposed depending on filtering, sorting or user permissions.

 

Microservices are not domain level APIs but at sub-domains or even sub-sub-domains. You should not be making small steps, instead, put on your seven-league boots!

American Trucking Industry 1952 Ad - Seven League Boots…

 

Tracking microservices

Consider creating a table where every database column is enumerated out and the microservice having CRUD over it is listed.

i.e.

  • Server.Database.Table.Schema.Column –> CRUD – >Microservice Name

 

The ideal (but likely impractical goal) is to have just one Microservice per specified column. That is a microservices may have many CUD columns, but a column will have only one CUD microservice ( N columns :: 1 Microservice).

 

Similarly, a table with

  • Server.Database.Table.Schema.Column –> R– >Microservice

can be used as a heat map to refactor as the migration occurs. We want to reduce hot spots (i.e. the number of Read microservices per column).

 

Building Microservices from Database Logins

Defining the actions that a microservice login can do cascades into a finite set of possible APIs. We are avoiding trying to define a microservice and then get the database access to support it. We are effectively changing the usual process upside down.

 

Instead of the typical path of asking the client what it needs for an API (to keep it’s life simple), we are insuring that there is a collection of APIs that satisfies its needs – although these may be complicated to call. What we need to return to the classical simplicity is intermediate APIs.

 

Intermediate APIs

Intermediate APIs are APIs are do not have explicit  database CUD rights. They are intended to be helper APIs that talk to the database microservices above and present a simpler API to clients. They will call the above APIs to change the database. They may also be caching APIs and database reporting APIs.

 

A Walk Thru

Using the university model cited above, the first naïve step could be to create a

  • Teacher API
  • Student API
  • Class API

If you bring in column permissions you find that these can be decomposed further. The reason that there may be a single row in the database for each of the above comes from Relational Database Design Normalization theory.  Instead, we should try to decompose according to user permission sets. For example:

  • Teacher API
    • Teacher MetaData API i.e. name,
    • Teacher Address Info API
    • Teacher Salary Info API
    • Teacher HR API
    • Teacher Card Access API
  • Student API
    • Student MetaData API, i.e. name,
    • Student Address Info API
    • Student Tuition API
    • Student Awards API
    • Student Card Access API

Our wishful state is that if you are authorized for an API, there is no need to check for further permissions. As I said, wishful. If you apply this concept strictly then you will likely end up with an unmanageable number of APIs that would be counter productive. This would be the case for an enterprise class system. For less complex systems, like customer retail systems, the number of permissions sets may be greatly reduced.

 

With the Blackboard system (when I was working on it), we were enabling support for hundred of thousands permission sets that often contains hundred of permission each (i.e. each person had their own set, each set contains permissions to access building, Uris, copying machines, etc).

 

An Intermediate API may be ClassAssignmentViewer. In this API, information from Student Metadata API, Teacher Metadata API and other APIs. Alternatively, it may be directly read only from the database.

 

Next Step

Once you have the microservices defined, you can start looking at segmenting the data store to match the microservices. When you leave a classic relational database, you may need to deal with issues such as referential integrity and foreign keys between microservices. If you have the microservice and the database login permissions pre-defined, then these issues are a magnitude simpler.

Bottom Line

The above is a sketch of what I discovered about migration process by trying several different approaches and seeing ongoing headaches, or, massive and risky refactoring.

 

With the above, you can start with a small scope and implement it. The existing system keeps functioning and you have created a parallel access point to the data. As functioning sets are completed, you can cut over to some microservices while the rest is running on the classic big api approach.  You can eventually have the entire system up in parallel and then do a cut over to these microservices stubs. Over time, you may wish to decouple the data stores but that can be done later. You need to isolate the CUD first into microservice to be above to do that step.

Saturday, May 28, 2016

Theory about Test Environments

Often my career has faced dealing with an arbitrary environment to test in. This environment preceded my arrival, and often was still there at my departure with many developers became fatalistic towards this arbitrary environment.  This is not good.

 

The Rhetorical Goal Recomposed

“We use our test environment to verify that our code changes will work as expected”

While this assures upper management, it lacks specifics to evaluate if the test environment is appropriate or complete. A more objective measurement would be:

  • The code changes perform as specified at the six-sigma level of certainty.

This then logically cascades into sub-measurements:

  • A1: The code changes perform as specified at the highest projected peak load for the next N year (typically 1-2) at the six-sigma level of certainty.
  • A2: The code changes perform as specified on a fresh created (perfect) environment  at the six-sigma level of certainty.
  • A3: The code changes perform as specified on a copy of production environment with random data at the six-sigma level of certainty.

The last one is actually the most critical because too often there is bad data from bad prior released code (which may have be rolled back – but the corrupted data remained!) . There is a corollary:

  • C1: The code changes do not need to perform as specified when the environment have had its data corrupted by arbitrary code and data changes that have not made it to production. In other words, ignore a corrupted test environment

 

Once thru is not enough!

Today’s systems are often multi-layers with timeouts, blockage under load and other things making the outcome not a certainty but a random event. Above, I cited six sigma – this is a classic level sought in quality assurance of mechanical processes.

 

“A six sigma process is one in which 99.99966% of all opportunities to produce some feature of a part are statistically expected to be free of defects (3.4 defective features per million opportunities).”

 

To translate this into a single test context – the test must run 1,000,000 times and fail less than4 times. Alternatively, 250,000 times with no failures.

 

Load testing to reach six-sigma

Load testing will often result in 250,000 calls being made. In some cases, it may mean that the load test may need to run for 24 hours instead of 1 hour. There are some common problem with many load tests:

  • The load test does not run on a full copy of the production environment – violates A3:
  • The same data is used time and again for the tests – thus A3: the use of random data fails.
    • If you have a system that has been running for 5 years, then the data should be selected based on user created data with 1/5 from each year
    • If the system has had N releases, then the data should be selected on user created data with 1/n from each release period

Proposal for a Conforming Pattern

Preliminary development (PD) is done on a virgin system each day. By virgin I mean that databases and other data stores are created from scripts and populated with perfect data. There may be super user data but no common user data.  This should be done by an automated process. I have seen this done in some firms and it has some real benefits:

  • Integration tests must create (instead of borrow) users
    • Integration tests are done immediately after build – the environment is confirmed before any developers arrive at work.
    • Images of this environment could be saved to allow faster restores.
  • Performance is good because the data store is small
  • A test environment is much smaller and can be easily (and cheaply) created on one or more cloud services or even VMs
  • Residue from bad code do not persist (often reducing triage time greatly) – when a developer realized they have accidentally jacked the data then they just blow away the environment and recreate it

After the virgin system is built, the developer’s “release folder scripts” are executed – for example, adding new tables, altering stored procedures, adding new data to system tables. Then the integration tests are executed again. Some tests may fail. A simple solution that I have seen is for these tests to call into the data store to get the version number and add an extension to NUnit that indicate that this test applies to before of after this version number. Tests can then be excluded that are expected to fail (and also identified for a new version to be written).

 

Integration development(ID) applies to the situation where there may be multiple teams working on stuff that will go out in a single release. Often it is more efficient to keep the teams in complete isolation for preliminary development – if there are complexities and side-effects than only one team suffers. A new environment is created then each teams’ “release folder scripts” are executed and tests are executed.

i.e. PD+PD+….+PD = ID

This keeps the number of moving code fragments controlled.

 

Scope of Testing in PD and ID

A2 level is as far as we can do in this environment. We cannot do A1 or A3.

 

SmokeTest development (STD) means that an image of the production data base is made available to the integration team and they can test the code changes using real data. Ideally, they should regress with users  created during each release period so artifact issues can be identified. This may be significant testing, but is not load testing because we do not push up to peak volumes.

Tests either creates a new user (in the case of PD and ID) or searches for a random user that was created in release cycle 456 in the case of STD. Of course, code like SELECT TOP 1 *… should not be used, rather all users retrieved and one randomly selected.

 

This gets us close to A3: if we do enough iterations.

 

Designing Unit Tests for multiple Test Environment

Designing a UserFactory with a signature such as

UserFactory.GetUser(UserAttributes[] requiredAttributes)

can simplify the development of unit tests that can be used across multiple environments. This UserFactory reads a configuration file which may have  properties such as

  • CreateNewUser=”true”
  • PickExistingUser=”ByCreateDate”
  • PickExistingUser=”ByReleaseDate”
  • PickExistingUser=”ByCreateDateMostInactive”

In the first case, a user is created with the desired attributes.  In other cases, the attributes are used to filter the production data to get a list of candidates to randomly pick from.

 

In stressing scenarios when we want to test for side-effects due to concurrent operation by the same user, then we could use the current second to select the same user for all tests starting in the current second.

 

Developers Hiding Significant Errors – Unintentional

At one firm, we successfully established the following guidance:

  • Fatal: When the unexpected happen – for example, the error that was thrown was not mapped to a known error response (i.e. Unexpected Server Error should not be returned)
  • Error: When an error happens that should not happen, i.e. try catch worked to recover the situation…. but…
  • Warning: When the error was caused by customer input. The input must be recorded into the log (less passwords). This typically indicates a defect in UI, training or child applications
  • Info: everything else, i.e. counts
  • Debug: what ever

We also implemented the ability to change the log4net settings on the fly – so we could, in production, get every message for a short period of time (massive logs)

Load Stress with Concurrency

Correct load testing is very challenging and requires significant design and statistics to do and validate the results.

 

One of the simplest implementation is to have a week old copy of the database, capture all of the web request traffic in the last week and do a play back in a reduced time period. With new functionality extending existing APIs then we are reasonably good – except we need to make sure that we reach six-sigma level – i.e.  was there at least 250,000 calls???  This can be further complicated if the existing system has a 0.1% error rate. A 0.1% error rate means 250 errors are expected on average, unfortunately this means that detecting a 1 error in 250,000 calls difference is impossible from a single run (or even a dozen runs). Often the first stage is to drive error rates down to near zero on the existing code base. I have personally (over several months) a 50K/day exception logging rate to less than 10. It can be done – just a lot of systematic slow work (and fighting to get these not business significant bug fixes into production). IMHO, they are business significant: they reduce triage time, false leads, bug reports, and thus customer experience with the application.

 

One of the issues is whether the 250,000 calls applies to the system as a whole – or just the method being added or modified? For true six-sigma, it needs to be the method modified – sorry! And if there are 250,000 different users (or other objects) to be tested, then random selection of test data is required.

 

I advocate the use of PNUnit (Parallel Nunit) on multiple machines with a slight twist. In the above UserFactory.Get() described above, we randomly select the user, but  for stress testing, we could use the seconds (long) and modular it with the number of candidate users and then execute the tests. This approach intentionally creates a situation where concurrent activity will generated, potentially creating blocks, deadlocks and inconsistencies.

 

There is a nasty problem with using integration tests mirroring the production distribution of calls. Marking tests appropriately may help, the test runner can them select the tests to simulate the actual production call distribution and rates. Of course, this means that there is data on the call rates and error rates from the production system.

 

Make sure that you are giving statistically correct reports!

 

The easy question to answer is “Does the new code make the error rate statistically worst?” Taking our example above of 0.1% error we had 250 errors being expected. If we want to have 95% confidence then we would need to see 325 errors to deem it to be worst. You must stop and think about this, because of the our stated goal was less than 1 error in 250,000 – and we ignore 75 more errors as not being significant!!! This is a very weak criteria. It also makes clear that driving down the back ground error rate is essential. You cannot get strong results with a high background error rate, you may only be able to demonstrate 1 sigma defect rate.

 

In short, you can rarely have a better sigma rate than your current rate unless you fix the current code base to have a lower sigma rate.

Thursday, May 12, 2016

The sad state of evidence based development management patterns

I have been in the development game for many decades. I did my first programs using APL/360 and Fortran (WatFiv) at the University of Waterloo, and have seen and coded a lot of languages over the years (FORTH, COBOL, Asm, Pascal, B,C, C++, SAS, etc).

 

My academic training was in Operations Research – that is mathematical optimization of business processes. Today, I look at the development processes that I see and it is dominantly “fly by the seats of the pants”, “everybody is doing it” or “academic correctness”. I am not talking about waterfall or agile or scrum. I am not talking about architecture etc. Yet is some ways I am. Some processes assert Evidence Based Management, yet fails to deliver the evidence of better results. Some bloggers detail the problems with EBM.  A few books attempt to summarize the little research that has occurred, such as "Making Software: What Really Works and Why we Believe It"

 

As an Operation Research person, I would define the optimization problem facing a development manager or director or lead as follows:

  • Performance (which often comes at increased man hours to develop and operational costs)
  • Scalability (which often comes at increased man hours to develop and operational costs)
  • Cost to deliver
  • Accuracy of deliverable (Customer satisfaction)
  • Completeness of deliverable
  • Elapsed time to delivery (shorter time often exponentially increase cost to deliver and defect rates)
  • Ongoing operational costs (a bad design may result in huge cloud computing costs)
  • Time for a new developer to become efficient across the entire product
  • Defect rate
    • Number of defects
    • ETA from reporting to fix
  • Developer resources
    • For development
    • For maintenance

All of these factors interact. For evidence, there are no studies and I do not expect them to be. Technology is changing too fast, there is huge differences between projects, and any study will be outdated before it is usable. There is some evidence that we can work from.

Lines of Code across a system

Lines of code directly impacts several of the above.

  • Defect rate is a function of the number of lines of code ranging from 200/100K to 1000/100K lines [source] which is scaled by developer skill level. Junior or new developers will have a higher defect rate.
  • Some classic measures defined in the literature, for example, cyclomatic complexity. Studies find a positive correlation between cyclomatic complexity and defects: functions and methods that have the highest complexity tend to also contain the most defects.
  • Time to deliver is often a function of the lines of code written.

There is a mistaken belief that lines of code is an immutable for a project. In the early 2000’s I lead a rewrite of a middle tier and backend tier (with the web front end being left as is), the original C++/SQL server code base was 474,000 lines of code and was the result of 25 man years of coding. With a team of 6 new (to the application) developers sent over from India and 2 intense local developer, we recreated these tiers with 100% api compliance in just 25,000 lines of code in about 8 weeks. 25 man years –> 1 man year. a 20 fold decrease in code base. And the last factor was an increase in concurrent load by 20 fold. 

 

On other projects I have seen massive copy and paste (with some minor change) that result in code bloat. When a bug is discovered it was often only fixed in some of the pastes. Martin Fowler describes Lines of Code as a measure of developer productivity as useless; the same applies to lines of code in a project.  A change of programming language can result in a 10 fold drop (or increase) in lines of code. A change of a developer can also result in a similar change – depending on skill sets.

 

Implementation Design

The use of Object-Relational Mapping (ORM) can often result in increased lines of code, defects, steeper learning curves and greater challenges addressing performance issues. A simple illustration is to move all addresses in Washington State from a master table to a child table. In SQL Server, TSQL – it is a one line statement, calling this from SQL it amounts to 4 lines of C# code. Using an ORM, this can quickly grow to 100-200 lines. ORMs came along because of a shortage of SQL developer skills. As with most things, it carry hidden costs that are omitted in the sales literature!

 

“Correct academic design” does not mean effective (i.e. low cost) development. One of the worst systems (for performance and maintenance) that I have seen was absolutely beautifully designed with a massive array of well defined classes – which unfortunately ignored the database reality.  Many calls of a single method cascaded through these classes and resulted in 12 – 60 individual sql queries being executed against the database.  Most of the methods could be converted to a wrapper on a single stored procedure with a major improvement of performance. The object hierarchy was flattened (or downsized!).

 

I extend the concept of cyclomatic complexity to the maximum stack depth in developer written code.  The greater the depth, the longer it takes to debug (because the developer has to walk through the stack) and likely to write. The learning curve goes up. I suggest a maximum depth of 7 (less than cyclomatic complexity), ideally 5. This number comes out of research for short term memory (wikipedia). Going beyond seven significantly increases the effort that a developer needs to make to understand the stack. On the one hand, having a deep hierarchy of objects looks nice academically – but it is counterproductive for efficient coding. Seven is a magic number to keep asking “Why do we have more than seven ….”

Developer Skill Sets

Many architects suffer from the delusion that all developers are as skilled as they are, i.e. IQs over 145.  During my high school teaching years, I was assigned both gifted classes and challenged classes – and learn to present appropriately to both. In some cities (for example Stockholm, Sweden) – 20% of the work force is in IT. This means that the IQ of the developers likely range from 100 upwards. When an application is released, the support developers likely will end up with an average IQ around 100. The question must be asked, how simple is the code to understand for future enhancements and maintenance?

 

If a firm has a policy of significant use of off-shore or contractor resources, there are  further challenges:

  • A high percentage of the paid time is in ramp-up mode
  • There is a high level of non- conformity to existing standards and practices.
    • Higher defect rate, greater time for existing staff to come up to speed on the code
  • Size of team and ratio of application-experienced versus new developer can greatly alter delivery scheduled (see Brook’s law

Pseudo coding different architecture rarely happens. It has some advantages – if you code up the most complex logic and then ask the question – “ A bug happens and nothing comes back, what are the steps to isolated the issue with certainty?” The architecture with the least diagnostic steps may be the more efficient one.

 

Last, the availability now and in the future of developers with the appropriate skills.  The industry is full of technology that was hot and promised the moon and then were disrupted by a new technology (think of Borland Delphi and Pascal!). I often do a weighted value composed of years since launch, popularity at the moment and trend to refine choices (and in some cases to say No to a developer or architect that want to play with the latest and greatest!). Some sites are DB-Engine Ranking and PYPL.  After short listing, then it’s a matter of coding up some complex examples in each and counting lines of code needed.

Specification Completeness And Stability

On one side, I have worked with a few PMs that deliver wonderful specifications (200-500 pages) that had no change-orders between the first line of code being written and final delivery a year later. What was originally handed to developers was not changed. Work was done in sprints. The behavior and content of every web page was detailed. There was a clean and well-reviewed dictionary of terms and meanings. Needless to say, delivery was prompt, on schedule, etc.

 

On the other side, I have had minor change-requests which mutated constantly. The number of lines of code written over all of these changes were 20x the number of lines of code finally delivered.

Concurrent Development

Concurrent development means that two or more set of changes were happening to the same code base. At one firm we had several git-hub forks: Master,Develop, Sprint, Epic and Saga. The title indicate when the changes were expected to be propagated to master. It worked reasonably, but often I ended up spending two days resolving conflicts and debugging bugs that were introduced whenever I attempted to get forks in sync. Concurrent development increases overhead exponentially according to the number of independent forks are active. Almost everything in development has exponential cost with size, there is no economy of scale in development.

 

On the flip side, at Amazon using the microservices model, there were no interaction between feature requests. Each API was self contained and would evolve independently. If an API needed another API changed, then the independent API would be changed, tested and released. The dependent API then was developed against the released independent API. There was no code-juggling act. Each code base API was single development and self-contained. Dependencies were by API not libraries and code bases.

 

Bottom Line

Controlling costs and improving delivery depends greatly on the preparation work IMHO -- namely:

  • Specification stability and completeness
  • Architectural / Design being well crafted for the developer population
  • Minimum noise (i.e. no concurrent development, change orders, change of priorities)
  • Methodology (Scrum, Agile, Waterfall, Plan Driven) is of low significance IMHO – except for those selling it and ‘true believers’.

On the flip side, often the business will demand delivery schedules that add technical debt and significantly increase ongoing costs.

 

A common problem that I have seen is solving this multiple dimension problem by looking at just one (and rarely two) dimensions and discovering the consequences of that decision down stream.  I will continue to add additional dimensions as I recall them from past experience.

Tuesday, May 10, 2016

Mining PubMed via Neo4J Graph Database–Getting the data

I have a blog dealing with various complex autoimmune diseases and spend a lot of time walking links at PubMed.com. Often readers send me an article that I missed. 

 

I thought that a series of post on how to do it will help other people (including MDs, grad students and citizen scientists) better research medical issues.

 

Getting the data from Pub Med

I implemented a simple logic to obtain a collection of relevant articles:

  • Query for 10,000 articles on a subject or key word

  • Retrieve each of these articles and any articles they referenced (i.e. the knowledge graph).
  • Keep repeating until you have enough articles or you run out of them!!

Getting the bootstrapping list of articles

A console application that reads the command line arguments and retrieves the list. For example,

downloader.exe Crohn’s Disease

which produces this URI

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=Pubmed&retmax=1000&usehistory=y&term=Crohn's+disease

This results in an XML file being sent

 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE eSearchResult PUBLIC "-//NLM//DTD esearch 20060628//EN" "
http://eutils.ncbi.nlm.nih.gov/eutils/dtd/20060628/esearch.dtd">
<eSearchResult><Count>44880</Count><RetMax>10000</RetMax><RetStart>0</RetStart><QueryKey>1</QueryKey><WebEnv>NCID_1_84230330_130.14.22.215_9001_1462926138_46088356_0MetA0_S_MegaStore_F_1</WebEnv><IdList>
<Id>27159423</Id>
<Id>27158773</Id>
<Id>27158547</Id>
<Id>27158537</Id>
<Id>27158536</Id>
<Id>27158345</Id>
<Id>27158125</Id>
<Id>27157449</Id>
<Id>27156530</Id>
<Id>27154890</Id>
<Id>27154001</Id>
<Id>27153721</Id>
<Id>27152873</Id>
<Id>27152872</Id>
<Id>27152547</Id>

So let us look at the code

class Program
    {
        static Downloader downloader = new Downloader();
        static void Main(string[] args)
        {
            if (args.Length > 0)
            {
                var search = new StringBuilder();
                foreach (var arg in args)
                {
                    search.AppendFormat("{0} ", arg);
                }
                downloader.TermSearch(search.ToString());
                downloader.ProcessAll();
            }
            downloader.Save();
        }
      }

The Downloader class tracks articles already downloaded and those to do next. It simply starts downloading and saving each article summary to an Xml file using the unique article Id as the file name. I wanted to keep the summaries on my disk to speed reprocessing if my Neo4J model changes.


using System;
using System.Collections.Generic;      
using System.Collections.Concurrent;
using System.Net;                 

using System.Linq;
using System.Threading.Tasks; 
using System.Xml;                    

using System.Text;    
using System.Configuration;
using System.IO;
namespace PubMed
{
    public class Downloader
    {
        // Entrez E-utilities at the US National Center for Biotechnology Information:
        static readonly String server = "
http://www.ncbi.nlm.nih.gov/entrez/eutils/";
        string dataFolder = "C:\\PubMed";
        string logFile;
        public System.Collections.Concurrent.ConcurrentBag<string> index = new ConcurrentBag<string>();
        public System.Collections.Concurrent.ConcurrentQueue<string> todo = new ConcurrentQueue<string>();
        public Downloader()
        {
            logFile = Path.Combine(dataFolder, "article.log");
            if (File.Exists(logFile))
            {
                var lines = File.ReadAllLines(logFile);
                foreach (var line in lines)
                {
                    if (!string.IsNullOrWhiteSpace(line))
                        index.Add(line);
                }
            }
        }
        public void Save()
        {
            File.WriteAllLines(logFile, index.ToArray());
        }

         public void ProcessAll()
        {

            var nextId = string.Empty;
            while (todo.Count > 0)
            {
                if (todo.Count > 12)
                {
                    var tasks = new List<Task>();
                    int t = 0;
                    for (t = 0; t < 10; t++)
                    {
                        if (todo.TryDequeue(out nextId))
                        {

                            tasks.Add(Task.Factory.StartNew(() => NcbiPubmedArticle(nextId)));
                        }
                    }
                    Task.WaitAll(tasks.ToArray());
                    Save();
                }
                else
                {
                    if (todo.TryDequeue(out nextId))
                    {

                        NcbiPubmedArticle(nextId);
                    }
                }
            }
        }

        public void TermSearch(String term)
        {
            var search = string.Format("
http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=Pubmed&retmax=1000&usehistory=y&term={0}", term.Replace(" ", "+"));
            new WebClient().DownloadFile(new Uri(search), "temp.log");
            var xml = new XmlDocument();
            xml.Load("temp.Log");
            foreach (XmlNode node in xml.DocumentElement.SelectNodes("//Id"))
            {
                var id = node.InnerText;
                if (!index.Contains(id) && !todo.Contains(id))
                {
                    todo.Enqueue(id);
                }
            }
        }

        public void NcbiPubmedArticle(String term)
        {

            if (!index.Contains(term))
            {
                try
                {
                    var fileLocation = Path.Combine(dataFolder, string.Format("{0}.xml", term));
                    if (File.Exists(fileLocation)) return;
                    var search = string.Format("
http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id={0}&retmode=xml", term);
                    new WebClient().DownloadFile(new Uri(search), fileLocation);
                    index.Add(term);
                    GetChildren(fileLocation);
                    Console.WriteLine(term);
                }
                catch
                {

                }
            }
        }
        private void GetChildren(string fileName)
        {
            try
            {
                var dom = new XmlDocument();
                dom.Load(fileName);
                foreach (XmlNode node in dom.DocumentElement.SelectNodes("//PMID"))
                {
                    var id = node.InnerText;
                    if (!index.Contains(id) && !todo.Contains(id))
                    {
                        todo.Enqueue(id);
                    }
                }
            }
            catch (Exception exc)
            {
                Console.WriteLine(exc.Message);
            }
        }
    }
}

Next Importing into Neo4J

An example of the structured data to load is shown below. Try defining your own model while you wait for the next post. 

 

<?xml version="1.0"?>
<!DOCTYPE PubmedArticleSet PUBLIC "-//NLM//DTD PubMedArticle, 1st January 2016//EN" "
http://www.ncbi.nlm.nih.gov/corehtml/query/DTD/pubmed_160101.dtd">
<PubmedArticleSet>
<PubmedArticle>
    <MedlineCitation Owner="NLM" Status="MEDLINE">
        <PMID Version="1">10022306</PMID>
        <DateCreated>
            <Year>1999</Year>
            <Month>02</Month>
            <Day>25</Day>
        </DateCreated>
        <DateCompleted>
            <Year>1999</Year>
            <Month>02</Month>
            <Day>25</Day>
        </DateCompleted>
        <DateRevised>
            <Year>2006</Year>
            <Month>11</Month>
            <Day>15</Day>
        </DateRevised>
        <Article PubModel="Print">
            <Journal>
                <ISSN IssnType="Print">0378-4274</ISSN>
                <JournalIssue CitedMedium="Print">
                    <Volume>102-103</Volume>
                    <PubDate>
                        <Year>1998</Year>
                        <Month>Dec</Month>
                        <Day>28</Day>
                    </PubDate>
                </JournalIssue>
                <Title>Toxicology letters</Title>
                <ISOAbbreviation>Toxicol. Lett.</ISOAbbreviation>
            </Journal>
            <ArticleTitle>Epidemiological association in US veterans between Gulf War illness and exposures to anticholinesterases.</ArticleTitle>
            <Pagination>
                <MedlinePgn>523-6</MedlinePgn>
            </Pagination>
            <Abstract>
                <AbstractText>To investigate complaints of Gulf War veterans, epidemiologic, case-control and animal modeling studies were performed. Looking for OPIDP variants, our epidemiologic project studied 249 Naval Reserve construction battalion (CB24) men. Extensive surveys were drawn for symptoms and exposures. An existing test (PAI) was used for neuropsychologic. Using FACTOR, LOGISTIC and FREQ in 6.07 SAS, symptom clusters were sought with high eigenvalues from orthogonally rotated two-stage factor analysis. After factor loadings and Kaiser measure for sampling adequacy (0.82), three major and three minor symptom clusters were identified. Internally consistent by Cronbach's coefficient, these were labeled syndromes: (1) impaired cognition; (2) confusion-ataxia; (3) arthro-myo-neuropathy; (4) phobia-apraxia; (5) fever-adenopathy; and (6) weakness-incontinence. Syndrome variants identified 63 patients (63/249, 25%) with 91 syndromes. With pyridostigmine bromide as the drug in these drug-chemical exposures, syndrome chemicals were: (1) pesticide-containing flea and tick collars (P &lt; 0.001); (2) alarms from chemical weapons attacks (P &lt; 0.001), being in a sector later found to have nerve agent exposure (P &lt; 0.04); and (3) insect repellent (DEET) (P &lt; 0.001). From CB24, 23 cases, 10 deployed and 10 non-deployed controls were studied. Auditory evoked potentials showed dysfunction (P &lt; 0.02), nystagmic velocity on rotation testing, asymmetry on saccadic velocity (P &lt; 0.04), somatosensory evoked potentials both sides (right P &lt; 0.03, left P &lt; 0.005) and synstagmic velocity after caloric stimulation bilaterally (P-range, 0.02-0.04). Brain dysfunction was shown on the Halstead Impairment Index (P &lt; 0.01), General Neuropsychological Deficit Scale (P &lt; 0.03) and Trail Making part B (P &lt; 0.03). Butylcholinesterase phenotypes did not trend for inherent abnormalities. Parallel hen studies at Duke University established similar drug-chemical delayed neurotoxicity. These investigations lend credibility that sublethal exposures to drug-chemical combinations caused delayed-onset neurotoxic variants.</AbstractText>
            </Abstract>
            <AuthorList CompleteYN="Y">
                <Author ValidYN="Y">
                    <LastName>Kurt</LastName>
                    <ForeName>T L</ForeName>
                    <Initials>TL</Initials>
                    <AffiliationInfo>
                        <Affiliation>Department of Internal Medicine, University of Texas Southwestern Medical School, Dallas 75235, USA.</Affiliation>
                    </AffiliationInfo>
                </Author>
            </AuthorList>
            <Language>eng</Language>
            <PublicationTypeList>
                <PublicationType UI="D016428">Journal Article</PublicationType>
                <PublicationType UI="D013485">Research Support, Non-U.S. Gov't</PublicationType>
            </PublicationTypeList>
        </Article>
        <MedlineJournalInfo>
            <Country>NETHERLANDS</Country>
            <MedlineTA>Toxicol Lett</MedlineTA>
            <NlmUniqueID>7709027</NlmUniqueID>
            <ISSNLinking>0378-4274</ISSNLinking>
        </MedlineJournalInfo>
        <ChemicalList>
            <Chemical>
                <RegistryNumber>0</RegistryNumber>
                <NameOfSubstance UI="D002800">Cholinesterase Inhibitors</NameOfSubstance>
            </Chemical>
        </ChemicalList>
        <CitationSubset>IM</CitationSubset>
        <MeshHeadingList>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D016022">Case-Control Studies</DescriptorName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D002800">Cholinesterase Inhibitors</DescriptorName>
                <QualifierName MajorTopicYN="Y" UI="Q000633">toxicity</QualifierName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D006801">Humans</DescriptorName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D008297">Male</DescriptorName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D018923">Persian Gulf Syndrome</DescriptorName>
                <QualifierName MajorTopicYN="Y" UI="Q000209">etiology</QualifierName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="Y" UI="D014728">Veterans</DescriptorName>
            </MeshHeading>
        </MeshHeadingList>
    </MedlineCitation>
    <PubmedData>
        <History>
            <PubMedPubDate PubStatus="pubmed">
                <Year>1999</Year>
                <Month>2</Month>
                <Day>18</Day>
            </PubMedPubDate>
            <PubMedPubDate PubStatus="medline">
                <Year>1999</Year>
                <Month>2</Month>
                <Day>18</Day>
                <Hour>0</Hour>
                <Minute>1</Minute>
            </PubMedPubDate>
            <PubMedPubDate PubStatus="entrez">
                <Year>1999</Year>
                <Month>2</Month>
                <Day>18</Day>
                <Hour>0</Hour>
                <Minute>0</Minute>
            </PubMedPubDate>
        </History>
        <PublicationStatus>ppublish</PublicationStatus>
        <ArticleIdList>
            <ArticleId IdType="pubmed">10022306</ArticleId>
        </ArticleIdList>
    </PubmedData>
</PubmedArticle>

</PubmedArticleSet>

Saturday, May 7, 2016

Microservices–Do it right!

In my earlier post, A Financially Frugal Architectural Pattern for the Cloud,  I advocated the use of microservices. Microservices are similar to REST, a concept or pattern or architectural standard, unlike  SOAP which is standards based. The modern IT industry trend towards “good enough”,  “lip-service” and “we’ll fix it in the next release”.  A contemporary application may use relational database software (SQL Server, Oracle, MySql) and thus the developers (and their management) would assert that their is a relational database system. If I move a magnetic tape based system into tables (one table for each type of tape) using relational database software – would that make it a relational database system? My opinion is no – never!!!

 

Then what makes it one? The data has been fully normalized in the logical model. Often the database has never been reviewed for normalization  despite such information being ancient (see William Kent, A Simple Guide to Five Normal Forms in Relational Database Theory, 1982), older than many developers. The implementation may be de-normalized in the physical model (if you have just a ‘database model’ and not separate physical and logical, then you are likely heading to trouble – in time (usually after the original developers have left!). For NoSql database, there is a lot of ancient literature out there dealing with both hierarchical databases and network databases which should also be used with MongoDB and Neo4j – but likely not.

 

My academic training is in mathematics and thus axioms and deriving theorems from them though rigorous logic.  The normalization of databases is immediately attractive to me. Knowing the literature (especially Christopher J.Date’s early writings from the 1970’s) is essential since “"Those who do not learn history are doomed to repeat it.”

 

Microservices Normalization Rules

Below are my attempt to define equivalent rules for microservices. They will likely be revised over time. They are very mathematical in definition by intent. Martin Fowler’s article is also a good read. Much of the discussion on the web is at a high level (the hand waving level), such as Microservices Architecture and Design PrinciplesMicroservices Design Principles, with some echoing some of the issues cited below Adopting Microservices at Netflix: Lessons for Architectural Design

 

A public REST API consumed over the internet is probably not a microservice. It may use many microservices and other composite APIs.

 

  • Composite API: A service that consumes other composite APIs and/or microservices but do not qualify below
  • Independent Microservice: A service that does not call any other microservices
  • Dependent Microservice: A service that calls Independent Microservices in parallel

An Independent Microservice

An independent microservice is the exclusive owner of a data store.

  • No other service or system may access the data store.
  • A microservice may change the software used to create the datastore with no consequences on any other system.
  • A microservice does not make calls to other services
    • An corollary of this is that microservices rarely use any libraries that are not generic across the industry
      • Exception: libraries of static functions that are explicit to a firm, for example, encryption of  keys (i.e. Identity Integers –> strings)
  • A microservice may contain publishers
    • The datastore that it controls may need to be pushed to reporting and other systems
  • A microservice may create log records that are directly consumed by other systems.
    • Non-blocking outputs from a microservice are fine
  • A microservice may make periodic calls.
    • A microservice may pull things off a queue, or push things to a queue
      • The nature of this data is transient. The queue services interaction must match the model of an API call and response. The call comes from one queue and written to another queue.
        • Ideally there will be no references to queues inside the microservice.
        • A call to the microservice would start reading a queue at a regular interval (the queue is in the call parameters)
        • The data on the queue would specify where the results should be sent
    • A microservice should not pull data from (non-microservice) data store.
      • Exception: a configurationless implementation such as described for queues above is fine.
  • A microservice configuration should never reference anything outside of it’s own world.
    • Configuration Injection is allowed. Microservice is deployed and loaded, then a call is done to supply it’s configuration.

A Dependent Microservice

  • Has exclusive ownership of it’s data store just like an independent microservice.
  • Calls dependent microservice to obtain data only (no update, delete or create)
    • Create Update Delete calls must always go to the independent microservice that owns it, no relaying should occur.
    • Calls are in parallel, never sequential
  • Note: Some types of queries may be inefficient, those should be directed at a reporting microservice (which independent microservices may publish to) or a composite service.

 

Control of Microservices

The best model that I have seen is one that some groups at Amazon used.

  • All calls to the microservice must have an identifier (cookie?) that identifies the caller.
    • The microservice determines if it is an authorized caller based on the identifier and possibly the IP address
  • The consumer of the microservice must be authorized by the owner of the microservice based on: a contract containing at least
    • Daily load estimates
    • Peak load estimates
    • Availability
  • The microservice may disable any consumer that exceeds the contracted load.
  • The consumer should be given a number from 1-100 indicating business importance.
    • If the microservice is stressed, then those services with lower values will be disabled.
  • There should always be SLA in place

Microservices should be executed on isolated VMs behind  a load distributor. The datastore should be also on a dedicated set of VMs, for example a set of VMs supporting a Casandra implementation.

 

More suggestions?

To quote Martin Fowler, “If the components do not compose cleanly, then all you are doing is shifting complexity from inside a component to the connections between components. Not just does this just move complexity around, it moves it to a place that's less explicit and harder to control.” I have seen this happen – with the appearance of microservices (because there are tons of REST APis on different servers) but behind this layer, there are shared DLL’s accessing multiple databases causing endless pains with keeping DLL current as features are added or bugs fixed. A bug in a single library may require the fix to be propagated to a dozen REST APIs’. If it must be propagated it is not a microservice.

 

My goal with this post is to define a set of objective check items that can be clearly determined by inspection. The ideal implementation would have all of them passing.

 

One of the side-effects is that the rules can often be inconvenient for quick designs. A rethinking of what you are trying to do often results – similar to what I have seen happen when you push for full normalization in a logical model. 

 

Do you have further suggestions?

Tuesday, April 26, 2016

Testing Angular Directives with Karma, Mocha, and Phantom

All Code on Github

The entire code from this article is on github.

Introduction

Angular Directives are part of the web part/components group of client-side  tools that allow quick additions to web pages. The idea is that with a very simple and short addition to an html page, complex functionality and UX are available.

Imagine a user login form with the traditional validation contained in a html template and Angular controller. The main, calling web page’s html could just include <login></login> to bring in that rich validation and display. Directives are wonderful for encapsulating the complexity away from the containing HTML.

Testing the directive is trickier. Granted the controller isn’t difficult to test. But the compiled html and scope are not so easy.

Perhaps it is enough to test the controller, and depend on the browser and the Angular library to manage the rest.  You could definitely make that case. Or perhaps you leave the compiled directive testing to the end to end (e2e) tests. That is also a fair. If either of these solutions doesn’t work for you, this article will explain 3 ways to test the compiled Angular directive.

Template
When you build the directive, you can choose static (hopefully short) html in the definition of the directive.  Notice that {{data}} will be replaced with the $scope.data value from the directive.
exports.user = function() {
  return {
    controller: 'userController',
    template: '<div class="user">{{data}}</div>'
    };
};
TemplateUrl
If you have more html or want to separate the html from the Angular directive, templateUrl is the way to go.
exports.menu = function() {
  return {
    controller: 'menuController',
    templateUrl: '/templates/menu.html'
  };
};
The html contained in menu.html is below:
<div class="menu">
    {{data}}
</div>

Compilation

Angular compiles the controller and the template in order to replace the html tag. This compilation is necessary to test the directive.

Prerequisites

This article assumes you have some understanding of javascript, Angular, and unit testing. I’ll focus on how the test was setup and compiled the directive so that it could be tested. You need node and npm to install packages and run scripts. The other dependencies are listed in the package.json files.

Karma, Mocha, Chai, Phantom Test System

Since Angular is a client-side framework, Karma acts as the web server and manages the web browser for html and javascript. Instead of running a real browser, I chose Phantom so everything can run from the command line. Mocha and chai are the test framework and assert library.

While the Angular code is browserified, that doesn’t have any impact on how the directive is tested.

The Source Code and Test

With very little difference, each of the 3 examples is just about the same: a very simple controller that has $scope.data, an html template that uses {{data}} from the scope, and a test that compiles the directive and validates that the {{data}} syntax was replaced with the $scope.data value.

Each project has the same directory layout:

/client angular files to be browserified into /public/app.js
/public index.html, app.js, html templates
/test mocha/chai test file
karma.conf.js karma configuration file
gulpfile.js browserify configuration
package.json list of dependencies and script to build and run test

Testing the static template


The first example tests a template using static html. 

The controller:
exports.userController = function ($scope) {
    $scope.data = "user";
};
The directive: 
exports.user = function() {
  return {
    controller: 'userController',
    template: '<div class="user">{{data}}</div>'
    };
};
The calling html page
<html ng-app="app">
  <head>
    <script type="text/javascript"
      src="https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular.js">
    </script>
    <script type="text/javascript"
      src="https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular-route.js">
    </script>
    <script type="text/javascript" src="/app.js"></script>
  </head>
  <body>
      <div>
          <user></user>
      </div> 
  </body>
</html>
The final html page will replace <user></user> with the compiled html from the template with the string “user” replacing {{data}} in the static template html. The mocha test
describe('userDirective', function() {
  var scope;

  beforeEach(angular.mock.module("app"));
  
  beforeEach(angular.mock.module('ngMockE2E'));

  beforeEach(inject(function ($rootScope) {
      scope = $rootScope.$new();
    }));
    
  it('should exist', inject(function ($compile) {
    element = angular.element('<user></user>');
    compiledElement = $compile(element)(scope);
 
    scope.$digest();

    var dataFromHtml = compiledElement.find('.user').text().trim();
    var dataFromScope = scope.data;
    
    console.log(dataFromHtml + " == " + dataFromScope);

    expect(dataFromHtml).to.equal(dataFromScope);
  }));
});
As part of the test setup, in the beforeEach functions, the test brings in the angular app, brings in the mock library, and sets the scope. Inside the test (the ‘it’ function), the element function defines the directive’s html element, and compiles the scope and the element. The compiled html text value is tested against the scope value – which we expect to be the same. The overall test concept is the same for each of the examples. Get the controller and template, compile it, check it. The details differ in exactly how this is done in the 2 remaining examples.

The 2 TemplateUrl Examples

The first example above relied on the directive definition to load the template html. The next 2 examples use the TemplateUrl property which has the html stored in a separate file so that method won’t work. Both of the next 2 examples use the templateCache to load the template, but each does it in a different way. The first example loads the template and tests the template as though the templateCache isn’t used. This is a good example for apps that don’t generally use the templateCache and developers that don’t want to change the app code in order to test the directive. The templateCache is only used as a convenience for testing. The second example alters the app code by loading the template in the templateCache and browserifying the app with the ‘templates’ dependency code. This is a good way to learn about the templateCache and test it.

Testing TemplateUrl – least intrusive method

The second example stores the html in a separate file instead of in the directive function. As a result, the test (and the karma config file) needs to bring the separate file in before compiling the element with the scope. The controller:
exports.menuController = function ($scope) {
    $scope.data = "menu";
};
The directive:
exports.menu = function() {
  return {
    controller: 'menuController',
    templateUrl: '/templates/menu.html'
  };
};
The template:
<div class="menu">
    {{data}}
</div>
The calling html page only different in that the html for the directive is
<menu></menu>
karma-utils.js
function httpGetSync(filePath) {
  var xhr = new XMLHttpRequest();
  
  var finalPath = filePath;
  
  //console.log("finalPath=" + finalPath);
  
  xhr.open("GET", finalPath, false);
  xhr.send();
  return xhr.responseText;
}

function preloadTemplate(path) {
  return inject(function ($templateCache) {
    var response = httpGetSync(path);
    //console.log(response);
    $templateCache.put(path, response);
  });
}
The file along with the template file are brought in via the karma.conf.js file in the files property:
// list of files / patterns to load in the browser
files: [
    'http://code.jquery.com/jquery-1.11.3.js',
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular.js',
   
    // For ngMockE2E
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular-mocks.js',
    'test/karma-utils.js',
    'test/test.directive.*.js',
    'public/app.js',
    'public/templates/*.html'
],
The mocha test
describe('menuDirective', function() {
  var mockScope;
  var compileService;
  var template;

  beforeEach(angular.mock.module("app"));
  
  beforeEach(angular.mock.module('ngMockE2E'));

  beforeEach(preloadTemplate('/templates/menu.html'));

  beforeEach(inject(function ($rootScope) {
      scope = $rootScope.$new();
    }));
    
  it('should exist', inject(function ($compile) {
    element = angular.element('<menu></menu>');
    compiledElement = $compile(element)(scope);
 
    scope.$digest();

    var dataFromHtml = compiledElement.find('.menu').text().trim();
    var dataFromScope = scope.data;
    
    console.log(dataFromHtml + " == " + dataFromScope);

    expect(dataFromHtml).to.equal(dataFromScope);
  }));
});
As part of the test setup, in the beforeEach functions, the test brings in the angular app, brings in the mock library, brings in the html template, and sets the scope. The template is brought in via a help function in another file in the test folder called preloadTemplate. Inside the test (the ‘it’ function), the element function defines the directive’s html element, and compiles the scope and the element. The compiled html text value is tested against the scope value – which we expect to be the same. This is the line of the test file that deals with the templateCache:
beforeEach(preloadTemplate('/templates/menu.html')); 
The app and test code do not use the templateCache in any other way. The test itself is almost identical to the first test that uses the static html in the directive.

Testing TemplateUrl – most intrusive method

In this last example, the app.js code in the /client directory is altered to pull in a new ‘templates’ dependency module. The templates module is built in the gulpfile.js by grabbing all the html files in the /public/templates directory and wrapping it in javascript that adds the templates to the templateCache. The test explicitly pulls the template from the templateCache before compilation. gulpfile.js – to build templates dependency into /client/templates.js
var gulp = require('gulp');
var browserify = require('gulp-browserify');

var angularTemplateCache = require('gulp-angular-templatecache');

var concat = require('gulp-concat');
var addStream = require('add-stream');

gulp.task('browserify', function() {
  return gulp.
    src('./client/app.js').
    //pipe(addStream('./client/templates.js')).
    //pipe(concat('app.js')).
    pipe(browserify()).
    pipe(gulp.dest('./public'));
});

gulp.task('templates', function () {
  return gulp.src('./public/templates/*.html')
    .pipe(angularTemplateCache({module:'templates', root: '/templates/'}))
    .pipe(gulp.dest('./client'));
});

gulp.task('build',['templates', 'browserify']);
templates.js – built by gulpfile.js
angular.module("templates").run([
    "$templateCache",  function($templateCache) {
     $templateCache.put("/templates/system.html","<div class=\"menu\">\n    {{data}}\n</div>");
    }
]);
app.js – defines template module and adds it to app list of dependencies
var controllers = require('./controllers');
var directives = require('./directives');
var _ = require('underscore');

// this never changes
angular.module('templates', []);

// templates added as dependency
var components = angular.module('app', ['ng','templates']);

_.each(controllers, function(controller, name)
{ components.controller(name, controller); });

_.each(directives, function(directive, name)
{ components.directive(name, directive); });

require('./templates')
The mocha test
describe('menuDirective', function() {
  var mockScope;
  var compileService;
  var template;

  beforeEach(angular.mock.module("app"));
  
  beforeEach(angular.mock.module('ngMockE2E'));

  beforeEach(inject(function ($rootScope) {
      scope = $rootScope.$new();
    }));
    
  it('should exist', inject(function ($compile, $templateCache) {
    element = angular.element('<system></system>');
    compiledElement = $compile(element)(scope); 

    // APP - app.js used templates dependency which loads template
    // into templateCache
    // TEST - this test pulls template from templateCache 
    template = $templateCache.get('/templates/system.html'); 
 
    scope.$digest();

    var dataFromHtml = compiledElement.find('.menu').text().trim();
    var dataFromScope = scope.data;
    
    console.log(dataFromHtml + " == " + dataFromScope);

    expect(dataFromHtml).to.equal(dataFromScope);
  }));
});

The Test Results

The test results for each of the 3 projects is checked in to the github project. Karma ran with debug turned on so that the http and file requests could be validated. When you look at the testResult.log files (1 in each of the 3 subdirectories), you want to make sure that the http and file requests that karma made were actually successful. The lines with ‘Fetching’, ‘Requesting’, and ‘(cached)’ are the important lines before the test runs.

36m26 04 2016 11:42:09.318:DEBUG [middleware:source-files]: [39mFetching /home/dina/repos/AngularDirectiveKarma/directiveTemplate/test/test.directive.template.js
[36m26 04 2016 11:42:09.318:DEBUG [middleware:source-files]: [39mRequesting /base/public/app.js?6da99f7db89b4401f7fc5df6e04644e14cbed1f7 /
[36m26 04 2016 11:42:09.318:DEBUG [middleware:source-files]: [39mFetching /home/dina/repos/AngularDirectiveKarma/directiveTemplate/public/app.js
[36m26 04 2016 11:42:09.319:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/node_modules/mocha/mocha.js
[36m26 04 2016 11:42:09.329:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/node_modules/karma-mocha/lib/adapter.js [36m26 04 2016 11:42:09.331:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/test/test.directive.template.js
[36m26 04 2016 11:42:09.333:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/public/app.js

During the test, you should see that the $scope.data value (different in each of the 3 tests) is equated to the compiled html as expected, for example, “user = user”:
LOG: 'user == user'
And finally, that the test passed:
Executed 1 of 1 SUCCESS