Thursday, May 12, 2016

The sad state of evidence based development management patterns

I have been in the development game for many decades. I did my first programs using APL/360 and Fortran (WatFiv) at the University of Waterloo, and have seen and coded a lot of languages over the years (FORTH, COBOL, Asm, Pascal, B,C, C++, SAS, etc).

 

My academic training was in Operations Research – that is mathematical optimization of business processes. Today, I look at the development processes that I see and it is dominantly “fly by the seats of the pants”, “everybody is doing it” or “academic correctness”. I am not talking about waterfall or agile or scrum. I am not talking about architecture etc. Yet is some ways I am. Some processes assert Evidence Based Management, yet fails to deliver the evidence of better results. Some bloggers detail the problems with EBM.  A few books attempt to summarize the little research that has occurred, such as "Making Software: What Really Works and Why we Believe It"

 

As an Operation Research person, I would define the optimization problem facing a development manager or director or lead as follows:

  • Performance (which often comes at increased man hours to develop and operational costs)
  • Scalability (which often comes at increased man hours to develop and operational costs)
  • Cost to deliver
  • Accuracy of deliverable (Customer satisfaction)
  • Completeness of deliverable
  • Elapsed time to delivery (shorter time often exponentially increase cost to deliver and defect rates)
  • Ongoing operational costs (a bad design may result in huge cloud computing costs)
  • Time for a new developer to become efficient across the entire product
  • Defect rate
    • Number of defects
    • ETA from reporting to fix
  • Developer resources
    • For development
    • For maintenance

All of these factors interact. For evidence, there are no studies and I do not expect them to be. Technology is changing too fast, there is huge differences between projects, and any study will be outdated before it is usable. There is some evidence that we can work from.

Lines of Code across a system

Lines of code directly impacts several of the above.

  • Defect rate is a function of the number of lines of code ranging from 200/100K to 1000/100K lines [source] which is scaled by developer skill level. Junior or new developers will have a higher defect rate.
  • Some classic measures defined in the literature, for example, cyclomatic complexity. Studies find a positive correlation between cyclomatic complexity and defects: functions and methods that have the highest complexity tend to also contain the most defects.
  • Time to deliver is often a function of the lines of code written.

There is a mistaken belief that lines of code is an immutable for a project. In the early 2000’s I lead a rewrite of a middle tier and backend tier (with the web front end being left as is), the original C++/SQL server code base was 474,000 lines of code and was the result of 25 man years of coding. With a team of 6 new (to the application) developers sent over from India and 2 intense local developer, we recreated these tiers with 100% api compliance in just 25,000 lines of code in about 8 weeks. 25 man years –> 1 man year. a 20 fold decrease in code base. And the last factor was an increase in concurrent load by 20 fold. 

 

On other projects I have seen massive copy and paste (with some minor change) that result in code bloat. When a bug is discovered it was often only fixed in some of the pastes. Martin Fowler describes Lines of Code as a measure of developer productivity as useless; the same applies to lines of code in a project.  A change of programming language can result in a 10 fold drop (or increase) in lines of code. A change of a developer can also result in a similar change – depending on skill sets.

 

Implementation Design

The use of Object-Relational Mapping (ORM) can often result in increased lines of code, defects, steeper learning curves and greater challenges addressing performance issues. A simple illustration is to move all addresses in Washington State from a master table to a child table. In SQL Server, TSQL – it is a one line statement, calling this from SQL it amounts to 4 lines of C# code. Using an ORM, this can quickly grow to 100-200 lines. ORMs came along because of a shortage of SQL developer skills. As with most things, it carry hidden costs that are omitted in the sales literature!

 

“Correct academic design” does not mean effective (i.e. low cost) development. One of the worst systems (for performance and maintenance) that I have seen was absolutely beautifully designed with a massive array of well defined classes – which unfortunately ignored the database reality.  Many calls of a single method cascaded through these classes and resulted in 12 – 60 individual sql queries being executed against the database.  Most of the methods could be converted to a wrapper on a single stored procedure with a major improvement of performance. The object hierarchy was flattened (or downsized!).

 

I extend the concept of cyclomatic complexity to the maximum stack depth in developer written code.  The greater the depth, the longer it takes to debug (because the developer has to walk through the stack) and likely to write. The learning curve goes up. I suggest a maximum depth of 7 (less than cyclomatic complexity), ideally 5. This number comes out of research for short term memory (wikipedia). Going beyond seven significantly increases the effort that a developer needs to make to understand the stack. On the one hand, having a deep hierarchy of objects looks nice academically – but it is counterproductive for efficient coding. Seven is a magic number to keep asking “Why do we have more than seven ….”

Developer Skill Sets

Many architects suffer from the delusion that all developers are as skilled as they are, i.e. IQs over 145.  During my high school teaching years, I was assigned both gifted classes and challenged classes – and learn to present appropriately to both. In some cities (for example Stockholm, Sweden) – 20% of the work force is in IT. This means that the IQ of the developers likely range from 100 upwards. When an application is released, the support developers likely will end up with an average IQ around 100. The question must be asked, how simple is the code to understand for future enhancements and maintenance?

 

If a firm has a policy of significant use of off-shore or contractor resources, there are  further challenges:

  • A high percentage of the paid time is in ramp-up mode
  • There is a high level of non- conformity to existing standards and practices.
    • Higher defect rate, greater time for existing staff to come up to speed on the code
  • Size of team and ratio of application-experienced versus new developer can greatly alter delivery scheduled (see Brook’s law

Pseudo coding different architecture rarely happens. It has some advantages – if you code up the most complex logic and then ask the question – “ A bug happens and nothing comes back, what are the steps to isolated the issue with certainty?” The architecture with the least diagnostic steps may be the more efficient one.

 

Last, the availability now and in the future of developers with the appropriate skills.  The industry is full of technology that was hot and promised the moon and then were disrupted by a new technology (think of Borland Delphi and Pascal!). I often do a weighted value composed of years since launch, popularity at the moment and trend to refine choices (and in some cases to say No to a developer or architect that want to play with the latest and greatest!). Some sites are DB-Engine Ranking and PYPL.  After short listing, then it’s a matter of coding up some complex examples in each and counting lines of code needed.

Specification Completeness And Stability

On one side, I have worked with a few PMs that deliver wonderful specifications (200-500 pages) that had no change-orders between the first line of code being written and final delivery a year later. What was originally handed to developers was not changed. Work was done in sprints. The behavior and content of every web page was detailed. There was a clean and well-reviewed dictionary of terms and meanings. Needless to say, delivery was prompt, on schedule, etc.

 

On the other side, I have had minor change-requests which mutated constantly. The number of lines of code written over all of these changes were 20x the number of lines of code finally delivered.

Concurrent Development

Concurrent development means that two or more set of changes were happening to the same code base. At one firm we had several git-hub forks: Master,Develop, Sprint, Epic and Saga. The title indicate when the changes were expected to be propagated to master. It worked reasonably, but often I ended up spending two days resolving conflicts and debugging bugs that were introduced whenever I attempted to get forks in sync. Concurrent development increases overhead exponentially according to the number of independent forks are active. Almost everything in development has exponential cost with size, there is no economy of scale in development.

 

On the flip side, at Amazon using the microservices model, there were no interaction between feature requests. Each API was self contained and would evolve independently. If an API needed another API changed, then the independent API would be changed, tested and released. The dependent API then was developed against the released independent API. There was no code-juggling act. Each code base API was single development and self-contained. Dependencies were by API not libraries and code bases.

 

Bottom Line

Controlling costs and improving delivery depends greatly on the preparation work IMHO -- namely:

  • Specification stability and completeness
  • Architectural / Design being well crafted for the developer population
  • Minimum noise (i.e. no concurrent development, change orders, change of priorities)
  • Methodology (Scrum, Agile, Waterfall, Plan Driven) is of low significance IMHO – except for those selling it and ‘true believers’.

On the flip side, often the business will demand delivery schedules that add technical debt and significantly increase ongoing costs.

 

A common problem that I have seen is solving this multiple dimension problem by looking at just one (and rarely two) dimensions and discovering the consequences of that decision down stream.  I will continue to add additional dimensions as I recall them from past experience.

Tuesday, May 10, 2016

Mining PubMed via Neo4J Graph Database–Getting the data

I have a blog dealing with various complex autoimmune diseases and spend a lot of time walking links at PubMed.com. Often readers send me an article that I missed. 

 

I thought that a series of post on how to do it will help other people (including MDs, grad students and citizen scientists) better research medical issues.

 

Getting the data from Pub Med

I implemented a simple logic to obtain a collection of relevant articles:

  • Query for 10,000 articles on a subject or key word

  • Retrieve each of these articles and any articles they referenced (i.e. the knowledge graph).
  • Keep repeating until you have enough articles or you run out of them!!

Getting the bootstrapping list of articles

A console application that reads the command line arguments and retrieves the list. For example,

downloader.exe Crohn’s Disease

which produces this URI

http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=Pubmed&retmax=1000&usehistory=y&term=Crohn's+disease

This results in an XML file being sent

 

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE eSearchResult PUBLIC "-//NLM//DTD esearch 20060628//EN" "
http://eutils.ncbi.nlm.nih.gov/eutils/dtd/20060628/esearch.dtd">
<eSearchResult><Count>44880</Count><RetMax>10000</RetMax><RetStart>0</RetStart><QueryKey>1</QueryKey><WebEnv>NCID_1_84230330_130.14.22.215_9001_1462926138_46088356_0MetA0_S_MegaStore_F_1</WebEnv><IdList>
<Id>27159423</Id>
<Id>27158773</Id>
<Id>27158547</Id>
<Id>27158537</Id>
<Id>27158536</Id>
<Id>27158345</Id>
<Id>27158125</Id>
<Id>27157449</Id>
<Id>27156530</Id>
<Id>27154890</Id>
<Id>27154001</Id>
<Id>27153721</Id>
<Id>27152873</Id>
<Id>27152872</Id>
<Id>27152547</Id>

So let us look at the code

class Program
    {
        static Downloader downloader = new Downloader();
        static void Main(string[] args)
        {
            if (args.Length > 0)
            {
                var search = new StringBuilder();
                foreach (var arg in args)
                {
                    search.AppendFormat("{0} ", arg);
                }
                downloader.TermSearch(search.ToString());
                downloader.ProcessAll();
            }
            downloader.Save();
        }
      }

The Downloader class tracks articles already downloaded and those to do next. It simply starts downloading and saving each article summary to an Xml file using the unique article Id as the file name. I wanted to keep the summaries on my disk to speed reprocessing if my Neo4J model changes.


using System;
using System.Collections.Generic;      
using System.Collections.Concurrent;
using System.Net;                 

using System.Linq;
using System.Threading.Tasks; 
using System.Xml;                    

using System.Text;    
using System.Configuration;
using System.IO;
namespace PubMed
{
    public class Downloader
    {
        // Entrez E-utilities at the US National Center for Biotechnology Information:
        static readonly String server = "
http://www.ncbi.nlm.nih.gov/entrez/eutils/";
        string dataFolder = "C:\\PubMed";
        string logFile;
        public System.Collections.Concurrent.ConcurrentBag<string> index = new ConcurrentBag<string>();
        public System.Collections.Concurrent.ConcurrentQueue<string> todo = new ConcurrentQueue<string>();
        public Downloader()
        {
            logFile = Path.Combine(dataFolder, "article.log");
            if (File.Exists(logFile))
            {
                var lines = File.ReadAllLines(logFile);
                foreach (var line in lines)
                {
                    if (!string.IsNullOrWhiteSpace(line))
                        index.Add(line);
                }
            }
        }
        public void Save()
        {
            File.WriteAllLines(logFile, index.ToArray());
        }

         public void ProcessAll()
        {

            var nextId = string.Empty;
            while (todo.Count > 0)
            {
                if (todo.Count > 12)
                {
                    var tasks = new List<Task>();
                    int t = 0;
                    for (t = 0; t < 10; t++)
                    {
                        if (todo.TryDequeue(out nextId))
                        {

                            tasks.Add(Task.Factory.StartNew(() => NcbiPubmedArticle(nextId)));
                        }
                    }
                    Task.WaitAll(tasks.ToArray());
                    Save();
                }
                else
                {
                    if (todo.TryDequeue(out nextId))
                    {

                        NcbiPubmedArticle(nextId);
                    }
                }
            }
        }

        public void TermSearch(String term)
        {
            var search = string.Format("
http://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=Pubmed&retmax=1000&usehistory=y&term={0}", term.Replace(" ", "+"));
            new WebClient().DownloadFile(new Uri(search), "temp.log");
            var xml = new XmlDocument();
            xml.Load("temp.Log");
            foreach (XmlNode node in xml.DocumentElement.SelectNodes("//Id"))
            {
                var id = node.InnerText;
                if (!index.Contains(id) && !todo.Contains(id))
                {
                    todo.Enqueue(id);
                }
            }
        }

        public void NcbiPubmedArticle(String term)
        {

            if (!index.Contains(term))
            {
                try
                {
                    var fileLocation = Path.Combine(dataFolder, string.Format("{0}.xml", term));
                    if (File.Exists(fileLocation)) return;
                    var search = string.Format("
http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id={0}&retmode=xml", term);
                    new WebClient().DownloadFile(new Uri(search), fileLocation);
                    index.Add(term);
                    GetChildren(fileLocation);
                    Console.WriteLine(term);
                }
                catch
                {

                }
            }
        }
        private void GetChildren(string fileName)
        {
            try
            {
                var dom = new XmlDocument();
                dom.Load(fileName);
                foreach (XmlNode node in dom.DocumentElement.SelectNodes("//PMID"))
                {
                    var id = node.InnerText;
                    if (!index.Contains(id) && !todo.Contains(id))
                    {
                        todo.Enqueue(id);
                    }
                }
            }
            catch (Exception exc)
            {
                Console.WriteLine(exc.Message);
            }
        }
    }
}

Next Importing into Neo4J

An example of the structured data to load is shown below. Try defining your own model while you wait for the next post. 

 

<?xml version="1.0"?>
<!DOCTYPE PubmedArticleSet PUBLIC "-//NLM//DTD PubMedArticle, 1st January 2016//EN" "
http://www.ncbi.nlm.nih.gov/corehtml/query/DTD/pubmed_160101.dtd">
<PubmedArticleSet>
<PubmedArticle>
    <MedlineCitation Owner="NLM" Status="MEDLINE">
        <PMID Version="1">10022306</PMID>
        <DateCreated>
            <Year>1999</Year>
            <Month>02</Month>
            <Day>25</Day>
        </DateCreated>
        <DateCompleted>
            <Year>1999</Year>
            <Month>02</Month>
            <Day>25</Day>
        </DateCompleted>
        <DateRevised>
            <Year>2006</Year>
            <Month>11</Month>
            <Day>15</Day>
        </DateRevised>
        <Article PubModel="Print">
            <Journal>
                <ISSN IssnType="Print">0378-4274</ISSN>
                <JournalIssue CitedMedium="Print">
                    <Volume>102-103</Volume>
                    <PubDate>
                        <Year>1998</Year>
                        <Month>Dec</Month>
                        <Day>28</Day>
                    </PubDate>
                </JournalIssue>
                <Title>Toxicology letters</Title>
                <ISOAbbreviation>Toxicol. Lett.</ISOAbbreviation>
            </Journal>
            <ArticleTitle>Epidemiological association in US veterans between Gulf War illness and exposures to anticholinesterases.</ArticleTitle>
            <Pagination>
                <MedlinePgn>523-6</MedlinePgn>
            </Pagination>
            <Abstract>
                <AbstractText>To investigate complaints of Gulf War veterans, epidemiologic, case-control and animal modeling studies were performed. Looking for OPIDP variants, our epidemiologic project studied 249 Naval Reserve construction battalion (CB24) men. Extensive surveys were drawn for symptoms and exposures. An existing test (PAI) was used for neuropsychologic. Using FACTOR, LOGISTIC and FREQ in 6.07 SAS, symptom clusters were sought with high eigenvalues from orthogonally rotated two-stage factor analysis. After factor loadings and Kaiser measure for sampling adequacy (0.82), three major and three minor symptom clusters were identified. Internally consistent by Cronbach's coefficient, these were labeled syndromes: (1) impaired cognition; (2) confusion-ataxia; (3) arthro-myo-neuropathy; (4) phobia-apraxia; (5) fever-adenopathy; and (6) weakness-incontinence. Syndrome variants identified 63 patients (63/249, 25%) with 91 syndromes. With pyridostigmine bromide as the drug in these drug-chemical exposures, syndrome chemicals were: (1) pesticide-containing flea and tick collars (P &lt; 0.001); (2) alarms from chemical weapons attacks (P &lt; 0.001), being in a sector later found to have nerve agent exposure (P &lt; 0.04); and (3) insect repellent (DEET) (P &lt; 0.001). From CB24, 23 cases, 10 deployed and 10 non-deployed controls were studied. Auditory evoked potentials showed dysfunction (P &lt; 0.02), nystagmic velocity on rotation testing, asymmetry on saccadic velocity (P &lt; 0.04), somatosensory evoked potentials both sides (right P &lt; 0.03, left P &lt; 0.005) and synstagmic velocity after caloric stimulation bilaterally (P-range, 0.02-0.04). Brain dysfunction was shown on the Halstead Impairment Index (P &lt; 0.01), General Neuropsychological Deficit Scale (P &lt; 0.03) and Trail Making part B (P &lt; 0.03). Butylcholinesterase phenotypes did not trend for inherent abnormalities. Parallel hen studies at Duke University established similar drug-chemical delayed neurotoxicity. These investigations lend credibility that sublethal exposures to drug-chemical combinations caused delayed-onset neurotoxic variants.</AbstractText>
            </Abstract>
            <AuthorList CompleteYN="Y">
                <Author ValidYN="Y">
                    <LastName>Kurt</LastName>
                    <ForeName>T L</ForeName>
                    <Initials>TL</Initials>
                    <AffiliationInfo>
                        <Affiliation>Department of Internal Medicine, University of Texas Southwestern Medical School, Dallas 75235, USA.</Affiliation>
                    </AffiliationInfo>
                </Author>
            </AuthorList>
            <Language>eng</Language>
            <PublicationTypeList>
                <PublicationType UI="D016428">Journal Article</PublicationType>
                <PublicationType UI="D013485">Research Support, Non-U.S. Gov't</PublicationType>
            </PublicationTypeList>
        </Article>
        <MedlineJournalInfo>
            <Country>NETHERLANDS</Country>
            <MedlineTA>Toxicol Lett</MedlineTA>
            <NlmUniqueID>7709027</NlmUniqueID>
            <ISSNLinking>0378-4274</ISSNLinking>
        </MedlineJournalInfo>
        <ChemicalList>
            <Chemical>
                <RegistryNumber>0</RegistryNumber>
                <NameOfSubstance UI="D002800">Cholinesterase Inhibitors</NameOfSubstance>
            </Chemical>
        </ChemicalList>
        <CitationSubset>IM</CitationSubset>
        <MeshHeadingList>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D016022">Case-Control Studies</DescriptorName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D002800">Cholinesterase Inhibitors</DescriptorName>
                <QualifierName MajorTopicYN="Y" UI="Q000633">toxicity</QualifierName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D006801">Humans</DescriptorName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D008297">Male</DescriptorName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="N" UI="D018923">Persian Gulf Syndrome</DescriptorName>
                <QualifierName MajorTopicYN="Y" UI="Q000209">etiology</QualifierName>
            </MeshHeading>
            <MeshHeading>
                <DescriptorName MajorTopicYN="Y" UI="D014728">Veterans</DescriptorName>
            </MeshHeading>
        </MeshHeadingList>
    </MedlineCitation>
    <PubmedData>
        <History>
            <PubMedPubDate PubStatus="pubmed">
                <Year>1999</Year>
                <Month>2</Month>
                <Day>18</Day>
            </PubMedPubDate>
            <PubMedPubDate PubStatus="medline">
                <Year>1999</Year>
                <Month>2</Month>
                <Day>18</Day>
                <Hour>0</Hour>
                <Minute>1</Minute>
            </PubMedPubDate>
            <PubMedPubDate PubStatus="entrez">
                <Year>1999</Year>
                <Month>2</Month>
                <Day>18</Day>
                <Hour>0</Hour>
                <Minute>0</Minute>
            </PubMedPubDate>
        </History>
        <PublicationStatus>ppublish</PublicationStatus>
        <ArticleIdList>
            <ArticleId IdType="pubmed">10022306</ArticleId>
        </ArticleIdList>
    </PubmedData>
</PubmedArticle>

</PubmedArticleSet>

Saturday, May 7, 2016

Microservices–Do it right!

In my earlier post, A Financially Frugal Architectural Pattern for the Cloud,  I advocated the use of microservices. Microservices are similar to REST, a concept or pattern or architectural standard, unlike  SOAP which is standards based. The modern IT industry trend towards “good enough”,  “lip-service” and “we’ll fix it in the next release”.  A contemporary application may use relational database software (SQL Server, Oracle, MySql) and thus the developers (and their management) would assert that their is a relational database system. If I move a magnetic tape based system into tables (one table for each type of tape) using relational database software – would that make it a relational database system? My opinion is no – never!!!

 

Then what makes it one? The data has been fully normalized in the logical model. Often the database has never been reviewed for normalization  despite such information being ancient (see William Kent, A Simple Guide to Five Normal Forms in Relational Database Theory, 1982), older than many developers. The implementation may be de-normalized in the physical model (if you have just a ‘database model’ and not separate physical and logical, then you are likely heading to trouble – in time (usually after the original developers have left!). For NoSql database, there is a lot of ancient literature out there dealing with both hierarchical databases and network databases which should also be used with MongoDB and Neo4j – but likely not.

 

My academic training is in mathematics and thus axioms and deriving theorems from them though rigorous logic.  The normalization of databases is immediately attractive to me. Knowing the literature (especially Christopher J.Date’s early writings from the 1970’s) is essential since “"Those who do not learn history are doomed to repeat it.”

 

Microservices Normalization Rules

Below are my attempt to define equivalent rules for microservices. They will likely be revised over time. They are very mathematical in definition by intent. Martin Fowler’s article is also a good read. Much of the discussion on the web is at a high level (the hand waving level), such as Microservices Architecture and Design PrinciplesMicroservices Design Principles, with some echoing some of the issues cited below Adopting Microservices at Netflix: Lessons for Architectural Design

 

A public REST API consumed over the internet is probably not a microservice. It may use many microservices and other composite APIs.

 

  • Composite API: A service that consumes other composite APIs and/or microservices but do not qualify below
  • Independent Microservice: A service that does not call any other microservices
  • Dependent Microservice: A service that calls Independent Microservices in parallel

An Independent Microservice

An independent microservice is the exclusive owner of a data store.

  • No other service or system may access the data store.
  • A microservice may change the software used to create the datastore with no consequences on any other system.
  • A microservice does not make calls to other services
    • An corollary of this is that microservices rarely use any libraries that are not generic across the industry
      • Exception: libraries of static functions that are explicit to a firm, for example, encryption of  keys (i.e. Identity Integers –> strings)
  • A microservice may contain publishers
    • The datastore that it controls may need to be pushed to reporting and other systems
  • A microservice may create log records that are directly consumed by other systems.
    • Non-blocking outputs from a microservice are fine
  • A microservice may make periodic calls.
    • A microservice may pull things off a queue, or push things to a queue
      • The nature of this data is transient. The queue services interaction must match the model of an API call and response. The call comes from one queue and written to another queue.
        • Ideally there will be no references to queues inside the microservice.
        • A call to the microservice would start reading a queue at a regular interval (the queue is in the call parameters)
        • The data on the queue would specify where the results should be sent
    • A microservice should not pull data from (non-microservice) data store.
      • Exception: a configurationless implementation such as described for queues above is fine.
  • A microservice configuration should never reference anything outside of it’s own world.
    • Configuration Injection is allowed. Microservice is deployed and loaded, then a call is done to supply it’s configuration.

A Dependent Microservice

  • Has exclusive ownership of it’s data store just like an independent microservice.
  • Calls dependent microservice to obtain data only (no update, delete or create)
    • Create Update Delete calls must always go to the independent microservice that owns it, no relaying should occur.
    • Calls are in parallel, never sequential
  • Note: Some types of queries may be inefficient, those should be directed at a reporting microservice (which independent microservices may publish to) or a composite service.

 

Control of Microservices

The best model that I have seen is one that some groups at Amazon used.

  • All calls to the microservice must have an identifier (cookie?) that identifies the caller.
    • The microservice determines if it is an authorized caller based on the identifier and possibly the IP address
  • The consumer of the microservice must be authorized by the owner of the microservice based on: a contract containing at least
    • Daily load estimates
    • Peak load estimates
    • Availability
  • The microservice may disable any consumer that exceeds the contracted load.
  • The consumer should be given a number from 1-100 indicating business importance.
    • If the microservice is stressed, then those services with lower values will be disabled.
  • There should always be SLA in place

Microservices should be executed on isolated VMs behind  a load distributor. The datastore should be also on a dedicated set of VMs, for example a set of VMs supporting a Casandra implementation.

 

More suggestions?

To quote Martin Fowler, “If the components do not compose cleanly, then all you are doing is shifting complexity from inside a component to the connections between components. Not just does this just move complexity around, it moves it to a place that's less explicit and harder to control.” I have seen this happen – with the appearance of microservices (because there are tons of REST APis on different servers) but behind this layer, there are shared DLL’s accessing multiple databases causing endless pains with keeping DLL current as features are added or bugs fixed. A bug in a single library may require the fix to be propagated to a dozen REST APIs’. If it must be propagated it is not a microservice.

 

My goal with this post is to define a set of objective check items that can be clearly determined by inspection. The ideal implementation would have all of them passing.

 

One of the side-effects is that the rules can often be inconvenient for quick designs. A rethinking of what you are trying to do often results – similar to what I have seen happen when you push for full normalization in a logical model. 

 

Do you have further suggestions?

Tuesday, April 26, 2016

Testing Angular Directives with Karma, Mocha, and Phantom

All Code on Github

The entire code from this article is on github.

Introduction

Angular Directives are part of the web part/components group of client-side  tools that allow quick additions to web pages. The idea is that with a very simple and short addition to an html page, complex functionality and UX are available.

Imagine a user login form with the traditional validation contained in a html template and Angular controller. The main, calling web page’s html could just include <login></login> to bring in that rich validation and display. Directives are wonderful for encapsulating the complexity away from the containing HTML.

Testing the directive is trickier. Granted the controller isn’t difficult to test. But the compiled html and scope are not so easy.

Perhaps it is enough to test the controller, and depend on the browser and the Angular library to manage the rest.  You could definitely make that case. Or perhaps you leave the compiled directive testing to the end to end (e2e) tests. That is also a fair. If either of these solutions doesn’t work for you, this article will explain 3 ways to test the compiled Angular directive.

Template
When you build the directive, you can choose static (hopefully short) html in the definition of the directive.  Notice that {{data}} will be replaced with the $scope.data value from the directive.
exports.user = function() {
  return {
    controller: 'userController',
    template: '<div class="user">{{data}}</div>'
    };
};
TemplateUrl
If you have more html or want to separate the html from the Angular directive, templateUrl is the way to go.
exports.menu = function() {
  return {
    controller: 'menuController',
    templateUrl: '/templates/menu.html'
  };
};
The html contained in menu.html is below:
<div class="menu">
    {{data}}
</div>

Compilation

Angular compiles the controller and the template in order to replace the html tag. This compilation is necessary to test the directive.

Prerequisites

This article assumes you have some understanding of javascript, Angular, and unit testing. I’ll focus on how the test was setup and compiled the directive so that it could be tested. You need node and npm to install packages and run scripts. The other dependencies are listed in the package.json files.

Karma, Mocha, Chai, Phantom Test System

Since Angular is a client-side framework, Karma acts as the web server and manages the web browser for html and javascript. Instead of running a real browser, I chose Phantom so everything can run from the command line. Mocha and chai are the test framework and assert library.

While the Angular code is browserified, that doesn’t have any impact on how the directive is tested.

The Source Code and Test

With very little difference, each of the 3 examples is just about the same: a very simple controller that has $scope.data, an html template that uses {{data}} from the scope, and a test that compiles the directive and validates that the {{data}} syntax was replaced with the $scope.data value.

Each project has the same directory layout:

/client angular files to be browserified into /public/app.js
/public index.html, app.js, html templates
/test mocha/chai test file
karma.conf.js karma configuration file
gulpfile.js browserify configuration
package.json list of dependencies and script to build and run test

Testing the static template


The first example tests a template using static html. 

The controller:
exports.userController = function ($scope) {
    $scope.data = "user";
};
The directive: 
exports.user = function() {
  return {
    controller: 'userController',
    template: '<div class="user">{{data}}</div>'
    };
};
The calling html page
<html ng-app="app">
  <head>
    <script type="text/javascript"
      src="https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular.js">
    </script>
    <script type="text/javascript"
      src="https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular-route.js">
    </script>
    <script type="text/javascript" src="/app.js"></script>
  </head>
  <body>
      <div>
          <user></user>
      </div> 
  </body>
</html>
The final html page will replace <user></user> with the compiled html from the template with the string “user” replacing {{data}} in the static template html. The mocha test
describe('userDirective', function() {
  var scope;

  beforeEach(angular.mock.module("app"));
  
  beforeEach(angular.mock.module('ngMockE2E'));

  beforeEach(inject(function ($rootScope) {
      scope = $rootScope.$new();
    }));
    
  it('should exist', inject(function ($compile) {
    element = angular.element('<user></user>');
    compiledElement = $compile(element)(scope);
 
    scope.$digest();

    var dataFromHtml = compiledElement.find('.user').text().trim();
    var dataFromScope = scope.data;
    
    console.log(dataFromHtml + " == " + dataFromScope);

    expect(dataFromHtml).to.equal(dataFromScope);
  }));
});
As part of the test setup, in the beforeEach functions, the test brings in the angular app, brings in the mock library, and sets the scope. Inside the test (the ‘it’ function), the element function defines the directive’s html element, and compiles the scope and the element. The compiled html text value is tested against the scope value – which we expect to be the same. The overall test concept is the same for each of the examples. Get the controller and template, compile it, check it. The details differ in exactly how this is done in the 2 remaining examples.

The 2 TemplateUrl Examples

The first example above relied on the directive definition to load the template html. The next 2 examples use the TemplateUrl property which has the html stored in a separate file so that method won’t work. Both of the next 2 examples use the templateCache to load the template, but each does it in a different way. The first example loads the template and tests the template as though the templateCache isn’t used. This is a good example for apps that don’t generally use the templateCache and developers that don’t want to change the app code in order to test the directive. The templateCache is only used as a convenience for testing. The second example alters the app code by loading the template in the templateCache and browserifying the app with the ‘templates’ dependency code. This is a good way to learn about the templateCache and test it.

Testing TemplateUrl – least intrusive method

The second example stores the html in a separate file instead of in the directive function. As a result, the test (and the karma config file) needs to bring the separate file in before compiling the element with the scope. The controller:
exports.menuController = function ($scope) {
    $scope.data = "menu";
};
The directive:
exports.menu = function() {
  return {
    controller: 'menuController',
    templateUrl: '/templates/menu.html'
  };
};
The template:
<div class="menu">
    {{data}}
</div>
The calling html page only different in that the html for the directive is
<menu></menu>
karma-utils.js
function httpGetSync(filePath) {
  var xhr = new XMLHttpRequest();
  
  var finalPath = filePath;
  
  //console.log("finalPath=" + finalPath);
  
  xhr.open("GET", finalPath, false);
  xhr.send();
  return xhr.responseText;
}

function preloadTemplate(path) {
  return inject(function ($templateCache) {
    var response = httpGetSync(path);
    //console.log(response);
    $templateCache.put(path, response);
  });
}
The file along with the template file are brought in via the karma.conf.js file in the files property:
// list of files / patterns to load in the browser
files: [
    'http://code.jquery.com/jquery-1.11.3.js',
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular.js',
   
    // For ngMockE2E
    'https://ajax.googleapis.com/ajax/libs/angularjs/1.4.0/angular-mocks.js',
    'test/karma-utils.js',
    'test/test.directive.*.js',
    'public/app.js',
    'public/templates/*.html'
],
The mocha test
describe('menuDirective', function() {
  var mockScope;
  var compileService;
  var template;

  beforeEach(angular.mock.module("app"));
  
  beforeEach(angular.mock.module('ngMockE2E'));

  beforeEach(preloadTemplate('/templates/menu.html'));

  beforeEach(inject(function ($rootScope) {
      scope = $rootScope.$new();
    }));
    
  it('should exist', inject(function ($compile) {
    element = angular.element('<menu></menu>');
    compiledElement = $compile(element)(scope);
 
    scope.$digest();

    var dataFromHtml = compiledElement.find('.menu').text().trim();
    var dataFromScope = scope.data;
    
    console.log(dataFromHtml + " == " + dataFromScope);

    expect(dataFromHtml).to.equal(dataFromScope);
  }));
});
As part of the test setup, in the beforeEach functions, the test brings in the angular app, brings in the mock library, brings in the html template, and sets the scope. The template is brought in via a help function in another file in the test folder called preloadTemplate. Inside the test (the ‘it’ function), the element function defines the directive’s html element, and compiles the scope and the element. The compiled html text value is tested against the scope value – which we expect to be the same. This is the line of the test file that deals with the templateCache:
beforeEach(preloadTemplate('/templates/menu.html')); 
The app and test code do not use the templateCache in any other way. The test itself is almost identical to the first test that uses the static html in the directive.

Testing TemplateUrl – most intrusive method

In this last example, the app.js code in the /client directory is altered to pull in a new ‘templates’ dependency module. The templates module is built in the gulpfile.js by grabbing all the html files in the /public/templates directory and wrapping it in javascript that adds the templates to the templateCache. The test explicitly pulls the template from the templateCache before compilation. gulpfile.js – to build templates dependency into /client/templates.js
var gulp = require('gulp');
var browserify = require('gulp-browserify');

var angularTemplateCache = require('gulp-angular-templatecache');

var concat = require('gulp-concat');
var addStream = require('add-stream');

gulp.task('browserify', function() {
  return gulp.
    src('./client/app.js').
    //pipe(addStream('./client/templates.js')).
    //pipe(concat('app.js')).
    pipe(browserify()).
    pipe(gulp.dest('./public'));
});

gulp.task('templates', function () {
  return gulp.src('./public/templates/*.html')
    .pipe(angularTemplateCache({module:'templates', root: '/templates/'}))
    .pipe(gulp.dest('./client'));
});

gulp.task('build',['templates', 'browserify']);
templates.js – built by gulpfile.js
angular.module("templates").run([
    "$templateCache",  function($templateCache) {
     $templateCache.put("/templates/system.html","<div class=\"menu\">\n    {{data}}\n</div>");
    }
]);
app.js – defines template module and adds it to app list of dependencies
var controllers = require('./controllers');
var directives = require('./directives');
var _ = require('underscore');

// this never changes
angular.module('templates', []);

// templates added as dependency
var components = angular.module('app', ['ng','templates']);

_.each(controllers, function(controller, name)
{ components.controller(name, controller); });

_.each(directives, function(directive, name)
{ components.directive(name, directive); });

require('./templates')
The mocha test
describe('menuDirective', function() {
  var mockScope;
  var compileService;
  var template;

  beforeEach(angular.mock.module("app"));
  
  beforeEach(angular.mock.module('ngMockE2E'));

  beforeEach(inject(function ($rootScope) {
      scope = $rootScope.$new();
    }));
    
  it('should exist', inject(function ($compile, $templateCache) {
    element = angular.element('<system></system>');
    compiledElement = $compile(element)(scope); 

    // APP - app.js used templates dependency which loads template
    // into templateCache
    // TEST - this test pulls template from templateCache 
    template = $templateCache.get('/templates/system.html'); 
 
    scope.$digest();

    var dataFromHtml = compiledElement.find('.menu').text().trim();
    var dataFromScope = scope.data;
    
    console.log(dataFromHtml + " == " + dataFromScope);

    expect(dataFromHtml).to.equal(dataFromScope);
  }));
});

The Test Results

The test results for each of the 3 projects is checked in to the github project. Karma ran with debug turned on so that the http and file requests could be validated. When you look at the testResult.log files (1 in each of the 3 subdirectories), you want to make sure that the http and file requests that karma made were actually successful. The lines with ‘Fetching’, ‘Requesting’, and ‘(cached)’ are the important lines before the test runs.

36m26 04 2016 11:42:09.318:DEBUG [middleware:source-files]: [39mFetching /home/dina/repos/AngularDirectiveKarma/directiveTemplate/test/test.directive.template.js
[36m26 04 2016 11:42:09.318:DEBUG [middleware:source-files]: [39mRequesting /base/public/app.js?6da99f7db89b4401f7fc5df6e04644e14cbed1f7 /
[36m26 04 2016 11:42:09.318:DEBUG [middleware:source-files]: [39mFetching /home/dina/repos/AngularDirectiveKarma/directiveTemplate/public/app.js
[36m26 04 2016 11:42:09.319:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/node_modules/mocha/mocha.js
[36m26 04 2016 11:42:09.329:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/node_modules/karma-mocha/lib/adapter.js [36m26 04 2016 11:42:09.331:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/test/test.directive.template.js
[36m26 04 2016 11:42:09.333:DEBUG [web-server]: [39mserving (cached): /home/dina/repos/AngularDirectiveKarma/directiveTemplate/public/app.js

During the test, you should see that the $scope.data value (different in each of the 3 tests) is equated to the compiled html as expected, for example, “user = user”:
LOG: 'user == user'
And finally, that the test passed:
Executed 1 of 1 SUCCESS

Friday, March 4, 2016

Capturing a Stripe Credit Card Charge

In this article, I'll show you the JSON objects and Angular/Node code to capture a credit card with the Stripe service. The code for this example project is on GitHub. It is a working project so the code may not be exactly like this article by the time you find it.

Introduction
Capturing credit card information for products, services, and subscriptions is easy with many tools provided by credit card processing companies. Stripe provides an html button that pops up a form to collect credit card information. It is incredibly simple and straightforward.



You can control the form to collect more data. You don't need to know how to program it, and it works. Yeah! This article doesn't cover the easy route of button and pop up because Stripe did a great job of that on their website.

If you would prefer to control the credit card experience from beginning to end, you may choose to build your own form and process the data you collect to meet your own business needs. If that is the case read on.

The Credit Card Token/Charge Experience
A credit card number should never make it to your server – where you are liable for fraud or theft. Instead, the client-side code sends the credit card information to the processor, in this case Stripe, and Stripe sends back a token to the client-side code.

The token isn't a charge to the card but more a promise to charge the card in the future. For the remainder of the customer's experience, you do not use the credit card information, only the token. The token travels with the rest of the customer information (name, address, items purchased) to the server. The server will make the actual charge and receive the resulting information from Stripe.

With stripe, all you really need is a way to make an https request to their Api server, passing JSON, then receive the JSON request. That is true for both the token request and the charge request. Stripe responses with JSON.

JSON is independent of the Technology
The rest of this article is broken into two different sections. The first section will review the JSON objects which have some requirements and a lot of optional fields. You can use any technology you want to make these, including curl. The second section will cover a specific code base of Angular on the client and Node/Express on the server. The code is very simple so there is little or no styling, validation, or error management.

The Stripe Key
Stripe needs your Publishable key to create the token. You can find this on the Stripe dashboard in your account settings.



If you are using curl, you can pass the key along with the token request. If you are passing it from code, it will need to be set before the token request. The curl example is:

curl https://api.stripe.com/v1/tokens \  -u sk_test_WqKypPUzUwdHSgZLT3zWZmBq: \  -d card[number]=4242424242424242 \  -d card[exp_month]=12 \  -d card[exp_year]=2017 \  -d card[cvc]=123

The key is the value after the –u param: sk_test_…

Notice that only the key and card information is passed in the above curl. You can and should capture and pass the billing address. This allows you to see the address in the Stripe Dashboard when the card is finally made.



Notice that you only see the last 4 digits of the credit card.

JSON to create a Stripe Token
The JSON object to request a token contains the credit card information. It should also contain the billing information for the customer.  A full list of card key/value pairs is listed in the Stripe docs.
client stripe token request =  
{   
   "number":"4242424242424242", 
   "cvc":"123", 
   "exp_month":"11", 
   "exp_year":"2016", 
   "name":"Barbara Jones", 
   "address_city":"Seattle", 
   "address_line1":"5678 Nine Street", 
   "address_line2":"Box 3", 
   "address_country":"USA", 
   "address_state":"WA", 
   "address_zip":"98105" 
}

The response will be a status code, success is 200 with a json object of data including the token.

client stripe token response.response = {   
   "id":"tok_17jylQJklCPSOV9aLkiy5879", 
   "object":"token", 
   "card":{   
      "id":"card_17jylQJklCPSOV9ahtXhPVB8", 
      "object":"card", 
      "address_city":"Seattle", 
      "address_country":"USA", 
      "address_line1":"5678 Nine Street", 
      "address_line1_check":"unchecked", 
      "address_line2":"Box 3", 
      "address_state":"WA", 
      "address_zip":"98105", 
      "address_zip_check":"unchecked", 
      "brand":"Visa", 
      "country":"US", 
      "cvc_check":"unchecked", 
      "dynamic_last4":null, 
      "exp_month":11, 
      "exp_year":2016, 
      "funding":"credit", 
      "last4":"4242", 
      "metadata":{     
      }, 
      "name":"Barbara Jones", 
      "tokenization_method":null 
   }, 
   "client_ip":"73.11.000.147", 
   "created":1456782072, 
   "livemode":false, 
   "type":"card", 
   "used":false 
}

For the rest of the credit card transaction, use the token only. You will need to pass it when you charge the customer's credit card.

JSON for a successful Stripe Charge
Now that you have the token, you can create a JSON object to represent the credit card charge.

client stripe token response.status = 200
server stripe charge request object = {    
   "amount":1000,  
   "currency":"usd",  
   "source":"tok_17jylQJklCPSOV9aLkiy5879",  
   "description":"Donation for XYZ",  
   "metadata":{    
      "ShipTo":"Bob Smith",  
      "BillTo":"Barbara Jones"  
   },  
   "receipt_email":"bob@company.com",  
   "statement_descriptor":"MyStripeStore",  
   "shipping":{    
      "address":{    
         "city":"Seattle",  
         "country":"USA",  
         "line1":"1234 Five Lane",  
         "line2":"Floor 2",  
         "postal_code":"98101",  
         "state":"WA"  
      },  
      "name":"Bob Smith",  
      "phone":""  
   }  
} 

Make sure the statement_descriptor has meaningful information to figure out the charge was valid– it shows up on the customer's bill. If you have information important to the transaction that Stripe doesn't collect, put those values in the metadata key. You can retrieve this information from Stripe to reconcile or fulfill the transaction on your end. Think of it as a backup – if your system goes down, Stripe still has enough information for you to rebuild the transaction.  The amount includes dollars and cents but no decimal. So "1000" is ten dollars, $10.00.

If you are new to Stripe, you may not be using their advanced features but if you collect the data now, converting to and seeding some of the advanced feature objects will be easy.

A successful charge response returns an null error object and a result object.

server stripe charge response.charge = {    
   "id":"ch_17jylQJklCPSOV9aHyof0XV8",  
   "object":"charge",  
   "amount":1000,  
   "amount_refunded":0,  
   "application_fee":null,  
   "balance_transaction":"txn_17jylQJklCPSOV9afi1bpfz5",  
   "captured":true,  
   "created":1456782072,  
   "currency":"usd",  
   "customer":null,  
   "description":"Donation for XYZ",  
   "destination":null,  
   "dispute":null,  
   "failure_code":null,  
   "failure_message":null,  
   "fraud_details":{    
   },  
   "invoice":null,  
   "livemode":false,  
   "metadata":{    
      "ShipTo":"Bob Smith",  
      "BillTo":"Barbara Jones"  
   },  
   "order":null,  
   "paid":true,  
   "receipt_email":"bob@company.com",  
   "receipt_number":null,  
   "refunded":false,  
   "refunds":{    
      "object":"list",  
      "data":[    
      ],  
      "has_more":false,  
      "total_count":0,  
      "url":"/v1/charges/ch_17jylQJklCPSOV9aHyof0XV8/refunds"  
   },  
   "shipping":{    
      "address":{    
         "city":"Seattle",  
         "country":"USA",  
         "line1":"1234 Five Lane",  
         "line2":"Floor 2",  
         "postal_code":"98101",  
         "state":"WA"  
      },  
      "carrier":null,  
      "name":"Bob Smith",  
      "phone":"",  
      "tracking_number":null  
   },  
   "source":{    
      "id":"card_17jylQJklCPSOV9ahtXhPVB8",  
      "object":"card",  
      "address_city":"Seattle",  
      "address_country":"USA",  
      "address_line1":"5678 Nine Street",  
      "address_line1_check":"pass",  
      "address_line2":"Box 3",  
      "address_state":"WA",  
      "address_zip":"98105",  
      "address_zip_check":"pass",  
      "brand":"Visa",  
      "country":"US",  
      "customer":null,  
      "cvc_check":"pass",  
      "dynamic_last4":null,  
      "exp_month":11,  
      "exp_year":2016,  
      "fingerprint":"uOlT1SgxEykd9grd",  
      "funding":"credit",  
      "last4":"4242",  
      "metadata":{    
      },  
      "name":"Barbara Jones",  
      "tokenization_method":null  
   },  
   "source_transfer":null,  
   "statement_descriptor":"MyStripeStore",  
   "status":"succeeded"  
} 

The two items you want to look for is status and paid.

Stripe keeps a log of your transactions as JSON objects including the request and response of both the token and the charge. You may want to store all the information on your server, but if you don't, you can get it from Stripe when you need to.

JSON for a failed Stripe Charge
If you customer's charge fails, the JSON for the charge is a bit different. You will want to present the customer with the meaningful error message to help them correct the problem on their end and complete the transaction successfully.

The charge object will be null and the status object will have the error message.

server stripe charge response.status = {    
   "type":"StripeCardError",  
   "stack":"Error: Your card was declined.\n    at Error._Error (/Users/dfberry/repos/stripe-express-angular/node_modules/stripe/lib/Error.js:12:17)\n    at Error.Constructor (/Users/dfberry/repos/stripe-express-angular/node_modules/stripe/lib/utils.js:105:13)\n    at Error.Constructor (/Users/dfberry/repos/stripe-express-angular/node_modules/stripe/lib/utils.js:105:13)\n    at Function.StripeError.generate (/Users/dfberry/repos/stripe-express-angular/node_modules/stripe/lib/Error.js:54:14)\n    at IncomingMessage.<anonymous> (/Users/dfberry/repos/stripe-express-angular/node_modules/stripe/lib/StripeResource.js:138:39)\n    at emitNone (events.js:72:20)\n    at IncomingMessage.emit (events.js:166:7)\n    at endReadableNT (_stream_readable.js:905:12)\n    at nextTickCallbackWith2Args (node.js:455:9)\n    at process._tickCallback (node.js:369:17)",  
   "rawType":"card_error",  
   "code":"card_declined",  
   "message":"Your card was declined.",  
   "raw":{    
      "message":"Your card was declined.",  
      "type":"card_error",  
      "code":"card_declined",  
      "charge":"ch_17jz4rJklCPSOV9aDPh2eooP",  
      "statusCode":402,  
      "requestId":"req_7zz28B9jlOpH1x"  
   },  
   "requestId":"req_7zz28B9jlOpH1x",  
   "statusCode":402  
} 

The message contains the information to display to the customer. While the transaction didn't complete, the log in the Stripe Dashboard will contain the same information. For a complete list of issues, look at the Stripe api for errors.

The Angular and Node/Express application
In order to use Stripe on the client, you need to pull in the Stripe javascript library.

You can get the Stripe library for the client from stripe

<script type="text/javascript" src="https://js.stripe.com/v2/"></script>

In order to use Stripe on the server, you need to install Stripe from NPM

npm install stripe –save-dev

The Donation Form
The example is a donation form. It collects the customer information and allows the customer to choose the donation amount. A shipping address is also collected, for a thank you card back to the customer.



The web page has very little styling, no validation, and only a single error message if the credit card charge is denied.

Angular Client Code
The Angular code includes a directive to display the html, a controller to collect the customer's information, and a service to post to the server. The token creation and charge are both handled in the service one call right after the other. This is definitely not ideal for a real-world situation.

The card, card, and customer information are kept in JSON objects in the models.js file. The Stripe publishable key is in config.js along with the cart item name. When the token and charge are made, the JSON object that Stripe expects for each of these is created.

As all of the Stripe work is in the service, that is the Angular code to review. The complete charge as 3 separate objects, cart, card, and customer.

exports.$myservice = function($http,$myappconfig){  
    var commit = function (completeCharge, callback){  
        var result = {};  
        // my stripe test key  
        Stripe.setPublishableKey($myappconfig.stripePublishableKey);  
        // credit card info passed   
        // billing address passed as part of charge.card  
        Stripe.card.createToken(completeCharge.card, function(status, response) {  
             if (status.error) {  
                console.log("stripe token not created");  
                result.error = status.error;  
                callback(result);  
            }   
            var chargeRequestObject = {   
                    stripeToken: response.id,   
                    cart: completeCharge.cart ,   
                    customer: completeCharge.customer  
            };  
            // token (not credit card) passed  
            // shipping address passed in charge.customer  
            $http.  
                post('/api/v1/checkout', chargeRequestObject).  
                then(function(data) { //success  
                    callback(null, data);  
                },  
                function(response){ //failure  
                    callback(response.data.error, response);  
                });  
            });  
        }  
    return {  
      commit: commit  
    };      
}

Node Server Code
The server code is a node app using Express. It only serves the html page above and has an api to process a charge -- very lean in order to drop it into any other web site.  Morgan is used for logging and Wagner is used for dependency injection.  You can start the app with npm start or node server/index.js.

Index.js
var express = require('express');  
var wagner = require('wagner-core');  
var path = require('path');  
require('./dependencies')(wagner);  
var app = express();  
app.use(require('morgan')());  
app.get(['/'], function (req, res) {res.sendFile(path.join(__dirname + '/../public/default.html'));});  
app.use('/api/v1', require('./api')(wagner));  
app.use('/public', express.static(__dirname + '/../public', { maxAge: 4 * 60 * 60 * 1000 /* 2hrs */}));  
app.listen(3000);  
console.log('Listening on port 3000!'); 

The dependencies file creates the wagner dependency injection for the config and stripe objects.

Dependencies.js
var fs = require('fs');  
var Stripe = require('stripe');  
var configFile = require('./config.json');  
module.exports = function(wagner) {  
  wagner.factory('Stripe', function(Config) {  
    return Stripe(Config.stripeKey);  
  });  
  wagner.factory('Config', function() {  
    return configFile;  
  });  
};

The config.json is simple the configuration json object:

{  
  "stripeKey": "sk_test_WqKypPUzUXXXXXXXT3zWZmBq",  
  "stripePublishableClientKey": "pk_test_ArJPMXXXXlF2Ml4m4e8ILmiP",  
  "Characters22_StoreName": "MyStripeStore"  
}

The main Stripe code of the application is in the api.js file to process the Stripe charge.
module.exports = function(wagner) {  
  var api = express.Router();  
  api.use(bodyparser.json());  
  /* Stripe Checkout API */  
  api.post('/checkout', wagner.invoke(function(Stripe) {  
    return function(req, res) {  
        // https://stripe.com/docs/api#capture_charge  
        // shipping name is in the metadata so that it is easily found on stripe's website   
        // statement_descriptor & description will show on credit card bill  
        // receipt_email is sent but stripe isn't sending receipt - you still have to do that  
        // shipping is sent only so you can pull information from stripe   
        // metadata: 20 keys, with key names up to 40 characters long and values up to 500 characters long  
        var stripeCharge = {  
            amount: req.body.cart.totalprice,  
            currency: 'usd',  
            source: req.body.stripeToken,  
            description: req.body.cart.name,  
            metadata: {'ShipTo': req.body.customer.shipping.name, 'BillTo': req.body.customer.billing.name},  
            receipt_email: req.body.customer.email,  
            statement_descriptor: config.Characters22_StoreName,  
            shipping: req.body.customer.shipping   
        };  
        console.log("server stripe charge request object = " + JSON.stringify(stripeCharge)+ "\n");  
        // Charge the card NOW  
        Stripe.charges.create(stripeCharge,function(err, charge) {  
            console.log("server stripe charge response.err = " + JSON.stringify(err) + "\n");        
            console.log("server stripe charge response.charge = " + JSON.stringify(charge) + "\n");   
            if (err) {  
                return res.  
                status(status.INTERNAL_SERVER_ERROR).  
                json({ error: err.toString(), charge: err.raw.charge, request: err.requestId, type : err.type});  
            }  
            return res.json(charge);  
        });   
     };  
  }));  
  return api;  
}; 

Summary
Captures credit cards with Stripe is simple. You can use their button and form or create your own system. If you create your own client and server, understanding the JSON objects for the token request and the charge request is important. You can pass just the minimum or all the data you have. Create the token on the client and pass the token and charge details to the server to complete the transaction.