Sunday, May 7, 2017

Agile Manifesto -- Some 16 years later

"In 2001, 17 individuals gathered in the Wasatch mountains of Utah to find common ground around Agile. After much skiing, talking, relaxing, and eating, they arrived at four common values that led to the development of the Agile Manifesto."[source] [Wikipedia]

Going to AgileManifecto.org, we see just 14 people listed, what appears to be their feeling on what promised so much hope some 15 years ago.



  1. Mike Beedle 
  2. Arie van Bennekum
  3. Alistair Cockburn
    1. " “Agile has become overly decorated. Let’s scrape away those decorations for a minute, and get back to the center of agile. The center of agile is … ” [2015]
  4. Ward Cunningham
    1. " "I have seen my ideas diluted as they diffused through the industry". You said "I’d much rather move to the next idea than struggle to keep the last idea pure." [2011]
  5. Martin Fowler
  6. Jim Highsmith - 
    1. Don’t “Control” Agile Projects
    2. Agile Bureaucracy: When Practices become Principles
  7. Andrew Hunt The Failure of Agile (2015)
  8. Ron Jeffries
    1. Do you want Crappy Agile? [2016] " it encourages you to track “metrics”. Do you want Crappy Agile? That’s how you get Crappy Agile, at least far too often."
    2. "Scrum is good, when done as intended. Otherwise it can be oppressive and dangerous to developers. Let's study: Defense Against the Dark Arts of Scrum." Dark Scrum
  9. Jon Kern
    1. "The only thing that has me mildly torn--and only mildly, because "to each his own" and "what-evs" come to mind--is the craze surrounding Scrum. Scrum, scrum, scrum, scrum. Part of me is happy that there are legions out there turning the sod under, trying to sow the agile seeds via Scrum. Another part of me thinks it is like Corn-to-Ethanol. A waste of energy for the dumbfounded among us, spurred on by lots of lobbying." [2011]
    2. Have not posted on his blog about agile since 2011
  10. Brian Marick "In his keynote at the Agile Development Practices conference, Brian Marick described values missing from the Agile Manifesto. His view is that the Manifesto was essentially a marketing document, aimed at getting business to give agile a chance." [2008]
    1. "As Agile moves into bigger companies and into less adventurous ones, those undocumented values are getting watered down. If that continues, I’m afraid Agile will be this decade’s fad that changes nothing."
  11. Robert C. Martin (aka Uncle Bob) moved on to "Clean Code" 
  12. Ken Schwaber owns Scrum.org
  13. Jeff Sutherland CEO of Scrum Inc
  14. Dave Thomas 

    Agile is Dead (Long Live Agility) - PragDave

Missing are: [source]

Question - Agile is good if selling it is your bread and butter

Many agile signatures appear to have become disillusioned by how Agile is commonly being adapted in the industry. Some people have stopped writing and posting on it. Others have proposed derivatives of it in the hope of getting back to core values. Others are making a good living from preaching the gospel of agile.

My view?  Simple, I started using RAD back in 1979 and been using the concepts (without the dogma) for the last 37 years.  The best implementation have been where the product owner/customer was very technical and detail orientated and put in the needed hours. The worst cases were scrum was magically expected to make management easy and not require their heavy engagement or hours  with the products they owned or manage.

I like the concepts and practice them instinctively, I hate the "rote-application of the manifesto" 

Monday, May 1, 2017

Running Tests in Docker for Front-end Developers (ng2+)

Unit tests (karma/jasmine) & End-to-end tests (protractor)

If you are comfortable with Docker and Angular 2, all you need is the Dockerfile. If you are new to testing in Docker, this short article will bring you up to speed. If you need more help getting up to speed with Docker, begin with my previous article: Docker for Angular 2 devs.

Installation 

In order to develop Angular 2+ in a container, you need to install Docker on a computer or VM. Docker takes a bit of memory and can take a lot of space so the biggest box you can give it is best. Once Docker is installed, start is up.

Make sure all your docker commands are run from the folder that contains the Dockerfile.

Dockerfile 

Docker works by reading the description of a Dockerfile (or several in conjunction), to build out an image. Once the image is running, it is called a container.

This particular Dockerfile is based on a Docker image that already has headless chrome, markadams/chromium-xvfb-js:7. There are several Dockerfiles at Docker hub for headless chrome. If this base image or my additions do not work, feel free to play around with other Dockerfiles. You will find references/websites at the bottom of this article.

What Does this Dockerfile do 

This Dockerfile installs the angular2-webpack-starter (and all dependencies). I use it as a development environment that I connect to and run the tests myself as I develop. I'm using the angular2-webpack-starter as my base test that the Dockerfile has all the needed dependencies to complete both the unit tests and the end-to-end tests successfully.

If the starter can run both types of tests, it should be fairly straightforward to add your own repository to the container for development and testing. The container has the full stack for this: Git, NPM, Yarn, Node, and Angular+, ng-cli, Webpack, Karma, Jasmine, Protractor, http-server.

Why use the angular2-webpack-starter?

I wanted an Angular 2+ project that used as much of the same stack as my current projects that also had unit tests and e2e tests. In order to validate I have the right Dockerfile configuration, all I have to do is run both types of tests in this repository. If they succeed, my own tests for my own projects should succeed as well.

Can I see it working?  

Yes, I have a short 4 minute video showing it from building the docker image to the final e2e test run. I've cut out 11 minutes of Docker building the image as there is a lot of installation of dependencies.



Using as a Development Environment 

I use this container as my main development & testing environment. I share my local repository folder into the container with the –v param and expose a small range of ports to allow for several front-end and api servers to run at once.

The angular2-webpack-starter runs on port 3000 so I open 3000-3005. This allows you to use your host computer to access the website (http://localhost:3000) as long as the port isn't in use on your host machine.

What are the important parts of the Dockerfile?  

There are definitely packages and settings in the Dockerfile you may not use, but there are a few you need to keep.

Protractor needs to know where the Chrome bin directory is. If you remove that environment variable, Protractor will not be able to run the tests. Protractor also needs the default-jre.

Getting karma/jasmine tests working in a container using headless chrome seems to be easy compared to protractor e2e tests. If you change the file, always use the protractor e2e test run as the verification that the change did no harm.

Git Commit Hash 

Since the repositories for both my Dockerfile and the angular2-webpack-starter will change over time, I'm noting the git commit hashes of the versions I used.

https://github.com/AngularClass/angular2-webpack-starter
commit e9521a42  - Sat Apr 22 19:27:33 2017 -0400

https://github.com/dfberry/DockerFiles/blob/master/headless-chromium/Dockerfile
commit 38b7554c Sun Apr 30 13:23:17 2017 -0700

References 

https://github.com/mark-adams/docker-chromium-xvfb
https://hub.docker.com/r/yukinying/chrome-headless/
https://hub.docker.com/r/justinribeiro/chrome-headless/

Thursday, October 20, 2016

Docker for Angular 2 devs

Docker is a Virtual Environment
Docker containers are great for adding new developers to existing projects, or for learning new technologies without polluting your existing developer machine/host. Docker allows you to put a fence around the environment while still using it.

Why Docker for Angular 2?
Docker is an easy way to get up and going on a new stack, environment, tool, or operating system without having to learn how to install and configure the new stack. A collection of docker images are available from Docker Hub ranging from simple to incredibly complex -- saving you the time and energy.

Angular 2 examples frequently include a Dockerfile in the repository which makes getting the example up and running much quicker -- if you don't have to focus on package installation and configuration.

The base Angular 2 development stack uses Node, TypeScript, Typings, and a build system (such as SystemJs or Webpack). Instead of learning each stack element before/while learning Angular 2, just focus on Angular 2 itself -- by using a Dockerfile to bring up a working environment.

The repositories for Angular 2 projects will have a package.json file at the root which is common for NodeJs/NPM package management projects. The Docker build will install the packages in the package management system as part of the build. The build can also transpile the typescript code , and start a static file web server -- if the package.json has a start script.

In order to get a new development environment up and a new project in the browser, you just need to build the Dockerfile, then run it. Running these two commands at the terminal/cli saves you time in find and learning the Angular 2 stack, and then building and running the project.

The Angular 2 Quickstart
For this article, I use the Angular 2 Quickstart repository including the Dockerfile found in the repository.

I use a Macintosh laptop. If you are using a Windows-based computer/host, you may have more or different issues than this article.

Docker via Terminal/Cli
I prefer the code-writing environment and web browser already installed and configured on my developer laptop/host. I configure the Docker container to share the hosted files. The changes are reflected in the container – and I run the Angular 2 project in watch mode so the changes immediately force a recompile in the container.

Viewing the Angular Project in a Browser
Since the Angular 2 project is a website, I access the container by the port and map the container's port to the host's port – so access to the running Angular project is from a web browser on the host laptop with http://localhost:3000.

Install Docker
Before you install Docker, make sure you have a bit of space on the computer. Docker, like Vagrant
and VirtualBox, uses a lot of space.

Go to Docker and install it. Start Docker up.

Check Docker
Open a terminal/cli and check the install worked and Docker started by requesting the Docker version


docker –v
>Docker version 1.12.1, build 6f9534c 

If you get a docker version as a response, you installed and started Docker correctly.

Images and Containers
Docker Images are defined in the Dockerfile and represent the virtual machine to be built. The instantiation of the image is a container. You can have many containers based on one image.

Each image is named and each container can also be named. You can use these names to indicate ownership (who created it), as well as base image (node), and purpose (xyzProject).

Pick a naming schema for your images and containers and stick with it.

I like to name my images with my github name and the general name such as dfberry/quickstart. I like to name the containers with as specific a name as possible such as ng2-quickstart.

The list of containers (running or stopped) shows both names which can help you organize find the container you want.

The Angular 2 Docker Image
The fastest way to get going with Docker for Angular 2 projects is to use the latest node as your base image -- which is also what the Angular 2 quickstart uses.

The image has the latest node, npm, and git. Docker hub hosts the base image and Node keeps it up to date.

Docker's philosophy is that the containers are meant to execute then terminate with the least privileges possible. In order to make a container work as a development container (i.e. stay up and running), I'll show some not-best-practice choices. This will allow you to get up and going quickly. When you understand the Docker flow, you can implement your own security.

The Docker Images
Docker provides no images on installation. I can see that using the command


docker images 

When I build the nodejs image, it will appear in the list with information about the image.



For now, the two most important columns are the REPOSITORY and IMAGE ID. The REPOSITORY field is the image name I used to build the image. My naming schema indicates my user account (dfberry) and the base image or purpose (node). This helps me find it in the image list.

The IMAGE ID is the unique id used to identify the image.

The Dockerfile
In order to create a docker image, you need a Dockerfile (notice the filename has no extension). This is the file the docker cli will assume you want to use. For this example, the Dockerfile is small. It has the following features:
  • creates a group
  • creates a user
  • creates a directory structure with appropriate permissions
  • copies over the package.json file from the host
  • installs the npm packages listed in the package.json
  • runs the package.json's "start" script – which should start the website

For now, make sure this is the only Dockerfile in the root of the project, or anywhere below the root.


# To build and run with Docker:
#
#  $ docker build -t ng-quickstart .
#  $ docker run -it --rm -p 3000:3000 -p 3001:3001 ng-quickstart
#
FROM node:latest

RUN mkdir -p /quickstart /home/nodejs && \
groupadd -r nodejs && \
useradd -r -g nodejs -d /home/nodejs -s /sbin/nologin nodejs && \
chown -R nodejs:nodejs /home/nodejs

WORKDIR /quickstart
COPY package.json typings.json /quickstart/
RUN npm install --unsafe-perm=true

COPY . /quickstart
RUN chown -R nodejs:nodejs /quickstart
USER nodejs

CMD npm start

The nodejs base image will install nodejs, npm and git. The image will just be used for building and hosting the Angular 2 project.

If you have scripts that do the bulk of your build/startup/run process, change the Dockerfile to copy that file to the container and execute it as part of the build.

Build the Image
Usage: docker build [OPTIONS] PATH | URL | -

In order to build the image, use the docker cli.


docker build –t <user>/<yourimagename> .
Example $: docker build –t dfberry/ng-quickstart .


If you don't want to annotate the user, just leave that off.


docker build –t <yourimagename> .
Example $: docker build –t ng-quickstart .

Note: the '.' at the end of the string is the url/location of the Dockerfile. I could have used a Github repository url instead of the local folder.

In the above examples, the REPOSITORY name is 'ng-quickstart'. If you don't use the –t naming param, your image will have a name of <none> which is annoying when they pile up on a team server.

The build will give you some feedback to let you know how it is going.


Sending build context to Docker daemon 3.072 kB 
Step 1 : FROM node:latest 

... 

Removing intermediate container 2cb50f334393 
Successfully built 1265b22b5b90

Since the build can return a lot of information, I didn't include the entire response.

The build of the quickstart takes less than a minute on my Mac.

The last line gives you the IMAGE ID. Remember to view all docker images after building to check it worked as expected.


docker images

Run the Container
Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...] 

Now that the image is built, I want to run the image to see the website.

If you don't have an Angular 2/Typescript website, use the ng2 Quickstart.

Run switches
The run command has a lot of switches and configurations. I'll walk you through the choices for this container.

I want to name the container so that I remember the purpose. This is optional but helpful when you have a long list of containers.


--name ng2-quickstart 

I want to make sure the container's web ports are matched to my host machine's port so I can see the website as http://localhost:3000. Make sure the port isn't already in use on your host machine.


-p 3000:3000 

I want to map my HOST directory (/Users/dfberry/quickstart to the container's directory (/home/nodejs/quickstart) that was created in the build so I can edit on my laptop and the changes are reflected in the container. The /home/nodejs/quickstart directory was created as part of the Dockerfile.


-v /Users/dfberry/quickstart/:/home/nodejs/quickstart 

I want the terminal/cli to show the container's responses including transpile status and the file requests.


-it 


The full command is:


docker run -it -p 3000:3000 -v /Users/dfberry/quickstart:/home/nodejs/quickstart --name ng2-quickstart dfberry/ng-quickstart

Notice the image is named dfberry/ng-quickstart while the container is named ng2-quickstart.

Run the "docker run" command at the terminal/cli.

The container should be up and the website should be transpiled and running.


At this point, you should be able to work on the website code on your host with your usual editing software and the changes will reflect in the container (re-transpile changes) and in the web browser.

List Docker Containers
In order to see all the containers, use


docker ps -a

If you only want to see the running containers, leave the -a off.



docker ps



At this point, the easy docker commands are done and you can work on the website and forget about Docker for a while. When you are ready to stop the container, stop Docker or get out of interactive mode, read on.

Interactive Mode (-it) versus Detached Mode (-d)
Interactive Mode means the terminal/cli shows what is happening to your website in the container. Detached mode means the terminal/cli doesn't show what is happening and the terminal/cli returns to your control for other commands on your host.

To move from interactive to detached mode, use control + p + control + q.

This leaves the container up but you have no visual/textual feedback about how the website is working from the container. You can use the developer tools/F12 in the browser to get a sense, but won't be able to see http requests and transpiles.

You are either comfortable with that or not.

If you want the interactive mode and the website transpile/http request information, don't exit interactive mode. Instead, use control + c. This command stops and removes the container from Docker, but doesn't remove the image. You can re-enter interactive mode with the same run command above.

If you are more comfortable in detached mode, where the website gives transpiles and http request information via a different method such as logging to a file or cloud service, change the docker run command.

Instead of using –it as part of the "docker run" command, use –d for detached mode.

Exec Mode to run commands on container
Usage: docker exec [OPTIONS] CONTAINER COMMAND [ARG...]

When you want to connect to the container, you the same -it for interactive mode but with "docker exec."  The end command tells the docker container what environment to enter in the container -- such as the bash shell.


docker exec –it ng2-quickstart /bin/bash

You can log in as root if you need elevated privileges.


docker exec –it –u root ng2-quickstart /bin/bash 

The terminal/cli should now show the prompt changed to indicate you are now on the container:


nodejs@faf83c87c12e:/quickstart$  

When you are done running the commands, use control + p + control + q to exit. The container is still running.

Sudo or Root
In this particular quickstart nodejs docker container, sudo has not been installed. Sudo may be your first choice and you can install. Or you could use the "docker exec" with root. Either way has pros and cons.

Stopping and Starting the Container
When you are done with the container, you need to stop it. You can stop, and restart it as you need by container id or name.


docker stop ng2-quickstart 
docker stop 7449222ec26b 

docker start ng2-quickstart 
docker start 7449222ec26b 

Stopping the container may take some time – be patient. Mine takes up to 10 seconds on my Mac. When you restart the container, it is in detached mode. If you really want interactive mode, remove the container, and use docker run again with –it.

Cleanup
Make sure to stop all containers when they are not needed. When you are done with a container, you can remove it


docker rm –fv ng2-quickstart 

When you are done with an image, you can remove that as well


docker rmi ng-quickstart 

Stop Docker
Remember to stop Docker when you are done with the containers for the moment or day.

Monday, September 5, 2016

An interesting Interview Question: Fibonacci Sequence

Write a function to calculate the nth Fibonacci Sequence is a common interview question and often the solution is something like

 

int Fib(int n)

{

   if(n < 1) return 1;

   return Fib(n-1) + Fib(n-2);

}

 

The next question is to ask for n=100, how many items will be on the stack. The answer is not 100 but actually horrible! It is closer to 2^100.

take the first call – we start a stack on Fib(99) and one on Fib(98). There is nothing to allow Fib(99) to borrow the result of Fib(98).  So one step is two stack items to recurse.  Each subsequent call changes one stack item into 2 items.   For example

  • 2 –> call [Fib(1), Fib(0)]
  • 3 –> calls [ Fib(2)->[Fib(1), Fib(0)], Fib(1) –> Fib(0) ]
  • 4 –> calls [ Fib(3)->[[[ Fib(2)->[Fib(1), Fib(0)], Fib(1) –> Fib(0) ]], Fib(2)->[Fib(1), Fib(0)], Fib(1) –> Fib(0) ]

Missing this issue is very often seen with by-rote developers (who are excellent for some tasks).

 

A better solution is to cache the values as each one is computed – effectively creating a lookup table. You are trading stack space for memory space.

 

Placing constraints on memory and stack space may force the developer to do some actual thinking. A solution that conforms to this is shown below

 

  private static long Fibonacci(int n) {
        long a = 0L;
        long b = 1L;
        for (int i = 31; i >= 0; i—)  //31 is arbitrary, see below

        {

            long d = a * (b * 2 - a);
            long e = a * a + b * b;
            a = d;
            b = e;
            if ((((uint)n >> i) & 1) != 0) {
                long c = a + b;
                a = b;
                b = c;
                }
           }
        return a;
    }

 

The output of the above shows what is happening  and suggests that the ”31”  taking the log base 2 of N can likely be done to improve efficiency

image

for 32:

image

for 65

image

for 129

image

 

What is the difference in performance for the naive vs the latter?

I actually did not wait until the naive solution finished… I aborted at 4 minutes

image

The new improved version was 85 ms, over a 3000 fold improvement.

Take Away

This question:

  1. Identify if a person knows what recursion is and can code it.
  2. Identify if he understands what the consequence of recursion is and how it will be executed(i.e. think about what the code does)
    1. Most recursion questions are atomic (i.e. factorial) and not composite (recursion that is not simple)
  3. Is able to do analysis of a simple mathematical issue and generate a performing solution.

Sunday, August 28, 2016

Apple Store Passbook UML Diagrams and Error Messages

While working on a recent project, a major stumbling block was a lack of clear documentation of what happened where. This was confirmed when I attempted to search for some of the messages returned to the Log REST points by iPhone.. There were zero hits!

 

image

 

In terms of a Store Card, let us look at the apparent Sequence Diagram

 

image

 

Log Errors Messages Seen and Likely Meaning

  • Passbook Inactive or Deleted or some one changed Auth Token
    • [2016-08-28 11:57:01 -0400] Unregister task (for device ceed8761e584e814ed4fe73cbb334ee9, pass type pass.com.reddwarfdogs.card.dev, serial number 85607BFE98D91A-765F7B05-D5E4-4B32-B16D-69C2038EF522; with web service url https://llc.reddwarfdogs.com/passbook) encountered error: Authentication failure
    • [2016-08-28 20:44:25 +0700] Register task (for device 19121d6b570b31a3fa56dbd45411c933, pass type pass.com.reddwarfdogs.card.dev, serial number 85607BFE98D91A-765F7B05-D5E4-4B32-B16D-69C2038EF522; with web service url https://llc.reddwarfdogs.com/passbook) encountered error: Authentication failure
    • [2016-08-24 10:04:38 +0800] Web service error for pass.com.reddwarfdogs.card.dev (https://llc.reddwarfdogs.com/passbook): Update requested for unregistered serial number 8C6772F099D51AA3-7A32F5FB-F7F8-4285-A2A2-79FC66DF942C
  • Bad Record Keeping in your application
    • [2016-08-23 19:58:35 -0700] Web service error for pass.com.reddwarfdogs.card.dev (https://llc.reddwarfdogs.com/passbook): Server ignored the 'if-modified-since' header (Tue, 23 Aug 2016 16:54:10 GMT) and returned the full unchanged pass data for serial number '8C6771F89ED51DAA-AAF3100E-C365-4CCD-8C95-ADC974F52894'.
    • [2016-08-23 16:49:38 -0700] Get pass task (pass type pass.com.reddwarfdogs.card.dev, serial number 8C6771F89ED31FAE-57ED753A-8464-408E-95EF-CEF75DBB30D6, if-modified-since Tue, 09 Aug 2016 21:57:32 GMT; with web service url https://llc.reddwarfdogs.com/passbook) encountered error: Received invalid pass data (The pass cannot be read because it isn’t valid.)
      • Cause: Corruption OR change of Certificate used to sign Passbook
    • [2016-08-23 13:56:44 -0700] Web service error for pass.com.reddwarfdogs.card.dev (https://llc.reddwarfdogs.com/passbook): Server requested update to serial number '8C6771F89ED41BAC-FFBF3B69-98F1-4F2A-A8B7-5AF457558EE7', but the pass was unchanged.
    • [2016-08-23 11:58:25 -0700] Web service error for pass.com.reddwarfdogs.card.dev (https://llc.reddwarfdogs.com/passbook): Device received spurious push. Request for passesUpdatedSince '20160823180851' returned no serial numbers. (Device = 2c04d18e5f8480f97bb9318b4065dba0)
    • [2016-08-08 10:23:57 -0700] Web service error for pass.com.reddwarfdogs.card.dev (https://llc.reddwarfdogs.com/v1/passbook): Response to 'What changed?' request included 1 serial numbers but the lastUpdated tag (20160808172351) remained the same.
      • Cause: Duplicate push notification sent to a device or logic error. If the tag is   1234, then the server logic should be > 1234 and NOT >=1234
  • Apple gives little guidance to status code and how the iphone will react
    • [2016-08-23 15:46:33 +0700] Get serial #s task (for device 6f175696d73dec465c561f4d3ee2dfe7, pass type pass.com.reddwarfdogs.card.dev, last updated (null); with web service url https://llc.reddwarfdogs.com/passbook) encountered error: Unexpected response code 504
    • [2016-08-23 01:42:53 -0700] Get serial #s task (for device 2c04d18e5f8480f97bb9318b4065dba0, pass type pass.com.reddwarfdogs.card.dev, last updated 20160823083910; with web service url https://llc.reddwarfdogs.com/passbook) encountered error: Unexpected response code 408
    • [2016-08-08 18:53:00 +0800] Get serial #s task (for device 726996d0f44f44b19f157aa0824f64cf, pass type pass.com.reddwarfdogs.card.dev, last updated (null); with web service url https://llc.reddwarfdogs.com/passbook) encountered error: Unexpected response code 596

I suspect there are more messages – I have just not stumbled across them yet.

Friday, August 26, 2016

Solving PushSharp.Apple Disconnect Issue

While doing a load test of a new Apple Passbook application, I suddenly saw some 200K transmissions errors from my WebApi application. Searching the web I found that a “high” rate of connect/disconnect to Apple Push Notification Service being reported as causing APNS to do a forced disconnect.

 

While Apple does have a limit (very very high) on the number of notifications before they will refuse connections for an hour, the limit for connect/disconnect is much lower. After some playing around a bit I found that if I persisted the connection via a static, I no longer have this issue.

 

Below is a sample of the code.

  • Note: we disconnect and reconnect whenever an error happens (I have not seen an error yet) 

 

using Newtonsoft.Json.Linq;

using PushSharp.Apple;

using System;

using System.Collections.Generic;

using System.Security.Cryptography.X509Certificates;

using System.Text;

namespace RedDwarfDogs.Passbook.Engine.Notification

{

    public class AppleNotification : INotification

    {

        private readonly IPassbookSettings _passbookSettings;

        private readonly ILogger_logger;

        private static ApnsServiceBroker _apnsServiceBroker;

        private static object lockObject = new object();

        public AppleNotification(ILogger logger,IPassbookSettings passbookSettings)

        {

            _logger= Guard.EnsureArgumentIsNotNull(logger, "logger");

            _passbookSettings = Guard.EnsureArgumentIsNotNull(passbookSettings, "passbookSettings");

        }

        public void SendNotification(HashSet<string> deviceTokens)

        {

            if (deviceTokens == null || deviceTokens.Count == 0)

            {

                return;

            }

            try

            {

                _logger.Write("PassbookEngine_SendNotification_Apple");

                // Create a new broker if needed

                if (_apnsServiceBroker == null)

                {

                    X509Certificate2 cert = _passbookSettings.ApplePushCertificate;

                    if (cert == null)

                        throw new InvalidOperationException("pushThumbprint certificate is not installed or has invalid Thumbprint");

                      var config = new ApnsConfiguration(ApnsConfiguration.ApnsServerEnvironment.Production,

                        _passbookSettings.ApplePushCertificate, false);

                    _logger.Write("PassbookEngine_SendNotification_Apple_Connect");

                    _apnsServiceBroker = new ApnsServiceBroker(config);

                    // Wire up events

                    _apnsServiceBroker.OnNotificationFailed += (notification, aggregateEx) =>

                    {

                        aggregateEx.Handle(ex =>

                        {

                            _logger.Write("Apple Notification Failed", "Direct", ex);

                            _logger.Write("PassbookEngine_SendNotification_Apple_Error");

                            // See what kind of exception it was to further diagnose

                            if (ex is ApnsNotificationException)

                            {

                                var notificationException = (ApnsNotificationException)ex;

                                var apnsNotification = notificationException.Notification;

                                var statusCode = notificationException.ErrorStatusCode;

                            }

                            _logger.Write("SendNotification", "PushToken Rejected", ex);

                            // We reset to null to recreate / connect

                            Restart();

                            return true;

                        });

                    };

                    _apnsServiceBroker.OnNotificationSucceeded += (notification) =>

                    {

                    };

                    // Start the broker

                }

                var sentTokens = new StringBuilder();

                lock (lockObject)

                {

                    _apnsServiceBroker.Start();

                    foreach (var deviceToken in deviceTokens)

                    {

                        if (string.IsNullOrWhiteSpace(deviceToken) || deviceToken.Length < 32 || deviceToken.Length > 256 || deviceToken.Contains("-"))

                        {

                            //Invalid Token, keep in Apple's good books                   

                            // We use GUID's thus - for faking pushtokens. Do not send them to apple

                            // We do not want to be get black listed

                        }

                        else

                        {

                            // Queue a notification to send

                            var nofification = new ApnsNotification

                            {

                                DeviceToken = deviceToken,

                                Payload = JObject.Parse("{\"aps\":{\"badge\":7}}")

                            };

                            try

                            {

                                _apnsServiceBroker.QueueNotification(nofification);

                                sentTokens.AppendFormat("{0} ", deviceToken);

                            }

                            catch (System.InvalidOperationException)

                            {

                                // Assuming already in queue

                            }

                        }

                    }

                    try

                    {

                        //duplicate signals may occur

                        _apnsServiceBroker.Stop();

                    }

                    catch { }

                }

                var auditLog = new Log

                {

                    Message = sentTokens.ToString(),

                    RequestHttpMethod = "Post"

                };

                _logger.Write("Passbook", PassbookLogMessageCategory.SendNotification.ToString(),

                    "PassbookAudit", "Passbook", auditLog);

                return;

            }

            catch (Exception exc)

            {

                // We swallow notification exceptions - for example APSN is off line. Allow rest of processing to work.

                _logger.Write("SendNotification", "One or more notifications via Apple (APNS) failed", exc);

                Restart();

                _apnsServiceBroker = null; //force a reset

            }

        }

        private void Restart()

        {

            if (_apnsServiceBroker != null)

            {

                try

                {

                    //duplicate signals may occur

                    _apnsServiceBroker.Stop();

                }

                catch { }

                _logCounterWrapper.Increment("PassbookEngine_SendNotification_Apple_Restart");

                _apnsServiceBroker = null;

            }

        }

    }

}

Sunday, August 7, 2016

Taking Apple PkPasses In-House–Working Notes

This year I had a explicit, yet vague, project assigned to me: Move our Apple PkPass from a third party provider to our own internal system. The working environment was the Microsoft Stack with C# and a little googling found that the first 90% of the work could be done by nuget, namely:

  • Install-Package dotnet-passbook
  • Install-Package PushSharp

Created a certificate file on the apple developer site and we are done … easy project… not quite

 

Unfortunately both in-house expertise and 3rd part expertise involved in the original project had moved on. Welcome to reverse engineering black boxes.

 

The Joy of Certificates!

Going to http://www.apple.com/certificateauthority/  open a can of worms. The existing instructions assumed you have a Mac not Windows 10.

The existing instructions found on the web(https://tomasmcguinness.com/2012/06/28/generating-an-apple-ios-certificate-using-windows/)  broke due to some change with Windows or Apple in April 2016 ( apple forum, stack overflow). The solution was Unix on windows via https://cygwin.com/install.html and going the unix route to generate pfx files.

 

The second issue was connected with how we run our IIS servers and the default instructions for installing certificate for dotnet-passbook were not mutually compatible. The instructions said that the certs needed to be install in the Intermediate Certification Authorities – after a few panic hours deploying to load hosts with problems, we discovered that we had to Import to Personal to get dotnet-passbook to work.

The next issue we encountered was that of invisible characters coming along when we copy the thumbprint to our C# code. We implemented a thumbprint check that verified both the length (40) and also walk the characters insuring that all were in range. After this, we verified that we could find the matching certificate. All of this was done on website load. . an error was thrown, the site would not load.

 

This saved us triage time on every new deployment:with an

  • We identify if a thumbprint is ‘corrupt’
  • We verified that the expected certificate is there

The last issue impacts big shops: The certificate should be 100% owned by Dev Ops and never installed on a dev or test machine. This means that alternative certs are needed in those environment. Each cert with have a different thumbprint – hence lots of web.config transformation substituting in the correct thumbprint for the environment. The real life production cert should be owned by dev ops (or security)  with a very strong password that they and they alone know.

 

The Joys of Authentication Tokens

Security review for in-house required that the authentication tokens be a one way hash (SHA384 or higher) and be unique per PkPasses. The existing design used Guids for serial numbers and thus we used a Guid for the authentication token when the pass was first created.  We can never recreate an existing PkPass because we do not know the authentication token, just the hash.  When a request comes in for the latest path, we hash the authentication token sent in the authentication header and compare it to the hash. We then persist it in memory and insert it into the PkPass Json,  then we Zip and Sign the new PkPass.  Security is happy.

 

Now when it comes to the 3rd party provider, we were fortunate that they stored the authentication tokens in plain text, so it was just a hash and save the hash into our database. If they had hashed (as they should have), then we would need to replicate their hash method. If it was a SHA1 and SHA-2 was required by our security, then we would need to do some fancy footwork to migrate the hash, i.e.

  1. add a “SHA” column iWn our table,
  2. when a new request comes in examine the SHA value
  3. if it is “1” then use the authentication token presented and authenticated to create a SHA-2 hash and update the SHA column to “2”
  4. if it is “2” then authenticate appropriately.

This will allow us to track the uplift rate to SHA-2. At some point security would likely say “delete the SHA1 PkPass records”. This is easy because we have tracked them.

 

Push Notifications

This went easy except for missing that a Push Certificate is NOT used for PKPass files. Yes, it is not used.  It is used for registered 3rd party developed Apple applications. The certificate used for connecting to the Apple Push Notification Service (APNS) is the certificate used to sign the PkPass files. There is no separate push notification certificate. Also, using PushSharp, you must set “validate certificate” to false, or an exception will be thrown.

 

The pushTokens are device identifiers and APNS does not provide feedback if the device still exists (one of my old phones exists, but is at the bottom of an outdoor privy in a national park…), is turned off, or is out of communication.  The author of PushSharp, Redth, has done an excellent description of the problem here. The logical way to keep the history in check is to track when each pass is last retrieved and then periodically delete the push notifications for devices where none of the associated passes have been retrieved in the last year.  You will have “dead” push tokens in some circumstances.

 

I have a pkPass, my iPhone got destroyed. I installed the pkPass on the new phone. The old iPhone push token will never be eliminated while I maintain my PkPass. Why? because we do not know which iPhone is getting updates!

 

Minor hiccup

The get serial number since API call had a gotcha dealing with modified since query parameters. Apple documentation suggest that a date be used and we originally code it up assuming that this was a http if-modified-since header. QAing on a iPhone clarified that it was a query parameter and not a http header. We simply moved the same date there and encountered two issues:

  • We had a time-offset issue, our code was working off the database local time and our code deeming it to be universal time…. (which a http header would be)
  • Our IIS security settings did not like seeing a “:” in a query parameter. We resolved by used “yyyyMMddHHmmss” format

The real gotcha that was stated in the apple documentation was that this is an arbitrary token  that is daisy chained from one call to the next. It did not need to be a date. A date is a logical choice, but it is not required to be a date.

 

The value received in the last get serial numbers response is what is sent in the next get serial numbers request. Daisy chaining. The iPhone does nothing but echo it back.

Avoiding a Migraine

The dotnet-passbook code puts into the Json, the pass type identifier name in the certificate regardless of what you passed in. This is good and wise and secure. It has an unfortunate side effect, the routing

devices/{deviceLibraryIdentifier}/registrations/{passTypeIdentifier} and passes/{passTypeIdentifier}/{serialNumber}

is determined by this pass type identifier. If you are running a site and passes come from passes/foobar/1234, but your certificate name is “JackShyte” then the Json in the pass returned would read JackShyte. When the iPhone gets a push token, it would then construct the url for the update as passes/JackShyte/1234 … which will likely return a 404. The PkPass will never be updated unless you create additional routings!!

 

The solution that I took was to compare the {passTypeIdentifier} in the routing to the certificate. If they did not match, then 404 immediately and log an exception. While it is technically possible to “unwind” such a foul up, the path is not pretty.

 

Migration

The key for migration is a stepped approach

  1. Deploy your new solution and test it, correct any issues that you find in the production environment
  2. Deploy the application or mechanism for creating new PkPasses (this could be part of 1), so all new passes use the in-house system
  3. Update your data from the third party provider with authentication tokens (or their hash) and serial numbers. You want to do this after 2, because you want this list to be closed (no new passes created on the third party system)
  4. Have the 3rd party provider change the WebServiceUrl to the in-house solution. In theory, a Moved response to the in house system would also work (I have not tested this with an iPhone).
  5. Since the 3rd party wants to shut down in time, then you must send out a push notification to every push token you have.  You will likely want to throttle this if you have a large numbers of push tokens (in my case, 30 million) because every push token could result in a request for a new PkPass file.
    1. This may need to be repeated to insure adequate coverage for devices off line or abroad without data plans

Bottom Line

The original design worked, but there was a ton of details that had to be sorted out. I have omitted the nightmares that QA had trying to validate stuff, especially the migration portions.