Sunday, August 13, 2017

From Developer to Architect: Patterns, Architecture Types, Soft Skills, and Continuous Delivery (Video Tutorial)

O’Reilly don’t just publish great technology books they also do some great video tutorials which are available from Safari.  I recently just finished the series: From Developer to Architect: Patterns, Architecture Types, Soft Skills, and Continuous Delivery which consists of tutorial style videos about architectural patterns and anti-patterns, soft skills and a run through of some DevOps ideas and best practices.  All the 17 tutorials are presented by two seasoned Architects Mark Richards and Neil Ford and take about
Neil Ford

Even though the course is more at the fundamental / introductory level there are some topics and nuggets of information that are still useful either because you never had exposure to them (for example, not many projects use event driven architectures extensively) or you have just forgotten them - in which case the course serves as an excellent refresher.

Some highlights:
  1. The differences between Application, Integration and Enterprise Architecture are well detailed.  
  2. The Expand and Contract pattern is one mechanism to get over the DB coupling the Shared Database pattern introduces.
  3. Shared database has a problem if the disparate applications all use their own caching. It becomes even more complex to determine when applications aren't looking at the latest data. 
  4. Even though ReST is generally stateless from the client's perspective, the resources have state and therefor using ETag is a good idea.  Consideration should also be given to use 409 / 412 HTTP status codes when clients are using the wrong version of the Resource.
  5. In a classical layered architecture, an open layer is one that can be by-passed e.g. Service Layer.  However, having too many open layers completely defeats the purpose of a layered architecture.
  6. Different patterns in Event Driven Architectures:
    • Event processor / Mediator topology: Useful when ordering is required, achieved through orchestration
    • Broker topology: No central mediator, custom process components receive events directly
  7. Long running feature branches are anti-thetical to CI.  In my own humble opinion this is another reason why if you want to really to do CI use a tool that is good for branching  / merging - GIT. 
  8. Dierzler's law this is an excellent application architecture which reminds people not to be fooled by the initial illusions of progress in rapid prototyping.   The user wants a full solution and doing a full solution is more difficult than the initial stages.  I really believe anyone involved with rapid prototyping, hackathons or any form agile development should know this law inside out / upside down. 
  9. The Anti Corruption layer is a good pattern to use a facade / adapter approach to hide away bad legacy code.
Mark Richards
A common theme throughout the course, is no matter what the architecture or technology you need to be prepared to change it and that means you need to be careful to get the level of coupling right.  I would add to that an architecture that has a lot of tight coupling will struggle to deal with technical debt because if you need to fix one little part you impact everything. 

So any criticisms?  Well not much.  Perhaps some of the explanations are too abstract. For example, the Space architecture would be better explained with more specfics using a NoSql database to achieve the data splits. Other than that, there are large amounts dedicated to soft skills.  While I don't doubt these to be important, you can get a lot of this by having a cup of coffee with a decent project manager or development manager and personally I prefer the technical stuff.  Overall it's a great series for an into to software architecture and perhaps it would be nice to see a follow up with more deep diving.  I would suspect that is exactly what the follow up series Software Architecture Fundamentails Beyond the Basics entails

Lastly, although this tutorial series is primarily aimed at Developers who want to be (or have become) Architects, I think anyone involved with any sort of SDLC can benefit from it.  Project, Dev Managers and Product Owners should all understand Dietzler's law for example.

  1.  Software Architecture Patterns   
  2.  Software Architecture Fundamentals - Neil Ford PDF  
  3.  Software Architecture Fundamentals beyond the basics

Sunday, July 16, 2017

Outputting the given, when, then, Extending Spock

Spock is a Java testing framework, created in 2008 by Peter Niederwieser a software engineer with GradleWare, which facilitates amongst other things BDD.  Leveraging this example, a story may be defined as:
Story: Returns go to stock

As a store owner
In order to keep track of stock
I want to add items back to stock when they're returned.

Scenario 1: Refunded items should be returned to stock
Given that a customer previously bought a black sweater from me
And I have three black sweaters in stock.
When he returns the black sweater for a refund
Then I should have four black sweaters in stock.

Scenario 2: Replaced items should be returned to stock
Given that a customer previously bought a blue garment from me
And I have two blue garments in stock
And three black garments in stock.
When he returns the blue garment for a replacement in black
Then I should have three blue garments in stock
And three black garments in stock.
Spock makes it possible to map tests very closely to scenario specifications using the same given, when, then format. In Spock we could implement the first scenario as:
class SampleSpec extends Specification{
    def "Scenario 1: Refunded items should be returned to stock"() {
        given: "that a customer previously bought a black sweater from me"
        // ... code 
        and: "I have three black sweaters in stock."
        // ... code
        when: "he returns the black sweater for a refund"
        // ... code
        then: "I should have four black sweaters in stock."
        // ... code

What would be nice would be to ensure accurate mapping of test scenario requirements to test scenario implementation. We could get someway down this path if we could output the syntax of the given, when, then from our test.  Spock allows us to add this functionality through its extension framework.

So, let's say our BA is really curious and wants more confidence from the developer that they stuck to the same given, when, then format and their code is in-sync. They want to get this information easily.   Developer could provide this information by first defining this annotation
import java.lang.annotation.*
import org.spockframework.runtime.extension.ExtensionAnnotation

@interface LogScenarioDescription {}
Followed by this implementation:
import org.apache.log4j.Logger
import org.spockframework.runtime.AbstractRunListener
import org.spockframework.runtime.extension.AbstractAnnotationDrivenExtension
import org.spockframework.runtime.model.FeatureInfo
import org.spockframework.runtime.model.SpecInfo

class LogScenarioDescriptionExtension extends AbstractAnnotationDrivenExtension; {
    final static Logger logger = Logger.getLogger("scenarioLog." + ReportExtension.class);

    void visitSpecAnnotation(Report annotation, SpecInfo spec) {
        spec.addListener(new AbstractRunListener() {
            void afterFeature(FeatureInfo feature) {
                if (System.getEnv("logScenarios")) {
          "***SCENARIO TEST:*** " +
                    for (block in feature.blocks) {
                        for (text in block.texts) {
This will then be applied to the test
class SampleSpec extends Specification{
When the test is executed it gives the following output:
***SCENARIO TEST:*** Scenario 1: Refunded items should be returned to stock
that a customer previously bought a black sweater from me
I have three black sweaters in stock.
he returns the black sweater for a refund
I should have four black sweaters in stock.
output to a specific logfile for scenario logging by using the following log4j:
log4j.rootLogger=INFO, stdout

log4j.logger.scenarioLog.extension.custom=INFO, scenarioLog


and now you have a logfile that your BA, QA can read! This helps foster an Agile culture of collaboration and ATDD where it possible to check that test scenarios implemented with those that were agreed.

Tuesday, June 13, 2017

CoderDojo, so what's the point?

Initially I was skeptical of CoderDojo. Here's another thing IT professionals are doing for free. Why isn't there Economics Dojo,
Medical Dojo, Legal Dojo?  Why is a profession that at times is
extremely competitive, has long demanding hours, low job security being glamorized beyond a point of credible fiction?  Why are people being told they need to code when there are plenty of careers where you will never need to code and always will be?  And shouldn't our kids be out getting exercise, exploring nature?  At least that's what Steve Jobs thought.

It didn't stop there.  There is this trap that all parents fall for that if they get their kids involved something young they'll be a step ahead and if they don't they'll be a step behind.  CoderDojo is not immune to this quasi chimera.  So, I started thinking over my many glorious years in the IT profession, I have worked with a range of coders from the exceptional to the cryptically insane.  If I was to differentiate between a good coder and a not so good coder the number one differentiator is habits.   Yes habits.  Good coders follow good habits:
  1. They write unit tests first.
  2. They are always seeking feedback on their code.
  3. If something complex is happening, they seek to build consensus.
  4. They follow agreed conventions and industry standards.
These good habits mean you end up with something maintainable.   On the other side, coders with bad habits:
  1. They never write adequate tests.
  2. They don't take feedback well (usually because they are not used to it).
  3. If something complex is happening, they hack something up and you find out about it later than you should.
  4. They don't follow conventions - just stick to they way they do things that only makes sense to them.
So someone with the same aptitude, the same I.Q., the same hairstyle ends up producing something
that is much more difficult to maintain.

Don't get me wrong, there was a time when aptitude was much more important.  Complex C++ memory management, any assembly code required serious aptitude and a base level will always be needed.  There are about 50 important principles from encapsulation to recursion that you need some sort of aptitude to get.  But now with Stackoverflow, Google, lots of great libraries and frameworks mean it is really more about habits.  Pick a technology everyone is using and you get lots of support for free.

So learning to code at age 9, is that going to do anything to help you get good habits? Of course it isn't.  So then what's the point?  The point is for one thing only: fun.

As a friend recently said, it is the new mechano and lego.  Let them code but make sure they still play with lego and mechano.

So in my case, my older son (age 7) recently came home from school crying because he wanted to learn scratch. My wife gave him an old laptop and after reading a few PDFs he was away coding a little game talking about loops, scripts and wanting to change the mp3 files.  I was a bit shocked watching him stare at the screen trying to figure out his own code, coming out with tech babble and being able to get something working all by himself.  Next his friend was over. Is this pair programming for kids?

Anyway, no doubt, this new generation will have more information available to them than any before. This provides all sorts of creative avenues not just in code.  However, I can't agree with articles such as this recent one from RTE, telling us why your kid should code?

Kids should do what is healthy, safe and fun.  If they find Economics, Law or Medicine more stimulating than coding great.  Perhaps, the experts in those professions could also run free classes for all kids.   Alas, I somewhat doubt that will ever happen...

Wednesday, June 8, 2016

Triggering a Client Cache Refresh

More and more web sites are using a single page style architecture.  This means there is a bunch of static resources (aka assets) including:
  • JS 
  • CSS
  • HTML templates
  • Images
  • ...
all residing in the client's browser.  For performance reasons, the static content is usually cached. And like all caches, eventually the data gets stale and the cache must be refreshed.  One excellent technique for achieving this in the web tier is Cache busting (see: here and here).
However, even using this excellent pattern there still needs to be some sort of trigger to indicate that things have gone out of date.  Some options:

HTML5 Sockets

In this case a server can send a request to any client to say go and get the new stuff.

Comet style polling

The Client consistently polls the server in the background asking if there is a new version available. When the server says there is, the client is triggered to start cache busting.

Both of these are good approaches but in some edge cases may not be available:
  1. Architecture may not allow HTML5 sockets - might be some secure banking web site that just doesn't like it.
  2. The Comet style polling might be too intensive and may just leave open edge cases where a critical update is needed but the polling thread is waiting to get fired. 

Client refresh notification pattern

Recently, on a project neither of the above approaches were available, so I needed something else which I shall now detail.  The solution was based on some existing concepts already in the architecture (which are useful for other things) and some new ones that needed to be introduced:
  1. The UI always knows the current version it is on.  This was already burnt into the UI as part of the build process.
  2. Every UI was already sending up the version it is on in a custom header. It was doing this for every request.  Note: this is very useful as it can make diagnosing problems easier. Someone seeing weird problems, nice to be able see their are on a old version straight away.
The following new concepts were then introduced:
  1. The server would now always store the latest client version it has received.  This is easy to do and again handy thing to have anyway.
  2. The server would then always add a custom header to every response to indicate the latest client version it has received.  Again, useful thing to have for anyone doing a bit of debugging with firebug or chrome web tools
  3. Some simple logic would then be in the client, so that when it saw a different version in response to the one it sent up, it knows there's a later version out there and that's the trigger to start cache busting!   This should be done in central place for example an Angular filter (if you are using Angular)
  4. Then as part of every release, just hit the server with latest client in the smoke testing. 
  5. As soon as any other client makes a request, it will be told that there is a later client version out there and it should start cache busting.

So the clients effectively tell each other about the latest version but without ever talking to each other it's all via the server.  It's like the mediator pattern. But it's still probably confusing. So let's take a look at a diagram.

With respect to diagram above: 
  • UI 1 has a later version (for example 1.5.1) of static assets than UI 2  (for example 1.5.0)
  • Server thinks the latest version static assets is 1.5.0
  • UI 1 then makes a request to the server and sends up the version of static assets it has e.g. 1.5.1
  • Server sees 1.5.1 is newer than 1.5.0 and then updates its latest client version variable to 1.5.1
  • UI 2 makes a request to the server and sends up the version of static assets it has which is 1.5.0
  • Server sends response to UI 2 with a response header saying that the latest client version is 1.5.1
  • UI 2 checks and sees that the response header client version is different to the version sent up and then starts busting the cache
Before the observant amongst you start saying this will never work in a real enterprise environment as you never have just one server (as in one JVM) you have several - true.  But then you just store the latest client version in a distributed cache (e.g. Infinispan) that they can all access. 

Note: In this example I am using two web clients and one back end server.  But note the exact same pattern could be used for back end micro-services that communicate with each other.  Basically anything where there is a range of distributed clients (they don't have to be web browsers) and caching of static resources is required this could be used. 

Until the next time, take care of yourselves.