Micro and Nano Services

Recently there has been a lot of hype around services. Why would there be any hype we have had the concept of Service Oriented Architectures for a long time. Well for some people SOA has not served them well. I think this just coms down to not understanding and the fact that SOA can be a complicated topic to get your head around.

With the advent of SOAP, RPC, DCOM, CORBA, etc made the SOA a very hard thing to get right. None of these technologies are compatible with one another or event interchangeable.

After going through some of these technologies we realised that maybe we should embrace HTTP as the transport. This gave rise to REST. A long with this realisation people started to remember what smart people before us have done to build software. If you look at the UNIX Philosophy we see some points that have really fascinated me:

1. Small is beautiful.
2. Make each program do one thing well.
3. Choose portability over efficiency.

I think REST and the UNIX philosophy gave rise to micro services.

Micro Services

I first heard of this term a couple of years ago. We have been building services for a very long time, though I always felt there is something wrong, oh thats right it was always a monolithic pile of crap I was building. The basic idea of a micro service is to break up your problem into smaller problems and the only coupling you have between them is the HTTP protocol (which really has proven to be stable). Micro services can be thought of as a the bounded context in DDD.

So what are some of the good properties that a micro service can bring us?

• We can implement the service in whatever language we want.
• We can deploy each service individually.
• We can scale each service individually.

I will try out a few things and see what is the best way to build these services. This was inspired by a great talk I heard at YOW 2013 by Sam Newman called Practical Considerations of Micro Services

Nano Services 

The concept of nano services was brought up when I was at YOW 2013 by Steve Vinoski. When he first raised it he mentioned that he found an article that described it as an SOA Anti-Pattern. Though what he meant was having the ability to build really small services within the language that you use.

These services are really about some of the new concurrency and parallelism patterns, such as:

1. Actor Model
2. Communicating Sequential Processes

A very interesting term indeed.

Git Flow

This post is not about git's branching model or how you should promote your code to different branches. Rather this is a post on how I have learnt to use git. Which I hope will serve as a starting point for other people.

If you are looking for a way to promote your code. Please have a look at the following pages:

1. A successful Git branching model
2. GitHub Flow

Background

When I first started using git I was just using for my own projects and like any other version control system. It was only until I actually used it in a team environment that I started to get the concepts. Luckily for me at the time I had a great teacher, that really helped me understand the basics of git.

Branching

The biggest thing I had to get used was creating a branch for every story/feature that we worked on. We never worked of the master branch. The reason for this will become more apparent when I describe the other features. Branching in git is so cheap and easy. To create a branch you perform the following command: 

git checkout -b branch_name

The name of the branch also needed to follow a convention. The start of the branch needed to start with your initials (in my case arf) followed by a small description (2 words or so) and maybe even the story number you might have been working on. This was so you know who was working on what branch without looking at the commit history.

At first I was against branching. As many of you I initially thought this goes against continuous integration. If you only work on a branch then you will never find out if it integrated properly or it will never run on CI as usually CI is configured to run against master. This was not the case there are many SAAS CI tools that will build all of your branches and if you constantly rebase master you will always see if it integrates correctly. This also allowed us to keep master clean.

Committing

When I first started committing code I was doing just like I was during my SVN days. Do a bunch of work commit it, oh I broke the build commit some more and on and on and on.

What is the problem with this style? 

Well it does not tell you the story of your implementation. Usually there are files committed together that don't belong with each other and the commits are fragmented.

When I was working on this project I got introduced to the concept of Single Responsibility for your commits. This was a powerful concept. Here were some of the basic rules that we followed:

1. Changes that are related to one another need to be in the same commit (to achieve that we squashed changes together if necessary)
2. Refactoring of code would be done on a separate commit. Meaning you would never change something and refactor it in the same commit.
3. Introduction of libraries (gems for our project) would be done on a separate commit.

Achieving some of these points was really context driven, luckily for me I had smarter people helping me.

Committing code with git is fairly easy. I would generally follow 2 patterns:

git add files

git commit -m "Descriptive message"

or (once I got better at single responsibility)

git commit -a -m "Descriptive message"

Now sometimes you find your self that you went trigger happy and changed the same file for multiple reasons. This violated the single responsibility. So git has wonderful command to deal with this:

git add -p

You can read about this command here.

Once we were done with the changes we would push the branch up to Github with the following command:

git push -f origin branch_name

You may have noticed that we used the force command. This is because when you rewrite history (squashing) git will not allow you to push your changes. Since we are on our own branch this is OK.

Rebasing

A while ago while reading about git I came across an interview with Linus Torvalds. Where he stated the following:

Eventually you’ll discover the Easter egg in Git: all meaningful operations can be expressed in terms of the rebase command. Once you figure that out it all makes sense.

Lets say you want to get the latest changes from master into your branch. You would issue the following command:

git fetch && git rebase origin/master

Lets say you wanted to rewrite some history you would use interactive rebase command:

git rebase -i how_far_back

This command is so powerful that I suggest you spend some time with it and think about how powerful it is.

Pull Request

This feature has to be one of my favourites. Since we are branching, once we think we are complete we issue a pull request. We would try to only issue a pull request if the build was green (this was not always the case). If you have used Github then you will know what I am talking about. Finally a tool that can be used for meaningful code reviews. This is so important.  

What do I mean abut this? 

How many times have you asked someone for a code review and because your commits were all over the place you just showed them what you wanted to show them. Well no more.

Here we get to see everything that we have done as a developer and we have to explain and justify our architectural decisions. I wont lie sometimes this go frustrating (this was mostly to do with my ego), however I actually learnt so much from those discussions that I presently miss them so much.

Once the reviewer was happy they would give it the thumbs up or the ship it sign.

Reverting

There would be the off chance that a merge back into master would cause some sort of issue. Which would mean that we would need to revert. I didn't get to do this much however there is a great guide from the man himself on how to do it.

Conclusion

Hopefully I have been able to show you how powerful git is and how you can use it to make sure you have a beautiful commit history. Git has multiple ways to accomplish things so if you have a better way please share it with me. Hope this was helpful.

Leiningen - HTTPS issues

Leiningen

As I have started my adventure in Clojure programming I wanted to start playing with the leiningen tool. 

Unfortunately where I was at has some strict policies around access. We have this crazy proxy so we had to do some magic to get it working. Since I haven't done any Java development in while I was lucky that I had a great friend at work to help me.

Error

I was trying to get lein working and was getting the following error:

Could not transfer artifact clojure-complete:clojure-complete:pom:0.2.3 from/to clojars (https://clojars.org/repo/): peer not authenticated

This could be due to a typo in :dependencies or network issues.

If you are behind a proxy, try setting the 'http_proxy' environment variable.

Solution

To get this to work we needed to do:

1. We need to get the certificate:

echo -n | openssl s_client -connect clojars.org:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > /tmp/clojars.cert

2. Import the key that we just extracted:

sudo keytool -importcert -alias 'clojars_cert'  -file /tmp/clojars.cert -keystore $(find $JAVA_HOME -name cacerts)

3. Make sure the key is there:

sudo keytool -list -v -keystore $(find $JAVA_HOME -name cacerts) | grep clojars

When you run the lein command it should work.

Hopefully this helps someone else in the future.

AWS Elastic Beanstalk

AWS Elastic Beanstalk

During my last adventure we decided to use AWS Elastic Beanstalk. This technology from Amazon is a way to provide developers an easier way to manager their deployments, meaning this a PaaS. We used it for a Ruby on Rails application.

You can control your deployments from git and magically it will send your code and deploy it to an instance. So lets decipher some of this magic.

To get started using EB you need to AWS Elastic Beanstalk Command Line Tool. To set them up on your mac all you need to do is:

brew install aws-elasticbeanstalk

Once you have that setup to get yourself started is quite straightforward. All you need to do is follow the instructions here. As you can see this is pretty straightforward and very easy if you want to launch a basic site. However as I started to go deeper with this platform I realised that I would need do some more work. Why is that you ask? Well for two reasons:

1. The flow of how you typically install a rails app is not exactly how the team at amazon designed it.
2. A deployment always brings the site down.

Extending EB

One of the things that we quickly realised is that we needed to extend the platform. Luckily this was easy to do, the downside is that we were changing the internals of the scripts that were provided by Amazon. Luckily other people have encountered similar problems. So what I decided to do is create a repository where I could add these extensions that we needed to do to make sure it would work.

Please have a look here. There are some interesting things that needed to be done in order to make it work. If you need a further explanation don't hesitate to reach out.

Deployments

As I mentioned deployments are really easy with EB if you don't mind bringing the website down. I fortunately don't believe that a site should go down for a deployment. What we needed to do is build a more robust deployment pipeline. Luckily for me I am a massive believer of Continuos Delivery. So we knew we had to build a deployment pipeline. We decided that we would build multiple stacks (QA, UAT, LIVE) each having 2 environments within them (A and B, this is modelled around Blue/Green Deployments)

To accomplish these deployments we built a tool called mist. This tool takes the pain away from the deployment and we were extremely proud of it as all the developers could easily deploy the latest or a specific version to an environment. I urge you to take a look at the code and tell me what you think.

The Future

As with any platform there is definitely room for improvement. EB is still considered beta (which in some peoples eyes this can be seen as not production ready). Here are some of the limitations that I hit:

1. An old version of Amazon Linux is used https://aws.amazon.com/amazon-linux-ami/2012.09-release-notes/. Unfortunately upgrading the AMI breaks EB.
2. The ruby version is ruby 1.9.3p286 (2012-10-12 revision 37165) [x86_64-linux]. This is quite old. Could not successfully upgrade this.
3. EB uses Phusion Passenger. The version is Phusion Passenger version 3.0.17. This again is quite old too. Could not successfully upgrade this.

We did explore other avenues like JRuby and I fell in love with the platform. Give enough time I wanted to get EB to work with JRuby.

Hopefully this is of help to anyone else out there that is thinking of using EB with Ruby on Rails.

Retina Images

Retina Images

On my recent project I got introduced to the concept of images that need to be used for retina devices. Basically it is an image that has twice as many pixels as the original image.

Converting Images

The convention for having a retina image is to have a file with the word @2x in it. Since the project was only using these types of images I needed to generate non retina images (in other words normal images). Luckily for us there is this awesome tool called mogrify. 

So here are the instructions to get you going:

• Install ImageMagick run: brew install ImageMagick
• Copy your @2x images to a safe location like /tmp/images
• To downsize your images run: mogrify -resize 50% -define png:format=png32 *.png
• To then rename your new images by removing the @2x run:
  
  
  for file in *; do
      mv "$file" "${file/@2x/}"
  done
• Copy your images back to the original folder.

Detecting Asset

Now that we have our assets converted we would like to detect which asset we would like to serve. This fortunately is pretty straightforward. All we need to do is detect from JavaScript (I prefer CoffeScript)

if window.devicePixelRatio == 2 

  # Server your @2x asset

else

  # Server your normal asset

That is it. I know I will use this technique in the future and hopefully you will too.

Simple Made Easy

Simple Made Easy

Recently I watched a great talk called Simple Made Easy by Rich Hickey. I was introduced to this video by a great developer. This talk covered some very interesting points. If you would like the slides they are here.

Simple

The definition of simple means one fold or braid. If you look it up on google I found composed of a single element; not compound. As he discusses one role, task, concept. I can see how we can relate this to SRP.

Easy

He describes easy as being near at hand or we can look at a different way not difficult to obtain. Another way he describes it is near our understanding / skill set / capabilities. This means that easy is relative.

What does this all mean. To me it clearly outlines that simple does not mean easy. Something that I think we get wrong from time to time, I know I do.

Change

Here is an interesting topic that he brings up. In order for us to make a change we need to be able to reason about our code. Often I had thought that because I had tests that I am free to change, I think there is some truth to it. However he made me think about the fact that we need to think about the change. He compares TDD to guard rail programming, meaning that tests will not always drive the right design. 

I was often told that you don't need to debug that the tests should be the only way to see what is wrong with your code. I really believe that is not possible all the time as there will be times where you just have no f**king idea of what is going on. This is because it is hard to reason about.

Anyway the point here is that nothing can replace the thought process.

Benefits of Simplicity

We have an ease of understanding, ease of change and debugging. If we are looking at a single thing it makes it simple as you don't have to think about other dependencies. Makes sense though I don't know how easy this is to achieve, no pun intended.

Conclusion

This is a very interesting presentation that I urge you watch and have a conversation with other people. Making things simple is extremely hard in my view and requires some upfront thinking. 

Law Of Demeter

Law of Demeter

When reading Practical Object-Oriented Design In Ruby, I got reminded of the Law of Demeter. To be honest I never knew it by that name. I always knew it by the name of Train Wreck, this is the way that it was explained to me.

So what is The Law of Demeter?

As described in this article:

The "Law of Demeter" (or LoD) as it is commonly called, is really more precisely the "Law of Demeter for Functions/Methods" (LoD-F). It is a design-style rule for object-oriented programs. Its essence is the "principle of least knowledge" regarding the object instances used within a method. The idea is to assume as little as possible about the structure and properties of instances and their subparts. Thus, it is okay for me to request a service of an objects instance, but if I reach into that object to access another sub-object and request a service of that sub-object, I am assuming knowledge of the deeper structure of the original object that was made available to me.

This basically says don't reach into objects that are internal to the object you are interacting with. For example, these violate the law:

customer.bicycle.wheel.tire
customer.bicycle.wheel.rotate

Fluent Interfaces

As described by wikipedia:

A fluent interface is normally implemented by using method chaining to relay the instruction context of a subsequent call (but a fluent interface entails more than just method chaining). Generally, the context is:

• defined through the return value of a called method
• self-referential, where the new context is equivalent to the last context
• terminated through the return of a void context.

I often confused this as being a violation of the Law of Demeter, however it is not a violation as we are acting on the same object. The only downside is that this could lead to long lines which I would still recommend putting into a method with a descriptive name.

Further Reading  

To anyone that wants to further explore these concepts, please have a read of the following:

• The Law of Demeter - The chapter from the book.
• Assuring Good Style for Object-Oriented Programs - The paper on the topic.
• A Ruby Library - This shows how this could be inplemented

Practical Object-Oriented Design in Ruby

Practical Object-Oriented Design in Ruby

I got recommended to this book by a group of developers that I highly admire. I think they were politely telling me that it will help my code if I read it. For a long time I have been writing OO code, however have I been writing it correctly? Turns out like many of you I believed that I was writing code in a OO way. Turns out that good OO is very rare.

This book takes a very pragmatic approach on the subject. Many of the concepts you would have heard, however the way they are described makes them seem so obvious. That is when I know it is going to be a good book.

Designing Classes with Single Responsibility

The book talks about 4 properties that will make your code easy to change:

1. Transparent - The consequence of change should be obvious in the code that is changing and in distant code that relies upon it. 
2. Reasonable - The cost of any change should be proportional to the benefits the change achieves.
3. Usable - Existing code should be usable in the new and unexpected contexts.
4. Exemplary - The code itself should encourage those who change it to perpetuate these qualities.

Creating code that is TRUE will guarantee that your class has a single responsibility. 

Why does this matter? 

A class that has more than one responsibility is harder to reuse.

How can we determine if a class has single responsibility? 

A way to find if the class should define the behaviour is to interrogate it. Ask your self "Please Mr/Mrs Class what is your behaviour?" If the question sounds odd, more than likely it is the wrong class. 

We can also try to describe what the class does. Pay attention to words like "and/or" these words are a giveaway that your class is doing too much. This applies to messages as well.

Writing Code That Embraces Change

1. Hide instance variables in a method - Don't use instance variables throughout your class.
2. Hide data structures - As data changes depending on data structures can become leaky. It is best to make sure that we return an object with well defined interface. In ruby we can use the Struct class.
3. Extract Extra responsibilities from Methods - For example if you are performing something in a loop, more than likely what is in the loop could be a method (or a few). Make sure move the smallest responsibility to a method.
4. Isolate extra responsibilities to classes - Have a look at your class and interrogate it. Chances are that some of your methods don't belong in that class.

Managing Dependencies

Well designed objects have a single responsibility, their very nature requires that they collaborate to accomplish complex tasks. To collaborate, an object must know about other objects. This knowing is what creates dependencies. Here is where we can get into trouble as some of these dependencies can strangle out application.

Understanding Dependencies

An object depends on another object if, when another object changes, the other might be forced to change in turn. 

How can we recognise dependencies:

• A name of another class is in your class
• Your class sends messages to other objects.
• The arguments that a message requires and the order.

Dependencies are inevitable. However we can keep them under control.

Writing Loosely Couple Code 

Every dependency is like a little dot of glue that causes your class to stick to the things it touches. How can we write loosely couple code?

Inject Dependencies

Having a name of a class causes a dependency in your class. By including a name of a class we introduce a static type in our class. The beauty of Ruby is that we don't need to depend on a type. What we want is to depend on a message. By injecting a dependency all we care about is the message the object can respond to.

Isolate Dependencies

Think of every dependency as an alien bacterium that's trying to infect your class. Give your class a vigorous immune system; quarantine every dependency.

How do we accomplish this?

• Isolate instance creation. Move the creation to its own method using ||= operator.
• Isolate vulnerable external messages. It is best to encapsulate an external message in its own method. If anything changes you isolate the change. 

Remove Argument-Order Dependencies

Knowing an object or message parameters and their order causes a dependency. It is best to try to avoid this order. In Ruby this can be achieved by the following. It is important to make sure that we provide sensible defaults. This can be achieved by using the following.

If you have a class that you can't control. It is best to create a small adapter class that makes sure there is no order dependency.

Creating Flexible Interfaces

Design is concerned with the messages that pass between objects. The conversation between objects takes place using interfaces.

Starting is hard, this is why TDD is hard to get right. The important thing to remember is to not focus so much on the domain objects, however focus on the messages. This means that rather than focus on the class focus on the behaviour. These messages are guides that lead you to discover other objects, ones that are just as necessary but far lest obvious.

How can we discover discover these messages:

• Use sequence diagrams. These diagrams provide a simple and lightweight way to experiment with different object arrangements and message passing schemes. This allows us to try a few things before committing to them.
• Asking for "what" instead of telling the "how". There is a big distinction between a message that asks for what the sender wants and a message that tells the receiver how to behave.

This is where I have found TDD to fail me as I did not put much thought into what the interface should look like. It is important to put some upfront analysis (thought). This helps us create a message-based application, which is the essence of an OO application

Conclusion

These are the messages from the book that I have found to be the most interesting. There are more however I didn't want to spoil it all. Some of the other topics I think warrant a blog post of their own. I urge you to get a copy of this awesome book.

Command Query Separation

Command Query Separation

In the past couple of months I have moved back to being a full time developer and as with anything that you do over the years you forget things. If you are anything like me you get excited about the concepts that you rediscover. This is how I felt with the concept of Command Query Separation.

The concept is quite simple and stated very well in wikipedia

It states that every method should either be a command that performs an action, or a query that returns data to the caller, but not both. In other words, asking a question should not change the answer. More formally, methods should return a value only if they are referentially transparent and hence possess no side effects.

Or simply in the book that coined the term:

Command-Query Separation principle - Functions should not produce abstract side effects.

Well why is it important. Well if you remember we write code for human beings.

I had a method that did the following:

def find_user
  User.find(@id).tap { |user|
     notify(user)
  }
end

Sure this code seems harmless, however this method violates a few principles:

1. The Single Responsibility Principle
2. It produces a Side Effect

Hopefully this serves as a good reminder to all of us OO programmers out there.

The art of naming

There has been numerous debates about naming things. In general, I like most of you find it difficult to name things correctly. It has been said that this is the hardest thing to do in our profession.

I have never really gave it much thought until I met one of my bosses. One of his pet peeves was things that were named incorrectly. For some time I thought what is the big deal, it's just a name and his response to that was if you can't name it you probably don't understand the problem. That statement got me thinking however not enough to really care about it. I really did wish we spent more time at the code level as I knew he had a wealth of experience.

Today I had a very interesting conversation. It all started around a code review of a feature that I was working on. This feature was around a user which had a mobile phone number and we wanted to validate that number by sending a unique code to the phone. Once they received it they would just enter that code. Easy enough.

We already had a User model so I decided to introduce two modules that would be mixed in to the User  model. These two modules were CodeGenerator and MessageSender. I thought this is great and it conveys the message quite well. During this code review in a very humble way it was brought to my attention that I have obviously forgot about "Object Thinking" (I blame this on DevOps and writing all those release management scripts :))

These two great articles really drove the message:

1. One of the Best Bits of Programming Advice I ever Got
2. Your coding conventions are hurting you

I really urge you all to read these great posts. It basically outlines trying to avoid the following in your names:

1. The -er suffix
2. The -able suffix
3. The -Object suffix
4. The I- prefix

So know with my "Object Thinking" cap on how would I approach the problem that I stated above. So my thoughts are the we could have a class called UniqueCode which could have the methods generate, validate. For sending we could have a class called SmsOutBox with a method send. All these classes could be composed with the User class to give it life. There could be better ways.

This is really fascinating and I thought since I have come across this twice, my life was trying to tell me something. Like my dad said to me growing up you will eventually understand what I was trying to tell you and I think this is one of those moments.

In-Memory POP3 Server in C#

Background

Today I had a requirement where we needed to download a message from an email server via POP3 and convert it to another message format to be used in a CMS.

As anyone that knows me testing is very important so I wanted to find a way to create an in-memory POP3 server. The reason for this was I wanted a way to test different kinds of messages that needed to be saved in the CMS and not rely on an actual physical server. So after searching for a while I realised that there wasn't much out there so I decided to write my own.

Implementation

I created a gist that shows you the specs that I wrote. An example spec looks like this:



It("should get a message with OpenPop.NET", () =>
    {
        using (var client = new OpenPop.Pop3.Pop3Client()) {
            client.Connect("localhost", 110, false);
            client.Authenticate("test", "test");

            var message = client.GetMessage(1).ToMailMessage();
            message.From.Address.Should().Be("dzs@iname.com");
        }
    });



The email message that I used to test looks like the following:


Return-Path: <dzs@iname.com>
Received: from iname.com (localhost [127.0.0.1])
        by localhost.localdomain (8.8.5/8.8.5) with ESMTP id RAA01110
        for <dzs@localhost>; Sat, 3 Jan 1998 17:46:55 -0800
Sender: dzs@localhost.localdomain
Message-ID: <34AEEA0E.71EA3BCC@iname.com>
Date: Sat, 03 Jan 1998 17:46:55 -0800
From: Doug Steinwand <dzs@iname.com>
To: dzs@localhost.localdomain
Subject: Test Message

Here's the body of my test message

 - Doug

The actual implementation is in the following gist. The library that I chose to use was LumiSoft.Net as it already had an implementation of a POP3 server (it just needed to be extended).

The interesting part is that for some reason the library AE.Net.Mail (which was the one we wanted to use at work) does not work. I get the following error:


1) should get a message with AE.Net.Mail
   Failure/Error: System.IO.IOException: Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.


So I will probably be replacing it for one of the other client libraries.

This is a quick way to enable a POP3 server in memory and I hope this helps someone else in a similar position.

Ruby on Rails

For years I have been building applications using C#. In recent years I have taken a keen interest in Ruby and Rails. I thought that I would blog about how to get started, to make it easier for me to remember. Hopefully you get a kick out of it.

Background

Ruby on Rails emphasises the use of well-known software engineering patterns and principles, such as active record pattern, convention over configuration, don't repeat yourself and model-view-controller.

Active Record Pattern

Active record is an approach to accessing data in a database. A database table or view is wrapped into a class. Thus, an object instance is tied to a single row in the table. After creation of an object, a new row is added to the table upon save. Any object loaded gets its information from the database. When an object is updated the corresponding row in the table is also updated. The wrapper class implements accessor methods or properties for each column in the table or view.

Convention Over Configuration

Convention over configuration is a software design paradigm which seeks to decrease the number of decisions that developers need to make, gaining simplicity, but not necessarily losing flexibility.

Don't Repeat Yourself

In software engineering, don't repeat yourself (DRY) is a principle of software development aimed at reducing repetition of information of all kinds, especially useful in multi-tier architectures. The DRY principle is stated as "Every piece of knowledge must have a single, unambiguous, authoritative representation within a system."

Model–View–Controller

Model–view–controller (MVC) is a software architecture pattern which separates the representation of information from the user's interaction with it.

1. A controller can send commands to its associated view to change the view's presentation of the model (e.g., by scrolling through a document). It can also send commands to the model to update the model's state (e.g., editing a document).
   
2. A model notifies its associated views and controllers when there has been a change in its state. This notification allows the views to produce updated output, and the controllers to change the available set of commands. A passive implementation of MVC omits these notifications, because the application does not require them or the software platform does not support them.
   
3. A view requests from the model the information that it needs to generate an output representation.

Getting Started

To get started is very easy. Perform the following steps:

• Install rails by running gem install rail.
• Run rails new <application_name> -T. The -T option means [--skip-test-unit] # Skip Test::Unit files
• Go to the application that you created cd <application_name
• Edit your Gemfile and add the following to the file:
  group :development do
  gem 'rspec-rails'
  end
• Since I don't like plain old HTML I prefer to use Slim. To make sure rails uses this template engine add the following to your Gemfile gem "slim-rails"
• Make sure now you run bundle install to make sure your new gem is up
• To get RSpec run the following command rails generate rspec:install
• Run rails server and go to http://localhost:3000/ to make sure everything is ruinning
• Once you are happy lets get it in version control:
• Run git init .
• Run git add -A
• Run git commit -m "Initial commit”
  
  

How easy was that.

Scaffolding

Rails uses the concept of scaffolding. Think of it as a way to generate some code for us. To get started we can create a new Post model

• rails generate scaffold Post name:string title:string content:text. This is such a powerful command as it goes and creates a new model, view, controller, specs and a db migration.
  
• To make sure we create a the data in the database lets run the migration run rake db:migrate
  
• Lets make sure that it all works by running rails server
  
• Go to http://127.0.0.1:3000/posts and play around with it

How easy was that to create a new Post model that has a view, controller that saves to the database.

Testing

As anyone that knows me knows I am a huge advocate of Behaviour Driven Development. The thing I love about ruby is that it makes it so easy. So let's write a test. Open up post_spec.rb and add this test


it 'should not save post without title' do
    post = Post.new
    expect(post.save).should be_false
end


To implement the solution we will add the following to the Post model and add


validates :title, :presence => true


Pretty easy, for more information please look at this excellent guide on testing.

Configuration

Configuration in rails is done through code and using YAML files. In your rails application all configuration is stored in the config folder. The most common configuration that you will deal with is:

• config/application.rb – Contains all the application configuration. Rather than putting everything in this file it is recommended to add all of your changes to config/initializers folder
  
• config/environments/* - Contains environment specific configuration. This configuration takes precedence to the application configuration.

For more information on configuration, this guide will help you out.Migrations


Rails abstracts the database nicely by introducing us to a concept of migrations. Think of it as a way to version your database. When you generate a model by running:


rails generate model Product name:string description:text


Rails will create a file under the db/migrations folder. This migration will look like:


class CreateProducts < ActiveRecord::Migration
   def change
     create_table :products do |t|
        t.string :name
        t.text :description
        t.timestamps
     end
  end
end


This is just a template and it is encouraged that you change it. As you should be thinking about the queries that you will be making against this model.


More information on migrations can be found here.

Asset Pipeline

The asset pipeline provides a framework to concatenate and minify or compress JavaScript and CSS assets. It also adds the ability to write these assets in other languages such as CoffeeScript, Sass and ERB.



The asset pipeline is a great idea. As backend developers we have taken this sort of thing for granted. This pipeline will allow us to work with JavaScript and CSS in a consistent way. So how does it work?



Well everything for the asset pipeline is stored under app/assets. The convention is that if the file starts with application all of your common assets will be loaded in that file. All of the assets are managed with a gem called sprockets. The format is as follows //= require sub/something, this is like the require statement in ruby. This allows you to manage all the dependencies for JavaScript and CSS.


Another cool aspect is that this pipeline allows you to chain different technologies together. For example if you wanted to use CoffeeScript you would name your file something.js.coffee and include the coffee-rails gem, now the asset pipeline will allow you to write all of you front end in CoffeScript. How cool is that?


For more information I recommend reading the official info on the asset pipeline.

Final Thoughts

This is just a basic introduction to the platform. As you can see Ruby on Rails is a fantastic platform. However realise that it is very opinionated, which is not a bad thing. What I love about rails is how easy it is to get going and testing is not an after thought. There is a lot more to rails which I will explore by writing an application that allows me to explore the concepts of lean projects.


To learn more about rails I recommend looking into the online guides and read the best book on the subject Agile Web Development with Rails (4th edition).

NoSQL Journey

NoSQL has been a hot topic for a lot of people. There has been lots of things that I have learnt about their architecture and how distributed systems are built. Why was it that other companies started looking at other solutions?

Why NoSQL?

The story is a typical that I think we can all relate to. As data and transaction volumes started to grow, companies realised that they needed to scale their solutions.

So companies tried to address these challenges by trying to make it fit the relational model:

1. Add more hardware or upgraded to faster hardware.
2. Simplifying database schema, denormalising the schema, relaxing durability and referential integrity.
3. Introducing various query caching layers, separating read-only from write-dedicated replicas.
4. Data partitioning.

These solutions drove the rise of what is now known as NoSQL. Scaling these kind of solutions we need to think about them in a different way. To understand large-scale distributed systems we need to understand the CAP theorem.

Brewer’s CAP Theorem

The theorem states that within a large-scale distributed data system, there are three requirements that have a relationship of sliding dependency: Consistency, Availability, and Partition Tolerance.

Consistency – All database clients will read the same value for the same query, even given con- current updates.

Availability – All database clients will always be able to read and write data.

Partition Tolerance – The database can be split into multiple machines; it can continue functioning in the face of network segmentation breaks.

Brewer’s theorem states that in any given system, you can strongly support only two of the three. So the traditional relational databases focus of Consistency and Availability, while the NoSQL movement tends to focus more on the Availability and Partition-tolerance (this is tunable in some of the systems). Due to this focus the NoSQL systems are said to be eventually consistent.

Eventually Consistent

Basically the storage system guarantees that if no new updates are made to the object, eventually all accesses will return the last updated value. If no failures occur, the maximum size of the inconsistency window can be determined based on factors such as communication delays, the load on the system, and the number of replicas involved in the replication scheme. The most popular system that implements eventual consistency is DNS (Domain Name System). Updates to a name are distributed according to a configured pattern and in combination with time-controlled caches; eventually, all clients will see the update.

When we look at all lot of our data needs one starts to wonder whether you really need it to be available then and there. How many of us have degraded a system because we were logging everything to the same database? Logs are a great example of data that can be eventually consistent (obviously depending what you are logging)

As stated in this great blog Eventual Consistency By Example

We may sum up the eventual consistency model in the following statement:

Given a total number on T nodes, we choose a subset of N nodes for holding key/value replicas, arrange them in a preference list calling the top node "coordinator", and pick the minimum number of writes (W) and reads (R) that must be executed by the coordinator on nodes belonging to the preference list (including itself) in order to define the write and read as "successful".

Here are the key concepts, extracted from the statement above:

• N is the number of nodes defining the number of replicas for a given key/value.
• Those nodes are arranged in a preference list.
• The node at the top of the preference list is called coordinator.
• W is the minimum number of nodes where the write must be successfully replicated.
• R is the minimum number of nodes where the read must be successfully executed.

More specifically, values for N, W and R can be tuned in order to:

• Achieve high write availability by setting: W < N
• Achieve high read availability by setting: R < N
• Achieve full consistency by setting: W + R > N

Final Thoughts

Hopefully you can se how powerful this information is and it really gets you thinking about the way you design your systems. I look forward going thought each of the NoSQL databases and see how these topics apply to them.

Web Development on Mono

Background

Today I decided that I wanted to try out deploying a ASP.NET MVC + Web Api application on Ubuntu.

Why do I want to do that? Well I have always loved C# and .NET, however I have never really enjoyed the Microsoft ecosystem. I am glad that Mono is around and have often wondered What if C# and .NET had targeted all platforms?

I wanted to get MVC 4 and Web API to work correctly. So after spending a few hours on it I thought I would share it with you all.

Process

Below will outline the steps that I took to get this working. I will leave the blood and tears to myself.

Preliminary

The first thing I did was to get vagrant to work. This makes it easier to provision a virtual machine on my mac book pro. The image that I used was Ubuntu Server 12.10 amd64 Minimal (VirtualBox 4.2.6). I created a Vagrantfile that contained the following code:


Vagrant.configure("2") do |config|
  config.vm.box = 'quantal64'
  config.vm.provision :shell, :path => 'bootstrap.sh'
  config.vm.network :forwarded_port, guest: 80, host: 9000
end

The bootstrap.sh contained the following:


#!/usr/bin/env bash

apt-get update
apt-get install -y git
apt-get install -y nginx
apt-get install -y g++
apt-get install -y gettext
apt-get install -y pkg-config
apt-get install -y automake

These packages will be used to build mono and the web-server  Once you have all this set up is just as easy as saying vagrant up and vagrant ssh

Compiling Mono

Once you have logged-in into the box you will run the following commands:

• wget http://download.mono-project.com/sources/mono/mono-3.0.7.tar.bz2
• tar -xvjpf mono-3.0.7.tar.bz2
• cd mono-3.0.7
• ./configure --prefix=/usr/local
• make
• sudo make install

Once all that finishes you will need to get web-sever to work:

• git clone https://github.com/mono/xsp
• cd xsp
• ./autogen.sh --prefix=/usr/local
• make 
• sudo make install

To test if it works create yourself a simple example project or use the one I made. To get the development web server up run sudo xsp4 --port 80 and go to http://localhost:9000/api/service/test. You should see the following:

<string xmlns="http://schemas.microsoft.com/2003/10/Serialization/">awesome</string>

Future

I will be going through some of the MVC 4 and Web API 4 features to make sure it all works. Once that is done I will try to do this on AWS by creating an AMI so that I can test some release processes. Thanks for reading and hope this was just as exciting for you as it was for me.

Lean Projects - Part 2

So some people will read the previous blog and say why are you being so negative? See I don't see it that way, I see it as a place that has a great culture that wants to embrace change and continuous improvement. I also see this as massive opportunity to try out some of the ideas that I will present. A lot of these ideas are not new topics, however they have been tailored for the environment that I am in.

The concept that I am calling this is Lean Projects. A way for us to embrace lean and agile methodologies in an agency environment.

Kanban

Kanban is a method for developing software products and processes with an emphasis on just-in-time delivery while not overloading the software developers. In this approach, the process, from definition of a task to its delivery to the customer, is displayed for participants to see and developers pull work from a queue.

The thing to note from the above statement is that Kanban works on the concept of a pull model.

Visualise Workflow

As stated above, one of the challenges that we face is to visualise our workflow. One way to get started is to map the value stream. The idea behind value stream mapping is to sketch out how you are working as this will help everyone to understand how the process works.

From this visualisation we can create what is called a Kanban board. The idea behind this board is to have a way to show your team what your value stream looks like. An example of a board looks like this:

This board is the single point of truth for everyone working on a product.

If you would like to know how to manage a process of the Kanban board, please read my previous blog Launching Australia's Biggest News Site

Limit Work In Progress

The most important part of the Kanban process is to limit your WIP.

In Kanban, we don’t juggle. We try to limit what we are doing, and get the most important things done, one by one, with a clear focus. It’s not uncommon to find that doing ten things at once takes a week, but doing two things at once takes hours, resulting in twenty things being done by the end of the week.

A great explanation can be found here Why Limit Work In Progress?

Measure and Manage Flow

The important aspect is to make sure that each of the features that is on the board actually goes through to the last step of your flow. We set limits to make sure we don't start picking up more features than the ones already started.

Some teams have the concept of stop the line, where the team gets together to make sure the pipeline clears up for the next features to be pulled in.

Fortunately Kanban has a few ways for us to measure the pipeline, as without measurement we can't improve on things.

Cumulative Flow Diagrams

This diagram is an area graph that depicts the quantity of work in a given state, showing arrivals, time in queue, quantity in queue, and departure. This is a great way to visualise how we are churning features through our work flow.

Lead Time

The term lead time (LT) is just a fancy way to say time to market. It’s essentially the time that elapses (on the calendar) from something is ordered until it is received. In software development, a feature is ordered when someone (like the product owner) asks someone else (the team) to implement it. The same feature is received when it is deployed to the production system and ready to be used by end users.

Why is this important? Well, we can track how long a feature took. This is a really powerful metric, as it can be used to eliminate the entering of hours spent into a system.

How can this be used in Lean Projects? Well when I estimate I always like to estimate in terms of effort, meaning that it is easier for me to think about how easy/hard something is rather than how long it will take. This effort for me is measured using t-shirt sizes, e.g. S, M and L. Now if we start tracking all the features that have been sized using a t-shirt size we can start measuring the lead time for each of these features. We can then take the average of the lead time and this amount can be used to drive the estimates of the next feature.

Make Process Policies Explicit

As mentioned on wikipedia:

Until the mechanism of a process is made explicit it is often hard or impossible to hold a discussion about improving it. Without an explicit understanding of how things work and how work is actually done, any discussion of problems tends to be emotional, anecdotal and subjective. With an explicit understanding it is possible to move to a more rational, empirical, objective discussion of issues. This is more likely to facilitate consensus around improvement suggestions.

LeanKit has a great example of how this can be achieved.

Feedback Loops and Improvement Opportunities

The way to improve this is to provide a way to have some form of feedback loop. I am big fan of Agile Retrospectives. The idea is to meet at a regular time to reflect on how everything is going. With Lean Projects we need to reflect on how efficient the process is and ask the right questions. The way that we can do that is by using the 5 Why's Technique. A good metric for improvement is reducing lead time, as this will help you deliver all features in a better way.

Remember to listen to your team as they are closer to the problem than we usually are. Usually issues tend to show that it is a process problem not a people problem. Lean Projects are about a bottom up approach.

Your people are your most important resource. Ensuring their support and cooperation is vital for all major projects.

Specification By Example

Specification by Example is a set of process patterns that facilitate change in software products to ensure that the right product is delivered efficiently.Wikipedia summarises this quite well:

Teams that apply Specification by example successfully commonly apply the following process patterns:

1. Deriving scope from goals
2. Specifying collaboratively – through all-team specification workshops, smaller meetings or teleconference reviews
3. Illustrating requirements using examples
4. Refining specifications
5. Automating tests based on examples
6. Validating the underlying software frequently using the tests
7. Evolving a documentation system from specifications with examples to support future development

So what does this all mean for Lean Projects? Well the basic idea is that we want to find a common language to describe the requirements through as a set of examples. For this work we need to do this collaboratively and to make sure that the requirements don't become stale as they will serve as documentation.

Before we can think of an example, we need to think of the value that we are adding. The best format I have found is as follows:

In order to [business value]

As a [role]

I want to [some action]

The first part is the most important and it should reflect one of these values:

• Protect revenue
• Increase revenue
• Manage cost
• Increase brand value
• Make the product remarkable
• Provide more value to your customers

Also think about the role that this feature is for. The reason I mention this is because sometimes we build a feature for the “Highest Paid Person's Opinion”, which most of the time adds no value. A scenario or example has the following format:

Given [context]

When I do [action]

Then I should see [outcome]

The Given step is where you set up the context of your scenario. Every scenario starts with a blank slate. The When step is where you exercise the application in order to accomplish what needs testing. Finally, the Then step is where you verify the result.

This way of working can be though of as outside-in development. So an example feature would look like this:

Feature: Login

In order to access the website content

As a website user

I need to log in to the website

Scenario: valid credentials

Given I am presented with the ability to log in

When I provide the email address "test@mywebsite.com"

And I provide the password "Foo!bar1"

Then I should be successfully logged in

Scenario Outline: invalid credentials

Given I am presented with the ability to log in

When I provide the email address <email>

And I provide the password <password>

Then I should not be logged in

Examples

| email | password |

| test@mywebsite.com | Foo!bar2 |

| test@mywebsite.com | Foo |

Lean Software Development

Lean software development is a translation of lean manufacturing and lean IT principles and practices to the software development domain. Adapted from the Toyota Production System, a pro-lean subculture is emerging from within the Agile community.

Lean software development plays by seven principles.

Eliminate Waste

Waste is anything that interferes with giving customers what they value at the time and place where it will provide the most value. Anything we do that does not add customer value is waste, and any delay that keeps customers from getting value when they want it is also waste.

Some forms of waste are:

1. unnecessary code or functionality
2. starting more than can be completed
3. delay in the software development process
4. bureaucracy
5. slow or ineffective communication
6. partially done work
7. defects and quality issues
8. task switching

Build Quality In

The idea here is to start with the frame of mind that quality is important and needs to be done from the start. Your goal is to build quality into the code from the start, not test it in later. You don’t focus on putting defects into a tracking system; you avoid creating defects in the first place.

Extreme Programming is an excellent example of building quality in, some examples are:

Pair Programming

Pair Programming is an agile software development technique in which two programmers work together at one workstation. One, the driver writes code while the other, the observer, reviews each line of code as it is typed in. The two programmers switch roles frequently.

Test Driven Development

Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards.

Continuous Integration

Continuous integration (CI) is the practice, in software engineering, of merging all developer workspaces with a shared mainline several times a day.

Refactoring

Code refactoring is a "disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behaviour", undertaken in order to improve some of the nonfunctional attributes of the software. Advantages include improved code readability and reduced complexity to improve the maintainability of the source code, as well as a more expressive internal architecture or object model to improve extensibility.

Create Knowledge

We have all been in the situation where we have only one person that knows the codebase. There will always naturally be individuals that are stronger than others in specific areas. This is what makes great cross functional teams, however having only one person know something is what is called a single point of failure.

It is important that the whole team has an understanding of how the system works, so that we can eliminate single points of failure. However this is easer said than done as not everyone is always interested in every aspect of every system.

The following provide great ways to create knowledge:

• Pair Programming
• Code reviews
• Lightning talks
• Training

Defer Commitment

Emergency responders are trained to deal with challenging, unpredictable, and often dangerous situations. They are taught to assess a challenging situation and decide how long they can wait before they must make critical decisions. Having set a time-box for such a decision, they learn to wait until the end of the time-box before they commit to action, because that is when they will have the most information.

Basically we want to decide as late as possible, specially for irreversible decisions.

Deciding too early, you run the very likely risk that something significant will have changed, meaning your end product might meet the spec, but it might still be the wrong product! This is one reason why so many projects fail.

Deliver Fast

If we can deliver software fast that means we can get real feedback from actual users, which is the reason you are building the feature in the first place. Another really great side effect of delivering software fast is that you would have had to solve the way you release your features. In order to release quickly means that you need to do it in a repeatable way, this means automation. The way to achieve this is through continuous delivery.

Continuous delivery is a pattern language in growing use in software development to improve the process of software delivery. Techniques such as automated testing, continuous integration and continuous deployment allow software to be developed to a high standard and easily packaged and deployed to test environments, resulting in the ability to rapidly, reliably and repeatedly push out enhancements and bug fixes to customers at low risk and with minimal manual overhead.

Respect People

I think it’s important to treat everyone with the same respect, whatever their job. It doesn’t matter whether they’re the CEO, a developer, project manager, the receptionist or the cleaner, respect everyone equally.

This comes back to the bottom up approach, giving people the responsibility to make decisions about their work. It is important to build knowledge and develop people who can think for themselves.

Respecting people means that teams are given general plans and reasonable goals and are trusted to self-organise to meet the goals. Respect means that instead of telling people what to do and how to do it, you develop a reflexive organisation where people use their heads and figure this out for themselves.

Optimise The Whole

A lean organisation optimises the whole value stream, from the time it receives an order to address a customer need until software is deployed and the need is addressed. If an organisation focuses on optimising something less than the entire value stream, we can just about guarantee that the overall value stream will suffer.

Looking at the whole picture is important. In the past I have spent time just optimising the development practices, however I believe that if you don't love what you are building then what is the point. We should all care about the overall value stream.

Final Thoughts

All of these thoughts have been going around in my head for years and I am truly excited to put these ideas to work in such a vibrant environment. I look forward sharing my findings as I love to see how all these thoughts change over time.