Technology Blog

It began with an off-hand comment on the ACCU General mailing list that at 7digital we release on average 50 times per week, across all of our products.  I thought nothing of it, virtually all of our products are web-based, which makes it relatively easy to deploy on a regular basis, but it seemed that others were interested in how we do it and so I was cajoled into giving my first presentation.

I began by explaining what we understand as Continuous Delivery – a set of repeatable, reliable steps which a change must go through before being released.  In our case most of these steps are automated.

I described where we came from and how we approached the change, in both a technical and business manner, and where we would like to see ourselves going.  I then included a flowchart of  A Day in the Life of a Change at 7digital, which Prezi allows me to ‘bounce’ around as we hit different stages in the pipeline.

I answered many questions clarifying how we handle rolling back a bad release (we actually ‘roll-forward’ with artifacts of the previous ‘good’ release), whether our QA team are comfortable with the process (yes, they help write the criteria),  and how large pieces of work are tackled (we try to break them down into deployable pieces no bigger than a day).

Here are the slides:

(cross-posted on my personal blog)

As craftsmanship

Lately there has been a push of software into craftsmanship, nurturing the skill and giving value to building software for the sake of it.

Not a long time ago the main trend was design patterns, inspired by real life architecture the Gang Of Four tried to look for common problems and summarize it into common solutions but these tend to be taught too early and developers lack the knowledge to know how to use them.

So we found Katas and other procedures trying to compensate for the lack of experience that is needed to understand the design patterns but with the experience we obtain the faultier the design patterns seem.

The reason behind this is the clunkiness and the one size fits all approach which seems to fail.

So focusing back on practice we went to the software as a craft, with each person having it’s own technique and experience and expecting that to be enough.

Wikipedia defines craft as ‘…a profession that requires some particular kind of skilled work. In a historical sense, particularly as pertinent to the Middle Ages and earlier, the term is usually applied to people occupied in small-scale production of goods. The traditional terms craftsman and craftswoman are nowadays often replaced by artisan and rarely by crafts-person (craftspeople).’

That sounds very extreme to compare software with something pertinent to the middle ages, I feel that something was missed. We have our personal experience and we know the patterns but there is a lack of knowledge on the direction to which we should mould the patterns to adapt them to our current problem.

As design

In The same wikipedia page we find a reference to the industrial revolution: ‘The mass production of goods by large-scale industry has limited crafts to market segments in which industry’s modes of functioning or its mass-produced goods would not or cannot satisfy the preferences of potential buyers.’

So what did we miss? We have the design patterns, we have the craftmaship but we lack the very important intermedium steps which are represented by the appearance of industrial design.

And back to wikipedia for the definition of industrial design as ‘…the use of a combination of applied art and applied science to improve the aesthetics, ergonomics, and usability of a product, but it may also be used to improve the product’s marketability and production. The role of an industrial designer is to create and execute design solutions for problems of form, usability, physical ergonomics, marketing, brand development, and sales.’

It’s interesting how easy it is to miss these steps but we have to remember that software development is a very new area and in my opinion it’s not harder than any other engineering or design areas, we just didn’t have enough time for it to figure out the big picture.

What can we do

Focusing now on Industrial design, what can we get out of it? Surely all the lessons about turning craft objects into something that can be used and manufactured in a larger scale will be useful. I will now focus on one of my favourite parts which is Dieter Rams.

You might already have heard of his work with Braun as a huge inspiration for Apple’s design but it’s inspiring on how much of these guidelines might also apply to software.

Rams’s Ten Principles of “Good Design” (Link)

Good Design Is Innovative. The possibilities for innovation are not, by any means, exhausted. Technological development is always offering new opportunities for innovative design. But innovative design always develops in tandem with innovative technology, and can never be an end in itself.
My opinion: Try out new tools and techniques but don’t make it be the priority.

Good Design Makes a Product Useful. A product is bought to be used. It has to satisfy certain criteria, not only functional but also psychological and aesthetic.Good design emphasizes the usefulness of a product while disregarding anything that could possibly detract from it.
My opinion: Having too many features, the wrong ones or having them built in a way that seems complicated is much more likely to be a problem with its design and not with the users.

Good Design Is Aesthetic. The aesthetic quality of a product is integral to its usefulness because products are used every day and have an effect on people and their well-being. Only well-executed objects can be beautiful.
My opinion: In the same way as the mantra ‘Clean code is good code’ a design that is almost intuitive will ease the work of the developers and make it more pleasant. I consider in this case the beauty of code to be an appreciation and a feel-good attitude when studying it.

Good Design Makes A Product Understandable. It clarifies the product’s structure. Better still, it can make the product clearly express its function by making use of the user’s intuition. At best, it is self-explanatory.
My opinion: A well done product doesn’t need extensive documentation and preparation. A very good example is the horrific interface of VCRs in the 80s and 90s versus current TV boxes.

Good Design Is Unobtrusive. Products fulfilling a purpose are like tools. They are neither decorative objects nor works of art. Their design should therefore be both neutral and restrained, to leave room for the user’s self-expression.
My opinion: Funny inside jokes or features that are only useful for the team should be left aside to make the real behaviour bubble up. There is room for a brand but there is no room for clever features that only work for the developer’s ego.

Good Design Is Honest. It does not make a product more innovative, powerful or valuable than it really is. It does not attempt to manipulate the consumer with promises that cannot be kept.
My opinion: Build it progressively and organically, don’t think too far ahead.

Good Design Is Long-lasting. It avoids being fashionable and therefore never appears antiquated. Unlike fashionable design, it lasts many years – even in today’s throwaway society.
My opinion: Avoid new trends that have not proven themselves to be useful. Avoid re-inventing the wheel. Ask why the current solutions don’t work and remember that there are reasons why they’ve worked so far.

Good Design Is Thorough Down to the Last Detail. Nothing must be arbitrary or left to chance. Care and accuracy in the design process show respect towards the consumer.
My opinion: Don’t try to force a simplified design if that is not what the user needs, the hard part is making is simple but complete.

Good Design Is Environmentally Friendly. Design makes an important contribution to the preservation of the environment. It conserves resources and minimises physical and visual pollution throughout the life cycle of the product.
My opinion: Amount of data, resources like memory, disk space or hardware are all invaluable and should be taken into consideration even with the cheap price of components nowadays, a brute force approach is not a replacement for a well though design.

Good Design Is as Little Design as Possible. Less, but better – because it concentrates on the essential aspects, and the products are not burdened with non-essentials. Back to purity, back to simplicity.

More inspiration

You just read my thoughts about design, I hope this inspires you to learn more for this and reach your own conclusions. I leave you know with a list of resources which I think are invaluable.

When looking into architecture, design, psychology or any other subject don’t read a book about a developers point of view on the subject but just get to the source.

Objectified – Industrial/Product design (Link)

Urbanized – Urban design/sustainable design (Link)

EAMES: The Architect and The Painter – Post war industrial design and innovation (Link)

MoMa exhibitions about minimalism (Link)

The Coming-of-Age of Software Architecture Research (Link)

Blank City - ”Do it yourself” independent film making. This is more inspiring about the Do-It-Yourself atittude. (Link)

Punk and Minimalism – (fanzine, need to look for a reference)

Bill Cunningham New York – Fashion photographer in NY. Also an inspiration about his emmersive atitude towards his work. (Link)

An interesting parallel occurred to me the other day during a conversation with a colleague: the mood of a team is subject to changes, just as that of an individual, and sometimes depression can set in. So, just as we can learn techniques to fight of depression in ourselves, maybe we can do the same within our teams.

It became clear to me a few weeks ago that my team was in the doldrums: we had come to the end of several significant pieces of work, but had released this work with very little fanfare, which led to a feeling of deflation; furthermore, our future workload was both daunting in size and vague in scope, with few clear short-term goals, which gave us a sense of listlessness; in the past few months, several really talented team members had left us, and we hadn’t yet managed to to find a new dynamic for the team, so we had lost the buzz that comes when collaboration becomes second nature; finally, a raft of factors beyond our control meant that we kept finding our work blocked, which just added another layer of frustration.

I had recently taken on the role of team lead, which meant not only that these issues became a particular problem for me, but also that I had an opportunity as primus inter pares to do something about this. In working to find a way out of this morass, I came to a realisation: I had been here before, and I already knew how to deal with it.

A rich seam of depression runs through my family, and it has been part of my life since childhood. In the past few years I have pushed it into remission, and it rarely raises its head now. However, from time to time I do catch the onset of a spell of depression, and I have identified a set of first-aid techniques that I can use to stop it developing further. Many of these techniques—regular exercise and fresh air, early nights, cutting down on alcohol, making time to read and relax—seem simple and obvious, but these are just the habits that can get sidelined when depression sets in, so I have to treat them as a strict regime and push myself to follow them. The result of this is incredibly effectively for me.

Here are my key actions in fighting the signs of depression:

• I have identified and look out for the early warning signs, so I know when I need to take action
• When I spot these signs, I acknowledge that the situation warrants special behaviour, even if this means putting other priorities on hold
• I have identified specific behaviours that I know help me recover
• I stick with these behaviours until I am back on an even keel

This brings me back to my team’s situation. Of course, the ideal is to develop good everyday habits that keep depression at bay, but while we work on that, we may well find ourselves slipping into the doldrums occasionally. It seems to me that all four of these actions can apply to a team just as readily as to an individual.

I would be interested to hear other people’s experiences of dealing with lapses in team mood, as well as thoughts on how these ideas fit with various codified working patterns. Please share your ideas!

On joining 7digital a year ago I published a short blog on my first fortnight with the new team. Now that our Technical Academy has been running for 2 weeks I’ve asked our 2 apprentices, Chris and Dan, to write a quick blog post about their experience so far. This is our first point in a series of regular feedback opportunities from our apprentices so we’re hoping to tailor the rest of their time here based on their positive and negative experiences so far.

The 7digital Apprentice Developers

Chris posted his “My First Fortnight” post on his personal blog: http://c24w.github.com/blog/2012/09/7digital-Technical-Academy-My-First-Fortnight/

Dan opted to post on the 7digital developers’ blog and his post is here: http://blogs.7digital.com/dev/2012/10/04/my-first-fortn…hnical-academy/

More on the Technical Academy

In early 2012 we were in a position to expand the development team but found that experienced developers, and especially those experienced in TDD, Agile, Lean and XP, were difficult to find. We also found that we were speaking to an increasing number of graduates and under-graduates when publicising the development team at various events, but often noticed a gap between the skills they were learning at University and the skills we’d like them to have. There has been some coverage around this topic recently with opinions from well known UK Software Craftsman Jason Gorman and mainstream media coverage from The Independent and The Next Web.

We decided to attempt to satisfy this need to train inexperienced new starters with a defined programme of classroom based sessions mixed in with the usual “on the job” training. You can see what we originally proposed to teach our new apprentices on the Technical Academy Jobs page although we’ve changed what we are doing based on feedback from teams. We like to continually improve our teams by inspecting and adapting all the practices we adopt and the Technical Academy is no different. This means we’ve added additional sessions on topics we thought we may not need to cover until later, but noticed that the apprentices had been involved in during their day to day work. Our emphasis is still very much on ensuring that apprentices can add business value right from the beginning so being able to adapt in this way ensures they do not slow down development in their respective teams. More on that in a later post.

In addition to the ability to hire less experienced programmers the academy also allows us to hire more diverse people. A good example is our current apprentice Dan, who studied Physics and Systems Biology, which brings a different view point on some of the practices we use, our product vision and the way we approach analytical tasks.

I have joined 7digital as part of the Technical Academy about two weeks ago and I started working in the API team. This is an appropriate stage to write down my first impressions here.

Setup

Setting up my work machine was not always straightforward, mostly with regards to networking and graphics. This is probably because I am working from a laptop, unlike most other Devs who work on desktops. I have made notes on the issues I encountered and will post them on the internal wiki so that new starters don’t have to go through the same difficulties I did. This would be in the spirit of continuous improvement at 7digital.

Classroom sessions

The main difference between the Technical Academy and a normal junior software development role is the inclusion of classroom sessions on the tools (such as C# and ReSharper) and development methods (such as TDD and Kanban) used at 7digital. The best point with these classes is how they are always relevant to the work performed. For example the class introducing mocking happened just after my pair programming partner of the day used it in code we were working on.

Open to different solutions and helpful environment

The team I got assigned to just started making a new internal website. We started to build it using a web framework none of us were very familiar with but was used by some other teams. Unfortunately it proved too difficult to use so we restarted with more familiar technology. Currently I feel like I have gotten up to speed with this project. This is due to the helpfulness of more senior members and the solid tests present in the codebase which allow us to easily verify changes.  Through the rotation of pair programming doubles, I find it interesting to see the small differences in the way people code with the two invariants being rigorous TDD and willingness to help.

Rollback

In another project my team released, we didn’t perform testing as thoroughly as we should have. This broke some of our client’s apps and required a late night rollback. Most of the following day my team worked a retrospective of the rollback. This retrospective involved finding out what caused the issues leading to the rollback and how similar issues could be avoided in the future. I found it great that the focus was on problem solving and improvement and not shifting blame.

Friendly office

The general workplace atmosphere is important and in that aspect 7digital does very well. Everyone is friendly and approachable and most interactions are informal. Furthermore the pace is set to be sustainable so working long hours is discouraged (but it doesn’t mean we don’t work hard).

Dan Kalotay

The pain

As the Ops team grew and Devs teams grew, it became a recurrent task to be asked by the Ops team to test infrastructural changes.

This didn’t use to be a problem, the teams were smaller, we had less features and the infrastructure was much simpler but they became a blocker for both teams as they have to be working simultaneously on the same task.

Even though we have Unit Tests, Integration Tests, Acceptance Tests and Smoke Tests and they run every time there is a new build (Continuous Integration) we feel the pain of infrastructural changes as these tests are not run for every change done by the Ops team.

Also, Ops working with every single Dev team made the problem even more serious with Dev teams becoming blocked due to the infrastructural work of other Dev teams which might not be related as well as Ops being used as a shared resource.

Even when both teams are synced to test a change there isn’t the knowledge on the Ops side to know how much testing is enough. This is due to the fast pace of development and Ops needing to be up to date with the knowledge of how the features work – this is a nice problem to have but needs to be solved.

Our idea

In a perfect scenario we would have enough people on the Ops side so that they can run side by side with Dev and be up to date with feature changes but that doesn’t happen and we can’t expect Ops to know every single feature and application. So a second option is making this as automated as possible to make problems be caught fast as they do on the Dev side but without having Ops do extra work.

Our solution for this was adding infrastructure to the continous delivery system but as we don’t need to know when these changes are done we use an active wait to run the necessary tests.

TL;DR: We made Smoke tests more concise, running continuously over fixed periods of time and easily accessible by everyone.

How we got there

A common scenario to test infrastructure would be running Smoke Tests to make sure the end to end path for a feature works. In my current team, it would be something like Downloading a music album or Streaming a track.

Reviewing the current tests there were some issues trying to simplify and slim down the tests:

Large number of Smoke tests - Might be a smell that the application is trying to do too much and it will also slow down the overall time to assess the state of the application.

Ambiguity between Smoke tests and Acceptance tests – a Smoke test is a feature/edge case with a happy path. In my opinion, a Smoke test is a specific type of Acceptance test. Smoke tests shouldn’t do tests for parameter validation, missing data or other types of test that require very specific set-up. Even worse, having a specific set-up for Production Smoke tests means we’ll have static test data as we don’t want dynamic test set-ups in that environment.

Slowness to test – If we already have Smoke tests down to a minimum but running the suite still feels slower than it should then maybe it is a performance issue. Is the performance being tracked?

Using Continuous Integration software for Smoke tests means that Ops has to use yet another tool. They should be able to have information about the state of an application without needing to go to the team.

Configuration is within the application – It is not obvious that if an application’s stack is changed that we can try to Smoke test that variation without a new build.

Status of Ops changes is not visible – Until an environment’s Acceptance and Smoke tests are run, there is no information about external changes. This actually makes us lose the advantages of Continuous Integration and move closer to the concept of a Nightly Build…

Some simple steps that are being done to try and fix it:

Review all Acceptance and Smoke tests to make sure they don’t overlap and that only essential Smoke tests are being performed.

Host information should be available in the runner so new variations can be tested within minutes. This also makes a case for keeping configuration outside a build.

Continuous run of Smoke tests in every environment including Production – it would be useful to know when a specific path breaks while working, giving all teams information about the stack as a whole.

Visibility of Smoke tests results - Moving closer to the concept of a status dashboard, the smoke tests should be able to be checked through an API and screen by all teams.

The increasing number of metrics being kept – Measuring API responses and number of errors will provide more information about response times, error baselines and more.

What we have so far:

Ops changes can be more aggressive, they don’t have to wait and we’re not afraid to fail as this is done quickly and very early in the infrastructure.

This isn’t new but it was good to bridge concepts from the Dev team into the whole of the stack and it has allowed us to move faster to bring more servers up and add new tools without hurting current infrastructural dependencies.