Technology Blog

Like many companies we’ve been struggling with a problematic pay review process. In our case the feedback mainly revolved around it feeling arbitrary and lacking transparency.

Around the time we were discussing this the Valve Handbook got posted, within which it talked about their peer review & stack ranking system:

“We have two formalized methods of evaluating each other: peer reviews and stack ranking. Peer reviews are done in order to give each other useful feedback on how to best grow as individual contributors. Stack ranking is done primarily as a method of adjusting compensation. Both processes are driven by information gathered from each other—your peers.”

Awesome, you get rated by your peers rather than a manager or HR person, who has no idea what you do (not that we did that anyway)! I liked this idea a lot and got to work on doing our own version.

I started with a trial peer review survey with one team. There were some positives, but it mostly went down badly. People really didn’t like the stack ranking and also that I only asked a few high level questions with the answer being a score out of 10.

So we went back to the drawing board. We got representatives from all our teams and held 4/5 sessions where we broke down the larger themes (Skill, Productivity, Communication, Team) into more detailed and objective questions. After a lot of persistence and effort we finally put this all together and we had the survey!

Which I promptly canned…

Trying to measure performance

The fundamental problem I was (naïvely) trying to solve with a peer review survey was to bring in a degree of measurement, which would hopefully mean people felt the pay review process had a quantifiable aspect and didn’t just come down to one person’s opinion. However we were getting into the terrain of incentivising our people based on individual optimisations (rather than organisational or team goals & objectives) and – most disturbingly – the anonymous feedback aspect just felt very wrong. It was and is completely contrary to our culture and the things we stand for. The trade-offs simply weren’t worth it.

Regardless of the unpleasantness of anonymous feedback, everywhere I’ve heard of using ranking/measurement schemes have really bad stories to tell, such as Microsoft and GE.

Warning signs everywhere.

Don’t mix pay reviews with feedback

Another problem is the survey would have been a kind of feedback mechanism. Imagine getting your results – all nicely presented in bar charts – and finding you scored really badly on one section. What the heck are you supposed to do with that?! I’m a really bad communicator? What do they mean by that? Who thinks that? Great, everyone thinks I’m rubbish but I’ve got no way of finding out why apart from going around everyone and asking them. Ouch!

I am a big believer in regular 1-2-1s (I’ve talked about them a bit here). As Head of Development I start every day with a 1-2-1 with one of my department (I see all 35+ people as regularly as I can). Each team also does 1-2-1s (usually with their Lead if they have one). Each new joiner gets a mentor who they have a monthly 1-2-1 with for their first 3-6 months.

By the time you get to a pay review their should be no surprises, no feedback that you haven’t already heard before.

Where we are now

Our latest attempt is heavily based (& in some parts very plagiarised, I have to admit) on the StackExchange compensation scheme.

The peer review survey wasn’t a complete waste. I adapted the themes and questions from the survey we built into a set of core values we desire from our colleagues to be used for guidance. This is still very new and as yet unproven, but it certainly feels a lot better than where heading previously. I could explain in more detail, but it would be duplicating what I’ve said in the document/guide, which you can download here:

7digital Dev & DBA Team Compensation

Finally, a word on annual performance appraisals

I’ve only been employed by one company who ran annual performance appraisals and was far from impressed. It’s something we’ve consciously avoided for our tech teams at 7digital. I could go into more details as to why they are wrong, but others much more qualified than me have already done so:

The Paradox of Performance Pay – a really great article/opinion piece by Dr Allan Hawke , a former chief of staff to Australian prime minister Paul Keating, a former federal departmental secretary and a former senior diplomat. http://www.canberratimes.com.au/national/public-service/the-paradox-of-performance-pay-20120430-1xtys.html

W. Edwards Demming talking about performance appraisals as one of the five deadly diseases of management. http://www.youtube.com/watch?v=ehMAwIHGN0Y&feature=youtu.be&t=5m

An article in HR magazine on a study of British workers showing huge dissatisfaction with how performance appraisals are run. http://www.hrmagazine.co.uk/hr/features/1015028/hr-directors-try-harder-engage-employees-appraisals-process

Over the last month we’ve started using ServiceStack for a couple of our api endpoints (go to the full ServiceStack story here) . We’re hosting these projects on a Debian Squeeze vm using nginx and Mono.

We ran into various problems along the way which we’ll explain, but we also managed to achieve some interesting things; here’s a summary. Hopefully you’ll find this useful.

Nginx

We’re using nginx and fastcgi to host the application. This is good from a systems perspective because our applications can run without root privileges.

For the communication between mono-fastcgi and nginx, we are using a unix socket file instead of proxying through a local port. This makes configuration much easier, as you map applications to files rather than port numbers, so the convention rules for this are much more straightforward. (Besides, you may be hit by a memory leak if you don’t use unix socket files.)

Furthermore, using files instead of ports has made our life easier for automated deployments because:

• We can just specify the socket file based on the incoming host header. This way you only need one app configuration in nginx for N sites. The key here is to use the variable $http_host in the fastcgi_pass setting of our nginx config file. Example:
• We can decouple deploy from release, and the latter will be an instant operation that doesn’t require a restart of the app or the webserver. The deployment procedure is therefore as follows:  
  1. Copy new release to servers.
  2. Start app server for new release listening on a different unix socket file than the current working release. i.e. /tmp/SOCK-myFqdn-newrelease. (We use nohup for this.)
  3. Do some test requests on the new release to verify that it works, using a custom host header that matches the unix socket file pattern used in step 2.
  4. Make a hard link (not symlink) on the current release to have a backup.
  5. Move the unix socket file of the new release into the the current release: i.e. “mv /tmp/SOCK-myFqdn-newrelease /tmp/SOCK-myFqdn”.
  6. Kill the old release.

The key step is (5) here. By moving the new release into the old release, without moving or removing the old release socket file before, we manage to decouple deployment operation from release, with an “atomic” mechanism which gives us zero downtime (we tested this with siege, a very interesting load testing tool, configured with 100 concurrent connections), without the need of adjusting load balancers as part of the deployment.

We’re still studying the best way to do step (6). For now, we just wait briefly to be sure the app has finished serving/processing the last requests. The problem with this approach is that we need to record the PID of the app to be able to kill it later (because lsof hasn’t worked in our scenario, after much testing, either after step 5, using the hard link, or before step 4, using the normal file; so we need to use pgrep after step 1 for now). We’re studying stopping the application via a final HTTP request (with the DELETE verb on the status endpoint maybe? sounds about right :) ), but calling Environment.Exit() from a web application doesn’t look right.

Capistrano

So, to be able to deploy to many servers at once, we use capistrano to launch the ruby scripts that control the 1-6 steps mentioned above. Another interesting thing we do with capistrano is configuration.

Many .NET developer teams are in the habit of managing configuration files at the app level, naming them with extensions that map to the environment. This is pretty handy because as a developer you have all the configuration values of all the environments right there in your IDE, in case you want to explore something.

But it has a big drawback: any configuration changes need to be done in the same repository as the code, which has the following consequences:

• Configuration changes end up in the long chain of continuous integration; you have to wait for building and testing infrastructure for your change to go through and be able to deploy it.
• Configuration ends up being mainly a developer-task. Sure, DevOps techniques now encourage this by trying to involve developers more in system administration and configuration, but in this case it is a drawback because it renders the systems team incapable of doing configuration changes because they are not normally familiar (and aren’t supposed to) know what’s the src/ sub-folder structure to locate the config files, and they don’t typically use the CI tool to see whether their configuration change has affected some test-suite.
• Sometimes configuration files end up being almost exactly the same between environments, only differentiated by one or two values. This makes locating errors in the config files a daunting task.

So we decided to implement the configuration at the deployment level: There is a base configuration file in the sources which contains default values. In a different repository, for each environment there is a ruby configuration file that only has the values that need to change, i.e.: production.rb, uat.rb, etc. This file simply contains variables whose name is the appSetting key and value needs to be replaced before deployment, or a hash map in which each key is an XPath expression to look for the proper node in the XML config file. Example:

This way we decouple implementation from configuration completely. If there are red builds all around your CI environment, you don’t need to do fancy things with stable branches to do configuration changes in production. Systems team could scale the application by adding new servers, without the developers knowing about it…

Mono

We’re using version 2.11.3. There are various bug fixes compared to the version that comes with Squeeze; namely, problems with min pool size specified in a connection string. Rule of thumb, if there’s a bug in mono then try the latest!

For build agents we use an even higher version, 2.11.5 (which has not been tagged yet and to get it, you need to clone the master branch), to be able to use xbuild with F#. There were some issues that we fixed to make this work both in the Mono repository and in the fsharp repository. (We found Mono to be faster than .NET when compiling builds and running tests!)

NB: Our systems team has helped a lot in ironing out the kinks of all this, kudos to them!

Over the last month we’ve started using ServiceStack for a couple of our api endpoints. We’re hosting these projects on a debian squeeze vm using nginx and mono. We ran into various problems along the way. Here’s a breakdown of what we found and how we solved the issues we ran into. Hopefully you’ll find this useful. (We’ll cover deployment/infrastructure details in a second post.)

Overriding the defaults

Some of the defaults for ServiceStack are in my opinion not well suited to writing an api. This is probably down to the frameworks desire to be a complete web framework. Here’s our current default implementation of AppHost:

For me, the biggest annoyance was trying to find the DefaultContentType setting. I found some of the settings unintuitive to find, but it’s not like you have to do it very often!

Timing requests with StatsD

As you can see, we’ve added a StatsD feature which was very easy to add. It basically times how long each request took and logs it to statsD. Here’s how we did it: It would have been nicer if we could wrap the request handler but that kind of pipeline is foreign to the framework and as such you need to subscribe to the begin and end messages. There’s probably a better way of recording the time spent but hey ho it works for us.

RestServiceBase, Exceptions and Error Responses

One of my biggest bugbears with ServiceStack was the insistence on a separate request and response object, the presence of a special property and that you follow a naming convention all in the name of sending error and validation messages back to the client. It’s explained at length on the wiki and Demis was good enough to answer our question.

RestServiceBase

The simple RestServiceBase that comes with provides an easy way of getting started, but there aren’t many hooks you can use to manipulate how it works. It would be nice if you could inject your own error response creator. We ended up inheriting from RestServiceBase and overriding how it works:

We basically chopped out the bits we’re not using and changed the thing that creates the Error Response:

This allows us to respond with an error no matter what the type is for the request or what response you are going to send back. It provides us with extra flexibility above what is provided out of the box. In a nutshell, if there’s an exception in the code we will always get a stack trace in the response if debug is on.

Validation

We had the same issue with the validation feature; if you don’t follow the convention you don’t get anything in the response body. So we followed the same practice and copied  the ValidationFeature and tweaked it how we wanted it.

Conclusion

I like ServiceStack; it’s really easy to get up and running and whilst it has it’s own opinions on how you should work, what framework doesn’t?

The article we posted on our development team productivity back in May got picked up by Derek Jones – a keen enthusiast of statistical analysis, R and how/if we can apply more empirical (and meaningful) measurement to software development, a worthy but daunting challenge! We were more than happy to give Derek the raw data behind our productivity stats and very keen see what he could do with it (particularly as my statistical analysis skills are questionable at best).

Derek has written up is findings on his blog, titled Descriptive statistics of some Agile feature characteristics. Some of this is pretty intense and will no doubt be causing flash backs for many (mild sweats & panic, recurring dreams of not having prepared for exams), however you can skip over the explanations for calculations (if you wish) and see some really interesting ways to model and represent the data. Unlike our report Derek has wisely chosen not to attempt to draw conclusions or make predictions, but merely pose data in interesting ways, allowing us to draw our own.

Derek’s analysis of our data will also be appearing in the book he is currently writing, Empirical Software Engineering with R.

If you have any comments please leave them on Derek’s blog – I know he’d really appreciate the feedback.