Technology Blog

In addition to JSON support we’ve also been hard at work improving the API itself. Here’s some of the changes that we’ve introduced to Catalogue API recently. But first 2 important upcoming changes:

“RRP” pricing on release/details

Historically the release/details endpoint provided recommended retail price (RRP) even for releases that didn’t actually have RRP supplied to us by content licensors (i.e. music labels). This typically applies to singles/EPs with 1-4 tracks who tend to have RRP specified only on track level.

In these cases the release RRP provided by the API was calculated as sum of track prices appearing on given release. But as this has been an ongoing source of confusion we’ve taken the decision that we will be more explicit and we will start indicating that a release doesn’t actually have a RRP by returning NULL as the RRP value.

This behaviour is consistent with our documentation  (see http://api.7digital.com/1.2/static/documentation/7digitalpublicapi.html#Price) but nevertheless we’ve set up a test endpoint where you can see how release with no RRP will look like in action:

http://api.7digital.com/1.2/release/details/beta?releaseId=2686113&oauth_consumer_key=YOUR_KEY_HERE

(Please note that this endpoint will be switched off once the changes have been made to the regular endpoint so make sure you don’t use it in any live applications)

Release “barcode” on artist/toptracks endpoint is being deprecated

We will also no longer be providing release barcodes in responses of artist/toptracks API endpoint (no other endpoints will be affected). In short term we will not be removing the entire “barcode” element from the response but it will be populated with an empty string. We recommend you to remove any dependencies on the barcode data for this endpoint as in the future we will be removing it completely (we will provide additional advance warning).

Both above changes are preliminarily scheduled to take place on June 3rd 2013. Please get in touch with us ASAP should you have any questions/concerns about this.

Release duration, release track count & track disc number

On the other hand we’ve been gradually adding the following information to relevant endpoints-

For releases:

• duration – total length in seconds of all tracks appearing on release
• trackCount - number of tracks appearing on release

At the moment these new fields appear at the end of release (or track) response but for readability we will be putting them in their logical order within the response,  e.g. so that trackNumber and discNumber appear next to each other. Now maybe a good time to double-check you don’t rely on order of the XML elements in API responses.

Pricing info available in more API endpoints

We’ve also added pricing information to more Catalogue API methods so that you don’t have to make multiple additional API calls e.g. to fetch prices for all tracks appearing in a chart.

All the following now provide prices:

artist/releases
release/details
release/tracks
release/search
release/chart
release/byDate
release/byTag/top
release/byTag/new
release/recommend
track/search
track/chart

More relevant API endpoints will be getting pricing info added in the near future.

Catalogue content filtering

Another functionality added to our API is content filtering which allows you to exclude certain types of content as you’re browsing the 7digital Catalogue API.

You can filter content from specific licensors or hide content not cleared for subscription based on-demand streaming services. Filtering can be handled by yourself using optional request parameters (e.g. see artist/releases documentation) or if you’re a Premium API user we can do the job for you and filter specific content for all requests by your API key directly on our servers.

We’re also planning to support additional content filters in the future (e.g. label, explicit content, format) please do let us know what you’d find most useful.

By popular demand we’re working on adding support for JSON response format to our API. We’re making a good progress and are now able to share some of it with you.

Please note that this is still work in progress and we strongly advise you not to use JSON responses in any production evironment yet as we might be making minor changes to the format of the responses based on your feedback. Please do let us know how you find it, either in our 7digital API Google Group or by contacting us directly at api@7digital.com

We’ll give everyone heads up once we’re happy the JSON support is stable enough to be used in live applications (at which point it will be also added to the API reference documentation).

Supported API endpoints

So far we have the following API endpoints already supporting JSON:

artist/chart
artist/browse
artist/releases
artist/search
release/byDate
release/chart
release/search
track/chart
track/search
tag
artist/byTag/top
release/tags

The remaining Catalogue API endpoints will be following shortly.  And based on your demand we will be adding JSON to other endpoints across the entire API through out 2013 (e.g. Locker API, Basket API, etc). Also any new API endpoints added to 7digital API will support JSON out-of-the-box.

Requesting JSON

You can ask for JSON responses by providing an Accept HTTP header with your API call, i.e.:

Accept: application/json

Curl example:

curl -H "Accept: application/json"
"http://api.7digital.com/1.2/artist/releases?artistId=1&oauth_consumer_key=YOUR_KEY_HERE"

Example structure of JSON responses: Don’t forget to let us know if you have any questions or feedback.

On Monday 3rd and Tuesday 4th December Paris held host to the first international API focused event in Europe – APIdays.io.  Myself, and my colleague Hibri, eagerly took part and we gave a short presentation on how 7digital grew their public API, the lessons we learned and the effect it had on the way we work.  You can view the slides at the end of this post.

We received great feedback from our slides – it felt as if many people are just getting started in the world of APIs whilst 7digital have had their public API for many years and that they were very interested in hearing our real-world story.

The format of the event was a little odd, with talks in slots of less than 30 minutes, which on the plus side meant that we got to see a lot of different viewpoints and experiences but that there wasn’t enough time for anyone to get deep into a topic.  I’d like to suggest that a technical track has available 1 hour slots for anyone who wants to host a full-on technical presentation and debate – it felt like we barely scratched the surface in less than 30 minutes. It was a great couple of days, and the first time I’d ever been to Paris.  I’m hoping that this event becomes a regular conference and that next time we can get far more technical with the content and swap the really gritty stories of lessons learned.

(Cross-posted from my personal blog here)

We will be performing an essential upgrade to our core network infrastructure on Thursday August 9th at 11am BST. This will affect the whole 7digital Platform and our services might be unavailable for up to 5 minutes. Whilst the upgrade is taking place the 7digital servers will be unreachable.

Apologies for any inconvenience caused.

We have just released a beta version of our new Track Search api endpoint. It can be accessed here:

http://api.7digital.com/1.2/track/search/beta?q=muse%20resistance&oauth_consumer_key=YOUR_KEY_HERE

With the same parameters as the current ~/track/search endpoint.

It uses Apache Solrs eDismax query parser and allows us to provide much more relevant search results than our current SQL Full Text Search based system.

It also supports the usage of basic syntax to hone down your search such as wrapping a phrase in “quotes” and + and -.

For example:

under the bridge -"All Saints"

We’re still tweaking it at the moment, and currently it will not be kept 100% up to date with the latest tracks, but it does have our entire catalogue. We would definitely appreciate some feedback on the tracks that are there!

After more than 5 years in service the time has come to retire the 7digital XML feeds. But this is good news as we have a brand new feeds system in place as a replacement, please read further down for more details.

THE MOST IMPORTANT PART FIRST:

If you’re currently using the old FTP-based XML feeds and haven’t received an email with upgrade instructions from us please contact api@7digital.com before Wednesday July 18th. The current FTP-based XML feeds will be permanently turned off on October 1st 2012.

*

The new 7digital Catalogue Feeds

Here are highlights of the most important changes:

• new format – instead of XML the new feeds are provided in much more compact CSV format. We’ve also switched from 7zip compression to gzip.
• incremental updates – in addition to weekly full catalogue dumps we are now also publishing daily feeds containing only catalogue content that has been updated in the previous day
• API compliant – the data field names now match our standard API naming conventions. FTP access is replaced with HTTP access with OAuth authentication so that you can use your standard API credentials

If you’re not an existing user and you’re interested in access to the new 7digital Catalogue Feeds please contact sales@7digital.com.

Why metrics?

Since I joined 7digital I’ve seen the API grow from a brand new feature side by side with the (then abundant) websites to be the main focus of the company. The traffic grew and grew and keeps on growing in an accelerated pace and that brings us new challenges.

We’ve brought the agile perspective into play which has made us adapt faster and make fewer errors but:

• We can do unit tests but they don’t bring out the behaviour.
• We can do integration tests but they won’t show the whole flow.
• We can do smoke tests but they won’t show us realistic usage.
• We can do load test but they won’t have realistic weighting.

Even when we do acceptance criteria we are actually being driven by assumptions, even with an experienced developer he is really just sampling all his previous work and as we move to a larger number of servers and applications it’s not humanly possible to take all variables into consideration.

It is common to hear statements like ‘keep an eye on the error log/server log/payments log when releasing this new feature’ but when something breaks it’s all about ‘what was released/when was it released/is it a specific server?’. As the data grows it becomes harder to sample and deduce from it quickly enough to feedback without causing issues, especially when agile tends to implement intermediary solutions which might have different behaviours from the final solution that have not been studied.

The truth is that nothing replaces real life data and statistics – including developers opinions – if it the issue is a black swan then we need to churn out usable information fast!

Taken from @gregyoung

This has been seen before by other companies; for example, Flickr on their Counting and Timing blog post. See also Building Scalable Websites by Flickr’s Cal Henderson.

This advice has been followed by other companies like Etsy on their Measure Anything Measure Everything blog post or Shopify on their StatsD blog post.

How to do it?

Decided to start with a winning horse I picked up the tools used by these companies:

StatsD is described as “a network daemon for aggregating statistics (counters and timers), rolling them up, then sending them to graphite”.

Graphite is described as “a highly scalable real-time graphing system. As a user, you write an application that collects numeric time-series data that you are interested i[...]. The data can then be visualized through graphite’s web interfaces.”

The way to implement these is available in several tutorials and I used StatsD own example C# client to poll our own API request log for API users, endpoints used, caching and errors.

In the future it would be ideal for the application to access StatsD itself instead of running a polling daemon.

There are a lot of usable features on Graphite. The ones I’ve used so far include Moving Average which will smooth out spikes in the graphs making it easier to see behaviour trends in a short time range and Sort by Maxima.

There are even tools to forecast future behaviour and growth using Holt Winters Forecasting Statistics and this is used by companies to understand future scalability and performance requirements based on data from previous weeks, months or years (seen in this Etsy presentation on Metrics)

How it looks and some findings

Right away I got some usable results. An API client had a bug in their implementation which meant they required a specific endpoint more often than they would use it – this data can help out with debugging and also prevent abuse.

Sampled and smoothed usage per endpoint per API user…

Another useful graph is error rates, which might be linked with abuse, deploying new features or other causes.

Error chart smoothed with a few spikes but even those are on the 0.001 % rate

Here is some useful caching information per endpoint to know how to tune up TTLs or look for stampede behaviour.

Sampled and smoothed Cache Miss per Endpoint

Opinion

After you start using live data to provide feedback for your work there is no going back. It is my opinion that analysis of short and long term live results of any type of work should be mandatory as we move out of an environment that is small enough to be maintained exclusively by a team’s knowledge.

Went to Amsterdam Music Hack Day and left with a working Tomahawk Resolver for 7digital previews, a spiked locker integration and some cool ideas on how to promote 7digital with Open Source

This will allow Tomahawk to use 7digital’s track search and listen to previews, in the future it will integrate buy buttons and help out Tomahawk to work with 7digital :-)

The locker integration is spiked using a local service which would be provided/maintained by either 7digital or a third party so it won’t be available immediately after the demo as there are feature, performance and security concerns.

What was used

http://developer.7digital.net/ 7digital API with the demo musichackday API client key

https://github.com/tomahawk-player Tomahawk resolver examples and documentation

API locker service built in Ruby with Sinatra and the 7digital ruby API client gem. Also uses the JSON gem.

Also built some local stubbed API responses using Ruby Sinatra as the connection was slow/failed sometimes :-)

Links

Don’t judge me on my JS..

https://github.com/goncalopereira/7digital-tomahawk-resolver working search with previews with musichackday key

https://github.com/goncalopereira/7digital-stubs stubs…

Service example code

This will authorise the user with a premium api account and get the locker, it is a slow implementation as locker search is not available and no caching was added but works.

require 'rubygems'
require 'sinatra'
require 'sevendigital'
require 'very_simple_cache.rb'
require 'json'

before do @api_client = Sevendigital::Client.new(:oauth_consumer_key =>’x’, :oauth_consumer_secret => 'x', :lazy_load? => true, :country => ‘GB’, :cache => VerySimpleCache.new, :verbose => "verbose" )

end

post '/authorise' do content_type :json

email = params["email"] password = params["password"] search = params["search"]

user = @api_client.user.authenticate(email,password) #not good, should cache more locker = user.get_locker() #also caching missing

valid_results = brute_force_search_on_string_compare(user, locker.locker_releases, search)

valid_results.to_json end

We will be making a change to the default image we serve when we do not have an a proper artist image available.

Currently we redirect requests, via an HTTP 302, to the following image: http://cdn.7static.com/static/img/artistimages/00/000/000/0000000000_200.jpg

This is due to change to: http://cdn.7static.com/static/img/artistimages/00/000/000/_defaultartist_286.png

No other change to behaviour is planned and we do not expect this to affect any API consumers or client applications. However, we advise checking that this won’t cause a breaking change in your application and amend as necessary. We recommend not relying on this URL in any manner, and simply check for the 302 redirect if you want to ascertain whether a non-default artist image is available.

Planned change date is Monday 27th February.

 

I’ve started to explore mono, with a view to moving some of our web applications to Linux. Used MonoDevelop  on OSX to  spike a simple HttpHandler to return a response.  I was more interested in how the hosting and deployment story worked with mono.

This is a little list of things I discovered as I went along.

http://www.mono-project.com/ASP.NET has  list of the hosting options available.  Went with the Nginx option.  Mono comes with xsp, which is useful for local testing.

Running a simple web application

To run xsp  /usr/bin/xsp –port 9090 –root  <path to your application>, and the application will be available on http://localhost:9090

 

To install Nginx on OSX,  get Homebrew.  And then simply  sudo brew install nginix

Follow the instructions here http://www.mono-project.com/FastCGI_Nginx to configure Nginix to work with Mono’s FastCGI server.

On OSX, the Nginix configs can be found in /usr/local/etc/nginx/nginx.conf

This is the configuration I tried for my testing,

In /usr/local/etc/nginx/nginx.conf

server{
   listen 80;
   server_name localhost;
   access_log /var/log/nginx/localhost_mono_access.log;
   location / {
        root /Users/hibri/Projects/WebApp/;
        index default.aspx index.html;
        fastcgi_index default.aspx;
        fastcgi_pass 127.0.0.1:9000;
        include /usr/local/etc/nginx/fastcgi_params;
   }
}

Add the following lines to /usr/local/etc/nginx/fastcgi_params

 fastcgi_param  PATH_INFO          "";
 fastcgi_param  SCRIPT_FILENAME    $document_root$fastcgi_script_name;

 

Start Nginx.

Start the Mono FastCGI server

fastcgi-mono-server2
     /applications=localhost:/:/Users/hibri/Projects/WebApp/ /socket=tcp:127.0.0.1:9000

And the application is available on http://localhost

Web Frameworks

We use OpenRasta for the services I want to run on Linux. OR didn’t work out of the box. This is something I’ll be exploring in the next few days.

Tried ServiceStack too, and was able to get one our projects (https://github.com/gregsochanik/basic-servicestack-catalogue) working on Mono as is.  Nancy is next on the list.