RESTful is hardly harmful.

A provocative essay came up on Hacker News today, entitled RESTful considered harmful.

The summary of the essay:

  • JSON is bloated in comparison to protobufs and similar binary protocols
  • There are no interface contracts or data schema
  • HATEOAS doesn’t work
  • No direct support for batching, paging, sorting, etc – eg no SQL semantics
  • CRUD is too limited
  • No, really, CRUD is too limited
  • HTTP Status codes don’t naturally map to business semantics
  • there’s no queueing, or asynchrony
  • There are no standards
  • Backward compatibility is hard

Let’s have a look at the validity of these concerns.

1. JSON is bloated in comparison to protobufs

The essay cites “one tremendous advantage of JSON”: human readability, and then completely discounts this advantage by saying that it’s bloated. It really is a tremendous advantage, which is why XML won over MQ’s binary protocol and the XDR from Sun RPC, and the NDR from DCE RPC, and every other frigging binary protocol. And readability is why JSON displaced XML.

Ask yourself this: what is the value of readability versus the performance advantages of the alternatives, like Thrift or protobufs? Is readability worth 1x as much as the improved efficiency you might get with protobufs? 2x? I believe that for many people, its worth 100x. It trumps all other. For uber-experts, it’s deceptively attractive to wave away the advantage of human-readability. For the rest of the world, for 97% of developers, it’s a huge, Huge, HUGE advantage. For high speed financial trades, JSON is wrong. For Google’s internal interfaces, wrong. For most of the world, RIGHT.

AND as the essay notes, REST doesn’t prescribe JSON. Or XML. Or anything. There’s a content-type header, and clients and servers can negotiate it. If the client says Accept: application/x-protobuf, and the server can send it, bliss for you. So this point – “JSON is bloated” – is not only not valid (false) in the first place, it’s also not an argument against REST.

2. There are no interface contracts or data schema

This is a feature. OMG, have we not tried this enough times? Did this guy skip his “History of IDL compilers” course in the Computer History department at school? Sun RPC IDL. DCE RPC IDL. Corba IDL. WSDL, ferpeetsake! XML Schema!!

It’s pretty straightforward to deliver plain-old-XML over HTTP, which is quite RESTful. More popular is JSON-over-HTTP. Either of those have schema languages. Few people embrace them, though. Why? Because IDLs and Schema languages are too much structure, and they handcuff people more than help them. We have fortunately learned from the past. There are more tools coming in this area, for those who wish to embrace them. See .

3. HATEOAS doesn’t work

Mmmmm, yep. No argument here. In my experience, nobody really uses this, in practice. Pragmatic REST is what people do, and it generally does not use HATEOAS.

4. no SQL semantics

Uhhuh, true. This has been addressed with things like OData. If you want SQL Semantics, seek solutions, don’t just complain.

5. CRUD is too limited

Really? This is a problem? That you might need a switch statement in your code to handle different types of events? Really?

6. CRUD is really too limited


Mmmmm, sorry. I have to stop now. I’m completely bored of responding to this essay by now. Except for one more:

10. Backward compatibility is hard

This has NOTHING to do with REST. This is just true. Back compat in any interface is tricky.

In summary, I don’t find any of the arguments compelling.

Let me draw an analogy. The position in this essay is like saying “Oil is no good as a transportation fuel.” Now, Oil has it’s drawbacks! Oil is dirty. We can imagine alternatives that are better in theory. Even today, in specific local situations (daily use, short trips, urban travel) electric cars are better, MUCH better, than fossil-fuel based cars. (An bicycles are even better than electric cars) But gasoline-powered cars deliver massive utility to billions of people. Gasoline refueling stations are everywhere. The delivery system for gasoline is mature and redundant. The World RUNS, very effectively, on gasoline-powered transport, by and large. Objectively, Oil is VERY GOOD as a transportation fuel.

Sure, we’ll evolve better approaches in the future. That’s great. And sure, we can imagine a world with electric-powered vehicles. But today, in the world of reality, Oil wins.

And likewise Pragmatic REST, HTTP, JSON, and schema-less interfaces are winning. We’ll evolve better approaches. But today, This platform wins.

HTTP, HTML, Javascript, and JSON are ubiquitous, are the foundation of the web, and are not going anywhere. Any architect is free to choose other options, and they might have good reasons for doing so. On the other hand the vast majority of installations won’t benefit from using protobufs or thrift, or some non-HTTP protocol. Pragmatic REST, JSON and HTTP are very very safe choices in the vast majority of scenarios.


API Growth, SOAP v REST, etc

From, John Musser’s GlueCon slides.  Interesting data pulled from .   As I understand it, ProgrammableWeb is mostly a repository of APIs. It’s free to register and I don’t believe there is any sort of authentication – in other words anyone can post any web API.  Each listing includes an API endpoint, a service.

The main point Musser makes is that APIs are continuing to grow exponentially.  He didn’t go so far as to coin a new law (“Musser’s Law”) for the growth rate, but the trend is pretty certain.

Some other interesting takeaways from the slides:

  • REST v SOAP is one of the themes.  PW’s data shows REST waaay outpacing SOAP and other alternatives.

(For the REST purists who say REST is an architecture and SOAP is a set of envelope standards, my response is, yeah, but… we all know that’s not really true. ) 

  • Musser shows how gosh-darned simple it is to express “Get the current IBM share price” in REST, vs how complex it is in SOAP.  This is true, but misleading if you are not careful.  Lots of API calls require a small set of input parameters. Some do not.  For a counter example look at the complexity of posting a Twitter messaging using OAuth, which I wrote about previously.
  • There are companies doing more than a billion API transactions per day.  Twitter, Google, Facebook – you figured those. Also AccuWeather, Sabre, Klout, NetFlix.  Those are just the ones we know about.

I remember registering for Programmable Web about 10 years ago.  There were 4 or 5 online book-sellers in those days, each had different prices. I put together a book price quote service, which would accept an ISBN, then screen-scrape HTML from each of them, and respond with three or four prices – not all vendors listed all books.  It was just a novel example.  I used it in demonstrations to show what an API could do.  It was never used for real commerce.

I published it.  The screen scraping was a brittle joke.  Every time Amazon updated their HTML, the service would break, and people would email me complaining about it.  Though that service died a long while ago, it remains listed on ProgrammableWeb today. Because of that I think it’s important to consider the real implications of the PW numbers – not all the listed services are alive today, and many of them are not “real” even if they are alive.

Even with all that, the steepening upward trend does show the rapid ramp up of API development and usage.  Even if those APIs are not real, there are real developers building and testing them.  And real companies making a business of it (Apigee, to name one).  Also,  it’s pretty clear that JSON is winning.

One can gain only so much insight from viewing slides. It would have been nice to hear Mr Musser’s  commentary as well.

Side note – The term “API” used to denote a programmer’s interface exposed in a library.  Developers would use an API by linking with a library.  The term has silently expanded its meaning to include application message protocols – in other words, how to format and send messages to a service across a network.  These things are very different in my mind, but yet the same term, API, is used to describe them both, these days.


HTTP apps? REST? JSON? XML? AJAX? Fiddler is invaluable

For developers, having access to and knowing how to use the proper tools is invaluable.  For any sort of communication application, I find Fiddler2 to be indispensable.  It is an “HTTP Debugging Proxy”, but ignore that – the main point is that it lets a developer or network engineer see the HTTP request and response messages, which means you can get a real understanding of what’s happening.  It’s WireShark for HTTP.

As an added bonus, in sniffs SSL traffic, and can transparently display JSON or XML in an appropriate human-friendly manner.

The name Fiddler refers to the ability to “fiddle” with requests as they go out, or responses as they arrive. This can be really helpful in trying what-if scenarios.

Free, too.  Thanks to EricLaw for building it.

I want to point out: this tool is not commercial, there’s no training course for it, there’s no vendor pushing it. It illustrates that developers need to train themselves, to stay current and productive. They need to keep their eyes open and add to their skills continuously, in order to stay valuable to their employers.

Twitter’s use of OAuth, part 3

In the prior post, I described the message flows required for granting a client application write access to a user’s Twitter status. In this post, I’ll cover the additional use case, posting a status message.

Use Case 2: Approved Client posts a status

The output of a successful authorization of a client application is an oauth_token and an oauth_token_secret. Twitter does not “expire” these items and so the client can re-use them forever.

Typically, clients (applications) will store the oauth_token and oauth_token_secret in a settings file. A .NET application on Windows might put the file in %AppData%, in other words, \users\Someone\AppData\Roaming\Vendor\Appname\Settings.xml.

To post status updates to a user’s Twitter stream, the client application can simply POST to /1/statuses/update.xml or /1/statuses/update.json, including an Authorization header constructed as described previously. In fact there is just one request+response for this use case. This header is formed in exactly the same way as described in the first use-case, except that there are now different required parameters. For regular status updates, the client needs to construct the signature base using:

  • all the oauth_ parameters (except the “secret” ones). This will include the oauth_token obtained from the approval dance, but not the oauth_verifier, which is a one-time thing, or the oauth_callback which is used only when requesting client approval.
  • the complete status parameter, URL-encoded, which must be inserted into the base in observance of lexicographic sorting order.

Again, the client must normalize the parameters – sort by name, encode the values, and concatenate together with ampersands – and then sign the UTF-8 encoded payload. The added twist here is that the payload to be signed must include named query string parameters. In the case of a status update to Twitter, there is just one query string parameter, ‘status’. That parameter must be included in the signature base, with the value OAuth-percent-encoded as described previously. The result must be inserted into the base in observance of the lexicographic sorting requirement.

The resulting signature base looks something like this:


Notice that the actual status string is double-URL-encoded. Yes, you must URL-encode it once, and append it to the sorted list of parameters. Then that sorted list gets URL-encoded again, before being concatenated with the method and URL, separated by ampersands. The result, the signature base, is what you see here. To sign, the entire signature base is URL-encoded again, before being UTF-8 encoded and signed. This is the kind of thing that is hard to discern in the scattered Twitter documentation. A simple example like this makes everything clearer, though not exactly clear.

The resulting message with the computed signature looks like this:

Authorization: OAuth

(That Authorization header is all on one line). The response message, if you are getting XML, looks like this:

  <created_at>Fri May 11 19:34:12 +0000 2012</created_at>
  <text>Twas brillig, and the slithy toves...</text>
  <source>Dino's TweetIt</source>
    <name>dino chiesa</name>
    <created_at>Wed Jul 22 15:20:26 +0000 2009</created_at>
    <time_zone>Pacific Time (US &amp; Canada)</time_zone>

If the client uses the .json suffix on the update URL, then it would get a JSON-formatted message containing the same information.

There may arise a situation in which the user has already approved the client application, but the client does not have access to the oauth_token and oauth_token_secret, and is therefore unable to properly construct the signature for the Authorization header. This might happen when a user gets a new PC or device, or tries to use the client application from a different PC, for example. In that case the client should simply request authorization again, as in Use Case #1, described in the previous post. If this happens, the /oauth/authenticate endpoint can automatically redirect back to your callback without requiring any user interaction or even showing any Twitter UI to the user.

Twitter’s use of OAuth, part 2

Last time, I covered the basics around Twitter’s use of OAuth.  Now I will look at the protocol, or message flows. Remember, the basic need is to allow a user (resource owner) employing an application (what OAuth calls the client) to post a status message to Twitter (what OAuth calls the server).

There are two use cases: approve the app, and post a status message. Each use case has an associated message flow.

Use Case 1: Initial client (application) approval

The first time a user tries to employ a client to post a status update to Twitter, the user must explicitly grant access to allow the client to act on its behalf with Twitter. The first step is to request a temporary token, what Twitter calls a “request token”, via a POST to like this:

Authorization: OAuth oauth_callback="oob",
Connection: Keep-Alive

In the above message, the Authorization header appears all on one line, with each named oauth parameter beginning with oauth_ separated from the next with a comma-space. It is shown with broken lines only to allow easier reading. That header needs to be specially constructed. According to OAuth, the signature is an HMAC, using SHA1, over a normalized string representation of the relevant pieces of the payload. For this request, that normalized string must include:

  1. the HTTP Method
  2. a normalized URL
  3. a URL-encoded form of the oauth_ parameters (except the “consumer secret” – think of that as a key or password), sorted lexicographically and concatenated with ampersand.

Each of those elements is itself concatenated with ampersands. Think of it like this:


Where the intended METHOD is POST or GET (etc), the URL is the URL of course (including any query parameters!), and the PARAMSTRING is a blob of stuff, which itself is the URL-encoded form of something like this:


Once again, that is all one contiguous string – the newlines are there just for visual clarity.  Note the lack of double-quotes here. 

The final result, those three elements concatenated together (remember to url-encode the param string), is known as the signature base.

Now, about the values for those parameters:

  • The callback value of ‘oob’  should be used for desktop or mobile applications. For web apps, let’s say a PHP-driven website that will post tweets on the user’s behalf, they need to specify a web URL for the callback. This is a web endpoint that Twitter will redirect to, after the user confirms their authorization decision (either denial or approval).
  • The timestamp is simply the number of seconds since January 1, 1970. You can get this from PHP’s time() function, for example. In .NET it’s a little trickier.
  • The nonce needs to be a unique string per timestamp, which means you could probably use any monotonically increasing number. Or even a random number. Some people use UUIDs but that’s probably overkill.
  • The consumer key is something you get from the Twitter developer dashboard when you register your application.
  • The other pieces (signature method, version), for Twitter anyway, are fixed.

That form must then be URL-encoded according to a special approach defined just for OAuth. OAuth specifies percentage hex-encoding (%XX), and rather than defining which characters must be encoded, OAuth declares which characters need no encoding: alpha, digits, dash, dot, underscore, and twiddle (aka tilde). Everything else gets encoded.

The encoded result is then concatenated with the other two elements, and the resulting “signature base” is something like this.


The client app must then compute the HMACSHA1 for this thing, using a key derived as the concatenation of the consumer secret (aka client key) and the token secret, separated by ampersands. In this first message, there is no token secret yet, therefore the key is simply the consumer secret with an ampersand appended. Get the key bytes by UTF-8 encoding that result. Sign the UTF-8 encoding of the URL-encoded signature base with that key, then base64-encode the resulting byte array to get the string value of oauth_signature. Whew! The client needs to embed this signature into the Authorization header of the HTTP request, as shown above.

If the signature and timestamp check out, the server responds with a temporary request token and secret, like this:

HTTP/1.1 200 OK
Date: Fri, 11 May 2012 18:04:46 GMT
Status: 200 OK
Pragma: no-cache
Last-Modified: Fri, 11 May 2012 18:04:46 GMT
Content-Length: 147


Here again, the message is shown on multiple lines, but in response it will actually be all on one line.  This response contains what OAuth calls a temporary token, and what Twitter has called a request token. With this the client must direct the user to grant authorization.  For Twitter, the user does that by opening a web page pointing to, inserting the proper value of oauth_token.

This pops a web page that looks like so:

When the user clicks the “approve” button, the web page submits the form to, Twitter responds with a web page, formatted in HTML, containing an approval PIN. The normal flow is for the web browser to display that form.  The user can then copy that PIN from the browser, and paste it into the client application, to allow the client to complete the message exchange with Twitter required for approval.   If the client itself is controlling the web browser, as with a Windows Forms WebBrowser control, then the client app can extract the PIN programmatically without requiring any explicit cut-n-paste step by the user.

The client then sends another message to Twitter to complete the authorization. Here again, the Authorization header is formatted specially, as described above. In this case the header must include oauth_token and oauth_verifier set to the token and the PIN received in the previous exchange, but it need not include the oauth_callback. Once again, the client must prepare the signature base and sign it, and then embed the resulting signature into the Authorization header. It looks like this:

Authorization: OAuth

(In an actual message, that Authorization header will be all on one line). The server responds with a regular HTTP response, with message (payload) like this:


(All on one line). The oauth_token and oauth_token_secret shown in this response look very similar to the data with the same names received in the prior response. But these are access tokens, not temporary tokens.  The client application now has a token and token secret that it can use for OAuth-protected transactions.

All this is necessary when using an application for the first time, and it has not been authorized yet.

In the next post I’ll describe the message flows that use this token and secret to post a status update to Twitter.

Twitter’s use of OAuth, part 1

Sending messages to Twitter from any application is straightforward, if you follow the protocol.  The problem is, Twitter does not explicitly define the protocol in any one place.

If you go to the developer center on Twitter’s site, there is a section that deals with how Twitter uses OAuth (which is defined in IETF RFC 5849), but Twitter doesn’t explain exactly how.  Instead, Twitter refers to external sources (one, two) for the definition of the general OAuth protocol, and it expends a bunch of effort explaining why apps cannot use Basic Auth.  Not helpful for an implementer. Twitter’s doc page then points developers to existing third-party OAuth libraries.  There is no documentation that I could find that explicitly describes or demonstrates what I would call the communications protocol: the message flows required to use OAuth to connect to Twitter, for the various scenarios – a web application, a “traditional” desktop app, a mobile app, and so on.

Beyond the communication protocol, there is no firm guidance from Twitter on User Interface models – what an application should look like or do.  There is no firm guidance around the issues surrounding the OAuth consumer keys and secret keys.

For various reasons a developer might want more detailed information.  For one thing, the developer may be developing their own Twitter-friendly library, for various reasons: licensing or other intellectual property concerns, efficiency, control.  If you are developing your own application that uses Twitter, the upshot is: the implementation is the documentation. Twitter provides some reference material, but it is not complete; when you get down to it, you need to try and see. Develop, test, diagnose, see what works. Developers will have to figure it out themselves.

I went through that process with a couple of applications based in C#, and here’s what I learned.  The basics:

  1. OAuth isn’t mysterious or mystical. It’s well-defined. Using the protocol with Twitter requires some exploration.
  2. OAuth defines 3 parties to the transaction: a server, a client, and a resource owner. The server is the manager of resources; in the case of Twitter, twitter is the server.  The resource owner is the user.  The client is the application that the user employs to connect to the server – for example, something that posts Tweets on behalf of a user. This can be a web application (another server), a desktop application, a mobile phone app, and so on.
  3. All OAuth 1.0 clients (and remember, client = application) have an associated “consumer key” and  “consumer secret”.  In later drafts of OAuth 1.0 these were together termed “client credentials”.  When a developer builds an app, he registers it with the OAuth provider – in this case, Twitter – and receives those two opaque strings. As part of the OAuth protocol, the application (aka client) sends the consumer key to Twitter at runtime, encrypted with his consumer secret, in order to identify itself. If Twitter can decrypt the message with the copy of the secret it holds in association to that consumer key, Twitter concludes the client to be authentic.
  4. When the client (application) identifies itself to Twitter, Twitter checks to see if the client is authorized to act on behalf of the user or “resource owner”.  If not, Twitter instructs the client to tell the user to approve the client for this interaction, via a web page interaction. In other words, Twitter tells the client, “Have the user visit this web page…” and when the client does so, Twitter asks the user, “Do you approve this app?”.  For this approval, the user (resource owner) must authenticate directly to Twitter with his username and password, and can then reply yes or no.  With an affirmative reply, Twitter marks the client (really, the application, identified by the consumer key it presents to Twitter) as authorized to act on behalf of this particular user.
  5. Now that the client is identified and authorized, it can send requests to Twitter to update the resources on behalf of the resource owner. In simple terms: the app can post tweets to the user’s status stream.
  6. If the user wants to later de-authorize the client application from acting on its behalf, it can visit to do so.

A Concern about Client Identity

The use of the consumer key or client credential as the basis for application identification means that the claimed identity of desktop applications cannot be trusted.  Here’s why:  OAuth assumes that an application that holds these credentials is the registered application.  Yet most traditional desktop or mobile applications will store this information in the compiled binary, or in some external store. This information will be accessible to other applications, which means the “client identity” can be easily forged, either mistakenly or on purpose. A developer just needs to grab those two items and embed them in a new app, and you’ve got a “stolen application identity” problem.  In short, anyone that uses a desktop application to connect to Twitter has the power to forge the identity of that application.  This is one of a set of concerns around Twitter’s use of OAuth, that have been previously articulated.  I don’t have a good answer for this problem.

In my next post I’ll describe the OAuth 1.0a protocol flows.