Pre-request script for Postman, to calculate HttpSignature

If you do REST, you probably have a favorite REST client testing tool.
Mine is Postman, a Google Chrome app.

Postman has a nifty feature called Pre-request scripts, which allows you to write some Javascript code that performs a calculation and then reads and writes the “environment” object for the request. This means you can calculate … hashes or digests or signatures or anything you like.

Here’s an example of a script that calculates an HMAC-SHA256 HttpSignature, using the keyId and secret-key embedded in the environment. It also computes a digest on the message payload. Postman helpfully includes CrytoJS in the JS sandbox, so it’s easy to do HMAC and SHA.

In this particular case, the HttpSignature verification on the server requires 2 headers (date and digest) plus the ‘(request-target)’ value. The digest is a SHA-256 of the payload, which is then base64 encoded.

Anyone can start with this and modify it to do other variations.
Good luck!

Addendum

I should have mentioned this: Postman, even the latest Chrome app version, uses XmlHTTPRequest to send out requests. XHR is purposefully limited, in the specification, to restrict some headers from being set explicitly on outbound requests. The list of restricted headers includes: Origin, Date, Cookie, Via, and others. The reason for this restriction: it is desired that the user-agent be fully in control of such request headers.

My example above uses an HttpSignature that signs the Date header. This means the code must also SET the Date header to a known value; in my case I am using a value generated by the pre-request script.

postman-set-headers

The value corresponds to “now”, the current moment. But the point is the value has to be absolutely known.

This is not possible in standard Postman, because of the XHR restriction. The effect you will see is that the intended Date header silently does not get sent with the request.

This may confound you if you don’t have an ability to trace the request on the receiving (server) side. In the context of a request that uses HttpSignature, the server will throw an error saying “Missing Date header”.

But! in Postman v0.9.6 and above, it is possible to configure Postman with something the Postman people call “the Intercptor plugin”, which then allows the lifting of this restriction. The Date header gets sent, and everything works.

If you don’t want to rely on the Interceptor plugin, and you want the HttpSignature to include the date value, then you’ll have to use a differently named header to hold the date. Use X-Date or anything other than “Date”. You need to change the client as well as the server of course, to make everything hold together.

Online calculator for SHA and HMAC-SHA

Here’s a thing I built. It’s just a webpage that calculates SHA-(1,224,256,384,512) and HMAC with the same algorithms.

I was using this to help with building a system that relies on HttpSignature. Developers need some help in constructing and validating their HMACs and SHAs.

The spec formerly known as Swagger is now OpenAPI

headline

Swagger has been renamed! Three weeks ago. I didn’t realize this, and (forgive me) I’ve been continuing to use the term “swagger” when I really should have been using “OpenAPI”, in the time since.

OAI Logo

Helpfully, Marsh, an esteemed colleague of mine, has produced a slackbot to remind me to use the world “OpenAPI” every time I type the… uh… old word… in slack chats. Now, I just need that slackbot to follow me around and remind me every time I *say* the old word.

There’s a new group, the OpenAPI Initiative, whose members include IBM, Google, Apigee, Intuit, Microsoft, Paypal… these members will govern the evolution of the spec.

REST Assured (hahahaha! ya get it?) that Apigee will be building some nice innovations on top of the OpenAPI spec. Exciting things coming soon. You can already see the beginnings at apistudio.io.

Apigee_API_Studio

It’s not difficult to imagine some interesting possible paths forward, from that tooling.

And, omigosh, I just realized that I haven’t posted an article here in about 6 months! Wow I must have been busy…

RESTful is hardly harmful.

A provocative essay came up on Hacker News today, entitled RESTful considered harmful.

The summary of the essay:

  • JSON is bloated in comparison to protobufs and similar binary protocols
  • There are no interface contracts or data schema
  • HATEOAS doesn’t work
  • No direct support for batching, paging, sorting, etc – eg no SQL semantics
  • CRUD is too limited
  • No, really, CRUD is too limited
  • HTTP Status codes don’t naturally map to business semantics
  • there’s no queueing, or asynchrony
  • There are no standards
  • Backward compatibility is hard

Let’s have a look at the validity of these concerns.

1. JSON is bloated in comparison to protobufs

The essay cites “one tremendous advantage of JSON”: human readability, and then completely discounts this advantage by saying that it’s bloated. It really is a tremendous advantage, which is why XML won over MQ’s binary protocol and the XDR from Sun RPC, and the NDR from DCE RPC, and every other frigging binary protocol. And readability is why JSON displaced XML.

Ask yourself this: what is the value of readability versus the performance advantages of the alternatives, like Thrift or protobufs? Is readability worth 1x as much as the improved efficiency you might get with protobufs? 2x? I believe that for many people, its worth 100x. It trumps all other. For uber-experts, it’s deceptively attractive to wave away the advantage of human-readability. For the rest of the world, for 97% of developers, it’s a huge, Huge, HUGE advantage. For high speed financial trades, JSON is wrong. For Google’s internal interfaces, wrong. For most of the world, RIGHT.

AND as the essay notes, REST doesn’t prescribe JSON. Or XML. Or anything. There’s a content-type header, and clients and servers can negotiate it. If the client says Accept: application/x-protobuf, and the server can send it, bliss for you. So this point – “JSON is bloated” – is not only not valid (false) in the first place, it’s also not an argument against REST.

2. There are no interface contracts or data schema

This is a feature. OMG, have we not tried this enough times? Did this guy skip his “History of IDL compilers” course in the Computer History department at school? Sun RPC IDL. DCE RPC IDL. Corba IDL. WSDL, ferpeetsake! XML Schema!!

It’s pretty straightforward to deliver plain-old-XML over HTTP, which is quite RESTful. More popular is JSON-over-HTTP. Either of those have schema languages. Few people embrace them, though. Why? Because IDLs and Schema languages are too much structure, and they handcuff people more than help them. We have fortunately learned from the past. There are more tools coming in this area, for those who wish to embrace them. See apistudio.io .

3. HATEOAS doesn’t work

Mmmmm, yep. No argument here. In my experience, nobody really uses this, in practice. Pragmatic REST is what people do, and it generally does not use HATEOAS.

4. no SQL semantics

Uhhuh, true. This has been addressed with things like OData. If you want SQL Semantics, seek solutions, don’t just complain.

5. CRUD is too limited

Really? This is a problem? That you might need a switch statement in your code to handle different types of events? Really?

6. CRUD is really too limited

….

Mmmmm, sorry. I have to stop now. I’m completely bored of responding to this essay by now. Except for one more:

10. Backward compatibility is hard

This has NOTHING to do with REST. This is just true. Back compat in any interface is tricky.


In summary, I don’t find any of the arguments compelling.

Let me draw an analogy. The position in this essay is like saying “Oil is no good as a transportation fuel.” Now, Oil has it’s drawbacks! Oil is dirty. We can imagine alternatives that are better in theory. Even today, in specific local situations (daily use, short trips, urban travel) electric cars are better, MUCH better, than fossil-fuel based cars. (An bicycles are even better than electric cars) But gasoline-powered cars deliver massive utility to billions of people. Gasoline refueling stations are everywhere. The delivery system for gasoline is mature and redundant. The World RUNS, very effectively, on gasoline-powered transport, by and large. Objectively, Oil is VERY GOOD as a transportation fuel.

Sure, we’ll evolve better approaches in the future. That’s great. And sure, we can imagine a world with electric-powered vehicles. But today, in the world of reality, Oil wins.

And likewise Pragmatic REST, HTTP, JSON, and schema-less interfaces are winning. We’ll evolve better approaches. But today, This platform wins.

HTTP, HTML, Javascript, and JSON are ubiquitous, are the foundation of the web, and are not going anywhere. Any architect is free to choose other options, and they might have good reasons for doing so. On the other hand the vast majority of installations won’t benefit from using protobufs or thrift, or some non-HTTP protocol. Pragmatic REST, JSON and HTTP are very very safe choices in the vast majority of scenarios.

Cheers

Chrysler is Internet-enabling your car as a way to accelerate death

From the holy-shit-how-did-they-not-test-this department, Fox News tells us that it is possible for hackers to seize control of a moving Chrysler automobile, fiddle with the radio, turning on the windshield wipers, or more ominously, controlling the transmission and the brakes. Considering the source (Fox Newsertainment), I am unsure whether to believe this. But there is also a piece on Wired. If true, seriously, Holy Shit.

Yes, APIs are everywhere.

Here’s an idea for the API team at Chrysler that has made the driveline remotely programmable – you guys should talk to the security team at Chrysler.

Update:Chrysler is recalling 1.4 million cars over this.

Sane names for screenshots on Mac OSX

Back when I was a Windows user and a .NET developer, I used a tool called Cropper to grab screen shots. That tool is great alone, and there are plugins for various photo destinations. I wrote and maintained a few plugins.

The basic capability to grab a portion of the screen into an image file is built-in to OSX, which is nice. I learned about it here.

I use the Command + Option + 4 sequence daily to grab interesting bits of the screen, for bug reports, demonstrations, illustrations, sharing information with friends, posting to Twitter, and so on.

But I miss the flexibility of Cropper. In particular, WTF is with the filenames, OSX?

OS X saves each screenshot with the name Screen shot [date] at [time]. As an example, a screenshot taken on July 9th, at 7:21 AM will be saved as Screen shot 2015–07–09 at 7:21 AM.png

Ugly! Lots of people ask how to change the filenames, and the standard answer isn’t quite satisfactory for most people. I use the terminal often, in addition to dired mode in emacs; because I fiddle around with files and directories outside of Finder, I’d like filenames that:

  • allow lexicographic sort order to also deliver a time-based sort
  • do not include spaces in the name

Not so much to ask, eh?

But the basic options to configure the names of the files are really poor. Basically I can move around the date and time portions of the file name, but I cannot change their formats to something more like ISO-8601, which is sortable. I don’t need strict ISO-8601, I just want something sortable.

The way I did it: I created a script that runs periodically, via launchd, checks for screenshot files with the ugly names in the Desktop folder, and then renames them appropriately.

You need 2 files to make this happen. One is the plist for the LaunchAgent. Create a file in /Users/YOURSELF/Library/LaunchAgents, I called mine local.screenshot.fixup.plist. The contents should look like this:

The second is the bash script that renames the files. Put this file anywhere (but you must reference that location as the Program string in the plist file above), and chmod it to be executable. The contents are like this:

After creating these files, you can either:

  • logout and log back in…
  • OR, run this command from the terminal:
    launchctl load ~/Library/LaunchAgents/local.screenshot.fixup.plist

… in order to get the new launchd to start renaming files.

When the screenshot is initially saved, it will have the original ugly name. But in one or two seconds, the watcher will run, find your screenshot file, and rename it.

Nice.

More info:

Naïve Data analysis leads to incorrect conclusions – WWII Bomber Plane edition

Raid by the 8th Air Force

Here’s a good story showing us how focusing on the data we have in front of us may lead us to incorrect conclusions.

The summary is this: In World War II, Allied bomber command undertook an analysis effort to determine how to optimally reinforce bomber planes to protect them from German anti-aircraft guns. They studied the bullet hole patterns in planes after they returned from missions, and the first impression was to re-inforce the planes where the bullet holes appeared most commonly.

But a smart guy named Abraham Wald pointed out that if the planes were returning with those bullet holes, then the bullets that caused those holes were not critical to avoid. The planes that didn’t return had likely sustained the bullet holes that demanded mitigation.

Are we talking about Bomber Planes or Software here?

Focusing on the data we have is a common approach that supports iterative improvement efforts. The Lesson from the Bomber Plane story is: there may be “invisible data” that needs consideration.

Clayton Christensen World Economic Forum 2013

Mr Christensen looking stern and wise.

This is the central problem explored by Clayton Christensen, in his classic book, The_Innovator’s Dilemma. What Christensen found was that companies can tend to focus on feedback they get from existing customers, while ignoring the needs of prospective customers. And this is dangerous.

We have a project going on right now in which we’re looking at the UI usage data for a particular tool in a web app. The web app tool has for some time now been stable. It was never very complete, and it hasn’t been extended actively. You could say it was stagnant. We know the reasons for that – we had other priorities and this web tool just needed to stay in the backlog for a while. Also, there were, and there still are, other ways to accomplish what was exposed by this web app tool; there are public APIs that expose the underlying function and people have the option to build tactical apps on those APIs. Now we want to enhance the web tool, and the web tool has recently been enhanced to at least collect data on how people are currently using it. How much should we rely on this usage data?

With the bomber planes, the data they had were the patterns of bullet holes in the returned planes. The data they didn’t have were the patterns of bullet holes in bomber planes that didn’t return.

With the web tool, the data we have will be the UI usage patterns of current users. The data we don’t have will be the usage patterns of people who abandoned the stagnant tool some time ago, and are relying on the underlying APIs and independent tools to accomplish their goals. The former set of users will have a relatively higher proportion of novice users, while the latter will have a higher proportion of experts.

So what’s the right path forward? Should we invest in improvements in the tool, solely based on the data we have? Should we guess at the data we don’t have? In the bomber plane example, the data we don’t have was clear by deduction – those planes were most likely shot down. In the web tool example, the data we don’t have are truly unknown. We cannot deduce what, exactly, people are doing outside the tool.

I think the only way to resolve this is to combine three factors: analysis of the API usage, analysis of the UI usage, and finally, perspective on the behavior of the expert users who have abandoned the tool, perhaps formed by direct observation or by intelligent inference. API analytics, UI analytics, and some intelligence.

Then we’ll need to apply some sort of priority to the different scenarios – do we need to help the experts more with their expert problems? or do we need to invest more in making the simple things even easier for the more novice users? Which is the more important audience, the novices or the experts? Which one is more important to serve, at this time in the product’s lifecycle?

Interesting questions! There’s no one right answer for every situation. I know which option I favor in our current situation.

Upgrade your nodejs on Mac OS X to v0.12.6

Per medium.com, there are critical vulnerabilities in nodejs and iojs. People running nodejs should upgrade to v0.12.6.

On Mac OSX, this takes just a few moments. Thanks to Stackoverflow for the tip.


~/ $ node -v
v0.10.22
~/ $ sudo npm cache clean -f
Password:
npm WARN using --force I sure hope you know what you are doing.
~/ $ sudo npm install -g n
npm http GET https://registry.npmjs.org/n
npm http 200 https://registry.npmjs.org/n
npm http GET https://registry.npmjs.org/n/-/n-1.3.0.tgz
npm http 200 https://registry.npmjs.org/n/-/n-1.3.0.tgz
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
n@1.3.0 /usr/local/lib/node_modules/n
~/ $ sudo n stable

install : node-v0.12.6
mkdir : /usr/local/n/versions/node/0.12.6
fetch : https://nodejs.org/dist/v0.12.6/node-v0.12.6-darwin-x64.tar.gz
installed : v0.12.6

~/ $ node -v
v0.12.6
~/ $

Web vs Native apps, part 12763

From Hacker News recently, Another posting, this one by Peter-Paul Koch, about the web-vs-native debate, dramatically entitled “Let’s concede defeat”!!

This is the latest installment in a long line of commentary on the topic. In short, the topic is: is it better to build and offer a native, platform-specific app, or to invest in building a web-based user experience? We’ve been having this debate for years, and the discussion just seems to spin round and round on the same points. readwrite.com is also chiming in on the “morass”. Everyone has a viewpoint. Web is better. Native is better. Hybrid is the future.

Lots of drama. A range of viewpoints. Many of the strongest opinions come from people with something to sell: they sell the vision of web as write-once-run-anywhere, or they sell a product that helps produce hybrid apps.

What’s the real story?

In my opinion, it’s not that complicated. Here you go: Web apps are super convenient to deploy, update, and access. There’s no “app store”, there’s no worry about if and when users will update. But, they’re not optimal for a crisp device-optimized experience. There are many web frameworks and choosing the right one is challenging. Attempting to “emulate” native app behavior can get you into trouble when trying to support N different device platforms.

The native app, on the other hand, is crisp and device optimized, and users like it. It’s easier to do offline-capable apps, and it works with system notifications. But, using native means you’re basically writing many apps, one version for each device. Also, the way users access the app is inconvenient, and there’s no guarantee users will update their apps (although there are mitigations for that).

But most apps will be hybrids, combining some web-app capability with some native capability.

Dzone has a nice table listing decision criteria between the three options. Kinlan has a nice, day-in-the-life journal that gives color to the tradeoffs, although looking for a web-based alarm clock to supplant the device-supplied one seems like tilting at windmills. No one needs a web-based alarm clock when every phone has several native apps to choose from. He made that point very effectively, though I don’t think he intended to do so.

We, as an industry, haven’t settled this issue yet. There’s still more talking we need to do, apparently.

Though there’s been much talk, there are still seemingly common perspectives that are just wrong.

  1. from ignorethecode, Native apps mainly benefit Apple and Google, not their users. It’s not in anyone’s interest to be locked into a specific platform, except for the platform’s owner.
    Wrong. This just completely disregards the platform-specific stuff developers can do with accelerometers, GPS, cameras, and screens. Baloney. Yes, you can get access to some of those things, some of the time, in web apps, but not in a reliably cross-platform way.
  2. from dzone, Native apps often cost more to develop and distribute because of the distinct language and tooling ecosystems,
    Wrong. This assumes the main cost is in the licensed tools, like Xcode and a deployment license from Apple. Sheesh. That is the tiniest component of app development and deployment cost.. C’mon now. Developers (people) and their time are expensive. Licenses are cheap.

I like what Mr Koch had to say in the quirksmode blog. Basically: different solutions will better suit different sets requirements, for commerce, news sites, and so on. Pragmatic. Realistic. Helpful.

Regardless whether people choose web apps, native apps, or hybrid apps, the backend is always APIs. Everybody needs APIs! And they need to be simple, easy to consume, well documented, secure, and managed and measured properly.

I don’t see the point in Revoking or Blacklisting JWT

I heard someone asking today for support for Revocation of JWT, and I thought
about it a little, and decided I don’t see the point.

Specifically, I don’t see the point of the process described in this post regarding “Blacklisting JWT in express-jwt“. I believe that it’s possible to blacklist JWT. I just don’t see the point.

Let’s take a step back and look at OAuth

For those unaware, JWT refers to JSON Web Token, which is a type of token that can be used in APIs. The format of JWT is self-describing.

Here’s the key problem tokens address: how does a server decide whether to honor or reject a request? It’s a matter of authorization. OAuthV2 has been proposed and is now being used by the industry as the model or framework for enabling authorization in API-oriented apps. Basically it says, “give apps tokens, then grant access based on the token.”

Often the way things work under the OAuth framework is:

  1. an app running on a mobile phone connects to a token dispensary (a server) to request a token
  2. the server requires the client (==app) to provide some credentials before generating and dispensing a token. Sometimes the server also requires user authentication before token delivering a token. (This is done in the Authorization Code grant or the password grant.)
  3. the client app then sends this token to a different server to ask for services.
  4. the API server evaluates the token before granting service. Often this requires contacting the original token dispensary to see if the token is good, and to see if the token should be honored for the particular service being requested.

You can see there are three parties in the game: the app, the token dispensary, and the API server.

One handy optimization is to put the API endpoint behind an OAuth-aware proxy server, like Apigee Edge. (Disclaimer: I work for Apigee). The app then contacts Edge for a token (via POST /token). If the credentials are good, Edge generates and stores an opaque token, which looks like n06ztxcf2bRpN42cDwVUNvroGOO6tMdt, and delivers it back to the app. The app then requests service (via GET /service, or whatever), passing the previously obtained token. Edge sees this request, extracts the token within it, evaluates whether the token is good, and either passes the request through to the API endpoint or rejects it based on the token status.

The key thing: these tokens are opaque. The app doesn’t know what that token is, beyond a string of characters. The app cannot tell what the token is good for, unless it asks the token dispensary, which is the final arbiter. Sometimes when dispensing the token, the token dispensary also delivers metadata about the token, like: expiry, scopes, and other attributes. But that is not required, and not always done. So, bearer tokens are often opaque, and they are opaque by default in Apigee Edge.

And by “Bearer”, we mean… an app that possesses a token is presumed to “own” the token, and should be granted service based on that token alone. In other words, the token is a secret. It’s like cash money – if you lose it, someone else can spend it. But not exactly like cash. An opaque token is more like a promissory note or an IOU; to determine if it’s worth anything you need to go back to the issuing party, to ask “are you willing to pay up on this note?”

How is JWT different?

JWT is a different kind of OAuth token. OAuth is just a framework, and does not stipulate exactly the kind of token that needs to be generated and delivered. One type of token is the opaque bearer kind. JWT is an alternative format. Rather than being an opaque string, JWT is a self-describing format for bearer tokens. Generally, a JWT includes an encoded payload that can be decoded and read by anyone, and that payload contains a bunch of claims. The standard set of claims includes: when the token was generated (“issued at”), who generated it (the “issuer”), the intended audience, the expiry, and other things. JWT can include custom claims, such as “the user is a good person”. But more often the custom claim is: “this user is authorized to invoke /serviceA at endpoint http://example.com”, although this kind of claim is shortened quite a bit and is encoded in JSON, rather than in English.

Optionally accompanying that payload with its claims is a signature, which can be verified by any party possessing the public key used to sign it, (or, when using secret key encryption, the secret key). This is what is meant by “self describing”. The self-describing nature of JWT is the opposite of opaque. [JWT can be unsigned, can be signed, or can be encrypted. The encryption part is an optional part of the spec.]

(Commercial message: I said above that Apigee Edge generates opaque bearer tokens by default. You can also configure Apigee Edge to generate signed JWT.)

Why Self-describing Tokens?

The main benefit of a model that uses self-describing tokens is that the API endpoint need not contact the token dispensary in order to determine if the token is good, not-expired, and if a request bearing such a token ought to be honored. In other words, JWT supports federation. One party issues the token, another party can verify it, without contacting the issuer. Remember, JWT is a bearer model, which means the possessor of the token is presumed to be authorized to get service based on what’s in the token. This is truly like cash money this time, because … when honoring a JWT, the API endpoint need not contact the issuer, just as when accepting a $20 bill, you don’t have to contact the US Treasury to see if the bill is worth $20.

So how ’bout Revocation of JWT?

This is a long story and I’m finally getting to the point: If you want JWT with powers to revoke the token, then you abandon the federation benefit.

Making the JWT self-descrbing means no honoring party needs to contact the issuer. Just verify the signature (verify the $20 bill is real), and then grant service. If you add in revocation as a requirement, then the honoring party then needs to contact the issuer: “I have $20 bill with serial number T128-DCQ-2872JKDJ; should I honor it?”

It means a synchronous call across the two parties. Which means federation is effectively broken. You abandon the federation benefit.

The corollary to the above is that you also still incur all the overhead of the JWT handling – the signing and verification. So you get all the costs of JWT and none of the benefits.

If revocation of bearer tokens is important to you, you could do the same thing with an opaque bearer token and eliminate all the fussy signature and validation stuff.

When you’re using an API Proxy server like Apigee Edge for both issuing and verifying+validating tokens, then there is no expensive additional remote call to check the revocation status. But you still lack the federation benefit, and you still incur this signing and verification nonsense.

I think when people ask for the ability to handle JWT with revocation, they don’t really understand what they’re asking.