How to Compute an HTTP Signature for Mastodon (and an example in NodeJS)

2023-April-7

I am reading this documentation from Mastodon.

And from it, I understand that Mastodon requires an HTTP Signature, signing at least these headers:
(request-target) host date digest

If your client is written in JavaScript and runs on Nodejs, an example for how to build a signature is given on the npmjs site.

But I believe this example is out of date. It does not use the (request-target) pseudo header. So that’s not gonna work.

So what must you do? Go back to the Mastodon documentation. Unfortunately, that too, is either out of date or confusing. For a post request, the mastodon documentation states that you must first compute the “RSA-SHA256 digest hash of your request’s body”. This is not correct. There is no such thing as “RSA-SHA256 digest”! RSA-SHA256 is not the name of a digest. Message Digests include: SHA1, SHA256, MD5 (old and insecure at this point) and others. According to my reading of the code, Mastodon supports only SHA-256 digests. The documentation should state that you must compute the “SHA256 digest”. (There is no RSA key involved in computing a digest).

Regardless of the digest algorithm you use, the computed digest is a byte array. That brings us to the next question: how to encode that byte array as a string, in order to pass it to Mastodon. Some typical options for encoding are: hex encoding (aka base16 encoding), base64 encoding, or base64-url encoding. The documentation does not state which of those encodings is accepted. Helpfully, the example provided in the documentation shows a digest string that appears to be hex-encoded. Unhelpfully, again according to my reading of the code, Mastodon requires a base64-encoded digest!

With these gaps and misleading things in the documentation, I think it would be impossible for a neophyte to navigate the documentation and successfully implement a client that passes a validatable signature.

  1. produce the POST body
  2. compute the SHA-256 digest of the POST body, including all whitepsace and leading or trailing newlines. Try this online tool to help you verify your work.
  3. Encode that computed digest (which is a byte array) with base64. This should produce a string of about 44 characters.
  4. Set the Digest header to be SHA-256=xxxyyyyy , where xxxyyyy is the base64 encoding of the SHA-256 digest.
  5. Set the http headers for the pending outbound request to include at least host, date, and digest.
  6. compute the signature following the example from the npmjs.com site, with headers of “(request-target) host date digest”, and using the appropriate RSA key pair.

If it were me, I would also include a :created: and an :expires: field in the http signature.

You can play around with HTTP Signatures using this online tool. That tool does not yet support computing a Digest of a POST body, but I’ll look into extending it to do that too.

Let me know in the comments if any of this is not clear.

I posted a working example for Nodejs as a gist on Github.

It depends only on nodejs and the builtin libraries for crypto and URL to compute the hash/digest and signature. It does not actually send a request to Mastodon; that is left for you to do.

Yarp vs Envoy proxy – build time comparison

2023-March-24

I’m doing some self-education these days, and was exploring YARP today. I learned about this via HackerNews some time ago, and only now got around to taking the time to explore in more detail. As Microsoft describes it, YARP is “a library to help create reverse proxy servers that are high-performance, production-ready, and highly customizable.”

It’s not a reverse proxy in its own right, but a library that you can embed into an ASPNET app to allow it to act as a reverse proxy. The “Yet Another” moniker is completely appropriate; there are many, many Reverse Proxies out there in various shapes, sizes and configurations. Why Microsoft wanted to build another one when there are good options out there – Envoy, nginx, haproxy, and many others – is perhaps a topic worth exploring (Google, my most recent employer, promotes the open-source Envoy proxy as a general-purpose RP, and also sells an API Platform, Apigee, that includes its own reverse proxy). It seems to me that Microsoft has large cloud investments, and wants to have control over this particular critical piece of widely-used infrastructure. Rather than compromise with something that’s already out there, go build something that fits the requirements for their massive cloud footprint, as well as for the shops other than Microsoft who are invested in .NET. I don’t think it’s worthy of too much more discussion than that.

With so many available options in reverse proxies, an interested observer might want to have some insight into comparisons between them. Now there are various criteria a person might want to investigate when comparing – features like hot-reload of configuration, the configuration model in general, support for “farms” of proxies all centrally managed, platform availability, performance…. Any reader could probably add two or three more items to that list.

All of that is interesting, but I don’t have time to conduct a thorough comparison at this time. But I will offer one quick observation. While exploring Envoy proxy back in November, I built it from source on my macbook pro. The build was a bear, and took maybe 90 minutes? Something like that. Basically it built every library that the envoy proxy depended on. I suspect most people don’t do that; they just use the docker container that the envoy project publishes.

Just for fun I cloned the yarp repo and ran a build. After sorting out some puzzles [1, 2] on my own, the build completed in about 55 seconds. That’s a pleasant surprise.

I know that with Bazel, the envoy build will be much faster on subsequent runs. But even so, building a YARP proxy is much much faster than building Envoy.

APIs, microservices, and the service mesh

2019-April-11

Got some time and want to learn about APIs, microservices, the service mesh, and how these pieces interplay in an enterprise?

Here’s a session Greg Kuelgen and I delivered at Google Next 2019.
Hero image

Youtube video

Summary: If you’ve got more than a handful of services intercooperating, you’re gonna want a service mesh infrastructure. And you will want to use API Management to share APIs outside of the team that developed them.

nodejs on Google App Engine – forcing HTTPS inbound, via HSTS

2018-June-20

How can I force my nodejs app running on Google App Engine, to always redirect to HTTPS ?

I have a pretty vanilla app that looks like this:

This thing is running in Google App Engine (GAE), and I’d like to make sure it listens only on HTTPS. There are standards like HSTS that can help. How can I use them?

This question and answer on Stackoverflow showed me the way. Basically, just add in a tiny module called yes-https. The new code looks like this:

Redeploying (no change to app.yaml) gets me the always-HTTPS behavior I want. When a client requests my service via http, it receives a 301 redirect pointing to the secure site.

HTTP/1.1 301 Moved Permanently
Date: Wed, 20 Jun 2018 16:27:56 GMT
Transfer-Encoding: chunked
X-Powered-By: Express
Location: https://foo-bar.appspot.com/
Via: 1.1 google

Nice, easy, clear.
Thanks to Justin for this handy module.

Jackson and XmlMapper – reading arbitrary data into a java.util.Map

2017-December-6

I like the Jackson library from FasterXML. Really handy for reading JSON, writing JSON. Or I should say “serialization” and “deserialization”, ’cause that’s what the cool kids say. And the license is right. (If you need a basic overview of Jackson, I suggest this one from Eugen at Stackify.)

But not everything is JSON. Sometimes ya just wanna read some XML, amiright?

I work on projects where Jackson is included as a dependency. And I am aware that there is a jackson-dataformat-xml module that teaches Jackson how to read and write XML, using the same simple model that it uses for JSON.

Most of the examples I’ve seen show how to read XML into a POJO – in other words “databinding”. If my XML doc has an element named “Fidget” then upon de-serialization, the value there is used to populate the field or property on the Java object called “Fidget” (subject to name remapping of course).

That’s nice and handy, but like I said, sometimes ya just wanna read some XML. And it’s not known what the schema is. And you don’t have a pre-compiled Java class to hold the data. What I really want is to read XML into a java.util.Map<String,Object> . Very similar to what I would do in JavaScript with JSON.parse(). How can I do that?

It’s pretty easy, actually.

This works but there are some problems.

  1. The root element is lost. This is an inadvertent side-effect of using a JSON-oriented library to read XML.
  2. For any element that appears multiple times, only the last value is retained.

What I mean is this:
Suppose the source XML is: 

<Root>
  <Parameters>
    <Parameter name='A'>valueA</Parameter>
    <Parameter name='B'>valueB</Parameter>
  </Parameters>
</Root>

Suppose you deserialize that into a map, and then re-serialize it as JSON. The output will be: 

{
  "Parameters" : {
    "Parameter" : {
      "name" : "B",
      "" : "valueB"
    }
  }
}

What we really want is to retain the root element and also infer an array when there are repeated child elements in the source XML.

I wrote a custom deserializer, and a decorator for XmlStreamReader to solve these problems. Using them looks like this: 

String xmlInput = "<Root><Messages><Message>Hello</Message><Message>World</Message></Messages></Root>";
InputStream is = new ByteArrayInputStream(xmlInput.getBytes(StandardCharsets.UTF_8));
RootSniffingXMLStreamReader sr = new RootSniffingXMLStreamReader(XMLInputFactory.newFactory().createXMLStreamReader(is));
XmlMapper xmlMapper = new XmlMapper();
xmlMapper.registerModule(new SimpleModule().addDeserializer(Object.class, new ArrayInferringUntypedObjectDeserializer()));
Map map = (Map) xmlMapper.readValue(sr, Object.class);
Assert.assertEquals( sr.getLocalNameForRootElement(), "Root");
Object messages = map.get("Messages");
Assert.assertTrue( messages instanceof Map, "map");
Object list = ((Map)messages).get("Message");
Assert.assertTrue( list instanceof List, "list");
Assert.assertEquals( ((List)list).get(0), "Hello");
Assert.assertEquals( ((List)list).get(1), "World");

And the output looks like this: 

{
  "Parameters" : {
    "Parameter" : [
      {
        "name" : "A",
        "" : "valueA"
      },{
        "name" : "B",
        "" : "valueB"
      }
    ]
  }
}

…which is what we wanted.

Find the source code here: https://github.com/DinoChiesa/deserialize-xml-arrays-jackson

Hat tip to Jegan for the custom deserializer.

medialize/URI.js – why’d you go and get all fancy?

2017-November-8

I have relied on URI.js from medialize for years.

I downloaded it a long time ago, and it just works. It’s handy for parsing and building URIs form within Javascript.
I happen to use nodejs often, but I also use a JavaScript engine that runs in the JVM (via Rhino or Nashorn). So I liked URI.js for its usability across those systems.

Recently I decided to download “the latest and greatest” URI.js, and what I found… did not make me jump for joy.

URI.js is no longer “just downloadable”.

Where before I could just download the raw JS file, URI.js now has a builder that allows me to select which options I wish to include. I get the concept, and it’s a nice idea, but when I de-selected every option, I got a minimized URI.js that I did not want. When I went to the source tree I found a URI.js that included all the require() statements for punycode, Second-Level Domains, and ipv6, all stuff I did not want.

*snif*

I couldn’t figure out how to get it to “just work” in nodejs without all of that, so I had to resort to manually changing the code. Basically I just removed all the require() statements for those unneeded / unwanted modules.

And it works.

It’s possible I’m missing something basic, but for sure, it got more complicated to get the simple solution. Seems like a step backward.

Do you use curl? Stop using -u. Please use .netrc

2017-March-21

An unsolicited tech tip.

Those of you who are API people, should exhibit good API hygiene.

One aspect of that is: “stop using curl -u” !!

Sometimes you have the urge to run a command like this:
curl -X POST -v -u 'yourusername:password' . https://foobar/slksls

Avoid this.

OK, ok, I know sometimes it’s necessary. But if you have an API endpoint that you often tickle with curl, and it accepts credentials via HTTP basic auth, you should be using .netrc to store the credentials.

The problem with using -u is that the password is shown in clear text on your terminal!

OK, I know, you’re thinking: but I’m the only one looking at my screen. . I can hear you thinking that right now. And that may be true, most of the time. But sometimes it’s not.

Sometimes you cut/paste terminal sessions into an email, or a blog post, or a bug report. And that’s when your password gets written down and shared with the world.

Treat Basic Authorization headers the same as passwords, because any observer can easily extract your password from that.

You might think that it’s ok to insert credentials in an email if it’s just being shared among your close work colleages. But that’s a bad idea also. Audit trails depend on the privacy of credentials. If you share them, the audit is gone. Suppose you have a disgruntled (ungruntled? never gruntled?) colleague who decides to take your creds and use them to recursively curl -X DELETE a whole bunch of resources. And the audit trail will show YOUR name on that act.

In short, it’s bad form. It could be forwarded or copy/pastad or it could leak into habit. It sets a terrible example for the children.

Here’s what I suggest:

Option 1: if you use curl

If you have a *nixy machine, create a ~/.netrc file and insert your creds there. See here for information.

chmod the file to 400. When you use the -n option, curl knows how to extract your creds from the file silently. You never have to type credentials on the command line again. I think you can do this on Windows too, but I don’t know curl on Windows.

If you build scripts that use curl, you should allow the user that same option. That way the user never keys in their creds to your script.

When you pass the -n option to curl, instead of -u USER:PASS, it tells curl, “if you ever connect with site.example.com, then use THESE creds” . This works with any HTTP endpoint curl can address via Basic Auth. I have creds for Jira, Heroku, and other systems all in my .netrc.

Hint: also don’t use curl -v, because that will show the basic auth header. You probably want -i anyway, which is less verbose than -v.

Option 2: don’t use curl

Use some other tool that hides the credentials completely.
I think Postman doesn’t quite hide the creds completely. So be careful!

Let’s all try to exemplify good security behavior.

It’s that time of year… when people think about exchanging JWT for opaque tokens

2016-December-28

Yes, it’s that time of year when people think about RFC7523, which describes how to exchange JWT for opaque OAuth tokens.

Right?

If you’re like me, the waves of acronyms, jargon, and IETF RFCs (see what I did there?) seem to never end. OAuth, JWT, RFC 7523, JTI, claims, RS256, PBKDF2…? I feel your pain.

But there is some good news… here’s something that will help clarify the ideas and use cases around RFC7523. I wrote a quick article, and also created an Apigee Edge API Proxy, that implements this for you. It illustrates exactly how to exchange JWT for opaque OAuth tokens, and I even include some commentary int he readme explaining why you’d want to do it. (Spoiler alert: It’s faster to verify opaque OAuth tokens). All available on the Apigee community site.

The way I think about RFC7523 – it is an alternative to the client_credentials “grant type”, described in IETF RFC6749, which is the document that describes the OAuth v2.0 Framework.

OK, I hear you saying it: “back up, Dino… What is this client_credentials thing?” Yes, there is an underscore there. The client_credentials grant type is designed to allow a client app to identify itself to a token dispensary. The client says “here’s my ID, and here’s a secret that only I (the client app) should know.” And the token dispensary can then look at those two pieces of information, and if they are valid (the client_id is not expired or revoked), then the token dispensary can issue a token. It’s like username + password authentication for a person, but client_credentials is used for identifying a client app. This grant type mostly useful in server-to-server communications, when one service is being used by another service. BUT, some people use client_credentials grants in their mobile apps, so that the API service can trust that the mobile app is who it claims to be. (There are some problems with this; basically the client_secret needs to be embedded in the client code, therefore it is accessible to hackers, and therefore it is not truly “secret”. We can talk about mitigations for this in a future blog post.)

So that’s the client_credentials grant type. As I said, RFC7523 is an alternative to the client_credentials grant. Basically, instead of sending in a client_id and client_secret, under the RFC7523 flow (which has the helpful and easy-to-remember moniker of “JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants”, seriously) the client app self-signs a JWT which includes the client_id as the issuer. The app sends that to the token dispensary. The token dispensary verifies the signature, verifies that the client_id is valid, and then issues an opaque OAuth v2.0 token.

Now, there are some interesting implications to this model. Maybe these are obvious to some of you, but I will state them anyway:

  1. the token dispensary and the client app have to conform to the same JWT signing convention. JWT can be signed with shared-secret (HS256) or with public/private key (RS256). Either way is fine, but the two sides must agree.
  2. regardless of the signing convention, it must be possible for the token dispensary to verify the signature. If HS256 is the agreed convention, this means the token dispensary and the client app must share a secret. (This can be the client_secret! if it has sufficient entropy, or it can be a key obtained from PBKDF2) If RS256 is the signing convention, it means the two parties must have a shared trust relationship, where the token dispensary has access to the public key of the client app. Bottom line, there is a little bit more overhead for you, setting up an JWT-for-opaque-token exchange mechanism, if you use RS256: specifically you need to provision a new RSA public/private keypair for the client, and the client needs to make the public key available to the token dispensary.
  3. the client app needs some extra intelligence, specifically a library that allows it to create a signed JWT. There are myriad options available regardless of the app platform + language you use, so in practice, this won’t be an obstacle, but it does mean there will be new code you must include in your client.

Once you get past those implications and the extra set-up overhead, the model in RFC 7523 is really nice because it’s extensible. That’s because the request-for-token is encapsulated in a JWT, and the JWT itself is extensible. You, as an API designer, can stipulate any arbitrary (custom) claims that clients must include in the JWT, in order to compose a valid request-for-token. And you can include restrictions on the standard claims or custom claims. Some examples:

  1. a proof-of-work string, something like a HashCash string or similar. Including proof-of-work would be a discouragement for bots.
  2. As another example, you can stipulate that the JWT be short lived. Verification of the JWT might include a proviso that rejects tokens that have a lifetime beyond 180 seconds, for example.
  3. you could institute a one-use policy on such JWT.
  4. you could require a “scopes” claim and validate the strings contained in that claim against the issuer (==client_id)

BTW, the example API Proxy I shared on Github shows how to implement the lifetime and one-use-only controls. (As with everything I publish on github, pull requests are welcomed!) If the inbound JWT that comprises the request-for-opaque-token does not pass these checks, a 401 Unauthorized is sent back.

BTW #2, did you know that Google services like Stackdriver and cloud storage use JWT-for-opaque-token exchange in order to enable service-to-service integration? Google also institutes the lifetime and one-use-only controls. The lifetime of the JWT must be less than 300 seconds.

Say, that reminds me!, Speaking of Google, did I mention that Google has acquired Apigee? Yes, I work for Google now! Part of the Apigee team within Google. w00t! I’m pumped, psyched, charged up, amped, and very pleased about this development.

So far, minimal changes for me, except for me I got a Chromebook! And yes, I authored this post from that very same device.

As always, I’m interested to hear your feedback on this. Let me know in the comments section.

Finally, I would like to wish all of you a Merry RFC7523 Season; and I wish you many Happy short-lived OAuth Tokens in the new year.

Stackoverflow and the early mover advantage

2016-September-26

Browsing HN this morning, I found an interesting piece discussing Stackoverflow. Apparently hackernoon wrote a piece entitled “The decline of Stackoverflow” and it got some attention on reddit.

Bozho’s perspective aligns pretty closely with mine. I’m in the top 0.1% overall on SO, but I have not contributed actively in years. Not because I perceived “a decline in the site” whatever that means, but because I got busy with other things, specifically Apigee.

I agree with Bozho that many of the easy questions have been answered. Sure, there are always new questions, and new technologies like Golang or some new iPhone feature, or a new version of Angular, will prompt a new class of questions. But, the basics around .NET GAC, or Java garbage collection, or how to do read-through caches in Java, or what is JSONP…. you know those are already answered.

And the “Early movers” on Stackoverflow – I guess I was among them – have garnered all the top scores, and continue to accrue points as new readers on stackoverflow upvote answers. So even though I haven’t posted a new, popular answer in a while, I still earn points every week. New arrivals to stackoverflow will probably never be able to attain the level of points I have. My point total grows more than theirs, even though I haven’t done any work, and they may be asking and answering new questions diligently. It’s very unequal.

Don’t get me wrong – I think Stackoverflow is super valuable. I agree with Bozho, that it serves its purpose well, which is to provide easily searchable answers to programming questions. But aside from that massive contribution of value to the programming community, if you just look at internet points, the points go to the early movers.

I’m looking forward to watching the first formal debate for the US presidential election, this evening. Coincidentally, last night I watched a Frontline episode from June, entitled Policing the Police – excellent work by Jelani Cobb btw. So in this moment the analogy that comes to my mind, for this phenomenon on stackoverflow, is economic.

Imagine the discovery and settlement of a new continent. The early movers come in and (after eradicating the natives, if any) lay claim to land. They work the land, and maybe expand. Later more people arrive, and they need to rent land from the earliest movers. It’s those early movers that continue to accrue $$ and interest over time. Later more people come in, and there is no opportunity to gain land. Their option is to work the land for someone else (upvote other questions and answers). I was born in the 1960’s, and all the land in the USA was already claimed. Large holders of assets including land at that time, have continued to benefit. (I am not complaining – I was born white, male, and healthy in one of the richest countries on the earth – I won the lottery at birth.) Or consider a real-estate marker in a geographically constrained area like Seattle or San Francisco. Early movers claim land and erect buildings, and later movers (like me) come in and rent space.

On Stackoverflow, it’s really not a big deal if a newcomer is blocked from attaining “internet points”. A newcomer can still benefit from the answers and discussions on the site. They’re digital assets! Sharing more does not decrease the value of the asset. Everyone can benefit. And internet points are worth approximately $0.00. In real life, it’s a huge deal if the top 1% of the population holds 40% of the wealth, and the trend is toward greater inequality. This wealth is tied to assets like land, housing, companies, etc., which act to extract more money from the people who are not in the 1%. Not sustainable, and more importantly, not moral.

I don’t think a massive, sudden transfer of wealth is the answer. That is the phenomenon we are watching in Iraq and Syria, where different parties are grabbing oil fields, or attempting to attain control of entire cities in order to tax the inhabitants. It is what happened when the Vikings invaded what is now France, and grabbed land (establishing Normandy) and started taxing citizens. Many people die in these sudden transfers of wealth. There is a great benefit in stability – it means people generally die of natural causes rather than war wounds.

On the other hand, substantive change is urgently necessary. Gradual, thoughtful, methodical change, ideally. Of the two candidates most likely to win the US presidential election, neither appear to be interested in changing things very much. Too bad.

Drupal 7, #states, and mutually exclusive checkboxes

2016-August-4

This post will be a bit techy. I confronted and solved a minor problem yesterday, and in the spirit of the internet, thought I’d share the solution, in case anyone else tries something similar.

This is about Drupal forms, and specifically within forms, the #states capability, which is a way that form designers can tell Drupal to do jQuery magic things on the form elements, enabling or disabling some of them based on the state or value of others.

The typical example is a checkbox, that when checked, will either enable (css ‘disabled: false’) or make visible (css ‘display: block’) a dependent textbox. Simple enough, right? and for that kind of simple case, it works well.

Drupal’s Forms API is described here, and the related
drupal_process_states here.

This is what it looks like to configure a Form in Drupal:

That says, show the textfield only when the referenced checkbox is checked. The reference to the checkbox is with a jQuery selector. This one works, really straightforward. And, the state is managed by Drupal in both directions. When the referenced checkbox is checked, then the textfield is visible. When the referenced checkbox is unchecked, then the textfield becomes not visible.

But what if you want a set of mutually exclusive checkboxes?

Mutually Exclusive Checkboxes

One approach is to just use the above model, and have each checkbox depend on the other. In other words, something like this:

This will not work. The reason this does not work, is that the state is managed by drupal in both directions. When checkbox #1 is checked, then checkbox #2 becomes unchecked. Which means checkbox #1 gets checked. Which means checkbox #2 becomes unchecked. And if you turn on the Firebug debugger, you can see the logical loop going round and round, endlessly.

There was an approach described here that suggested using two conditions in the array. But that didn’t work for me; I still had the endless loop. After fiddling with this for an hour, searching around for hints, I decided to just do it myself with my own jQuery. The logic was simple to write. And, I didn’t want to fight the Drupal Forms API any longer.

So here’s the solution. Include this JavaScript in your module:

As you can see, it registers a ‘change’ hook for a specially-marked checkbox. And when the checkbox is affirmatively checked, it unchecks the other checkbox. When the checkbox is unchecked, it does nothing.

How does that JS get loaded? In the Drupal module code, do this:

And finally, how do we set up the checkboxes in the Forms API? Like this:

And that gets the desired behavior: It is possible for zero or one of those checkboxes to be checked, but not both.

It took more time to write this post than it took to build the solution shown here! And of course I never did manage to figure out how to do the same just using the Forms API. This is an example of an API, the Forms API in Drupal, that does some things well, and this one thing….? Not so well. Much easier to just jump out and solve it this way.

Maybe this will help some one else!

By the way, this is included in a Drupal module that allows administrators to verify / validate user registration.