Got some time and want to learn about APIs, microservices, the service mesh, and how these pieces interplay in an enterprise?
Here’s a session Greg Kuelgen and I delivered at Google Next 2019.
Summary: If you’ve got more than a handful of services intercooperating, you’re gonna want a service mesh infrastructure. And you will want to use API Management to share APIs outside of the team that developed them.
How can I force my nodejs app running on Google App Engine, to always redirect to HTTPS ?
I have a pretty vanilla app that looks like this:
This thing is running in Google App Engine (GAE), and I’d like to make sure it listens only on HTTPS. There are standards like HSTS that can help. How can I use them?
This question and answer on Stackoverflow showed me the way. Basically, just add in a tiny module called yes-https. The new code looks like this:
Redeploying (no change to app.yaml) gets me the always-HTTPS behavior I want. When a client requests my service via http, it receives a 301 redirect pointing to the secure site.
HTTP/1.1 301 Moved Permanently
Date: Wed, 20 Jun 2018 16:27:56 GMT
Transfer-Encoding: chunked
X-Powered-By: Express
Location: https://foo-bar.appspot.com/
Via: 1.1 google
Nice, easy, clear.
Thanks to Justin for this handy module.
I like the Jackson library from FasterXML. Really handy for reading JSON, writing JSON. Or I should say “serialization” and “deserialization”, ’cause that’s what the cool kids say. And the license is right. (If you need a basic overview of Jackson, I suggest this one from Eugen at Stackify.)
But not everything is JSON. Sometimes ya just wanna read some XML, amiright?
I work on projects where Jackson is included as a dependency. And I am aware that there is a jackson-dataformat-xml module that teaches Jackson how to read and write XML, using the same simple model that it uses for JSON.
Most of the examples I’ve seen show how to read XML into a POJO – in other words “databinding”. If my XML doc has an element named “Fidget” then upon de-serialization, the value there is used to populate the field or property on the Java object called “Fidget” (subject to name remapping of course).
That’s nice and handy, but like I said, sometimes ya just wanna read some XML. And it’s not known what the schema is. And you don’t have a pre-compiled Java class to hold the data. What I really want is to read XML into a java.util.Map<String,Object> . Very similar to what I would do in JavaScript with JSON.parse(). How can I do that?
It’s pretty easy, actually.
This works but there are some problems.
What I mean is this:
Suppose the source XML is:
<Root>
<Parameters>
<Parameter name='A'>valueA</Parameter>
<Parameter name='B'>valueB</Parameter>
</Parameters>
</Root>
Suppose you deserialize that into a map, and then re-serialize it as JSON. The output will be:
{
"Parameters" : {
"Parameter" : {
"name" : "B",
"" : "valueB"
}
}
}
What we really want is to retain the root element and also infer an array when there are repeated child elements in the source XML.
I wrote a custom deserializer, and a decorator for XmlStreamReader to solve these problems. Using them looks like this:
String xmlInput = "<Root><Messages><Message>Hello</Message><Message>World</Message></Messages></Root>";
InputStream is = new ByteArrayInputStream(xmlInput.getBytes(StandardCharsets.UTF_8));
RootSniffingXMLStreamReader sr = new RootSniffingXMLStreamReader(XMLInputFactory.newFactory().createXMLStreamReader(is));
XmlMapper xmlMapper = new XmlMapper();
xmlMapper.registerModule(new SimpleModule().addDeserializer(Object.class, new ArrayInferringUntypedObjectDeserializer()));
Map map = (Map) xmlMapper.readValue(sr, Object.class);
Assert.assertEquals( sr.getLocalNameForRootElement(), "Root");
Object messages = map.get("Messages");
Assert.assertTrue( messages instanceof Map, "map");
Object list = ((Map)messages).get("Message");
Assert.assertTrue( list instanceof List, "list");
Assert.assertEquals( ((List)list).get(0), "Hello");
Assert.assertEquals( ((List)list).get(1), "World");
And the output looks like this:
{
"Parameters" : {
"Parameter" : [
{
"name" : "A",
"" : "valueA"
},{
"name" : "B",
"" : "valueB"
}
]
}
}
…which is what we wanted.
Find the source code here: https://github.com/DinoChiesa/deserialize-xml-arrays-jackson
Hat tip to Jegan for the custom deserializer.
I have relied on URI.js from medialize for years.
I downloaded it a long time ago, and it just works. It’s handy for parsing and building URIs form within Javascript.
I happen to use nodejs often, but I also use a JavaScript engine that runs in the JVM (via Rhino or Nashorn). So I liked URI.js for its usability across those systems.
Recently I decided to download “the latest and greatest” URI.js, and what I found… did not make me jump for joy.
URI.js is no longer “just downloadable”.
Where before I could just download the raw JS file, URI.js now has a builder that allows me to select which options I wish to include. I get the concept, and it’s a nice idea, but when I de-selected every option, I got a minimized URI.js that I did not want. When I went to the source tree I found a URI.js that included all the require() statements for punycode, Second-Level Domains, and ipv6, all stuff I did not want.
*snif*
I couldn’t figure out how to get it to “just work” in nodejs without all of that, so I had to resort to manually changing the code. Basically I just removed all the require() statements for those unneeded / unwanted modules.
And it works.
It’s possible I’m missing something basic, but for sure, it got more complicated to get the simple solution. Seems like a step backward.
An unsolicited tech tip.
Those of you who are API people, should exhibit good API hygiene.
One aspect of that is: “stop using curl -u” !!
Sometimes you have the urge to run a command like this:
curl -X POST -v -u 'yourusername:password' . https://foobar/slksls
Avoid this.
OK, ok, I know sometimes it’s necessary. But if you have an API endpoint that you often tickle with curl, and it accepts credentials via HTTP basic auth, you should be using .netrc to store the credentials.
The problem with using -u is that the password is shown in clear text on your terminal!
OK, I know, you’re thinking: but I’m the only one looking at my screen. . I can hear you thinking that right now. And that may be true, most of the time. But sometimes it’s not.
Sometimes you cut/paste terminal sessions into an email, or a blog post, or a bug report. And that’s when your password gets written down and shared with the world.
Treat Basic Authorization headers the same as passwords, because any observer can easily extract your password from that.
You might think that it’s ok to insert credentials in an email if it’s just being shared among your close work colleages. But that’s a bad idea also. Audit trails depend on the privacy of credentials. If you share them, the audit is gone. Suppose you have a disgruntled (ungruntled? never gruntled?) colleague who decides to take your creds and use them to recursively curl -X DELETE a whole bunch of resources. And the audit trail will show YOUR name on that act.
In short, it’s bad form. It could be forwarded or copy/pastad or it could leak into habit. It sets a terrible example for the children.
Here’s what I suggest:
If you have a *nixy machine, create a ~/.netrc file and insert your creds there. See here for information.
chmod the file to 400. When you use the -n option, curl knows how to extract your creds from the file silently. You never have to type credentials on the command line again. I think you can do this on Windows too, but I don’t know curl on Windows.
If you build scripts that use curl, you should allow the user that same option. That way the user never keys in their creds to your script.
When you pass the -n option to curl, instead of -u USER:PASS, it tells curl, “if you ever connect with site.example.com, then use THESE creds” . This works with any HTTP endpoint curl can address via Basic Auth. I have creds for Jira, Heroku, and other systems all in my .netrc.
Hint: also don’t use curl -v, because that will show the basic auth header. You probably want -i anyway, which is less verbose than -v.
Use some other tool that hides the credentials completely.
I think Postman doesn’t quite hide the creds completely. So be careful!
Let’s all try to exemplify good security behavior.
Yes, it’s that time of year when people think about RFC7523, which describes how to exchange JWT for opaque OAuth tokens.
Right?
If you’re like me, the waves of acronyms, jargon, and IETF RFCs (see what I did there?) seem to never end. OAuth, JWT, RFC 7523, JTI, claims, RS256, PBKDF2…? I feel your pain.
But there is some good news… here’s something that will help clarify the ideas and use cases around RFC7523. I wrote a quick article, and also created an Apigee Edge API Proxy, that implements this for you. It illustrates exactly how to exchange JWT for opaque OAuth tokens, and I even include some commentary int he readme explaining why you’d want to do it. (Spoiler alert: It’s faster to verify opaque OAuth tokens). All available on the Apigee community site.
The way I think about RFC7523 – it is an alternative to the client_credentials “grant type”, described in IETF RFC6749, which is the document that describes the OAuth v2.0 Framework.
OK, I hear you saying it: “back up, Dino… What is this client_credentials thing?” Yes, there is an underscore there. The client_credentials grant type is designed to allow a client app to identify itself to a token dispensary. The client says “here’s my ID, and here’s a secret that only I (the client app) should know.” And the token dispensary can then look at those two pieces of information, and if they are valid (the client_id is not expired or revoked), then the token dispensary can issue a token. It’s like username + password authentication for a person, but client_credentials is used for identifying a client app. This grant type mostly useful in server-to-server communications, when one service is being used by another service. BUT, some people use client_credentials grants in their mobile apps, so that the API service can trust that the mobile app is who it claims to be. (There are some problems with this; basically the client_secret needs to be embedded in the client code, therefore it is accessible to hackers, and therefore it is not truly “secret”. We can talk about mitigations for this in a future blog post.)
So that’s the client_credentials grant type. As I said, RFC7523 is an alternative to the client_credentials grant. Basically, instead of sending in a client_id and client_secret, under the RFC7523 flow (which has the helpful and easy-to-remember moniker of “JSON Web Token (JWT) Profile for OAuth 2.0 Client Authentication and Authorization Grants”, seriously) the client app self-signs a JWT which includes the client_id as the issuer. The app sends that to the token dispensary. The token dispensary verifies the signature, verifies that the client_id is valid, and then issues an opaque OAuth v2.0 token.
Now, there are some interesting implications to this model. Maybe these are obvious to some of you, but I will state them anyway:
Once you get past those implications and the extra set-up overhead, the model in RFC 7523 is really nice because it’s extensible. That’s because the request-for-token is encapsulated in a JWT, and the JWT itself is extensible. You, as an API designer, can stipulate any arbitrary (custom) claims that clients must include in the JWT, in order to compose a valid request-for-token. And you can include restrictions on the standard claims or custom claims. Some examples:
BTW, the example API Proxy I shared on Github shows how to implement the lifetime and one-use-only controls. (As with everything I publish on github, pull requests are welcomed!) If the inbound JWT that comprises the request-for-opaque-token does not pass these checks, a 401 Unauthorized is sent back.
BTW #2, did you know that Google services like Stackdriver and cloud storage use JWT-for-opaque-token exchange in order to enable service-to-service integration? Google also institutes the lifetime and one-use-only controls. The lifetime of the JWT must be less than 300 seconds.
Say, that reminds me!, Speaking of Google, did I mention that Google has acquired Apigee? Yes, I work for Google now! Part of the Apigee team within Google. w00t! I’m pumped, psyched, charged up, amped, and very pleased about this development.
So far, minimal changes for me, except for me I got a Chromebook! And yes, I authored this post from that very same device.
As always, I’m interested to hear your feedback on this. Let me know in the comments section.
Finally, I would like to wish all of you a Merry RFC7523 Season; and I wish you many Happy short-lived OAuth Tokens in the new year.
Browsing HN this morning, I found an interesting piece discussing Stackoverflow. Apparently hackernoon wrote a piece entitled “The decline of Stackoverflow” and it got some attention on reddit.
Bozho’s perspective aligns pretty closely with mine. I’m in the top 0.1% overall on SO, but I have not contributed actively in years. Not because I perceived “a decline in the site” whatever that means, but because I got busy with other things, specifically Apigee.
I agree with Bozho that many of the easy questions have been answered. Sure, there are always new questions, and new technologies like Golang or some new iPhone feature, or a new version of Angular, will prompt a new class of questions. But, the basics around .NET GAC, or Java garbage collection, or how to do read-through caches in Java, or what is JSONP…. you know those are already answered.
And the “Early movers” on Stackoverflow – I guess I was among them – have garnered all the top scores, and continue to accrue points as new readers on stackoverflow upvote answers. So even though I haven’t posted a new, popular answer in a while, I still earn points every week. New arrivals to stackoverflow will probably never be able to attain the level of points I have. My point total grows more than theirs, even though I haven’t done any work, and they may be asking and answering new questions diligently. It’s very unequal.
Don’t get me wrong – I think Stackoverflow is super valuable. I agree with Bozho, that it serves its purpose well, which is to provide easily searchable answers to programming questions. But aside from that massive contribution of value to the programming community, if you just look at internet points, the points go to the early movers.
I’m looking forward to watching the first formal debate for the US presidential election, this evening. Coincidentally, last night I watched a Frontline episode from June, entitled Policing the Police – excellent work by Jelani Cobb btw. So in this moment the analogy that comes to my mind, for this phenomenon on stackoverflow, is economic.
Imagine the discovery and settlement of a new continent. The early movers come in and (after eradicating the natives, if any) lay claim to land. They work the land, and maybe expand. Later more people arrive, and they need to rent land from the earliest movers. It’s those early movers that continue to accrue $$ and interest over time. Later more people come in, and there is no opportunity to gain land. Their option is to work the land for someone else (upvote other questions and answers). I was born in the 1960’s, and all the land in the USA was already claimed. Large holders of assets including land at that time, have continued to benefit. (I am not complaining – I was born white, male, and healthy in one of the richest countries on the earth – I won the lottery at birth.) Or consider a real-estate marker in a geographically constrained area like Seattle or San Francisco. Early movers claim land and erect buildings, and later movers (like me) come in and rent space.
On Stackoverflow, it’s really not a big deal if a newcomer is blocked from attaining “internet points”. A newcomer can still benefit from the answers and discussions on the site. They’re digital assets! Sharing more does not decrease the value of the asset. Everyone can benefit. And internet points are worth approximately $0.00. In real life, it’s a huge deal if the top 1% of the population holds 40% of the wealth, and the trend is toward greater inequality. This wealth is tied to assets like land, housing, companies, etc., which act to extract more money from the people who are not in the 1%. Not sustainable, and more importantly, not moral.
I don’t think a massive, sudden transfer of wealth is the answer. That is the phenomenon we are watching in Iraq and Syria, where different parties are grabbing oil fields, or attempting to attain control of entire cities in order to tax the inhabitants. It is what happened when the Vikings invaded what is now France, and grabbed land (establishing Normandy) and started taxing citizens. Many people die in these sudden transfers of wealth. There is a great benefit in stability – it means people generally die of natural causes rather than war wounds.
On the other hand, substantive change is urgently necessary. Gradual, thoughtful, methodical change, ideally. Of the two candidates most likely to win the US presidential election, neither appear to be interested in changing things very much. Too bad.
This post will be a bit techy. I confronted and solved a minor problem yesterday, and in the spirit of the internet, thought I’d share the solution, in case anyone else tries something similar.
This is about Drupal forms, and specifically within forms, the #states capability, which is a way that form designers can tell Drupal to do jQuery magic things on the form elements, enabling or disabling some of them based on the state or value of others.
The typical example is a checkbox, that when checked, will either enable (css ‘disabled: false’) or make visible (css ‘display: block’) a dependent textbox. Simple enough, right? and for that kind of simple case, it works well.
Drupal’s Forms API is described here, and the related
drupal_process_states here.
This is what it looks like to configure a Form in Drupal:
That says, show the textfield only when the referenced checkbox is checked. The reference to the checkbox is with a jQuery selector. This one works, really straightforward. And, the state is managed by Drupal in both directions. When the referenced checkbox is checked, then the textfield is visible. When the referenced checkbox is unchecked, then the textfield becomes not visible.
But what if you want a set of mutually exclusive checkboxes?
One approach is to just use the above model, and have each checkbox depend on the other. In other words, something like this:
This will not work. The reason this does not work, is that the state is managed by drupal in both directions. When checkbox #1 is checked, then checkbox #2 becomes unchecked. Which means checkbox #1 gets checked. Which means checkbox #2 becomes unchecked. And if you turn on the Firebug debugger, you can see the logical loop going round and round, endlessly.
There was an approach described here that suggested using two conditions in the array. But that didn’t work for me; I still had the endless loop. After fiddling with this for an hour, searching around for hints, I decided to just do it myself with my own jQuery. The logic was simple to write. And, I didn’t want to fight the Drupal Forms API any longer.
So here’s the solution. Include this JavaScript in your module:
As you can see, it registers a ‘change’ hook for a specially-marked checkbox. And when the checkbox is affirmatively checked, it unchecks the other checkbox. When the checkbox is unchecked, it does nothing.
How does that JS get loaded? In the Drupal module code, do this:
And finally, how do we set up the checkboxes in the Forms API? Like this:
And that gets the desired behavior: It is possible for zero or one of those checkboxes to be checked, but not both.
It took more time to write this post than it took to build the solution shown here! And of course I never did manage to figure out how to do the same just using the Forms API. This is an example of an API, the Forms API in Drupal, that does some things well, and this one thing….? Not so well. Much easier to just jump out and solve it this way.
Maybe this will help some one else!
By the way, this is included in a Drupal module that allows administrators to verify / validate user registration.
Bryan Kirschner of Apigee on what it takes for a CEO to pass the laugh test when it comes to digital strategy.
He proposes a handful of metrics that CEOs should be managing; and says that observers should judge CEOs on their progress on these metrics.
Makes a lot of sense to me.
I have a bit of java code that handles JWT. It generates a MACVerifier and then uses that to verify a signature. Someone commented that it was taking more time than they expected. I didn’t see a ton of opportunity for optimization, but I thought I might wrap the generation of the MACVerifier in a cache.
At first I tried EHCache. EHCache is the gold standard as far as Java caching. There are sooo many options, and there is sooo much flexibility. Write through caches, read-through caches, caches with persistence that is configurable in ways you had not imagined you needed. Java Attributes to add caching to servlets or JAX-RS. EHCache has it all.
So I figured it would be a safe choice. But after a little bit of fiddling with it, I decided EHCache was too much. To me, EHCache violates the “do one thing well” principle of design, or if you like, the Single responsibility principle (As applied to the module, if not a particular class), or, just unsatisfying documentation which is a common problem even among “successful” open source projects.
Why is there a CacheManager? What if I create a Cache and don’t register it with a CacheManager – what happens? What do I lose? Why do I want a CacheManager? Why are there names for both managers and caches? What would happen if I registered a Cache with multiple managers? What if I don’t want persistence? What if the Cache itself goes out of scope – will it be garbage collected?
I couldn’t find ready answers to these questions and the whole experience left me lacking confidence whether the cache would do the right thing for me. In the end I concluded that EHCache was more, much more than I needed, and would require more time than I wanted to invest, to get a cache. I just wanted a simple in-memory Cache in Java with TTL support (where TTL also implies time-since-last-access or time-to-idle). And what do you know! Google Guava provides that!
At first it was unclear how to best exploit it. But a little reading showed me that Guava has a clever design that allows the cache itself to load items into it. I don’t need to write MY code to check for existence, and then create the thing, and then put it into the cache. Guava has a LoadingCache that does all this for me. I just call cache.get() and if the item is present, it is dispensed. If it is not in the cache, then the cache loads it and gives it to me. Read-Through cache loveliness. So simple and easy.
This is my code to create the cache:
And to use the cache, I just call cache.get(). Really slick. Thanks, Google!