REST vs RPC

Internal vs External comms in a
microservices architecture

A Microservices architecture advocates the proliferation of many small inter-communicating services. As a result, using such an architecture can result in an increased overhead in the number of network bound, 'internal' inter-service communications.

This is in addition to the 'external' existing network comms of serving the clients accessing our service. I will refer to these two forms of communication as internal and external respectively. These two forms of communication can be observed to possess differing characteristics.

Internal comms are high frequency in nature and completely under our control, that is that we can coordinate upgrade of the client and server as necessary.

External comms are of lower frequency, can come from varied sources and backwards compatibility / versioning is more of a concern as we do not have control over the clients.

REST and RPC are two technologies that can be used for network bound communications. In my experience REST is better suited to use in an external API and RPC to use in internal comms, primarily for these reasons:

REST for external comms
- easy to version using headers
- well understood and ubiquitous, ease of client implementation
- json much easier to parse and debug than a binary format

RPC for internal comms
- low cost, around 20 times less than REST
- less boilerplate required for calls (which will be frequent), it is autogenerated
- provides easy streaming

In RPC changes to models require synchronised client and server releases, thus models often become expand (ie. add fields) only. This can lead to a lot of deprecated fields and lockstep client and server releases when models are altered, making RPC a little brittle for external APIs.

The boon of low latency comms that rpc provides cannot be understated and provides us the freedom to disregard much of the costs of network bound requests and fully subscribe to a microservices architecture.

Go 1.9 Released

Go 1.9 was released yesterday. Release notes are available here.

This release has couple of useful changes in addition to the introduction of type aliases.

1. ./... no longer matches vendor directories. This means that running go test ./... or say go lint ./... from the top level of your repo in order to run on all your packages no longer annoyingly matches your vendored dependencies. Previously one would be forced to run 'go test $(go list ./... | grep -v /vendor/)', and now one can simply run 'go test ./...'. Wooo!
   
   
2. go vet and go tool vet now support all of each other's build flags. This is a welcome change that provides some uniformity in their usage, as although go tool vet was designed for linting by package and go vet for linting by files, as mentioned in a previous post certain flags such as -shadow were not available in 'go vet'.

Congrats to the go team on another successful release!

Mocking Large Interfaces Using Embedding

Introduction

On the topic of embedding, Sean Kelly gave a great talk on embedding at this year's golang uk conference, it was quite dense but lightened with constant pictures of his pet corgi!

If you are not familiar with embedding in go, it is sort of like inheritance. Only it is based upon a 'has a' rather than the 'is a' relationship of traditional inheritance in object orientated languages, this can be seen as a sort of 'sideways inheritance' or composition of one object upon another.

See example: https://play.golang.org/p/e_I2Eg9a_D

In the above example we can see struct E embedded in struct S. This allows struct S to inherit members of E into its own namespace.
This is useful as it enables us to reuse functionality provided by an embedded type across different structs as though this struct provided this functionality. This allows us to reuse objects across classes in the same way that we would functions.

Mocking Large Interfaces using Embedding

When we embed an interface in a struct the compiler is satisfied that the struct satisfies the interface. This allows us to create mocks with much less boilerplate than stubbing every member of the interface. The caveat is that if we don't implement a member of the interface on the mock and it is called then that will generate a nil pointer runtime error - a panic. See example: https://play.golang.org/p/REIuxJG2x6

An alternative is to use a mock generation tool such as counterfeiter which generates the mock code itself given the interface definition.

Bear in mind when mocking a large interface we should ask ourselves if a smaller one would suffice. As the go proverb says 'The bigger the interface, the weaker the abstraction'. Larger interfaces are less expressive and less composable. I have had to deal with large interfaces mainly generated from protobuf service definitions. But if you must mock a large interface embedding or generation may save you time.

Back From the Golang UK Conference 2017

So the blog has been quite quiet recently, this has mainly been due to having other things occupying my time of late as I have been going through some big changes. I'm in London! I moved down recently having spent the last eight years in Leeds.

Never fear, this was partly motivated by my desire to work on new exciting projects in go which I will be sure to discuss in time. In the shorter term expect some posts related to talks and conversations I had at the Golang UK Conference. I was lucky enough to spend two days at the conference and see some great talks, meet fellow go developers and learn about how go is used in other companies across the world. I had a really great time and I would like to thank all the speakers, delegates and especially the organisers for making it happen!

Watch this space

Golang gotcha #5: launched timers cannot be garbage collected till stopped or fired

You may run into this gotcha if you run timers with long timeouts in a tight for select loop. It is idiomatic to use a timer in order to provide a timeout on a channel receive. It is common to use time.After for this, because time.After is very easy to use as it provides a '<-chan Time', equivalent to 'NewTimer(d).C', but there is no way to stop it. And as the godoc says:

The underlying Timer is not recovered by the garbage collector until the timer fires. If efficiency is a concern, use NewTimer instead and call Timer.Stop() if the timer is no longer needed.

This example demonstrates such a leak when using time.After:
timer_leak.go
https://play.golang.org/p/XLY-isFtnn

This example demonstrates that the timers are not garbage collected even after the function they are launched in returns, as some may expect:
timer_leak_func.go
https://play.golang.org/p/W3rjZ5ziFb

This example demonstrates that stopping the timers resolves the leak:
timer_no_leak.go
https://play.golang.org/p/5CThl1JbaS

This may seem like an unnecessary optimisation but in the right circumstances, a tight for loop with a long lived timer, these can really add up.

Golang gotcha #4: Implicitly ignored return values

It is possible to implicitly ignore all values returned by a go function without any compilation or even vet errors. This is demonstrated in this playground example:

https://play.golang.org/p/V0fmYKADFx

Now, I am imagine that you are asking "Why is this ever allowed? Madness, madness, insanity and lies!" or something less dramatic. Well consider the fmt.Printf usage in the example and lets take a look at the fmt.Printf signature.

'if ignorance is bliss then knock the smile off my face' - Rage Against the Machine

 

Yes, we're only bloody, implicitly ignoring that too aren't we? You see, a fair amount of the standard library relies upon this behaviour so disallowing it would break backwards compatibility :(. Though I would argue that this implicit ignoring is bad practice and should not be explicitly utilised or encouraged when it comes to user defined functions, it is an easy avenue for bugs to creep in.

In-depth discussion here.

Golang gotcha #3: Accidentally shadowed variables

A common mistake that I have often seen cause exasperation to many programmers is that of accidental variable shadowing, this is one of the first things that I look for when asked to help debug misbehaving code. This occurs via misuse of the short variable declaration clause :=.

Let's recap declaration, assignment and what the shorthand does:

So the := clause is shorthand for a declaration and an assignment, where the type of the declaration is implicitly inferred from the assignment value. It is very useful and the lack of verbosity feels almost like you are using a dynamic language.

Now when does this get us into trouble? The problem occurs when we accidentally declare and assign to a new variable in a different scope rather than assign to an existing variable in an outer scope.

'Short cuts make long delays' - Frodo Baggins

https://play.golang.org/p/Pset590PA2

This example demonstrates an accidental variable shadowing bug in code used to determine the highest integer in a slice. The variable 'highest' is redeclared and assigned to in the scope of the if statement, shadowing the 'highest' variable declared in the scope of main. Whereas desired behaviour is assignment to the highest variable declared in the scope of the main function. Here we say that the variable highest is 'shadowed' as a result of this redeclaration. Try modifying line 14 of the code to 'highest = v' and note the change in behaviour.

Now, it is a good question as to why this is allowed, I believe that it is primarily to allow flexibility and to protect existing code from things like imports as explained here.

This is catchable by using go vet with the experimental option -shadow enabled.

Note that it is necessary to invoke vet via 'go tool vet' rather than 'go vet' in order to enable flags, see this issue.

For those interested, more in depth discussion can be seen here.