Better error logging with with error wrapping

This December, why not give the gift of better error logging?

For a while I have thought that the functionality provided by the stdlib errors package in the standard library was insufficient. One issue with errors is that they lack the context, namely stack information, that say, an exception would provide in another language. With the stdlib package, you can see the line number where an error is logged but you cannot see where the error was generated which is usually more important. This is because good error handling practice is to log the error at its highest level of propagation, but it is here where it is most far removed from the site of the error generation. You do get the string that is set when the error is generated but these can often be dynamically generated or reused in multiple places so they are no replacement for a stack trace.

This problem can be resolved by use of the third party pkg/errors package github.com/pkg/errors. This package was designed to be used as a drop in replacement for the stdlib errors package, providing a superset of the stlib package's functionality.

pkg/errors provides some functions that can annotate errors with stack information and messages at certain levels of the stack. New, Wrap and WithStack all annotate an error with stack information whilst Wrap also enables you to provide a message relevant to that level of the stack. WithStack and Wrap are smart enough, using the runtime package to examine an error and work out where it was generated even after that function has returned, which is useful if you have no control over the code that generates the error. Where you do it is best to annotate the error at generation to be sure that you don't forget to do it.

Here is an example which demonstrates annotating an error with stack information at generation and providing a contextual message at another point of the call stack. What's so great about pkg/errors is that it is instantly swappable with the stdlib package so you can enrich your errors with a one line import change!







Pretty cool, huh, happy wrapping!

Initialising String Literals at Compile Time in Go

Recently I was working on a service and realised that we had no established way of querying it for its version information. Previously, on an embedded device, I have written a file at build time to be read by the service at run time, with the location set by an env var. It would also be possible to set the version in an env var itself but these are overridable

  However a colleague suggested injecting the version string into the binary itself at compile time so I decided to investigate this novel approach.

go tool link specifies that the -X argument allows us to define arbitrary string values

go tool link
...
  -X definition
        add string value definition of the form importpath.name=value
...

go build help explains that there is an 'ldflags' option which allows the user to specify a flag list passed through to go tool link

go build help
...
    -ldflags 'flag list'
        arguments to pass on each go tool link invocation.
...

So we can pass a string definition through the go command (build, run etc)!



In the above program we can see that we define a variable foo in package main.
Thus the fully qualified name of this variable is main.foo, thus this is the name we pass.

$ go run main.go
injected var is []

$ go run -ldflags '-s -X main.foo=bar' main.go
injected var is [bar]

This can be used in novel ways to inject the output of external commands at compile time such as the build date.

$ go run -ldflags "-X 'main.foo=$(date -u '+%Y-%m-%d %H:%M:%S')'" main.go
injected var is [2017-09-13 13:44:59]

This is a nice feature that I'm sure has applications beyond my reckoning at this time. In my use case it makes for a compact, neat solution, however it does cause some indirection when reading the code making it harder to follow so I would argue should be used sparingly. It has the nice quality of being immutable over env vars which could potentially be rewritten. Anyhow it is a pretty cool linker feature!

Go 1.9 Released

Go 1.9 was released yesterday. Release notes are available here.

This release has couple of useful changes in addition to the introduction of type aliases.

1. ./... no longer matches vendor directories. This means that running go test ./... or say go lint ./... from the top level of your repo in order to run on all your packages no longer annoyingly matches your vendored dependencies. Previously one would be forced to run 'go test $(go list ./... | grep -v /vendor/)', and now one can simply run 'go test ./...'. Wooo!
   
   
2. go vet and go tool vet now support all of each other's build flags. This is a welcome change that provides some uniformity in their usage, as although go tool vet was designed for linting by package and go vet for linting by files, as mentioned in a previous post certain flags such as -shadow were not available in 'go vet'.

Congrats to the go team on another successful release!

Mocking Large Interfaces Using Embedding

Introduction

On the topic of embedding, Sean Kelly gave a great talk on embedding at this year's golang uk conference, it was quite dense but lightened with constant pictures of his pet corgi!

If you are not familiar with embedding in go, it is sort of like inheritance. Only it is based upon a 'has a' rather than the 'is a' relationship of traditional inheritance in object orientated languages, this can be seen as a sort of 'sideways inheritance' or composition of one object upon another.

See example: https://play.golang.org/p/e_I2Eg9a_D

In the above example we can see struct E embedded in struct S. This allows struct S to inherit members of E into its own namespace.
This is useful as it enables us to reuse functionality provided by an embedded type across different structs as though this struct provided this functionality. This allows us to reuse objects across classes in the same way that we would functions.

Mocking Large Interfaces using Embedding

When we embed an interface in a struct the compiler is satisfied that the struct satisfies the interface. This allows us to create mocks with much less boilerplate than stubbing every member of the interface. The caveat is that if we don't implement a member of the interface on the mock and it is called then that will generate a nil pointer runtime error - a panic. See example: https://play.golang.org/p/REIuxJG2x6

An alternative is to use a mock generation tool such as counterfeiter which generates the mock code itself given the interface definition.

Bear in mind when mocking a large interface we should ask ourselves if a smaller one would suffice. As the go proverb says 'The bigger the interface, the weaker the abstraction'. Larger interfaces are less expressive and less composable. I have had to deal with large interfaces mainly generated from protobuf service definitions. But if you must mock a large interface embedding or generation may save you time.

Golang gotcha #5: launched timers cannot be garbage collected till stopped or fired

You may run into this gotcha if you run timers with long timeouts in a tight for select loop. It is idiomatic to use a timer in order to provide a timeout on a channel receive. It is common to use time.After for this, because time.After is very easy to use as it provides a '<-chan Time', equivalent to 'NewTimer(d).C', but there is no way to stop it. And as the godoc says:

The underlying Timer is not recovered by the garbage collector until the timer fires. If efficiency is a concern, use NewTimer instead and call Timer.Stop() if the timer is no longer needed.

This example demonstrates such a leak when using time.After:
timer_leak.go
https://play.golang.org/p/XLY-isFtnn

This example demonstrates that the timers are not garbage collected even after the function they are launched in returns, as some may expect:
timer_leak_func.go
https://play.golang.org/p/W3rjZ5ziFb

This example demonstrates that stopping the timers resolves the leak:
timer_no_leak.go
https://play.golang.org/p/5CThl1JbaS

This may seem like an unnecessary optimisation but in the right circumstances, a tight for loop with a long lived timer, these can really add up.

Golang gotcha #4: Implicitly ignored return values

It is possible to implicitly ignore all values returned by a go function without any compilation or even vet errors. This is demonstrated in this playground example:

https://play.golang.org/p/V0fmYKADFx

Now, I am imagine that you are asking "Why is this ever allowed? Madness, madness, insanity and lies!" or something less dramatic. Well consider the fmt.Printf usage in the example and lets take a look at the fmt.Printf signature.

'if ignorance is bliss then knock the smile off my face' - Rage Against the Machine

 

Yes, we're only bloody, implicitly ignoring that too aren't we? You see, a fair amount of the standard library relies upon this behaviour so disallowing it would break backwards compatibility :(. Though I would argue that this implicit ignoring is bad practice and should not be explicitly utilised or encouraged when it comes to user defined functions, it is an easy avenue for bugs to creep in.

In-depth discussion here.

Golang gotcha #3: Accidentally shadowed variables

A common mistake that I have often seen cause exasperation to many programmers is that of accidental variable shadowing, this is one of the first things that I look for when asked to help debug misbehaving code. This occurs via misuse of the short variable declaration clause :=.

Let's recap declaration, assignment and what the shorthand does:

So the := clause is shorthand for a declaration and an assignment, where the type of the declaration is implicitly inferred from the assignment value. It is very useful and the lack of verbosity feels almost like you are using a dynamic language.

Now when does this get us into trouble? The problem occurs when we accidentally declare and assign to a new variable in a different scope rather than assign to an existing variable in an outer scope.

'Short cuts make long delays' - Frodo Baggins

https://play.golang.org/p/Pset590PA2

This example demonstrates an accidental variable shadowing bug in code used to determine the highest integer in a slice. The variable 'highest' is redeclared and assigned to in the scope of the if statement, shadowing the 'highest' variable declared in the scope of main. Whereas desired behaviour is assignment to the highest variable declared in the scope of the main function. Here we say that the variable highest is 'shadowed' as a result of this redeclaration. Try modifying line 14 of the code to 'highest = v' and note the change in behaviour.

Now, it is a good question as to why this is allowed, I believe that it is primarily to allow flexibility and to protect existing code from things like imports as explained here.

This is catchable by using go vet with the experimental option -shadow enabled.

Note that it is necessary to invoke vet via 'go tool vet' rather than 'go vet' in order to enable flags, see this issue.

For those interested, more in depth discussion can be seen here.

Golang gotcha #2: Methods without pointer receivers cannot modify the actual receiver object

One gotcha that most newcomers to the language run into involves pointer receivers. A method on a struct is declared with a receiver such as those above. Meow is a method on struct type Cat with a receiver of c. c here is similar to self in Python, though we name it by convention as an acronym of the struct name rather than self. In the first example, when Meow() is called, c, the instance the method is called on, is passed as though it were an argument of the method.

If the method were instead defined with a pointer receiver as in the second example, then c would evaluate to a reference to the Cat instance.

Thus the impact of using non-pointer receivers is:
1) Additional memory overhead from copying.
2) Immutability of the actual receiver. This is the major gotcha, this means that methods declared with a non-pointer receiver cannot persist changes to the instance outside the scope of the method.

See this example for a demonstration:
https://play.golang.org/p/aRSViFKlGn

I think that non-pointer receiver methods are more trouble than they are worth. It may seem wise to use them on methods that do not mutate the instance such as Get() and the like but there is a real danger it will come back to bite you in the form of a hard to diagnose bug.

Notable mention in golang docs:
https://golang.org/doc/faq#methods_on_values_or_pointers

Golang gotcha #1: Taking references to loop variables

The indomitable Smiler at Alton Towers
Holder of the world record for the most inversions, at 14

A colleague of mine ran into a bit of a golang gotcha recently. This relates to taking references / pointers to loop variables. I was already aware of the danger of this in the usage of closures. Closures access all variables in scope, implicitly by reference, this is in contrast to a function argument which is passed by value, this is demonstrated in the following example:

https://play.golang.org/p/M5hAoE3XpQ

The closure in the for loop evaluates a reference to i rather than a copy of it, thus as i changes, the value that each goro has a reference to changes. In contrast, passing i as an argument to a function takes a copy. In general I dislike closures, they have a dirty scope, it is unlikely they use all variables in their scope, thus they have low cohesion and are harder to reason about. In Golang it is idiomatic to favour the explicit over the implicit, if you need access to a variable, you are better off explicitly passing it as an argument.

Now, onto the example which inspired this post:

https://play.golang.org/p/YVhysSXzZD

In this case, similarly to the previous example we take a reference to a loop variable. However, the loop variable is reused in the further iterations of the loop. Thus our reference is now to a different value. It pays to be careful with loop variables and not to take references to them! An alternative solution as demonstrated in the example is to take the index or to take a copy.

Hopefully this post has illustrated some of the dangers of taking references to loop variables.

PS. I went on the Smiler last year, it really is quite good.

On Golang and Maintainability

I have talked a bit before, mainly in this post, about how Golang as a language tends to expose complexity and excludes some features that while useful can serve to hide complexity. In this post I'm going to explore this topic in more depth and explain why I think this contributes to Golang being a language better suited to writing maintainable code than Python.

Any sufficiently advanced technology is indistinguishable from magic - Arthur C Clarke

Where Python favours the implicit, Golang favours the explicit. And, where Python hides complexity in 'magic' language features, Golang forces you to go the long way round. Some language features in Python that I consider suitably magic are: decorators, properties and list comprehensions. Decorators and properties are mechanisms of indirection, and all these listed features provide handy shortcuts for developers. List comprehensions themselves are fine but nesting or using them for their side effects can quickly result in difficult to read code.

 Short cuts make long delays - Frodo Baggins

The interactive capabilities of the python interpreter can encourage a user to build up multiple lines of Python code into a single complex expression. Case in point, nested list comprehensions, these are usually the result of the condensation of a couple of loops into a one line wonder. And, programmers tend to love one line wonders, they exude elegance, and removing all those lines makes you feel warm and fuzzy inside, because readability and conciseness are easily confused.

Given the fact that it took some thought and tinkering to determine how to compress some readable for loops into such a concise representation, it is likely that the next person to come along, in the absence of the context of the expression's formation, will struggle to decode the compressed representation. In fact they may even try and rewrite it long-form in order to unravel its secrets. List comprehensions that are used for their side effects are full of even more implicit nastiness.

Maintainability comprises a number of factors but a key one is the ability of another programmer (or even you!), to come along and understand the intention of your program. Readability is not inversley proportional to LoC (number of lines of code), mistakenly in this belief programmers can be inclined to do things in complex rather than intelligble ways. The problem is that it can be difficult to distinguish the two. Perhaps a misunderstanding of the code is a indicator of a flaw of the reader or perhaps it is because a simpler representation would suffice. In the former case the writer could be forced to writing a lowest common denominator. In the latter case it pays to consider a language feature's potential cost as well as its benefits.

Language features are like power tools, we come up with excuses just to use them

Golang forgoes many shortcut features resulting in more explicit and maintainable code. I have found that whilst no means necessary, static typing also helps manage complexity and thus improve maintainability in a large application. And optimising for maintenance can be a good idea as this is often where we spend most of our time as developers.

Improvements in go 1.8

This post represents notes collected on the new go release and from the state of go talk of Feb 2017, on changes in go 1.8.

Video of the talk can be found here.
Slides of the talk can be found here.

 General Improvements

• ignore struct tags in type conversions (easier type conversions)
• 32-bit mips support
• osx 10.8+ supported
• go 1.8 is last version to support ARMv5E and ARMv6 processors
• go 1.9 will require ARMv6K 
• go vet (sort of compiler warnings) now detects closing http.Response.Body before checking error
• default gopath $HOME/go on unix
• go bug command opens a bug on github.com/golang/go with version/machine information
• Compiler backend improvements (SSA) sees cpu usage reductions of 20-30% on arm and upto 10% on x86 (SSA was already part-implemented on x86).

 Performance Improvements

• build times faster than go 1.7 but slower than go 1.4
• improved -race detection
• mutex contention profiling `go test bench=. -mutexprofile=mutex.out`, can provide data on whether you should lock in a less or more granular manner, sequential could even be faster.
• sub-millisecond (~100 microsecond) GC pause times, costing an extra 1/2% cpu.
• defer is a 1/10th to a 1/3rd faster, but still not that fast, for example...
• cgo is 50% faster, mostly due to removing high frequency defer calls

Additions to the Standard Library

• sort.Slice() introduced, provides easier slice sorting
• plugins introduced (linux only linux atm), load shared libraries at runtime, enables hot code swapping
• added Shutdown method to http.Server, was previously very hard to stop previously, personally I had to resort to https://github.com/hydrogen18/stoppableListener
• HTTP/2 support introduced

Full go 1.8 release notes are here.

go 1.8 is set to be released on February 16th 2017.

Golang UK conference is on August 16th to 18th 2017.

Thoughts on Two Years in Golang

In my last post, I talked/ ranted a little bit about not being swept up in new trends or languages without proper analysis of their pros/cons and suitability for use in certain scenarios. Hence after having learnt Golang from scratch two years ago and having been programming in it day in day out its about time that I collected my thoughts on it.

Now a lot can be said about the cost of learning a new language, that time spent learning the basics, making the right of passage mistakes and getting up to speed with the tooling. However, I think that Golang recognises these costs and does what it can to mitigate these for a new developer, not to say that there isn't still a cost. But, I know that for many companies, mine included, the ease at which a Golang programmer can be converted is a signifcant consideration in the choice of the language.

C and Python had a love child and they called it Golang

Strict, opinionated and boring

I think of Golang as a strict, opinionated and boring language. Now, I know that the word 'boring' has many negative connotations. But when I invoke it here I mean that it lacks many of the features that tittilate academics and occupy the minds of advanced programmers. I discussed the exclusion of exceptions in a previous article. Other non-existent features include some I miss: Generics, operator overloading, primitive sets, assertions. And some I don't: nested functions, inheritance.

I have often heard people say Golang ignores the last X years of language development. Of course there are some useful features missing but in order to keep the language small and simple you have to be strict, and evaluate the costs and benefits of adding a new feature. Terseness can be considered as a feature in and of itself. In other languages the plethora of features can be bewildering and take an age to master, with the extra folds hiding more pitfalls and stumbling blocks.

Inheritance is a big ticket item but I have found that Interface gets you most of the benefits of duck typing without dragging in the massive amount of complexity and metadata fiddling inheritance brings.

Golang has some really nice features. Goroutines are great, these are lightweight concurrency primitives, basically multiplexing upon threads. There are also channels for communicating between goroutines. It is really great that Go can do concurrency so well out of the box and I find it much more clear than Python's generators.

Importantly Golang is very quick to compile and run, out-performing C Python easily and many other Python implementations. This is an oft cited reason for switching from Python to Golang. Most of my work with Golang has been on embedded devices and this was the reason Python was never in the running. There were concerns about its GC (Garbage Collection) latency but great work has been done to bring this to sub-millisecond levels in go 1.8

It has nice concise syntax, something akin to a cross between Python and C, which is nice as I am fond of Python syntax, Java syntax makes me queasy.

Ecosystem

Probably the best feature is the tooling available and the strength of the ecosystem in general, it is fairly comprehensive and has a strong standard library which is something I really miss in Python. It tries hard to get things right the first time and mostly succeeds.

govet and golint are great static analysis tools and gofmt and goimports can format your source code on save in compliance with the style guide, saving time and bikeshedding. Golang really benefits from the strictness here, introduced at such an early stage that everyone is forced to get on board. I am so used to auto code formatting that I also set up auto pep8 formatting in Python and didn't look back.

The source tree layout and the build process are also standardised and there are great tools for running, building, testing and generating coverage stats in a standardised way with very little effort. You get deployable static binaries with little hassle which I always found a struggle with Python. This layout and process is strictly dictated which I know will rub some people the wrong way but in my opinion it saves a lot of turmoil for a little sacrifice in freedom.

It is very easy to pull dependencies `go get github.com/username/foo`, and you're there. However the lack of versioning and no way of telling how popular a library is are problematic. There are some third party solutions to the former problem, personally I use godep and there was some attempt to fix versioning with vendoring, but I don't feel as this is a complete solution and poses its own questions. However I am always a bit horrified by the multitude of tools when I have to pull dependencies in Python  {pip, easyinstall, setuptools}, I don't think go does too bad in comparison.

Gripes

Now for some gripes.

Non-pointer receiver methods, this is often a common pitfall for new go programmers. In using a method with a non-pointer receiver, the receiver itself is copied by value meaning that changes to that receiver after the function call are not persisted. See this code example.

Lack of a generic max function, this is quite embarrassing for the language as it is something that newcomers will run into fairly early. Due to the lack of generics there is no max function for all numeric types and seemingly as a result of this no max function for any numeric type, err, yea, I know.

Sensible slicing syntax, now the syntax we have is quite nice for some use cases and is appreciated but I still have to resort to slice tricks.

Being strict and opinionated has downsides, on some issues the exclusion of certain features and lack of support for certain use cases makes it seem as though some problems are being wilfully ignored, namely, generics and dependency versioning.

Summary

I find Golang a great place on the ladder of abstraction, garbage collected and static typed. I can develop faster in Python but I am more confident of my Golang code's correctness as Python hides complexity, tries to be smart and lacks the safety of the compiler. However Golang does lack some of the libaries and stacks for widespread adoption on the server though this is improving everyday. And its memory requirements may be too demanding for some extremely resource constrained embedded environments, however it has performed admirably for our embedded use case thus far. After two years I like Golang as a language, there's much much more that I have to say about it. But it suffices to say that its a language that I am now very comfortable with and productive in and I feel more confident writing maintainable and efficient code in than Python.

On Golang and Exceptions

I have been programming professionally in Golang for a couple of years now and I have to say that I really like the language. My first experience of Golang was a bit of a drop in the deep end, coming into a new job where I would be using Golang as my main language with no real experience. Yet, despite this, it did not take very long before I became productive. I believe that this is partly due to the simplicity of golang and its density/ economy of language features.

Golang was designed to be a small, strict and opinionated language. The small size reduces the required learning time, strictness ensures users do not form harmful habits such as ignoring warnings or leaving unused variables lying around and its opionatedness puts an end to bikeshedding about things like brace placement. This is in contrast to a language such as C++, massive and sprawling and certainly intimidating to a newcomer. The size and complexity of C++ provides many places for the concealment of pitfalls. And an effective understanding of the quirks and gotchas of the lanugage is deservedly highly valued in the corporate world. The problem with giving you this much rope is that it is long enough to hang yourself many times over. Sure, it is powerful but it is shows little respect for your sanity if you are not well directed in your work. Golang also tries to avoid introducing magic where possible. By magic I mean, a feature that hides a sufficient amount of complexity so as to appear 'magic' to the uninformed.

'Any sufficiently advanced technology is indistinguishable from magic' - Arthur C Clarke

One of the magical language features that got the chop in Golang is exceptions. Recently, when doing some work in Python I noticed that I didn't really miss exceptions, they complicated the control flow a lot and caused me much fear and consternation. This is because exceptions are magic, they can cause unexpected jumps in your code based on non-local conditions and inject complexity. They are another thing that you constantly have to think about when writing code. I find that multiple return values, available in both Python and Golang, is a much more intuitive and useful feature that largely subverts the need for exceptions.

'But it doesn't even have exceptions' - reaction of an old workmate when I told him I was now working in Golang.

I see how exceptions can be useful in standardising error reporting, which is great. We've all had to deal with a function with obscure error reporting, that say returns an int value, and we end up asking, does 0 denote an error, what do negative values mean? etc. However Golang also standardises this by providing the error type and interface providing a standard with room for extensibility.

I understand that not allowing exceptions complicates the success case code as often 'if err != nil {...)' is liberally applied. However one really needs to consider if these minor gripes are worth adding extra complexity to the language and burdening the programmer with as an extra concern.