Friday, 18 September 2020

Is software running before we can walk?

 Even the update to Blogger reminds us that people feel the constant need to update things. There is nothing obvious added to the interface and no additional features it is just one of those things that companies do because they don't have any actual improvements to make so why not a "freshen up".

Now I appreciate that you cannot leave old software running forever. OSs and frameworks change (rightly or wrongly) and if you want to update to get security fixes, you might lose support for old functionality and have to do some re-factoring but lots of changes are UI fluff and often make the interface slower and more clunky generally, have issues across browsers since they will either use semi-compatible features or build a massive bundle that is transpiled for everyone + dog.

And the issue isn't just web apps being updated. I am frightened at the number of times mistakes are made in workplaces due to a combination of either poorly written software, software that is far too complicated and then the people using these systems who have very little technical understanding.

I remember my sister trying to change her name on a utility bill and it took something 10 times. At one point, they managed to change her gas bill address but not her electricity. Now, you would think that a system that manages customer accounts would be pretty basic. "Change Details" => "Update" and it would just work? Of course not.

A good example is Salesforce. It is a massively popular and profitable business which is CRM as-a-service, which is great for them but I assumed it was popular because it was super slick and easy, a bit like GMail when it first came out. But no. It is frighteninly complicated with hundreds of screens - probably trying to be all things to all people but having terrible UI as a result. One thing I noticed was that it wasn't even nice "Bootstrappy" controls but something out of a Windows NT horror film.

There are loads of systems like this and the problems boil down to:

  • Software customers who don't know how to specify good software
  • Suppliers who don't know how to coach customers to specify good software
  • Suppliers who care a lot more about the money they can make instead of writing software to be proud of (you know who you are!)
  • Lack of formal training/expertise requirement in the supplier industry. Anyone can legally write code for virtually anything!
  • Lack of investment in UI/UX. I have met several people who do this and even there, the understanding is not always as high as you would hope. I saw a great example of UX design once (can't find the link) where a card payment page was being designed. Very simple questions/considerations about not wasting key strokes, not making the user click into boxes, not making them choose pointless things like what card type it was. By the way, a lot of card payment screens from people like worldpay and sage are shocking. You should be ashamed.
  • Users who do not need any formal training and are let loose on these systems. I worked with a (grown but not elderly) man who used to double-click hyperlinks on web pages. Goodness knows what damage he has inflicted over the years.
  • A general lack of the ability to correctly spot critical problems and resolve them. For example, the second time my sister had to call up to change her name should have rung an alarm. Either the previous call handler doesn't know what they are doing or the system is broken. Both scenarios are business critical but it seems most people only think to say, "sorry, let's try again".

What can we do? Of course, there is no set opinion on this. I would like to see the formalisation of the software industry, even if it only means one accredited person signs off on the work. This approach could give some slack like it does in Law and Medicine for areas that are personal preference, but it could also mandate things like acceptable password hashing algorithsm, approved libraries for certain things, frameworks which have passed muster and those that haven't etc.

I think we also need to start giving the general public mandatory IT training at school like a "driving licence". A poorly maintained computer can lead to virus infection and being part of a botnet of some sort, why would we let people just own these without any controls?

I also thing service providers could be more proactive, especially when known malware is detected from an IP address: block its DNs and return a standard page to the customer saying, "This line has been blocked due to malware detected, call this number to discuss options" or whatever. It could also just throttle the line so it cannot contribute much to a DDoS attack. Even if this was opt-in, lots of people would presumably rather know.

Lots of problems but until there is a recognised software institution, it is almost impossible to make this happen any time soon.

Thursday, 10 September 2020

Octopus Deploy with MicroK8S - You must be logged in to the server (the server has asked for the client to provide credentials)

 One of the most annoying errors is when you don't think you have changed anything and you suddenly see an error!

We have a microservices deployment on Octopus Deploy that points to a MicroK8S cluster on Ubuntu for development and an Azure Kubernetes Service for production. We had deployed successfully on Friday to development but today (the following Thursday), I noticed that it couldn't connect to the dev cluster.

The error in the log was: You must be logged in to the server (the server has asked for the client to provide credentials)

I obviously wondered about credentials expiring (they didn't), changes to the network/firewall (no changes) and then got a bit side-tracked with the fact that client and server versions of K8S need to be close: the client needs to be within 1 minor version above or below the server version. I spent an hour upgrading AKS to 1.17 since the dev cluster is already 1.19 and then downloaded and installed kubectl for 1.18 onto the Octopus but it still didn't work - AKS was fine but the dev cluster was not.

Eventually, after setting up a local cluster config and connecting to the cluster manually (and thank goodness it eventually worked), I realised that something was different on the dev cluster. Instead of username/password, the admin account was now showing as username/token. 

I found that by running microk8s config on the dev VM and it prints it in plain text!

Fortunately, Octopus supports tokens so I had to add the named token, updated the connection to the infrastructure and it worked!

I still do not know exactly what has happened but I can only assume that microk8s was updated automatically in the last week and the upgrade must have changed the password to a token, perhaps for security reasons or feature changes, who knows.

Monday, 7 September 2020

What makes a great developer?

 Anyone with enough experience should be able to answer this question. Anyone who has worked across a number of different organisations and types of organisations; who has worked in a number of different but related roles; someone who has managed people; who has fired people; who has produced great product and who has seen and made many mistakes along the way!

Someone like me.

Paul Graham says that the people you need to employ are "Smart and get sh@t done" which is a massive simplification but gets at the root of the issue. A great developer is clever enough to make short work of hard problems and then get's stuff done so that they can add value to your company. There are plenty of smart people who are not pragmatic and there are plenty of pragmatic people who are not very clever.

Clever people might see great solutions but might take too long to realise them, they might also be perfectionists. There are too many problems to be clever without pragmatism. 

On the other hand, pragmatists who are not clever might be too quick to "hack and whack". Who might think that any solution is a good solution, who are not good at seeing a big picture, or who cannot spot inconsistencies or unmaintainable code.

So here are some scenarios where I have spotted the good from the bad:

1) People who can very quickly debug a problem. The worst kind of problems are those that we know are affecting a production system and possibly affecting thousands of people. Despite that, there are those who can isolate and identify the problem in minutes and get a patch or even a proper fix out to production - and those who can't. Those who can't do this are often good at getting stressed and confused but who simply do not have acceptable debugging skills to use tooling to get logs/callstacks/repeatable local scenarios etc.

2) People who don't need you to write everything out in detail. There are people who I have worked with that I can disucss, say, an API with. With a few simple sentences, they understand enough to go away and produce exactly what I want. They understand enough to know that the stuff we didn't discuss is still important - assumptions like secure, performant, easy to read and maintain etc. Other people take much more time and input and even then do not have enough skill and produce code which is OK at best and incomplete or broken at worst.

3) People who can balance the work arising with staying on task. Some people get easily distracted wile implementing feature X. What they see might be definitely needed or it might be a nice-to-have but good developers know when something is a quick 5 minutes of extra quality, when it might be done but later, and when it is something that requires a quick, "I noticed X was missing, shall I do it" to the Team Leader.

4) People who are very self-aware of their skill level. It is hard to know how good you are at your job when there are so many variables. Programming is like medicine and law in that there is no single measure of ability and much is based on the context but what we do want are people that know that in the world of e.g. .Net, I am a 5 out of 10 or an 8 etc. Usually, the very young are better at this because they know that they don't much and they ask questions which hopefully get fewer and more advanced over time. There are, however, those who think they are an 8 when they are really a 4. What they produce is slow, sub-optimal and hard to maintain. They might have misunderstood the requirements but haven't asked for help because "they're an 8!". These are hard to work with because it's hard correcting something who already thinks they know it all!

5) People who understand architecture. Although we shouldn't pre-optimise code, we should understand that certain things are likely to be extended and certain things are not. If we have no need for multi-branding, we don't need an architecture that supports it. On the other hand, if we are building a plugin system, it should be written to be extensible due to its nature. Some devs don't understand enough and only ever see the code at a micro level.

6) People who both give and receive knowledge. It is nice when people offer knowledge to the group for the "greater good". I don't like asking people to give training but if you are a good developer, it should be part of your mindset. You learn something like OWASP Top 10 because it is really important so why wouldn't you want everyone else to know about it? Not everyone likes giving training but if you are clever, you should be able to work out a good presentation pretty easily.

7) People who live for the "continuous improvement". Good people I have worked with are the sort who say, "look how cool CI/CD is" or "dotnet core is so much faster than netfx, let's look to migrate" etc. I like it because it should be, "why not?" whereas others are so insecure, they prefer living with what they have and know. Obvisously we don't follow every latest trend but in most cases you have to run to stand still in the software world and it is frightening to me that some companies are still using apps that were written 20 years ago and are definitely not patched or up-to-date.

8) People who instinctively get the trade-offs in our profession. A good person would know that introducing Java to a .Net company (as a replacement) is unlikely to ever happen, not because Java is not as good but because there are so many dependencies on the tech stack from expertise, cost of hosting etc that the only reason you might ever do it would be if Java released something that was twice the speed of dotnet core out of the box - maybe it would be worth it. Lots of developers have very obvious preferences for certain things, which might be partially justified or might even be based on something you think you know but don't. These opinions come out in all sorts of ways (everyone else uses NUnit but I use XUnit; Everyone uses and understand Visual Studio but I have use Rider because it's slightly nicer; You have to use Unit Tests even if they don't make sense)

Did you notice what wasn't there? I didn't specify that a great developer knows how to solve a programming problem. Why? Because that is assumed. It is programming 101, even a junior shouldn't take too long to get their head round what is happening and why. If you can't Google something like "How to create dictionary from Linq in .net" you shouldn't be a programmer, honestly! The funny thing is that many developers think that it is precisely this that singles them out from the ordinary devs!

Now for the bad news.

These people are hard to find. They are probably not as rare as you might think but picking them out of the crowd is hard. Part of the problem is that some work for companies where their true selves have been beaten out of them and given the chance, they would thrive in a high quality environment.

Wednesday, 26 August 2020

TeamCity or Jenkins for CI with .Net?

 Here it is at last, an unofficial and very poor quality comparison between two of your favourite CI build tools Jenkins and TeamCity. Which one should you choose and why?

Introduction

Jenkins is free and open source and written in Java. It is probably the most widely known and used CI build system and has been around since 2011.

TeamCity by Jetbrains is a commercial product and is also written in Java using a SQL Server backend. It has been around longer than Jenkins at 2006 but both are clearly established systems.

Pricing

When you are setting up your devops environment, you might only need an agent or two but once you start growing, some of the commercial offerings start getting expensive fast. With Jenkins that is not an issue, which is great.

TeamCity starts with a free tier that gives you 3 agent licences and 100 build configurations. Certainly enough for most people when starting out but as soon as you need additional agents, they come in at around $300 each. Now if you need that many, you should be making some money enough to pay for these but that is certainly not guaranteed and the worst thing, of course, is that you might only need the additional agents very infrequently so paying the extra is slightly annoying. This pricing really doesn't scale. For a large outfit needing 100 agents, you are paying something like $25K, which seems a lot.

If you want support, you need to buy the enterprise licence, which for 3 agents is around $2000 and includes a year of upgrades and email support.

Support

Jenkins does not provide official support channels other than listing a number of companies who can support you. This probably works OK since I suspect that having a more local company or one in your market vertical is probably better than a single support channel. The pricing will obviously vary depending on what is on offer and what you need.

Support is included for both systems on forums but if you want email support with TeamCity, you need the enterprise licence.

Personally I have not used Jenkins support but I have used TeamCity. I find the response times very erratic, sometimes not receiving an answer for weeks and other times very quickly. The staff do seem to know what they are talking about once you do get an answer.

Configuring for .Net

TeamCity definitely has the better .net support. You can install the .net plugin and install MS build tools on the agent and it is pretty easy to add and configure .Net steps.
 
Jenkins on the other hand has minimal .Net support. The whole interface, to be honest, is very unappealing lacking basics like directory browsers for directory fields. In most cases, you need to use the "Windows batch command" box and just type in command line entries, which is annoying, because invariably you would need to copy and paste these as opposed to a nice plugin that helps you out by asking for what you need and passing to the command line correctly.

The UI Generally

I would also say that TeamCity wins in general. The layout is generally logical but there are definitely some real areas of concern. Firstly, the UI is very slow and unresponsive. Pages can easily take seconds to load, which is a shame when you are trying to update a lot of things very quickly. It also has issues with asynchronous updates sometimes working and sometimes not. For example, you are looking at a project page and someone starts the build, the right-hand side should update with the new build appearing. Sometimes it does, sometimes it doesn't.

The administration of TeamCity is also clunky. They mix features in the main content area with dropdown menus in some places and the fact that to edit some things, you need to go to the correct level in administration. You can't, for example, copy a build configuration from inside that build, you have to go up to the parent group and copy it from there.

Jenkins has avoided some of this by keeping really simple. In place of some of the nicer functionality that TeamCity has, Jenkins just has boxes. Lots of boxes. Whereas in TeamCity you add steps in a nice list that you can go into the steps or go back up to the list, Jenkins just has a massive long list of each step detail in one place. Even the navigation at the top of the build config just scrolls you down to the bottom. To be fair, I have not used it much so it might have been due to the specific test I was running but the interface on the whole looks like it did on day 1 in 2011, which looks like Java from 1998. TeamCity, also runs on Java but at least they make it look more like a web app.
 
Jenkins also has a funny approach to build variables. If you want to inject, e.g. the working directory into a command, you need to use different syntax depending on whether you are running on Linux or Windows and also whether you are running CMD or Powershell. Honestly, the UI could have hidden this but to add to the confusion, some plugins need the variables in a certain style whatever the underlying OS (perhaps what the server should do automatically!)

Another feature which seems very lacking in Jenkins is the ability to set the working directory of each build step. In a Windows batch command, you can CD into a directory but that is very clunky. You should be able to set the directory like you can in each type of TeamCity step because it just makes sense that way.

Plugins

Both systems have many plugins available. Lots of TeamCity's are produced by Jetbrains so they are quite decent. There are also lots included out of the box like DotCover to measure test converage as well as many .Net friendly things like Nuget, Nunit etc. You will probably not need to install any/many additional plugins.
 
On the other hand Jenkins is all about plugins. The server itself provides very little out of the box. You can't have, say, a .Net Build step without a plugin and then all you can find is an Msbuild one which doesn't install any MS stuff, just sets up the command and calls CMD with it. Because all of them are community, the quality is highly variable as you would expect. Some are out of date and unlikely to be updated, others are probably really good. You will need to install plugins to do most things in Jenkins so be warned!

Adding Agents

Like most systems that run Java, adding additional agents always seems a little long-winded. In Team City, you get some instructions that you run after installing Java and some other pre-requisites on the agents. Sometimes you can log in to the agents page on the server and download what you need and you can even, theoretically, push an agent to another machine but of course the setup for that with permissions and ports is not worth the hassle. I'm glad you don't do this very often.
 
In Jenkins, I was confused after first setting it up assuming that the server is not setup as an agent as it is in TeamCity. The UI implied I needed to add one so I did, pointing to localhost, which really screwed up the UI. I think it could easily have dealt with that and checked for localhost or 127.0.0.1 to tell you that you do not need to do this for localhost. Looking at the instructions for adding another agent, it looks like you need to install Java, download a war file and execute it. I don't know how easily that will work.

Performance

Well to me, this one is so important when you have lots of builds. How long does it take to run a build? You would think that running e.g. a .net build on any CI would take roughly the same amount of time because MsBuild runs as fast as it can run but you would be wrong.

To compare Jenkins and TeamCity, I decided to create a Jenkins server, add a build that matched what we already have configured on TeamCity and see whether there was any marked difference. I didn't expect much but there was a lot of difference! A build that was taking around 4m40s on TeamCity only took 2m50 on Jenkins! Yes a whopping 40% speed increase on Jenkins. That is despite the fact that the Team City server is 8 cores and 16GB and the Jenkins one only 4 cores and 8GB RAM - Msbuild only uses 1 core by default as far as I know.

Now we have had suspicious performance issues on TeamCity for a while, I just couldn't understand why unit tests that take, maybe 30 seconds on my developer machine, take a whole minute on TeamCity. Of course, we reached out to support and get sent a load of server tweaking stuff, some of which we did, some of which we didn't understand. That is not really an acceptable response for something that costs this kind of money. I would expect the people who write this to understand it enough to write a tool that prints a report saying "You need to increase that or decrease that" and ideally it should all happen during installation. We ran a trace on the build server and found their SQL database is being hit multiple times per second, causing significant CPU usage for SQL Server even when you are just running a basic build - we found a forum post from someone else with the response, "That is an important part of the system". Great.

The thing is, we are now using the 3 free agents and have some builds that take 45 minutes so reducing that to 25 minutes would be really significant and since build times affect us far more than UI issues, it really is something worth jumping ship for and going for Jenkins.

Conclusion

TeamCity is not crazy expensive if starting out and the UI is much more friendly, particularly for .Net but their performance is unimpressive and the pricing won't scale.

Jenkins on the other hand certainly looks older and lacks some of the niceties of TeamCity but if you know the basics of calling dotnet or msbuild on the command line, it probably won't take you long to create a suitable build so if you are up for that, I would probably recommend starting with Jenkins - you might never leave.

Wednesday, 19 August 2020

Random CORs errors on Chrome using Cloudfront and S3

 Well this was a journey.

TLDR; It happens when you make a non-CORs request that is cached by the browser. A subsequent CORs request for the same object finds the cached item and tries to use it - it fails CORs because the first request had neither CORs headers or a VARY header. This affects Chrome but not currently other browsers for accidental reasons.

So this is the story, we have a web app making a cross-domain call to a CDN that contains static assets. With me so far?

That CDN is Cloudfront on AWS which uses S3 as an origin. Cheap and relatively easy to setup (I still find AWS a bit like learning a new programming language but I digress...).

I set up CORS, which is mandatory when making cross-origin calls and it worked. Well most of the time.

Sometimes, very occasionally, we would get CORs errors in the console and the resource wouldn't load. Kind of a deal-breaker when you are loading site.css from CDN! The error message? "Access to CSS stylesheet at blah from origin blah has been blocked by CORS policy: No "Access-Control-Allow-Origin" header is present on the requested resource.

This is another classic example of the error is completely meaningful but only after you have learned the meaning. As far as I understood, the file definitely had the correct headers because it worked most of the time.

Google showed me lots of people with similar random problems and no clue about what was causing it. Some people suggested setting up CloudFront correctly, which is not useful if the errors are random. Others came out with the usual snake oil and "this worked for me" but in this case, they were mostly wrong. The OP was making changes, it seemed to work because the error was random and then some time later, "The errors came back, what else could it be".

Fortunately there are some people who do know what is happening (Thank you Michael - Sqlbot).

First you need to understand a bit about CORs - one of the most misunderstood controls in HTTP. Like many mechanisms, it strived not to break existing systems which led to some of its complexity. It is also a security mechanism that doesn't actually provide any security except where clients opt-in to the functionality!

Presumably you know that a web page can include links from anywhere. Many of us include jQuery or Bootstrap etc. into pages and often we get these from CDNs that exist on a different origin to the page. This is a cross-origin request and is normal. However, in some cases, people expose resources on origins that they want to use on their website but they don't want all the Haters embedding the resources in their sites and getting the victim to pay for bandwidth and hosting costs or losing valuable assets to be embedded into someone else's site (e.g. a news feed or photo list). This is where CORs comes in.

When using CORs, the client sends the origin that is requesting the resource in the "Origin" header to the e.g. CDN. In other words, shared.cdn.com/something is being requested from a page at origin www.stealingotherpeoplesstuff.com. Now this is where it gets weird. CORs doesn't say that the shared resource should not be returned, it says to return headers confirming or denying that the client is allowed to proceed. Why? because they didn't want to break existing browsers that wouldn't be sending these new headers and were already using a setup that worked OK. I personally think the w3c should provide deadlines for new functionality to be implemented otherwise we live with backwards compatibility forever but maybe they are nicer than me. So the client (let's assume a browser) then looks at the returned headers and looks specifically for the header "Access-Control-Allow-Origin" to include either a * (all origins are welcome!) or otherwise a specific origin that matches the one that was sent in the request meaning "you are specifically allowed". Note that the protocol says that if there are multiple allowed origins, only the one in the request is sent back in the header unless all origins are allowed in which case * is sent. If the header is present, the browser carries on as usual, if not, the browser blocks the use of the content and puts an error in the console explaining why.

So this all makes sense. It is slightly more complicated if the request is not "simple" in other words, something that is likely to have an effect on the endpoint you are calling like a POST or DELETE. In the case of these unsafe requests, a "preflight" options request is made to basically ask the question in advance of making the call. The response contains the same CORs headers (or not) but this time the browser will not make the call to the unsafe method if it fails because if it did, something bad might happen.

All of this can be bypassed without CORs. If you type in the URL to a protected resource in the URL bar and press enter, it simply loads. Why? It is not a cross-origin request! The URL you typed is the same as the "page" you are in so no CORs headers are sent and the resource is returned. A bit weird but like I said, this was a deliberate decision and this is where the original "bug" comes from.

OK, what happens when we introduce cache to the browser? You have to decide when to use cache and when the cached item needs to be different and another call is made to the origin server. The most obvious part of a cache-key is the host and path of the resource. In general if a second call is made to the same item and it is in cache, you get the cached item. Now there are many caches and they don't all exist within the browser. What happens, for instance, if you cached something that used gzip encoding but it is then requested from a client that doesn't support (or doesn't want for some reason) gzip encoding? Just using the path would fail because the second request would get the cached gzip one and it wouldn't work. So another piece of data is added to the cache key: the Vary header contents. The vary header says, "you can use this item from cache as long as the values of these headers match the original request". For example, if the vary header contained "Content-Encoding" then the item would cache not only the URL but also that the content-encoding was gzip. If another request to the same item requiring, say, deflate was made, the cache would know that the cached item is not valid and will make another request. It is up to the origin to send this header based on anything in the request that could cause a different item to be returned. Often the web server will add the vary headers for content encoding.

All good so far? The bug is actually an edge case but one that is valid according to the specification and occurs when you first make a call to a CORs related item but without CORs headers e.g. you enter the URL into the URL bar and press enter. Since no CORs headers are sent, the CDN does not return any cors responses. Since it is not a CORs request, the Origin header is not sent. Since Origin is not sent, S3 doesn't reply with Vary: Origin. So what is cached? A basic item with the path and a vary header that will probably include only the content encoding and also no cors headers. This is all valid although arguably it causes the whole problem.

Next, you make a CORs request. Does it match the path of the cached item? Yes. Does it match the content-encoding in the vary header? Yes. Great, we can use this. The CORs check is then made on the object that has no CORs headers so of course, it fails.

There are some workarounds but no-one is necessarily doing it wrong although somebody could do something to make it work:

The browser probably can't do much, although it might say "we have an origin header in the request and the cached item doesn't contain origin in the vary header so this isn't the right object lets call the network again"

S3 doesn't add a vary: origin response header even though by definition here, it is running as a CDN origin and requires this header by definition. At least if it was a switch to enable it, the problem would go away.

Cloudfront is probably not doing anything wrong. It passes the headers to S3 that it should and returns what it gets back. Ironically, this is the place that is suggested as the workaround because it supports scripts and the script to add (in the StackExchange link) is to say if there is no vary in the response from S3, add it in so that Chrome will cache it correctly.

The other browsers haven't fixed it as such, it's just their cache works slightly differently so it accidentally works for them!

aarrrgggh


Thursday, 13 August 2020

Exam Results are not fair!

 That's the cry in the newspapers today. In lieu of exams in the UK, which were cancelled due to Covid 19 risks, the education departments had to decide how to award exam results to people who did not sit them. 

In Scotland, they took the estimated grades from teachers, applied a normalisation function to the results (as they actually do each year in fact) to try and account for unreasonable fluctuations and to ensure that the results have some meaning when compared to the previous and subsequent years. This meant that many results were downgraded from what teachers predicted, leading to an outcry, leading to the Scottish government changing their tack that no grades would be awarded that were lower than the teacher's prediction, much joy ensued...

Except not everywhere. Hang on a second! I worked really hard and was definitely an A student and now I've found out that another student who was a B at best was awarded As because the teacher said so? That's not fair!

Of course it isn't fair. It isn't fair if you use teacher's grades because some are more generous than others. Some might strive for accuracy, others to inflate their school's overall grades. Some might be looking to give the benefit of the doubt and others are much more religious in their gradings. 

So presumably we should just go with the original idea and adjust everything according to past results? Well that's not fair! Maybe our school improved so much that we would have received higher grades than the system allows for since the affect of the normalising will punish outliers.

OK, let's just use mock results. That's not fair! I wasn't going 100% for my mocks and was planning to work extra hard before my real exams, especially since my memory isn't great and I needed to cram over a much shorter period. I lost that chance and am now being punished for it.

Of course it's not fair. None of it is. It is unfair that people are getting grades lower than they deserve or would have got but it is also unfair that some people who wouldn't have done well are being rewarded by generous teacher grades and are causing the genuine clever clogs to look less differentiated.

The point is, that having a country suffer from a pandemic is not fair. It's not fair that some elderly folks (and some younger) have possibly lost a few years or more due to a disease that we haven't seen before. It's not fair that some businesses have gone bust. That people have lost thousands on holidays and weddings that couldn't take place. Coming to think of it, it is unfair that some people die young, that many do not live out their dreams or are abused or neglected in some way. It is unfair that people in poorer countries can be clever and motivated and still fail to achieve greatness due to lack of opportunity while others in richer countries have incredible amounts of wealth and waste it on cars, phones and TVs that they don't need.

Do you know what, don't dwell on what has taken place. Take what you are handed in life and make the most of it. Many of us missed out on opportunities we thought were a given. Maybe we had someone to blame, maybe we didn't, but we all learned that life does not end when you don't make the course that you intended or meet the grades for the career you were looking at. We moved on, we worked hard and we got to where we are today.

So be content, count your blessings and take the flack on the chin. You'll be much happier for it.

Tuesday, 21 July 2020

Recruitment Advice from an amateur

I am a professional development manager but an amateur recruiter. I have been trying to recruit between 2 and 4 developers for the past 3 months and it is hard. I have learned quite a lot and tried to automate as much as possible but the domain is still under-served by well thought-out companies or products that provide some benefit but still leave many of the problems unsolved. Except by our time!

That is really the ultimate problem. I am paid because I am good at managing software development and with other software related tasks so it is not good use of money for me to spend hours and hours per week processing applications or otherwise finding candidates. It would be great to have an HR administrator to do a lot of this for me but firstly we are too small to need one and also they won't have the specialism that can spot a good senior dev from someone who has been writing software for 15 years and assumes that this means they are senior.

So what is recruitment? We are trying to find somebody who is a good fit for our company and who can add value in a certain area. The amount of value they will add is firstly about their seniority and therefore how much they will cost. A good senior should be able to produce more quality work than the equivalent number of juniors so we should never be employing "the best we can afford" if that person is not at the level we need them to be. For example, if we need a Marketing Manager then we need somebody who can do that job well, not someone who thinks they can do the job but they cost £10K less. There are always exceptions but unless we know somebody, we need someone who can, not someone who might.

£10K might sound like a lot but our sales are something like £2M/year so will £10K break the bank? A marketing or sales manager will increase income by way more than £10K per year so although it will take some time between hire and results, if you have not seen the results within a year, they are the wrong hire (or maybe you are in the wrong business!). Business is like whack-a-mole, as you see weaknesses, you address them. If you employ a marketing manager who doesn't deliver results because there aren't enough sales people to handle the leads, you either need to employ more sales people or you whacked the wrong mole and should have put something else in place first.

So back to my developers. The problem with saying that we need somebody who is a good fit is that it raises some reality. We don't always know what we mean by a good fit, we don't know how to decide whether the cnadidate is a good fit, the recruiter might know some things but likely lacks technical ability to discern good fit from fit and then the candidate themselves won't be the best qualified to say they are a good fit. They might promise the world but there is no cost to them to say they will perform well. If they don't, they can leave and we can get rid of them but then we are 3 months down the line and still a developer down, we might also be a lot of money out of pocket with recruiter fees.

So we need to know what a good fit is, how to judge the fit of a candidate and to make sure they can evaluate themselves. All of this with the least amount of effort for the company who are often processing 10s or 100s of applications, the candidate might only be applying to a handful and a recruiter might be doing it for them so, again, no cost to the candidate can make them lazy.

Your recruitment needs to be a funnel. Early stages need to be quick, easy and ideally automated so that anyone who is way off the mark can be dismissed early-on before they take our time. As the candidate progresses, we need to decide what other boxes they need to tick and measure them as objectively and consistently as possible. The slowest or most labour-intensive tasks , like interviews, should be left until the end. The funnel is narrower here, which means less interviews, and also the people who have lasted this long are worth the more in-depath conversation.

So what does our Developer recruitment funnel look like at SmartSurvey?

Auto Response

All applicants should get an automatic response. Ideally, avoid an email address because once the recruiters get it or guess it, you will be inundated, an online form is better. The automatic response tells them their application has been received and under what conditions they can expect a response and when. For example, you might respond to everyone initially to say, "sorry you don't have the right skills" but eventually you will get so many waste of time applications that it might be easier to say, "If you don't hear from us within 2 weeks, your application has not been successful at this time".

Initial Survey

We ask some very broad questions in the first place via an online survey. We ask things like "Do you meet all of the requirements of the job spec " and "Are you legally allowed to live and work in the UK". These examples might not quite work for you but it is a quick and easy way of not wasting time. You might want to confirm their level of experience, that they have done this specific role before etc. (before you have to read a CV). You should also ask about salary expectation and notice. If 3 months is too long for you to wait, don't find this out after the interview - then everyone's time is wasted. If someone is interested in the role, they shouldn't care about spending 2 minutes answering some simple questions. Create a survey at SmartSurvey if you want!

Self-Assessment

A really important question I ask in the developers survey is for the applicant to measure their skill in certain subjects, say SQL Server and Cloud Systems. This does two things. Firstly, it provides another filter. If someone wants £45K, then that is OK, but if they score themselves 1 or 2 on each skill, they are unlikely to be worth £45K! Secondly, it directs the technical screening tests and ultimately interview questions that I will ask to qualify it. If someone doesn't claim to know much about HTML, I don't have a reason to doubt it and there is no point asking them questions about it. If on the other hand they think they are a 5/5 on SQL Server, you can bet they will get some SQL questions.

Technical Test

So by now, I have done the rough sort and it hasn't taken too much effort. Anyone who remains on the list thinks they meet all the requirements and has scored themselves on each skill so now I send them an online technical test. Again, relatively easy for me to initiate but another drop-off point from the funnel. I tailor these to the skillsets the candidate identifies and we run them from CoderByte which is good value for money and has a lot of pre-defined tests (in some areas more than others). Some people don't bother doing the test - great. That means they already have another role or they are simply not that interested. Others don't do very well. The tests are not mind-numbingly hard but I will read into a candidate who takes a long time to carry out a very simple task - especially if they think they are senior! There is no fixed pass mark but I would expect them to finish all the tests in the hour if they are senior and since the record is about 30 minutes, that isn't a big ask.

If the role is less technical and more subjective, like design, you can perform either a long-answer test or ask for a sample of work or even to carry out a specific task but remember we should not expect too much of their time unless we are to pay them.

Not everyone agrees with technical tests but certainly for my seniors, I want somebody who can think clearly with the clock ticking and someone who can perform simple tasks from memory. So many people say that they "can learn whatever else I don't know" but the truth is an experienced person can do something 10 times quicker than someone who doesn't so the fact that "we have Google" doesn't excuse people with no head knowledge.

CV Check

I don't bother checking CVs usually until this point because this can easily take 5 or 10 minutes per CV, not something you want to do with 50 CVs per week until later in the funnel. At this stage, they have demonstrated some technical ability but the CV is a good source or certain data. Although I occasionally ask the candidate to tidy their CV and make it clear, sometimes I won't bother if it is that poor, I won't assume they just haven't updated it, I will assume either they didn't bother checking it (poor attention to detail) or it is actually as poor as it looks.

So what can we tell from a CV that might not be obvious from elsewhere?
  1. Length of role. What if the candidate has only ever worked somewhere for 1 year at a time? Might be OK for sales but not so much for development. In my case, I want people to have seen the lifecycle of an application, which usually takes much longer. They might also have commitment issues - how much investment will you lose if they only stay 1 year?
  2. Inconsistent types of role. Are they unable to decide what they want to do or have they not done well enough to chose what they want, instead accepting whatever they can get? There might be personal reasons and you might be able to ask them but otherwise it makes me uncomfortable.
  3. When they got their skills. Someone claims 20 years in development but worked in a shop for the past 5 years is not the same as someone whose 20 years is up until now! Likewise for certain technologies, using SQL Server 10 years ago is not recent enough to claim it as a skill.
  4. Communication skills. Does the CV look tatty and lacking care and attention? They might be more technical people but are there typos? Small fonts? Poor typesetting?
  5. Does the CV spell out what they did? So often I reject a CV because it says something like, "the company produces an application that..." rather than "I was the main developer for the database design and produced a bespoke application that..." The CV should sell the person.
Apart from filtering, this is now the final information that I will use to dictate what we need to talk about at the interview. Some of these might be simple "If you can clarify..." e.g. why is there a 2 month gap in your CV, others might require more conversation, "Can you explain how you have management experience when none of your roles mentions this".

Interview

The idea is that you have automated out most of what you wanted to know which should help focus the interview. You already know they meet a technical level (but you might want to discuss their solutions), and that their CV is either OK or there are some points you need to clarify.

Although you can get clarification on the CV earlier, if it might be deal-breaking, otherwise, they make good opening questions for the interview. This is probably the first time you have met and they might be nervous so we try and show our friendliness and make sure we smile, remember during the interview not to make any negative responses that might be read as failures and also remind them that any further technical questions are not about passing and failing as much as the suitability for the role in question.

They should have a good understanding of the company by now and if they care, they will have visited the web site and found out as much as possible. If they haven't, I sometimes give a very subtle hint like, "that's interesting, any reason you haven't?" just to remind them that they need to come across well in the interview.

If possible, it is good to have a score card. You will already have scored them on certain things (pass fail or out of 5) but keeping this on record is good if anyone was ever to make an accusation about prejudice or discrimination on the process but another good thing about automating things is that you won't generally know about race/disability and possibly not gender until the interview so that helps show that it is a neutral process.

We are judging soft skills in the interview like confidence, being engaging, having passion, being able to communicate clearly. Some people have 1 word answers, some talk for ages without answering the question but although we try to help them, we are also imagining what it would be like working with them. There is no right or wrong, it might be better that they have fewer words or better that they are chatty and outgoing.

There shouldn't be too many outstanding issues by the interview because unless you are a company that everyone is desperate to work for, you can only make so many demands on the candidates time. You might have some follow-up technical questions or need to ask questions which are not answered easily on "paper" like "give me an example of where you had to deal with team conflict".

The Technical Test

Most of the time, the interview is enough to give an idea whether an offer should be made and for how much. Normally, I would not offer somebody below their asking salary, I would simply say they are not worth that money in my opinion, if someone would accept £50K, why are they asking for £60K?

In some cases though, the interview still leaves us feeling a bit undecided. Maybe some good things, maybe some bad, maybe the experience is not strong in any one area and Generalists are not usually good for a small company, you want some unique skills to make the team stronger. In this case, we set a paid technical test of about a weeks work, based on a real type of task, to see whether what they produce is very close to perfect or has lots of issues with it. This would only be for seniors who would be expected to produce good qork quickly. We would then say that an offer is dependent on the task being completed to a usable standard but they will be paid if they put the work in. There is some legal work and money required for this but if someone is able to produce well at the end of the day, that is what you want them for.