Monday, 29 October 2018

Certbot renew fails. unauthorized Invalid response from...

So LetsEncrypt is obviously pretty cool for free and mostly automatic installation of TLS certificates but on a site that is still under development (and not touched for over a month), the 90 day expiry is (imho) a bit of a problem and I hadn't installed a cron job to auto renew the cert. I'm also nervous about such a wide-spread automation feature that, if compromised, would immediately affect hundreds of thousands of servers around the world.


Since the cert had expired, FF wouldn't let me onto the site so I logged into the server, made sure everything was up to date (Ubuntu/RaspPi) and tried to run certbot renew and then saw the error "unauthorized", "Invalid response from URL"!

It's always a bit annoying when something that is supposed to just work just doesn't so onto Google for some help.

The "challenges" were not present on the server, but apparently, certbot deletes them after it passes or fails so that was no help but interestingly, the acme-challenge folder was not present. I re-created it to make sure I hadn't accidentally deleted it but still no dice.

Certbot should be able to follow redirects (my site is setup https only) so I added a test file into /acme-challenge and curled it - it worked fine after the 301 but still no joy!

I then modified the config for the site so that http request for .well-known would not be redirected (although Lets Encrypt is supposed to ignore cert errors like expired certs). I tested that it worked with curl but certbot was still failing, always showing the same error.

I then tried with the verbose flag set (-vvvv) and all I could see was this:

Creating root challenges validation dir at /home/pi/www/frontend/web/.well-known/acme-challenge
Creating root challenges validation dir at /home/pi/www/frontend/web/.well-known/acme-challenge
Attempting to save validation to /home/pi/www/frontend/web/.well-known/acme-challenge/E6qaqKvXDbzhpEzuzJfZWfHDjQS69gUVoCURymPU2cY
Attempting to save validation to /home/pi/www/frontend/web/.well-known/acme-challenge/BUxHgfEjQOCve47PhkkjATwyzSxYocl7giIK_XcU-zQ

Where the word "attempting" was not exactly helpful!

That's when I spotted that the site had been moved! certbot was not looking in the correct location, which is possibly because it was the only site on the box to begin with but was then moved into a sub folder. Nginx had been updated so the site worked but certbot had lost its way.

What I did to get round it, was simply to run sudo certbot certonly --standalone, enter the site name again and get it to generate me the new certs (which I now specified with www in the domain name) so they were stored in a separate folder. Then all I had to do was update the nginx config to point to the new certs and...hey presto.. all back up and running!

Thursday, 25 October 2018

Rewriting is hard, even for small applications

I came across an application that would suit my interest in model railways as a computerised control panel written as a Windows Forms application in VB .Net. Think CAD with control inputs that can connect to a hardware interface. Sadly, the original author had died young leaving the code in limbo for over 10 years. I had the idea to take it, open source it and give it some more life but I couldn't really do it straight-away because:

  1. It was written in Vb .Net, which encourages some poor design choices and is a lacking skillset in the community
  2. It was written in .Net framework 1.1, a very outdated framework (2003), with a large set of features that are really useful but didn't exist back then.
  3. There were a number of design choices that showed lack of experience (but the author was not a Software graduate so that is understandable)
That said, it was complete, it worked and it was a laudable effort considering the amount of code involved.

Rewriting is hard, though. In our head, we think it will be great to be able to build from scratch, therefore leaving behind the bad choices or replacing old design ideas with new ones. Of course, it is never that easy, you generally end up meeting difficult choices, that were made in the old code and now lost in time, and also you meet compromises that are hard to make a call on.

The first step in a good rewrite is asking, firstly, "Do I need to rewrite this" and this boils down to the question of, "What is my end game here?". If my end game is simply to use the functionality that already exists then there is no reason to rewrite it. It works, it might be in a language I don't like but why waste days of time fixing it if I don't need to change it?

In my case, since the end game was open source, a future and further development, my main rewrite aim was simply to bring it as up-to-date as possible and replace some of the poorer design choices with neater code, to make it easier for contributers to help out. Some of these poor design choices included:

  1. Using the "Add/Edit Item" form to delete items - this meant all the form references existed for the life of the application.
  2. Use of module variables (VB .Net language for Globals)
  3. Lots of duplication of code e.g. input validation for forms was generally copied and pasted into each form
  4. Very complex over-engineered use of collections to store the individual items, such as its screen position, its type and other properties. This meant operations like Add and Delete have to modify multiple collections to keep data integrity.
  5. Large files including multiple classes instead of separating them out into separate files
  6. Some good use of inheritance, some poor use of it.
Anyway, as I said before, the code was way better than most people new to coding would have achieved and was substantial for a one-person project so I am not complaining. Even if a professional had written it, several of these things would still be present due to changes in frameworks and feature updates.

Porting VB .net directly to C# is not 100% possible because there are certain things that do not exist in C# such as module variables (they would need a static class). Some of the types are in different namespaces in VB since they were added for backwards compatability and were not added to C#. In general, however, it is relatively straight-forward to turn a VB class into a C# class, "New" into a Constructor, module variables into static classes etc and in fact, both types can exist in the same project - the type of project (vcproj or vbproj) is merely about what types get offered by default in the "New" dialog!

So how to start? If it was a 100 line program, it wouldn't matter. If it was 50000 lines, it would have required a lot of functionality described on paper and understanding gained before work started but this was one of those in-between programs.

Method 1: You can simply start a completely new project and pull in functionality as you go. For example, you start with the main form and add handlers for e.g. New, Open, Close etc to build up a basic working app (You can copy and paste form controls from one project to the other!).

Method 2: You delete one vb type and replace it with a C# type either in the same project or in a separate library. Theoretically, you keep the whole thing working at each stage.

Each has its pros and cons. In Method 1, you get a clean foundation and it is perhaps easier to not put in functionality that is not designed well as you go along. At the same time, it will take you a long time to get something that even resembles a fully-featured app and you might not even spot some major bugs until much later on. One example is when I was simplifying the way the Add/Edit dialog used a class variable to keep track of changes made in the dialog. "We don't need that", I thought, replacing it with a direct reference to the object in the main collection, until later, something crashed and I realised that it was done that way so you can cancel out of the dialog without making the changes. Kind of obvious but easily missed when you only have a skeleton of a program.

Method 2 sounds more friendly but remember the 80/20 rule. 80% of it is easy. Replacing a "Signal" in VB with a "Signal" in C# is pretty easy, copy  the properties, reformat for C#, add some braces and semi-colons and it just works. Except of course when it doesn't. When you notice some weird way that a global has been passed around and as soon as you try and move the global to a static class, you suddenly get 500 build errors. Do you spend a load of time and fix them all or might you end up removing a load of them in the wash anyway? Another example was calling the model to validate itself and then the model calling MessageBox.Show() if there is an error. Not something you would normally call from a model. Do we leave it in there for now? Delete it with a TODO? Decide on a refactoring pattern now and start doing that? Fortunately, there is a lot of commonality in the classes which makes it fairly easy to remember how we removed that code before but once you have 5 or more refactorings ongoing, it gets very hard.

So started with Method 2 but decided it did not actually give the benefit of an always-working application, so I went back to Method 1. I have the original code to use as a reference, a new project with just new code in it and a "Scratch" project that I am deleting things from as I go along so I can see my progress.

Two other areas that I will need to battle (but have mostly left for now) is the tight-coupling of a) The hardware interface - which currently uses MERGs RPC and b) The file storage mechanism, which is currently plain text with tabs (what could possibly go wrong!) but which should both be abstracted away. I have made a start on file saving and reading, especially since it would be good to maintain backwards compatibility but I have put this in a separate Serializer class that uses the Visitor pattern to keep it away from the models.

The moral to the story? I don't know. Try a bit, decide whether this needs to be done at a much higher (functional) level or whether you can just hack and whack. Don't be afraid to change tack if it gets too hard and keep regular labelled commits into source control so you can backpedal if you bite off more than you can chew!

Wednesday, 24 October 2018

The Development Industry needs to take CPD and Seniority more seriously

I often liken Software Development to being a Doctor. There are some fairly simple basic principles that everyone can learn easily but they don't make you a doctor. There are more advanced fringe cases, interesting symptoms and even conditions that have rarely if ever even been seen before. A Doctor has to deal with these but that still would not qualify somebody to be a Consultant, something which takes a number of years, specific training, often across more than one hospital, examinations and an ability to answer the questions that noone else can answer. It is obvious that this is important because we are talking about people's lives. But just because we aren't, is Software, and other Industries within their rights to have a completely casual approach to professional development?

We lack anything remotely similar to this in the Computer Science world. You might have a degree, you might not and then you might have Industry certifications or informal qualifications from online training providers but it seems that it is far too easy for someone to call themselves a Senior Engineer - in fact, it costs nothing to give somebody the title but it undermines the people who really are gifted and have worked hard to achieve their status for someone who has worked at a single company for 5 years to be able to be called a Senior Developer.

Sure, there are plenty of "unqualified" Devs who are good at their job but there are also very many "Senior Devs" who lack a very broad experience, who only understand a very specific subset of computer science and who might also lack the people skills that need to be the requirement of somebody who is hopefully passing on their skills to younger/newer Developers and leading the charge on technology upgrades/new ways of working/new languages etc.

When I used to interview for Senior Devs, I gave them a grilling. I asked them what they thought a Senior Developer was (the answers were generally ill-thought out and wishy washy). I then told them what I thought a Senior Developer should mean - things I mentioned above - and asked them for examples of when they had done that. When had they gone above and beyond? Why was their code any better than what a normal dev could produce? Did they understand the latest trends in Software Development so they know whether e.g. Microservices is useful for certain scenarios or not.

The reality is, we have something that people think is easy and which there doesn't appear to exist any industry-led formal training path like you do for many other professions including other things that you might call "easy" for some gifted people like Accounting or Architecture. I think it is time that the industry bodies like the Chartered Institute for IT and others, start to formalise what it should mean to be a Developer. Sure, you could be an Associate Developer if you lack formal training, but you should not be allowed to proceed without doing the additional tasks/projects/exams that prove you have reach a standard.

I know this is contraversial, but what do others think?

Friday, 19 October 2018

Making Cyber Security more practical for small businesses

I went to an interesting talk the other day about the role of the CISO. One particular speaker, Nic Miller from Aedile Consulting spoke about the burden of large accreditations like ISO 27001 on small businesses and the reality is they are not designed to provide specific advice for practical security controls and so an organisation can end up with a badge and still be massively exposed to attack.

I wrote a blog post earlier about this: Should I use ISO 27001 for a small business?

The problem is that most people in the Information Assurance world are, well, information people. Their solution to everything is information, process, paperwork and badges that might win you a contract but mean very little in practice.

Most small companies are not interested in needless paperwork but they are interested in staying safe and understanding the practical steps you can take to reduce your attack surface and reduce the risk of a hack or other malware attack. The National Cyber Security Centre has some great resources, and these also overlap with existing guidance, which are described on the NCSC but to find the simple steps can still be a little daunting. Some guidance is very specific ("Advice for British Airways Customers") and this should not be in the same pot of documentation as "Supply chain security collection" and "Design Principles: Designing to avoid disruption", which are all concerned with very different environments and for different end users.

I found the Australian Cyber Security Centre (part of the Australian Signals Directorate) Essential 8 much more practical and useful for small companies. Instead of trying to get really abstract about risk management and reducing risk with controls etc. something that most small companies have no experience of, they instead specify that things like disabling Flash and Java in browsers and disabling automatic macro functionality in Office software can buy you a lot for the effort required. In fact, you don't even need to understand why.

I then decided to write a quick and easy (and free) cyber security audit tool which is based on the ASD 8 as well as couple of other areas and then gives you a rough score as to how well you are doing. As the site says, 100% doesn't mean you are immune from attack and < 50% doesn't mean you are going to be attacked soon but it does give some ballpark idea.

If anyone wants to take it for a spin and give it some feedback, please do!

If anyone wants help to perform an audit, please contact Cotswold IT Consulting on 01242 500028 or

Tuesday, 9 October 2018

SEO for the new company!

I hadn't really thought much about getting business for my new company, I just assumed networking would make it all happen but my website is still very background on Google, even if you search for it by name!

Anyway, to hopefully help with the SEO score, here is a link to Cotswold IT Consulting, a local (Cheltenham, UK) and independent company helping small and medium businesses with marketing, strategy and software issues.

I will be blogging on that site too although they are likely to be shorter and less detailed than this blog.

Modifying SASS options when building from NetBeans

Sometimes I am amazed that documentation for something as ubiquitous as SASS is hard to find. There is plenty of stuff about how to write sass itself, but actually using the scripts, specifically the command-line and it basically mentions that you can use the normal sass options, doesn't explain how, and none of the examples seem to show much more than just input file:output file!

Specifically, I wanted to alter the compression being used and it was not clear how to pass this to the Windows batch file that Dart uses to call sass. Is it --option:something --option=something --option something?

I tried a few variations and got errors like "Positional and ":" arguments may not both be used" and "Only two positional args may be passed".

Answer: You need to use

sass.bat --flag --otherflag inputfile.scss output.css --parameter value --otherparameter value

I eventually worked this out using a trick that I had read somewhere about how to get Netbeans to call the sass file correctly. The problem is that Netbeans assumes the arguments to pass to sass and it includes --debug-info and --cache-location which do not match Dart sass. Whether they match another flavour or are just out of date, I don't know: the options should have underscores (and neither are actually needed anyway).

The trick was to edit the sass.bat file provided in the Dart sass windows command line utility and do some batch magic to alter the command line before sass sees it.

Netbeans calls it like this (newlines added for clarity):

"--cache-location" "C:\Blah\Cache\8.2\sass-compiler" 

Inside the batch file, lines 3, 4 and 5 remove the unecessary parts from the arguments by using a batch replacement function:


i.e. take arguments and replace X with Y. In the first instance, X is "--cache-location " and Y is nothing (the part after the equals and before the % closing tag):

set SCRIPTPATH=%~dp0
set arguments=%*
set arguments=%arguments:--cache-location =%
set arguments=%arguments:C:\Blah\Cache\8.2\sass-compiler =%
set arguments=%arguments:--debug-info =%

The next lines add the style option after the input and output files and then runs sass using dart and passing the full arguments to it:

set arguments=%arguments% --style compressed
 "%SCRIPTPATH%\src\dart.exe" --no-preview-dart-2 "-Dversion=1.13.4" "%SCRIPTPATH%\src\sass.dart.snapshot" %arguments%

Friday, 21 September 2018

Using variables in Powershell in Bamboo deployment project

Another case of poor documentation = time wasted, this time, courtesey of Atlassian's Bamboo.

I need to make a deployment project that runs some powershell scripts - OK so far. They have a script task, which can run powershell - still OK.

But..In order to make the environments different, I wanted to inject some variables. They have a variables button in Edit Environment, which I clicked and there was 1 annoying thing and 1 good thing. The good thing was that they said on Windows, you use %NAME% in scripts and the annoying thing was that it was unclear why the Scripts examples were capitalized and the variable names were not. I could work that out but really, they should have been consistent.

Anyway, I could NOT get these to work. I would try things like  

az group create %bamboo_DEPLOYMENT_NAME%

(using capitals just in case) and no dice. All kinds of random errors and weirdness, I eventually worked out that it must not be substituting them properly, but without any useful echoing from Bamboo, I tried all kinds of combinations. Fortunately, the log message proved that the environment variables were there and were correctly named.

I eventually realised that the help in the variables page was incomplete, their online docs were poor and even some forum questions were not answered  and was really only referring to command line usage, not Powershell usage. SO instead of az group create %bamboo_DEPLOYMENT_NAME%

it should be:

az group create $Env:bamboo_DEPLOYMENT_NAME

And then it all worked!