Monday, 26 September 2016

Help, my servo isn't working!

I bought some remote-control servos from China recently - got stitched up a bit with VAT getting added later and DHL charging me £11 for the processing, adding a total of 50% to my order so they ended up working out about £2.50 each. I think you can probably get them cheaper if you buy them in smaller numbers so they fly in under the radar as it were (legally!).

Anyway, I knew how they were supposed to work, connected one to my PIC prototype board didn't work! Nothing.

While Googling the potential problem, I have learned the following, some of which is generally good advice for most things!

1) Start with the simplest PIC program you can and if possible, add some checks to make sure it is working as expected. For instance, I made a program that set two servo positions and changed every 2 seconds. I then connected the timer to another LED so I could make sure that the program was running at the correct rate and probably working.
2) Remember that the servo will draw a significant amount of power when it is moving (as little as 45mA but potentially much more). Most controller outputs cannot source that level of current so don't do what I did by connecting the power wire to a switched output of the PIC (I did it so the plug would fit somewhere useful on my board where I couldn't get 5V GND and a digital IO next to each other!)
3) If it moves but doesn't go where you want it, it is possible the pulse width to position is not what you think. See if you can find the spec for your servo - it is not always that easy. You could also experiment with different values.
4) Check (carefully) that the arm is not stuck at one of the extremes. Although the gearing will make the arm hard to move, even under no load, if you are careful, you should be able to move it away from either end.
5) It obviously could be broken so having a second spare one is obviously helpful.
6) Be careful with oscillator settings. If you have accidentally got the timings wrong, the pulses could be 1000s of times too long or short. Again, it is worth connecting a spare LED to an output to ensure the cycle times are as expected.

Monday, 19 September 2016

Git stops working with VSTS - Authentication failed

Annoying bug with no obvious cause!

I have been using Git (via Tortoise Git) to push code to VSTS. I've been using it all day and then suddenly, "Git did not exit cleanly- Authentication failed".

After re-entering my credentials about 5 tims, each time taking even more care that I was typing them in correctly, it wasn't making any difference.

I fixed it in the end by changing my alternate credentials in VSTS to the same password as before but it looks like the act of saving it had refreshed the privileges and then it worked.

You can get to that by clicking your name in the top-right of VSTS and choosing Security->Alternate Credentials.

Migrating Azure Cloud Services to App Services is not a 5 minute job!

So we use Cloud Services, they were the most PaaS you could get 4 years ago when PixelPin started but they are a bit old hat. They are thinly veiled VMs, which are managed by Microsoft but they don't permit an auto scale to happen very quickly and because they don't make good use of shared hardwre, they are not the most cost effective.

Enter App Services, which supports API, mobile apps and web apps but for less money. They also, more importantly, provide a very quick therefore effective way to scale from 2 to 20 or 20 instances in only a few seconds, using hot standby services.

Perfect, so why not simply copy and paste your app from Cloud Services to App Services. Here's why:

  1. You have to learn the concepts and setup of App Services on Azure. You need to learn about plans - shared and dedicated and how you can then have multiple apps hosted on the same plan (which is cool, because you aren't paying more to host more apps, just adding more load to what you are already paying for).
  2. You can theoretically deploy directly from Visual Studio using "Publish" but this always crashes on my machine, which leaves either publish to Git and get App Services to pull in the changes (works OK but you have to use Git, which I hate, as well as spend time working out the deployment workflow). You can also use VSTS, but only if you use Git or otherwise you have to push the deployment from VSTS onto App Services, again, more learning required.
  3. You cannot run your startup script like you used to before because you are basically sharing an environment docker style. For instance, we setup cipher suites by running our startup script on Cloud Services but we are paying for the whole system so that's OK. You can't do that on App Services, you can only control your web.config settings (and I can't guarantee that some of these might not work as expected/be overridden/locked by the parent config). You must CAREFULLY consider what your startup script does and whether it can be done another way or you can accept the risk of not doing it.
  4. Things like certificates and app settings are done differently. App settings work in a similar way, except you no longer access them with the Azure Runtime functions but instead, simply access them as App Settings. It also means if you want to deploy them, they are specified as App Settings in config instead of settings in your cscfg file. There might be security considerations because of this and there are various ways to workaround that.
  5. Certificates are more a pain. You no longer specify them in your cscfg files and you don't have direct access in the same way. What you have to do is add an App Setting to expose the certificate to your app, which is then done in the CurrentUser store (since you are sharing hardware!). An article describes it here: but it looks like you can either expose 1 or all certificates using the App Setting.
  6. All of the stuff you used to set in your cloud project (VM sizes, numbers etc.) is done in the portal or using an Azure Resource Manager (ARM) Template. This decouples the development process from needing Visual Studio and a specific project type etc. and opens up the use of things like Kudu or docker to deploy code directly to Azure.
  7. If you used cloud services for session then you will also need to change that to use Redis, which to be honest is probably cheaper and easier than the Azure Cloud Service provider anyway.
  8. No doubt I will find some more things that are different but be warned! Spend some time on it, attempt to move a really cut down version of the app so you can see the sorts of issues you are having and then spend some time planning how you will need to change things to move forwards.
That said, I am excited about moving to App Services and think we will eventually get a much more responsive and cheaper system as a result.

Friday, 16 September 2016

Git, maybe one day you will be my friend

I have had a terrible week at work. I have basically done nothing for 4 days due to a whole raft of issues related to mostly Microsoft Technologies. Partly due to bugs and partly due to lack of understanding (or both). What people don't always realise is that bugs are annoying to experts but fatal to newbies because when you don't know what something is supposed to do, you cannot spot the bug.

My problems this week have included:

  1. Attempting to get a Load Balancer and Scale Set configured on Azure except the interface was allowing invalid configurations and there is no obvious feedback on healthcheck probes. Needed MS support.
  2. Trying to deploy directly from Visual Studio to Azure App Services except VS publish crashes due to a weird dll reference problem (its looking for an old method in a new DLL). If MS hadn't CHANGED the interface to the dll, there wouldn't be a problem. Spent a DAY removing old visual studio installs and installing VS 2015 again and still no dice. Broken out of the box.
  3. Trying to deploy to Azure instead via VSTS. This doesn't support TFVC, although the docs don't say this, you just wonder why your project is not listed in the portal. Roll on a few days and have finally worked out that you have to use Git on VSTS.
  4. Create a Git repo, check in all my code and get the system to auto deploy to Azure. Deployment error - nice and esoteric, basically means it can't access my private repos but this is not obvious. I can't even remember how I realised this.
  5. Tried to work out how to get Nuget working properly in my project. The project seemed to be OK according to the 10 different versions of documentation, decided I needed feed credentials that were not encrypted to use on App Services but the VSTS portal has changed and the Get Credentials button isn't there any more, despite the documentation being based on the old dialog which worked. So no credentials and no articles describing what must be App Services 101 - using custom NuGet packages. Decided I would have to copy the NuGet dlls into the project manually and reference them locally to the project for now.
  6. Finally get my app up on App Services and a big fat error. No site, just a random 500 error. Tried to remotely debug as per the helpful guide. Doesn't work, can't connect.
  7. DNS outage on Azure for about 4 hours affecting our production system!
  8. Decide the only way this is going to work is to start from a vanilla app in Visual Studio and add the stuff in bit-by-bit until something breaks. Of course, I can't use the Publish mechanism, that's still broken even on a new project.
  9. The vanilla app loads OK, I set up SSL and a public DNS so this should be good right? Add some files, try and do as few as possible but of course the references quickly spider and I end up with half the app in place. Deploy it - compile error. This time, the generously supplied gitignore file for Visual Studio excludes pfx files, which seems reasonable except my system uses them so I had to include them. The app still displays (it doesn't do anything but hey).
  10. Copy the rest of the app over and commit, error 500, 502, all kinds of stuff like that. Custom errors off (doesn't work) turn on all the supposedly wonderful diagnostics, none of them say anything useful. Try and upload debug version etc. eventually find that there is a duplicate web config entry. Why does this work locally? So it will work now? Nope, just more and more server errors and no way to debug them. Remote debugger still doesn't work and all attempts to enable logging and everything else is futile. I find a nice article about debugging errors using App Services SCM endpoint but these also don't match the reality. The "capture now" button simply doesn't exist.
  11. I decide there is only one thing to do. Revert back to the last good deployment that was still looking like the default MVC app (without CSS!) and add in items in much smaller stages, if possible. It's clearly a configuration type issue to make it 500 instead of getting an Application error so this should be doable.
And then there was GIT. I'd forgotten how much I hated it and locking horns at 8pm on Friday evening is not comforting to the soul! Subversion and even TFS are easy enough to revert but, remember, I HAD to use Git to deploy to Azure App Services so what do I do?

I'll show the log first (Tortoise Git) to see where to revert to. Choose the right point and "Reset master to this". OK, soft, medium or hard? No idea but let's copy the entire folder first - I have been burned before. OK, medium looks good.

Hmm, it says it's done something but all my files are still here - what a pile of crap. Why would I ever want to reset something and leave the files here? OK, let's try again. Reset hard right? OK, some things seem to have been reverted but I still have a load of files present that were not present when this was committed!! Why are they here? Copy the entire directory again, I know I am going to be loud swearing soon!

Try and open the project into Visual Studio to see what it is looking like and it won't open - some IIS config problem. Why? It was working fine when I committed this code. Fixed that and it looks OK but still has all these uncontrolled files. Why are they there? Git - I hate you.

Manually delete the files I don't need, commit the ones that should belong to this build. It looks like a pretty early cut but I can handle that, I have all the files elsewhere (please God, let me have the files elsewhere and Git not trying to also reset the other files since I copied a git folder!).

Now. Git push. FAIL. Oh for f**k sake. Pushed files were rejected because your branch is based on an earlier part of the remote branch. Oh really? Well I did f**king reset it so perhaps you could simply ask me, like a sane product, do you really want to overwrite everything with this. Nope. Talks about another load of crappy commands I might have to use to get this to work.

Git I hate you. After looking at the (futile) attempts at newbie Git tutorials, which still leave me utterly confused, even the ones with nice pictures, do you know what I am going to do (and this is VERY damming of the cult of Git and its contempt of people like me who don't get it): I am going to start again. That's right. I am going to delete the entire Git repo, create a new one and copy files in manually. Won't that take a while? I've already lost about 4 hours and have moved on precisely zero knowledge so why not burn 4 hours doing something manually that VCSs are supposed to automate.

Git, maybe one day you will be my friend - but I doubt it.

Source Control from Visual Studio is still terrible!

Sourcesafe used to just about work as long as you used it in the Sourcesafe interface. Trying to use Sourcesafe from inside Visual Studio was fraught and it still is. TFS, VSTS, TFVC, whatever you want to call them are basically still Sourcesafe, you can see by the files that are dumped in controlled directories and the weird entries that are added to project and solution files.

As with many decisions in life, to know we are doing it properly, we need to ask the most pure questions: What is source control supposed to do?

Answer: It is supposed to provide protection for code

It also provides a record of changes but fundamentally it is about protecting code from accidental changes - which can be reversed if needed and also from intentional changes that can be traced easily by check in comments. If your Source Control does not ALWAYS protect code, it is not good.

Let's look back to good old Microsoft who have somehow managed to mangle Sourcesafe into several different forms and somehow keep its geriatric body alive. How does Source Control work in Visual Studio?

1) Connecting a project to source control. This is a complete ball ache in Visual Studio. In VS2015, there is a Team Explorer, a Source Control Explorer, Context Menus, Menus under File->Source Control and workspace mappings. Do I create the project first in VSTS and then connect or can I simply "create new project" from my uncontrolled source? This is so hard to follow that even though I have done it about 20 times on different projects, I still can't do it right first time. Also, why is it that sometimes you "Add solution to source control" and not all of the projects show that little green plus symbol? Answer - no idea.
2) Disconnecting a project from source control. The short answer is that I think this is largely impossible UNLESS you disconnect, move the project to another folder and rename the destination project in Source Control. You can select File->Source Control->Advanced! and then choose to unbind the project. Why this is advanced, I do not know. Unbind it and guess what? There are still source control binding files on the disk, it is just pretending to be unbound. Try and bind it to another project in Source Control? Good luck, you will get that infamous message, "The item blah is already under source control at the selected location". What? I just disconnected it! No you didn't, we were just pretending.
3) Deleting a project on the server. Be VERY careful when you do this. Do you know what happened last time I did that? Visual Studio decided that since I deleted the server project, I obviously didn't want any of my local files. Yes, it deleted everything on my local disk, which is the OPPOSITE of what anyone ever wants. At minimum, you would ask, by default it would keep whatever was there. Another gotcha is basically Visual Studio being completely unable to handle the deleted project when there are local changes - how does it know there are local changes if the project was deleted in SCC? It basically forces you to undo the pending changes, even though they are irrelevant and if you don't? You get errors every time you try and edit workspace mappings. What happens if you undo your changes? You guessed it, you lose all the changes which you cannot undo because the project is now deleted!
4) Moving a project from one SCC binding to another? Forget it. Best way is to great a new project and manually copy stuff over. Honestly, that is Source Control 101 but for some reason, MS still haven't got the basics right.

Other options? You can use the command line or some tool outside of VS to manage source control, it's annoying but it seems to work much better than VS does and makes more sense because everything is in the same place.

The truth is, if you do this all the time, you will HOPEFULLY not delete anything but for the rest of us, be prepared to lose your job...oh yeah and ALWAYS make backup copies of folders before doing anything other than checking-in changes to files.

Thursday, 15 September 2016

Load Testing is not as easy as it sounds

So we have an authentication-as-a-service product called PixelPin. We want to load test it so we can a) tell our customers that it can definitely handle their X,000 customers logging in at the same time, and also to benchmark performance to find any bottlenecks and let us see the effect of architecture changes.

So far, so good.

When it comes to planning the load test, it is actually really hard. Setting up the tests is easy enough using Visual Studio Team Services and Visual Studio Enterprise Edition but actually working out what to test and how is hard.

Why? Because there are so many variables. Various cloud services, bandwidth, CPU, memory and even dependent services that are used for testing like a dummy OAuth login site. If the test site cannot handle enough user load then the load test won't be stressing the tested system anywhere near enough.

In fact, our first step is to try and work out which variables can be ignored. If a certain system is hardly struggling to manage the load, let's not spend any time with that and instead work on the really slow parts.

Another question that can be hard is how to simulate a realistic load test. There's no point saying we can handle 5000 requests per second hitting the home page when that is not what people will be doing. We need to think about the mix of journeys, the effect that caching and browser mix might have on things and a whole host of other reality checks.

You also get another problem if you are not careful about setting up your test environment. We created 5000 test users and put them into a spreadsheet that the tests use to fire at the test site but if you aren't careful (and we weren't), then the same user account is hit twice in VERY quick succession and causes an error that would never happen in real life (unless a real user and an attacker happened to do that). You start having to really understand what the load test is doing under the covers, how is it selecting the users from the spreadsheet? Can I have 2 test agents running at the same time or will that cause parallel errors that shouldn't happen?

It is going to take weeks to sort through the random errors (errors which are usually caused by an underlying problem, the error itself doesn't show that) and then to decide how to measure what we are trying to measure in a suitable way.

Load Testing is not as easy as it sounds!

Wednesday, 14 September 2016

Making Azure Virtual Machine Scale Sets work!


So we are trying to create a test environment for load testing our application. Unfortunately, we use older resources (Cloud Services and VMs) and these were all originally deployed manually so I have been trying to create a close to exact replica of our production system bit-by-bit.

Resource Manager is the new thing in Azure and it is basically a concept and set of APIs that allow you to group resources together and to script the deployment, which will be very useful going forwards for our multiple deployments.

One of the things I had to replicate was a VM set, which we were using to run a PHP app. The reason we used a VM was that the cloud service tools for PHP had various shortcomings and were not suitable so I went old-school. Although there are now App Services Web Apps for PHP, we want to load test the original architecture so that we can then benchmark that against changes, including moving to App Services.

Virtual Machine Scale Sets

To use the new portal, you have to use Resource Manager and resources that are compatible and the closest to classic VMs are called Virtual machine Scale Sets which are basically the same thing but with more configuration.

I assumed this would be a relatively quick and easy setup like it was on classic but the problem with all configuration-heavy features is that if it doesn't work - which it didn't - it is hard to know which bit is wrong.

I got it to work eventually so I thought I would describe the process.

Create the Scale Set

This is like normal. Go to your subscription in the new portal, click Add and choose Virtual Machine Scale Set. It will list several, I am using the Microsoft one. Select it and press Create.

Fill in the details. The password is what RDP is set to use. It is useful to choose a new Resource Group since there will be about 10 resources created so it is easier to group them into 1.

The next page gives you options for the IP address (you can use the default), a domain name, which must be globally unique and options like whether you want to auto-scale. Auto-scale is only useful if you are using an image that will automatically install and run when the instances are increased. In my case, I am installing the code manually so auto-scale isn't much use!

I think that the only way to correctly template the installation is to use a Resource Manager template which is not supported in the portal - so the portal just gives you the vanilla setup.

Setup your instances

By default, the load balancer is setup with 2 inbound NAT rules for port forwarding RDP to your specific instances. You should probably change the port numbers because they are always 50000 series. With these port numbers you can simply RDP to your public IP address (shown in the overview for the load balancer and several other places) and a specific port number. Obviously for each instance, they should be setup the same since they are to balance the load.

This can obviously take time, especially with things like PHP to install, but as with all installations, test it as you go along to narrow down what might be wrong. Once you've finished, access the web apps locally to ensure they work and another cool trick is to point once instance at the other in the hosts file and attempt to access one instance from the other just to make sure firewall ports are open. This takes place on the virtual network so won't give any public access yet.

Setup a probe

In order to load balance instances, the load balancer will need a way to probe the health of your instance. Your choices are HTTP or TCP and this probe(s) is setup in the Probes menu for the load balancer.

When you are first testing it, it might be tempting to use HTTP simply to your web site but be warned that there are some cases where this won't work and you will not know anything except it doesn't work! I'm not sure that https is supported (although you can type it into the Path box) and it doesn't appear to be happy with host headers.

Unfortunately, there is no obvious to test the probes in real-time, you can only enable diagnostics and hope for some useful information but it is very user unfriendly.

You can always start by trying an HTTP probe, if it doesn't work try the following trick.

Download psping from sys internals onto each instance and run it as administrator from the command line with the arguments psping -4 -f -s and it will run a TCP server that simply pings a response to a request on that port. You can then setup a TCP probe to point at the port you specified which worked in my case showing me that the HTTP part was the problem.

If you must use HTTP which is the most useful in most cases, you might need to create a default web site with no host header and put your probe endpoint in there. That worked for me (I used iisstart.htm, which was already in wwwroot). Note that ideally the probe would do more than just see a web page load but would also carry out other basic tests to make sure the instance is healthy - not too heavy though it will be called every few seconds.

Setup a Load Balancing Rule

The Load Balancing Rule is for endpoints that are load balanced across the instances (e.g. HTTP and HTTPS). To create the rule, you must have a probe but otherwise it is straight-forward. A name, a front and back port (these are likely to be known ports for public sites like 80 and 443). By default, there will only be 1 pool and the probe you just created to choose from and unless you need sticky sessions (best to design your system NOT to need them) disable session persistence which will round-robin requests to each server.

Other Things to Check

As with all of these things, don't forget things like public DNS settings to point to the correct location, SSL certificates and consideration of how you will update/fix/maintain your instances. It is possible for these instances to die so ideally you should have an automated deployment although for now, I am not going to bother spending time on that!