Thursday, 20 October 2016

Azure App Services not ready for the big time yet?

I really like the idea of App Services for web apps. It basically looks like cloud services but with shared tenancy so lower cost per "server" than a cloud service.

The other really big winner for me is the much quicker scalability when scaling out. Cloud services understandably take a long time to scale out and cannot handle short traffic bursts for that reason.

App Services on the other hand can scale out in seconds. I think they do this by sharing the code and just increasing the number of virtual servers.

It looks great on paper and deployment is as easy as Right-Click->Publish from visual studio, using web deploy and therefore taking seconds. Cloud service deployment is still cryingly slow!

So that's the good news, what's the bad news?

It doesn't seem to work properly!

We set up an app service plan (Standard) and deployed a web application that calls a web service, which accesses a database. We also use Azure Redis Cache.

We fired a load test at the system to see what sort of load we could handle and setup auto-scale for a CPU of 80% to allow form 2 (minimum) to 6 (maximum) instances.

So what happens when we run the test? Loads of Redis timeout errors. Hmm, the requests per second are only about 100/150 and the response times are less than 1 second so why is the redis 1.5second timeout occurring?

We had a few pains, where re-deployments didn't seem to fix the problem and even though we could see the remote files easily using the Server Explorer in Visual Studio, the presence of the machine key element didn't seem to remove the errors related to "could not decrypt anti-forgery token". Hmmm.

To be fair, it did expose a problem related to the default number of worker threads. In .Net, this is set to the number of cores (1 in our case) and any new threads can take a minimum of 500mS time to create, easily tipping the request over the 1.5 seconds. So I increased the number of threads available but still we were getting timeouts even when the number of threads was way below the threshold for delays.

Maybe we were pegging CPU or memory causing the delays? No idea. The shared tenancy of App Service hosting does not currently allow the real time metrics to display CPU or memory for the individual virtual server so that's no help. It also wasn't scaling up so I guessed it wasn't a problem with resources and why should it be at such a low user load?

I finally got fed up and created a cloud service project, set the instances to the same size as the app service instances, using the SAME redis connection and ran the same test. Only a single error and this pointed to an issue with a non-thread safe method we were using (which is now fixed). No redis timeouts and a CPU and memory barely breaking a sweat.

We ended up in a weird situation where we couldn't seem to get much more than about 100 requests per second from our AWS load test but that is another problem.

Why does a cloud service running the same project work where the same thing on App Services doesn't? I couldn't quite work out what was wrong. A problem with Kudu deployment? Some kind of low-level bug in App Services? A setting in my project that was not playing nice with App Services?

No idea but there is NO way I am going to use App Services in production for another few months until I can properly understand what is wrong with it.

These PaaS tools are great but when they go wrong, there is almost no way to debug them usefully. The problem might have been me (most likely!) but how can I work that out?

Cannot answer incoming calls on Android

Weird problem: Phone rings and/or vibrates but the screen doesn't change. All you can see is the home screen. How can you answer the call? You can't.

Well you can but you need to know what you did!

In my case, I am on the Three mobile network, which is, basically, terrible around where I live and work. They have an app that uses the Internet for calls and texts if you are not in mobile range. If you are, all kinds of pain occurs because texts sometimes go to the app and not the phone etc.

Anyway, like many annoying apps, decide that they can fill up your notification area with a permanent message saying that Wi-Fi calling is ready. Amazing!

Anyway, I had disabled notifications for the Three app to get rid of that annoying message but the thing I hadn't realised is that it will also disable the incoming call notification. For some reason, the phone still rings and vibrates but you can't do anything.

I had to re-enable notifications and then, of course, I can answer the phone! I also have to live with the annoying permanent notification.

The same has also occurred with Google Dialer according to forums but I don't have myself.

Friday, 14 October 2016

Azure App Services (Websites) - Cannot find the X.509 certificate using the following search criteria

Deployed a brand new WCF .Net web service to Azure App Services and when trying to load up the svc file in the browser, got the message above.

Here's what you need to know:

  1. You need to upload the required certificates to the Azure portal
  2. You need to make sure that you are referencing the certs from the CurrentUser store, NOT the LocalMachine store. App Services uses shared hardware so you can only access the CurrentUser location.
  3. You need to add an App Setting to tell the service which certs to make available to the web service. You only need to do this for certs referenced in the app, not for certs you are only using for an SSL endpoint. The key is WEBSITE_LOAD_CERTIFICATES and the value is 1 or more comma-delimited thumbprints for the certs you want to load.
  4. You CANNOT add this only in the web config file, despite Azure merging portal and web config values, it MUST be added in the portal to the Application Settings tab.

Thursday, 13 October 2016

Encrypting/Decrypting Web.config sections with PKCS12

What is it?

I'm not sure why I haven't blogged this before, since it can be quite confusing but this is a cool way of protecting web.config secrets from being seen casually by Developers.

The basic idea is that you use an SSL certificate as the key and then the sections in web.config are not readable. They will decrypt automatically, however, when used by IIS if the certificate is installed locally.

It is worth mentioning that if a developer can run the code locally, they will still be able to find out the secrets, it is more of a protection from people who can see code in a repository but not run it.

You can use any SSL/PKCS12 certificate for encryption/decryption but I recommend using a self-signed certificate that should be in-date (since some services e.g. Azure App Services will not allow upload of expired certificates). If you use a self-signed certificate, you get to control its lifetime more easily if you are worried about it being hacked. If you share one of your public certs, you will probably have more work to do when the cert expires.

You should be able to encrypt all web config sections but although this works in theory, certain sections cannot be encrypted due to when they are accessed by the system. I can't find a useful resource so you will have to try it and see. It will be obvious if a section fails to decrypt.

So how do you do it?

First, you should reference or install Pkcs12ProtectedConfigurationProvider, which does exist as a NuGet package.

You then use aspnet_regiis.exe, which lives in the .Net framework directory(ies) in C:\Windows\Microsoft.Net.

So how does it know which key to use to encrypt? You need to create a section in your web config that references the thumbprint of the certificate you want to use. It looks like this:

        <add name="CustomProvider" thumbprint="d776576a90b5c345f8a9d94e732c86c1076eff78" type="Pkcs12ProtectedConfigurationProvider.Pkcs12ProtectedConfigurationProvider, PKCS12ProtectedConfigurationProvider, Version=, Culture=neutral, PublicKeyToken=34da007ac91f901d">

aspnet_regiis performs various other functions as well but the two options we are interested in are:

-pe to encrypt a section
-pd to decrypt it.

If you use the f option as well, you can specify a physical location for a web.config, which I have found is easier than using the VirtualPath (and in some cases you won't have virtual path).

The -f should point to the directory containing the web.config, not the web.config file itself. You will end up with a command like this:

aspnet_regiis.exe -pef "connectionStrings" "C:\Users\luke\Documents\Visual Studio 2015\Projects\SolutionFolder" -prov "CustomProvider"

Note that prov specifies the name of the key entry that you want to use to encrypt the section.

If you are encrypting a custom section, you might have some fun and games trying to get aspnet_regiis to find and load the dll that defines the config section. You can use gacutil to add these dlls to global assembly cache and then use the correct version and public key (which you can find in a tool like ILSpy) so that aspnet_regiis knows to look for the dll in the gac.

If your section is not at the top level, you need to specify it like system.web\mysection

Sometimes, it pretends to work but doesn't. Run it again, it sometimes catches up! Google any errors, they are fairly easy to work out.

You will then end up with a section of the same name but with a reference to the customProvider you are using and an XML encrypted packet.

Remember to upload the certificate you are using to any systems that will need to read the section otherwise you will quickly get an error!

And as always, start with a really simple example and make sure it works before going crazy and trying to encrypt everything.

Wednesday, 28 September 2016

.Net MVC5 - RedirectToAction not Redirecting in the browser

I had a funny one on a test site I was using. Very simple MVC app, a page that requires a login and then a logout link that does the following:

public ActionResult Logout()
    return RedirectToAction("Index");

However, after calling this code, a 302 is returned to the browser for the redirect but it doesn't include the Location HTTP header pointing to the new location, so the browser does not redirect. Instead, the user has the click the link on the displayed page.

I thought this was very strange and dug into the code that produces the Redirect inside System.Web.HttpResponse and found that the Location was always added using the same url that was correctly inserted into the response body. Clearly something was removing the header after it was added and before the redirect took place.

After I commented out SignOut(), the redirect worked correctly so somehow, for some reason, IAuthenticationManager.SignOut() is removing the Location header, but AFTER it has been added in the next line.

I haven't found a reason on stack overflow but I might dig a bit deeper into SignOut to find out what is happening.

Monday, 26 September 2016

Help, my servo isn't working!

I bought some remote-control servos from China recently - got stitched up a bit with VAT getting added later and DHL charging me £11 for the processing, adding a total of 50% to my order so they ended up working out about £2.50 each. I think you can probably get them cheaper if you buy them in smaller numbers so they fly in under the radar as it were (legally!).

Anyway, I knew how they were supposed to work, connected one to my PIC prototype board didn't work! Nothing.

While Googling the potential problem, I have learned the following, some of which is generally good advice for most things!

1) Start with the simplest PIC program you can and if possible, add some checks to make sure it is working as expected. For instance, I made a program that set two servo positions and changed every 2 seconds. I then connected the timer to another LED so I could make sure that the program was running at the correct rate and probably working.
2) Remember that the servo will draw a significant amount of power when it is moving (as little as 45mA but potentially much more). Most controller outputs cannot source that level of current so don't do what I did by connecting the power wire to a switched output of the PIC (I did it so the plug would fit somewhere useful on my board where I couldn't get 5V GND and a digital IO next to each other!)
3) If it moves but doesn't go where you want it, it is possible the pulse width to position is not what you think. See if you can find the spec for your servo - it is not always that easy. You could also experiment with different values.
4) Check (carefully) that the arm is not stuck at one of the extremes. Although the gearing will make the arm hard to move, even under no load, if you are careful, you should be able to move it away from either end.
5) It obviously could be broken so having a second spare one is obviously helpful.
6) Be careful with oscillator settings. If you have accidentally got the timings wrong, the pulses could be 1000s of times too long or short. Again, it is worth connecting a spare LED to an output to ensure the cycle times are as expected.

Monday, 19 September 2016

Git stops working with VSTS - Authentication failed

Annoying bug with no obvious cause!

I have been using Git (via Tortoise Git) to push code to VSTS. I've been using it all day and then suddenly, "Git did not exit cleanly- Authentication failed".

After re-entering my credentials about 5 tims, each time taking even more care that I was typing them in correctly, it wasn't making any difference.

I fixed it in the end by changing my alternate credentials in VSTS to the same password as before but it looks like the act of saving it had refreshed the privileges and then it worked.

You can get to that by clicking your name in the top-right of VSTS and choosing Security->Alternate Credentials.