Thursday, 20 October 2016

Azure App Services not ready for the big time yet?

I really like the idea of App Services for web apps. It basically looks like cloud services but with shared tenancy so lower cost per "server" than a cloud service.

The other really big winner for me is the much quicker scalability when scaling out. Cloud services understandably take a long time to scale out and cannot handle short traffic bursts for that reason.

App Services on the other hand can scale out in seconds. I think they do this by sharing the code and just increasing the number of virtual servers.

It looks great on paper and deployment is as easy as Right-Click->Publish from visual studio, using web deploy and therefore taking seconds. Cloud service deployment is still cryingly slow!

So that's the good news, what's the bad news?

It doesn't seem to work properly!

We set up an app service plan (Standard) and deployed a web application that calls a web service, which accesses a database. We also use Azure Redis Cache.

We fired a load test at the system to see what sort of load we could handle and setup auto-scale for a CPU of 80% to allow form 2 (minimum) to 6 (maximum) instances.

So what happens when we run the test? Loads of Redis timeout errors. Hmm, the requests per second are only about 100/150 and the response times are less than 1 second so why is the redis 1.5second timeout occurring?

We had a few pains, where re-deployments didn't seem to fix the problem and even though we could see the remote files easily using the Server Explorer in Visual Studio, the presence of the machine key element didn't seem to remove the errors related to "could not decrypt anti-forgery token". Hmmm.

To be fair, it did expose a problem related to the default number of worker threads. In .Net, this is set to the number of cores (1 in our case) and any new threads can take a minimum of 500mS time to create, easily tipping the request over the 1.5 seconds. So I increased the number of threads available but still we were getting timeouts even when the number of threads was way below the threshold for delays.

Maybe we were pegging CPU or memory causing the delays? No idea. The shared tenancy of App Service hosting does not currently allow the real time metrics to display CPU or memory for the individual virtual server so that's no help. It also wasn't scaling up so I guessed it wasn't a problem with resources and why should it be at such a low user load?

I finally got fed up and created a cloud service project, set the instances to the same size as the app service instances, using the SAME redis connection and ran the same test. Only a single error and this pointed to an issue with a non-thread safe method we were using (which is now fixed). No redis timeouts and a CPU and memory barely breaking a sweat.

We ended up in a weird situation where we couldn't seem to get much more than about 100 requests per second from our AWS load test but that is another problem.

Why does a cloud service running the same project work where the same thing on App Services doesn't? I couldn't quite work out what was wrong. A problem with Kudu deployment? Some kind of low-level bug in App Services? A setting in my project that was not playing nice with App Services?

No idea but there is NO way I am going to use App Services in production for another few months until I can properly understand what is wrong with it.

These PaaS tools are great but when they go wrong, there is almost no way to debug them usefully. The problem might have been me (most likely!) but how can I work that out?

Cannot answer incoming calls on Android

Weird problem: Phone rings and/or vibrates but the screen doesn't change. All you can see is the home screen. How can you answer the call? You can't.

Well you can but you need to know what you did!

In my case, I am on the Three mobile network, which is, basically, terrible around where I live and work. They have an app that uses the Internet for calls and texts if you are not in mobile range. If you are, all kinds of pain occurs because texts sometimes go to the app and not the phone etc.

Anyway, like many annoying apps, decide that they can fill up your notification area with a permanent message saying that Wi-Fi calling is ready. Amazing!

Anyway, I had disabled notifications for the Three app to get rid of that annoying message but the thing I hadn't realised is that it will also disable the incoming call notification. For some reason, the phone still rings and vibrates but you can't do anything.

I had to re-enable notifications and then, of course, I can answer the phone! I also have to live with the annoying permanent notification.

The same has also occurred with Google Dialer according to forums but I don't have myself.

Friday, 14 October 2016

Azure App Services (Websites) - Cannot find the X.509 certificate using the following search criteria

Deployed a brand new WCF .Net web service to Azure App Services and when trying to load up the svc file in the browser, got the message above.

Here's what you need to know:


  1. You need to upload the required certificates to the Azure portal
  2. You need to make sure that you are referencing the certs from the CurrentUser store, NOT the LocalMachine store. App Services uses shared hardware so you can only access the CurrentUser location.
  3. You need to add an App Setting to tell the service which certs to make available to the web service. You only need to do this for certs referenced in the app, not for certs you are only using for an SSL endpoint. The key is WEBSITE_LOAD_CERTIFICATES and the value is 1 or more comma-delimited thumbprints for the certs you want to load.
  4. You CANNOT add this only in the web config file, despite Azure merging portal and web config values, it MUST be added in the portal to the Application Settings tab.

Thursday, 13 October 2016

Encrypting/Decrypting Web.config sections with PKCS12

What is it?

I'm not sure why I haven't blogged this before, since it can be quite confusing but this is a cool way of protecting web.config secrets from being seen casually by Developers.

The basic idea is that you use an SSL certificate as the key and then the sections in web.config are not readable. They will decrypt automatically, however, when used by IIS if the certificate is installed locally.

It is worth mentioning that if a developer can run the code locally, they will still be able to find out the secrets, it is more of a protection from people who can see code in a repository but not run it.

You can use any SSL/PKCS12 certificate for encryption/decryption but I recommend using a self-signed certificate that should be in-date (since some services e.g. Azure App Services will not allow upload of expired certificates). If you use a self-signed certificate, you get to control its lifetime more easily if you are worried about it being hacked. If you share one of your public certs, you will probably have more work to do when the cert expires.

You should be able to encrypt all web config sections but although this works in theory, certain sections cannot be encrypted due to when they are accessed by the system. I can't find a useful resource so you will have to try it and see. It will be obvious if a section fails to decrypt.

So how do you do it?

First, you should reference or install Pkcs12ProtectedConfigurationProvider, which does exist as a NuGet package.

You then use aspnet_regiis.exe, which lives in the .Net framework directory(ies) in C:\Windows\Microsoft.Net. If you use the Visual Studio Developer Command Prompt, then this program should be in the path already.

Make sure you run the command prompt as Administrator, otherwise you will not be able to run gacutil and asp_regiis will return the error "Keyset does not exist" because it will not have permission to access the local certificate store.

So how does it know which certificate to use to encrypt? You need to create a section in your web config that references the thumbprint of the certificate you want to use. It looks like this:

<configprotecteddata>
    <providers>
        <add name="CustomProvider" thumbprint="d776576a90b5c345f8a9d94e732c86c1076eff78" type="Pkcs12ProtectedConfigurationProvider.Pkcs12ProtectedConfigurationProvider, PKCS12ProtectedConfigurationProvider, Version=1.0.0.0, Culture=neutral, PublicKeyToken=34da007ac91f901d">
    </add></providers>
</configprotecteddata>

aspnet_regiis performs various other functions as well but the two options we are interested in are:

-pe to encrypt a section
-pd to decrypt it.

If you use the f option as well, you can specify a physical location for a web.config, which I have found is easier than using the VirtualPath (and in some cases you won't have virtual path).

The -f should point to the directory containing the web.config, not the web.config file itself. You will end up with a command like this:

aspnet_regiis.exe -pef "connectionStrings" "C:\Users\luke\Documents\Visual Studio 2015\Projects\SolutionFolder" -prov "CustomProvider"

Note that prov specifies the name of the key entry that you want to use to encrypt the section.

If you are encrypting a custom section, you might have some fun and games trying to get aspnet_regiis to find and load the dll that defines the config section. You can use gacutil to add these dlls to global assembly cache and then use the correct version and public key (which you can find in a tool like ILSpy or by browsing to the Gac at c:\windows\microsoft.net\assembly\gac_msil and looking at the folder name for your dll e.g. v4.0_1.0.0.0__34da007ac91f901d) so that aspnet_regiis knows to look for the dll in the gac.

If your section is not at the top level, you need to specify it like system.web\mysection. Also, some sections cannot be encrypted, which I think is because they are needed before it would be possible to run the decryption in IIS.

Sometimes, it pretends to work but doesn't. Run it again, it sometimes catches up! Google any errors, they are fairly easy to work out.

You will then end up with a section of the same name but with a reference to the customProvider you are using and an XML encrypted packet.

Remember to upload the certificate you are using to any systems that will need to read the section otherwise you will quickly get an error!

And as always, start with a really simple example and make sure it works before going crazy and trying to encrypt everything.