Friday, 25 October 2013

The Myth of Code Portability and "lack of vendor lock-in"

This is a Friday-style essay looking at something that is mentioned time and time again in articles about, "how we moved from .Net to Scala" or, "breaking free from the Microsoft stack". What I see time and time again is that people seem to say that they prefer one language or framework over another simply because it is open source/non-commercial. Naturally, open source can mean many things but the basics of the argument are that locking into a single vendor is a mistake that cannot be undone later but I don't agree with this for various reasons, described below.

The reality of virtually every system I have ever worked on or heard about is either, they only live for perhaps 5-10 years or otherwise they reach a point where they are left alone and not maintained unless absolutely necessary. From the customer point-of-view, this is fine because once it basically works, they want it left alone. By and large, if that customer decided, 10 years later, that they wanted a load more functionality, they would almost certainly get a completely new system based on completely different components and also, very possibly from a new supplier (because most customers get fed up with suppliers over time!).

So let us consider "product lock in" against this background. Scenario 1: Supplier creates a product based on a full Microsoft Stack, which gets installed at a customer site.

Issue 1: Maintenance.
No problems here, the .Net stack hasn't fundamentally changes in the past 15 years. C# looks pretty similar, albeit with some new functionality in later releases. Updating server OSs to new versions of Windows hasn't caused many headaches as far as I know, in fact, getting software and OS from the same vendor is good in terms of compatibility testing and support from that supplier. Consider what would happen if your open source product didn't work properly on Red Hat any more - you might be able to fix it but contributing to open source projects is not as easy as it sounds on paper and that assumes that you have the skills to even know how to fix it - you probably don't!

Issue 2: Major update. Again, no real issues here. This will be a new system that may or may not be Microsoft-based. If this is big enough, the customer is probably comfortable with the idea of replacing servers or hosting or OSs to suit whatever technology you decide to use for the new system.

Issue 3: A change is required (perhaps major) to a piece of your software - a module or service perhaps. Well, on the one hand, If you use the MS stack, it is likely that it either all works well or if it doesn't, it is a problem with the implementation, in which case, a technology change is not the issue, just a rewrite of that part of the code. On the other hand, many important customers are not comfortable with major changes anyway and the supplier certainly wouldn't want to do this unless they were paid for it. Using the MS stack doesn't actually preclude any of these things. The Vendor lock-in is not really in play and after all (God forbid) you can call PHP web services and MySQL databases from .Net if that was your core problem.

So what is the lock-in that people are concerned about? I think people are often talking about money rather than lock-in. I don't like the idea that because my software runs on Windows, MS have some kind of financial grip on me. But do they really? If you pay your licences up-front for Windows Server, how often do you have to upgrade and how much does it cost in the scheme of it? People still run 10 year old servers and even if licences are £500 a pop, compare that to the rest of the IT costs in an organisation and it really is peanuts.

What would using Linux, PHP, MySQL etc. actually give you? Really? Free updates - amazing, they are free from MS too except for major upgrades (which are hardly deadly expensive). A database engine that is still not as mature as SQL Server - again, if that's what you want, fine, but I am happy to pay a few hundred quid more for something that won't easily spill my user details over my web app. The same goes for .Net and Visual Studio with Azure. The cheapest? Nope but in my opinion, well worth the money.

If you want to use PHP - great. MySQL? Fine, Scala - do whatever floats your boat, but the decision, in my opinion, should be very heavily weighted towards the skillsets you have or can easily come-by and not some idealistic opinion about the latest and greatest languages, whether they are open-source or not. If your software is tested correctly, you should already know if the language/framework/database is suitable for the scale of work that the application needs to do.

Much debate is centred around endemic and often thinly veiled hating of corporations. It is easy to hate a company that makes lots of money but in most cases, they have lots of money because they produce something that people want. Some of the assumptions about the motivations of these companies are also out-dated (and might not have been true in the first place) - again, it is easy to assume that the only thing that a large company cares about is money but I think MS (I don't know much about Apple and Oracle etc) have made large improvements over the past 5 years which many haters still don't even know.

The moral? Don't change your system just to be "free", wait for it to need an upgrade and make changes then, based on what you need for your system at that point in time.

Invalid service path! Cannot locate ServiceDefinition.csdef in current folder or parent folders

This was really confusing me, I got this error attempting to add a PHP worker role to my cloud service and the ServiceDefinition.csdef was DEFINITELY in the correct directory!

Anyway, I had to download the source code for the Azure CmdLets and spent an hour working out how to get it to debug and use the modified source to log more verbose information (anyone else find that the verbose is not very verbose!).

...anyway, this error means one of two things, firstly, it means what it looks like it means but secondly, once it finds the ServiceDefinition.csdef file and knows the correct directory, it then parses the ServiceConfiguration.Cloud.cscfg and reads all the roles that exist and then looks for their directories being present, if it doesn't find all the role directories, it assumes the csdef is not correct and rather than recursing like it does when it first looks for the csdef file, it just fails with the same error text.

This was caused, in my case, because I had added a non-PHP worker role (assuming it would be C# - it wasn't!), deleted the folder and ran Add-AzurePHPWorkerRole at which point I got the error. It would be nice to have a more specific error here, otherwise, there is no point in displaying what it does!

I edited the cscfg files and the csdef file, removed the old role sections and ran the cmdlet again and it was all good!

Dropbox folder keeps appearing, can't delete it - solved

This was annoying and slightly confusing. Some folders would delete OK (and sync online) but others would re-appear. This was on Windows 7.

Solution: Right-click and "View on" which opens your browser. It turns out, in my case, there were two files in the folder which, for whatever reason, hadn't sync'd to my computer and made the folder look empty. I deleted them using the browser interface and was then able to delete the folders permanently from my computer.

Tuesday, 22 October 2013

Finally - Azure with client certificates!


The basic setup is that I have a web application which is public and a web service which is private. Since they are both hosted on Azure, I thought I would use client certificates as a way to ensure only the web application can access the web service. I had untold problems with it and got fed up with the odd IIS type errors about mismatches and authentication modes so I kind of gave up and hosted the web service on an Azure virtual machine so I could configure it all.

Anyway, there were a few problems with the raw virtual machine approach. It is harder to scale up (although I think it is possible with Azure VMs), it costs slightly more to run and although it was quick to deploy using SubVersion directly onto the box, I had so many problems trying to update the web site without it causing the web application to stop communicating with it properly. I decided enough was enough and tried to set up the client certificates in a cloud service again.

Client Certificates

You probably already know what these are if you are reading this article but if you don't, the basic idea is to use an X509 (SSL/TLS) type certificate to prove to the server that you are an authorised client. Using this mechanism is a good way of protecting your public facing but private services/applications from prying eyes. Note, that although you can possibly use these without https, I would not recommend it.

You can self-sign certificates but that opens a can of worms regarding how to check whether the certificate is genuine and having to override various errors but in my case, I am using SSL certificates from a root authority, which are pretty cheap really for the extra security.

Server End


The web service is pretty much a standard WCF web service but you have to be a little careful with namespaces and stuff (that is not related to client certificates but I might as well include it all to make sure it works). I HIGHLY recommend testing the basic web service without too much security just to make sure it works. The security settings can drive you mad but your problem might be related to something much simpler.

So... the service interface itself looks like this:

namespace com.mycompany.WebServices
    [ServiceContract(Namespace = "")]
    public interface IMyWebService
    { //etc...

Naturally, the exact namespaces are not too important (and don't need to use https in the name) but best to keep the standard format. The implementation is this:

namespace com.mycompany.WebServices
    [ServiceBehavior(Namespace = "")]
    public class MyWebService : System.Web.Services.WebService, IMyWebService
    { //etc...

This is pretty standard stuff and although you can often live without the namespaces, it can make it really difficult if you start referencing more than one web service from your client with type conflicts etc.


There are obviously lots of little bits, many of these are specific to my implementation (buffer sizes etc) but again, I will include it all so that you get an example that definitely works. The configuration for the service looks like this:

      <service behaviorConfiguration="defaultBinding" name="com.mycompany.WebServices.MyWebService">
        <endpoint address="standard" binding="wsHttpBinding" bindingConfiguration="standardConfig"
          name="standard" bindingName="standard" bindingNamespace=""
          contract="com.mycompany.WebServices.IMyWebService" />
        <binding name="wsdlBinding">
          <textMessageEncoding messageVersion="None" />
          <httpsTransport requireClientCertificate="true" />
        <binding name="standardConfig" maxBufferPoolSize="10485760" maxReceivedMessageSize="10485760">
          <readerQuotas maxDepth="32" maxStringContentLength="10485760" maxArrayLength="10485760" maxBytesPerRead="10485760" maxNameTableCharCount="10485760" />
          <security mode="Transport">
            <transport clientCredentialType="Certificate" />
            <message clientCredentialType="None" />
        <behavior name="defaultBinding">
          <serviceMetadata httpGetEnabled="false" httpsGetEnabled="true" httpsGetBinding="customBinding" httpsGetBindingConfiguration="wsdlBinding"/>
          <serviceDebug includeExceptionDetailInFaults="true"/>
              <certificate findValue="6FE3C33463GF87GFF94DC52479E72BFF60A1C34B" x509FindType="FindByThumbprint" storeLocation="LocalMachine" storeName="My"/>
            <serviceCertificate findValue="AB656DEE53E920801D23A5CC90250CB11F88D62E" storeLocation="LocalMachine" storeName="My" x509FindType="FindByThumbprint"/>
    <serviceHostingEnvironment aspNetCompatibilityEnabled="true" multipleSiteBindingsEnabled="true"/>

There are not too many things to worry about here but note that I have a service certificate for the https and another certificate that is what the clients use to connect and authenticate. Both of these are added into the Azure configuration as per normal certificates (both of them in LocalMachine/My). I have also included the certificate for my certificate provider to provide the full trust chain and this is also added in LocalMachine/My. I have not needed to do this for the root certificate which is obviously included on Azure. All of these certificates need to be uploaded to the Azure management interface so they are available to the project during deployment.

I then have to unlock the access section of the web config to force the use of SSL and require a client certificate. You can create a startup file like this: and inside that file, you need to run "%windir%\System32\inetsrv\appcmd.exe unlock config /section:system.webServer/security/access" without the quotes.

Then inside your web config, add the following section inside system.webServer:

   <access sslFlags="Ssl,SslNegotiateCert"/>

Client End

The client needs some similar configuration but note that using "Add Service Reference" inside visual studio did not work as expected, so I used SvcUtil.exe (which is part of the Windows SDK if you need to download it) and running SvcUtil.exe against the wsdl like this:


produced the relevant client code (a .cs file) and an example configuration to use for the client. I already had the client configuration setup so I didn't use this.

I had to include the .cs file from SvcUtil into my project and then create the client configuration in my web config in much the same way as it looks in the web service end:

        <binding name="standard">
          <security mode="Transport">
            <transport clientCredentialType="Certificate" />
            <message clientCredentialType="None" />
        <binding name="webHttpBinding" />
      <endpoint address=""
        behaviorConfiguration="CCBehaviour" binding="wsHttpBinding" 
        bindingConfiguration="standard" contract="IMyWebService" name="standard" />
        <behavior name="CCBehaviour">
            <clientCertificate findValue="6FE3C33463GF87GFF94DC52479E72BFF60A1C34B" storeLocation="LocalMachine" storeName="My" x509FindType="FindByThumbprint" />

As before,the certificate referenced in the clientCertificate element was added to the client's Azure configuration in the LocalMachine/My store and also uploaded to the Azure control panel.


It seemed easy once I had it all working but I'm not sure where I got it wrong before. I'm always suspicious of any caching mechanisms or the classic of mine which is, "break something just before you fix something" that just keeps pushing the problem to somewhere else! Hopefully this will help you get started with the client certificates functionality, a great way to secure communications with a web service.

Thursday, 17 October 2013

I really can't stand Java

Let me be up-front. I ONLY use Java because it is the only way I can write apps for Android. There are other promising frameworks that allow you to use other languages but I am betting that most of the low-level functionality I need is not available under those frameworks, some of it is barely there in the Android Java libraries!

My big gripe, though, is that using Java is such a struggle and I am a senior developer with over 10 years of commercial experience in all kinds of languages and frameworks. I am not a big fan of PHP but I can see where it works well. I wouldn't write most of my code in C but for some things it has an elegant beauty. Java on the other hand seems to have the worst of most worlds rolled into one.

  1. The language itself has weird shortcomings, that could have been fixed, but haven't and which are a real pain for those of us who are used to C#. Having to declare or catch every exception is nonsense. On paper it sounds good but, in reality, it means that exceptions are taken for granted and everything put into large "catch all" blocks rather than the intention of making people treat errors deliberately. If I am parsing a string into a number and I have already validated the input, why do I have to bloat my code with error handling for InvalidFormatException.
  2. Trying to compare strings with == compiles OK but doesn't do what virtually every other language would do - a string comparison of the two strings. In Java, it compares objects which, in my experience is totally useless and just asking to cause errors - subtle ones! The same goes for other basic types and you can't even overload operators to overcome this.
  3. Lots of the core libraries seem unnecessarily verbose. Compare the default HttpClient usage with the Apache one. No comparison.
  4. Various inconsistent naming and casing for classes throughout the core libraries.
  5. C# has introduced var for an implied typed variable. If you use var something = new Class(), you know that var equates to Class and this makes declarations so much shorter and clearer. In Java, no such luck, some names, especially with generics end up causing a constructor call taking up a whole line.
  6. Performance is basically terrible for most scenarios - another difference between the theory (write once, run anywhere) and the reality (virtual platforms suck). At least .Net can be compiled to a more useable and much faster byte code. The fact that it can perform acceptably well is not an excuse. I can make anything run fast by massive optimising or lots of hardware but these things should be like that by default.
  7. Libraries, jars, source paths, class paths etc. Is a NIGHTMARE in Java. It's like extern "C" in the old days which said "you should find this at runtime but if not, blow up". Why? If I have the library added to a project, shouldn't it all just work? This has done my nut trying to get a custom SSL manager working which compiles fine and goes bang at runtime. This is further worsened in the Android world since Google have handicapped some of the basic Java libraries for various (presumably) security and performance reasons.
  8. Using the file system directories to mimic the package names is complete nuts and doesn't (in my opinion) really give you anything other than headaches. Adding jars (glorified zip files) doesn't really change this. If anything, jars make it even more confusing since you have to browse through them to find some of your classes.
  9. Separation of docs, classes and libraries is just stupid. Another problem that rears it's head when you are trying to browse docs or code when debugging.
  10. Trying to understand the toolchain is also very hard for newbies. JDKS, JREs, J2EE - nice abbreviations for geeks but not useful for people trying to enter the world of Java.
  11. It doesn't do much of any of it's design goals. It's cross platform is crappy, it's embedded use is almost non-existent since it is so heavy weight and all of the failed goals came with a cost of a poorly designed language that is hard to use.
  12. There are too many third-party libraries that are considered standard - very worrying for a framework provider. Apache web client instead of the Java one, Bouncy Castle instead of the encryption classes in and even a cut-down version for Android that looks the same but will fail (sometimes at runtime) for this reason.
  13. One class per file (except inner classes) is another pointless-ism. An IDE is perfectly capable of parsing all files under a directory and finding where all the classes live. If I have a bunch of small classes, do I really need these massive directories full of files when one file called messages.class would make more sense?
I would think that many of these things could actually be fixed without any major problems. Stop requiring exceptions to be thrown or caught (but offer the option if you really want). Perhaps, allow code to be either compiled to native (since in many cases, people don't want to run the code on lots of different platforms), be more deliberate in taking good third-party code into the main set of packages so people who want fairly pure programs can rely on these features and not have to ship loads of random libraries that may or may not be maintained.

I think in it's purest sense - the basic language - is not that bad, it mostly looks like C# and C++ but with all the additional annoyances makes it worse than the alternatives. C# has been evolving and provided features that are very useful - although optional. Using var for types you don't care about, (since you might just dump them in a data table for instance), linq for SET based data manipulation makes some functionality so much briefer (a foreach loop on one line which reads really well) and libraries which are not just maintained in one place, for consistency, but which have also integrated work that others have done such as DotNetOpenAuth.

To me, the ONLY reason I use Java is because I have to. I think in every way, C# and probably other languages are superior (most other languages are not exactly equivalent so this is harder to judge).

If Java died, I wouldn't cry!

Tuesday, 15 October 2013

SSL/TLS for Developers or IT Admins

TLS, do I care?

BEAST attacks, dodgy RC4 algorithms, government spying etc. etc. etc. SSL (or TLS as it should now be known) is quickly becoming big news. It is one of the technologies that in the past, we were happy enough for it just to work, but now we are told it is very easy to mis-configure and like many technologies must be renewed over time as older mechanisms become vulnerable to attacks.

Do I Even Need It?

This is a fair question but for many situations, the answer is a simple, "yes". Setting up TLS is not very difficult and the prices of certificates are cheap (unless you self-sign which is free!). Unless the data you are sending and receiving is completely public already and no authentication is taking place, you should probably use it.

Traditional views about the burden of SSL are largely irrelevant on todays machines, except the most fragile units like mobile phones, and I read the other day that Google's average CPU load went up by a mere 1% when SSL was enabled for all gmail accounts.

So What Is It?

If you want to encrypt your data, you are trying to prevent anyone who doesn't have the "key" from knowing what the data is in its "plain text" form. This is fine if you are sending encrypted data between two pre-established points, which can already have agreed on a key but for many situations where there are many users and most or all of these are not known to the web application, it is both impractical but also insecure to distribute the key to all of these people in advance. What TLS does is provide a mechanism that allows for a client to agree on a key to use with the server and afterwards to use this key for the entire session. This key will not be the same as the key for any other client and will not be the same one used if the client comes back another time to connect to the server.

If you are interested, wikipedia has a whole load of documents describing how this takes place, starting here: We will cover how the basics work.

Secrecy and validity

The first important question is how the client establishes that it is indeed talking to who it thinks it is. It does not want to agree encryption keys with a spy server pretending to be the real server. For this, we have the whole Certificate Authority scheme which is a hierarchy of Trusted certificate issuers who basically use a secure system to assure you that if a certificate claims to be from domain, then the server really is because theoretically, no-one else would be able to sign a certificate claiming to be from Sadly, this is not actually true, since any trusted issuer can issue a certificate for any domain, which is great for competition but not so great for security. If someone steals a signing key from one of these companies (or someone bad works there) then fraudulent certificates can be generated. Currently, the only way to prevent this is to do some kind of pinning of certificates to the issuer that is known to have issued them. We won't discuss this further since the proposed solutions are not widely adopted.

Once the client sees a certificate claiming to be from and decides it can trust this certificate, it can then use the public key contained in the certificate (which is a slow encryption method called asymmetric encryption) to encrypt some data and send it back to the server. The server can only decrypt this with its private key. Even if someone stole the certificate, without the private key they could not determine what this data is. This data is used to encrypt some other data and send it back to the client to prove that the server did receive and decrypt the data from the client - the circle is complete.

What happens next is that the client can send a list of TLS protocols that it supports to the server and the server can choose one of these to decide the details of the rest of the handshake and the methods used to encrypt the session keys. Usually, it will choose the first one in the list it supports.

Once the session keys have been generated and established at both ends, the client and server can now start using symmetric encryption which is much faster - the overhead is not noticeable except at very high server levels.

At the end of the session, one or both parties can signal to the other that the connection needs to be torn down, at which point, the machines can clean up their encryption keys.

Setting It Up

Setting up TLS is pretty easy, certainly on Apache, nginx and IIS. In IIS, for instance, you can request a certificate within the GUI, which produces a request file. This file is sent to a certificate issuer who will give you a certificate from as little as $10 (free from some places if you are not-for-profit), which you then add back into IIS with the click of a mouse. At this point, you can enable SSL for a given website, add the binding to port 443 (or whichever) and then tell it which certificate to use.

Apache and nginx are pretty similar but mostly done in the configuration file, pointing to the locations of the certificate files.

If you only did that, you would get a system that works, that gives you a green padlock and a warm fuzzy feeling that you are nice and secure so what's the problem?

Configuration and Weaknesses

There are some problems with these default setups. Some of it is understandable, other parts are suspicious, all of it should be checked and as a developer (or an IT admin), you should be familiar with the latest issues found in SSL and various algorithms therein. You should also be aware that some tools are very useful for helping you but others might also give false positives or false negatives so try and cover as much as possible and record what you know.

There is a difference between locking down systems where you have control over both ends (a mobile app connecting to your web server) as opposed to websites that you want to reach as many people as possible. TLS provides a big headache when it comes to older browsers and we will see why.

The main areas of issue are the relative weaknesses of TLS algorithms (sometimes vs speed) and the weakness in some libraries to correctly handle the SSL handshake and verification of Certificates. A very useful tool can be found here: which will connect to the specified server and check for known issues, spelling them out in useful detail. You are unlikely to score more than 90s across the board unless it is your own server but this is usually at the cost of older browsers not being able to connect (including newer versions of FireFox and IE!).

The History Weaknesses

History provides weaknesses in certain ways. As you might imagine, although TLS version 1.2 is reasonably well understood, many browsers either do not support it at all, or more confusingly do not have it enabled by default. For this reason, your servers must still support TLSv1 (and 1.1) and this, although reasonably secure, is certainly not ideal.

History has also shown that certain algorithms are no longer secure enough, perhaps just because they are old (and key lengths need to increase to make them harder to brute force) or because time has revealed weaknesses which can be exploited. Although most servers allow you to configure which algorithms you support, usually, the ordering is not definable since the client tells the server which algorithms it supports in order of preference. These also need to be balanced with the fact that you may need to support older algorithms for older browsers. By supporting older algorithms, even if the order puts the best ones first, it is sometimes possible for an attacker to force the client and server to both downgrade their algorithm to a weak version which can more easily be hacked or exploited. The general instruction here is to put the strong ones first (DHCE and DHE) and some middle-ones afterwards that are reasonably strong and not to include any of the most weak versions - ones that use things like MD5 hashes. (There are various articles on Google about which to use)

The third way in which history hurts is that the longer that these have been exposed to the wild west of the internet, the longer attackers have had to try and exploit them, in which case certain modes like block chaining and RC4 encryption have supposed weaknesses that mean that although these attacks might be unlikely or hard, these systems are worth avoiding.

The Trade-off Weakness

Another type of weakness relates to the fact that some algorithms are much slower than others for given key sizes. This means that although a client might want a strong algorithm, a server is more likely to prefer one that is faster, even if a little less secure. The advice here is to start with the strong ciphers and worry about the performance when it becomes a problem. Load-test the systems to compare different ciphers before making the systems weaker.

Client Weaknesses

Client weaknesses fall into two camps. The browser weaknesses, the most common method used to connect to web applications, but also any net-aware client applications, assumed to be written in-house but the same issues apply anyway.

For a browser, your hands are tied. Unless you are one of the open-source browser developers, you have limited control over what is enabled/disabled, what cipher suites are supported, which are preferred and how well they are implemented. The plus-side is that the weaknesses are usually well known and well documented. Another issue with browsers is that you have no control over older versions that still exist and from which you might well want people to still connect to your system. Forcing people to update browsers is not easy, even if they are direct customers. The advice, know the limits of each browser, have an idea of which older versions you won't support (based on usage stats) and keep aware of the latest changes. Once a good percentage of browsers support, for instance, TLS 1.1, you might disable TLS 1.0. Also, try and capture browser stats for your own sites. General browser stats are handy but it is better know what your users connect with to decide how many will be affected by tightening anything up.

Client apps (and net-aware mobile apps) suffer the same potential problems as browsers but these should be more within your control. The following section discusses certain things that you should check before you trust these apps to communicate effectively.

Client App Weaknesses

Like most developers, I found a library for my Android app (The Apache HttpClient library), plugged it in and away I went. There are, however, a whole load of assumptions I have made about this library which I have not currently checked but definitely should.
  1. Does the library check that the host name on the SSL certificate matches the host name I am using the client to connect to?
  2. Do I get an error if the certificate is self-signed?
  3. Do I get an error if the chain of trust cannot be established on the SSL certificate?
  4. If I revoke a certificate, will the library ever detect this and block the connection? If so when?
  5. What cipher suites does the library send to the server and in what order? How can I limit this, especially if I only talk to my own servers?
  6. What protocol version does it use by default and which does it support? SSL? TLS 1.0? TLS 1.2?
  7. Does the library correctly implement security features such as blocking client renegotiation?


TLS is a minefield but although it is quite a large field, fortunately, the information is all there on the net to make informed decisions. Sites like stackoverflow or serverfault are frequented by clever people. The Qualsys site is fantastic for the latest best-practice, and the people who run the lab are very active on the forums there.

Spend some time learning about how TLS hangs together and remember, most of the time, something is better than nothing. Start with the first steps and try and test and fix as time goes by so that at least your system will be hard enough to crack, which will cause the attackers to try somewhere else instead!

Monday, 14 October 2013

CodeIgniter: You have specified an invalid database connection group.

We are currently going through a range of frameworks trying to create and test plugins for our authentication system at Out of habit, we have been creating a database and user for each of the frameworks and plumbing them in to the framework. CodeIgniter was not happy.

My colleague was working through a tutorial and trying to display a list of news items, bringing up the page caused a few errors including the one above. This was because in the config.php file, the value of $active_group was set to 'test' even though the database connection settings were all set against 'default' (there were no settings for 'test').

That fixed however, the page still did not work but fixing the first problem had only uncovered something else. Using a set of "echo 'Hello'; die();" lines throughout code, we worked out that a line in the controller that was supposed to return the query results (as an array) failed.

We tried to connect to the database manually with the user we had created for the framework and realised that somehow the permissions were not set up correctly. This might have been because we recreated the database but not the user and although the user still looked to have permissions to the DB, in fact, it didn't.

We re-ran the permission code and it all worked. We had assumed that CodeIgniter uses the database by default since the site was working, but clearly, it only uses the database for developer-created tables that might or might not be required as opposed to things like wordpress which are data driven right from the off.

Moral: Development is messy and hard!

Using FastCGI instead of ModPHP

(Updated for Apache 2.4)

On a test server, I wouldn't normally care in too much detail about things like php modules and the like. Apache works out of the box with mod-php, so why bother?

One thing that can be annoying is that by default, the user www-data owns the web root so if I need to write any files to it, I have to use sudo, at which point they become owned by root and not accessible by the web server. Every file write is followed by sudo chown.... to change ownership to www-data.

A way round this is to use fast-cgi which allows the files to be run as their owner and which means I can write files into the web root without using sudo and without running chown after I make every addition. It also makes it much easier to use FTP/WinSCP to copy files to the server when connected as a user other than www-data (i.e. every time).

So how do we change Apache to run fast-cgi instead of mod-php?

Install fastcgi

sudo apt-get install libapache2-mod-fastcgi

Note that if this is not found, you might have to un-comment the multiverse repository in /etc/apt/sources.list and run apt-get update. Once installed, create a file named php-fastcgi.conf inside /etc/apache2/conf.d (<= Apache 2.2) or in /etc/apache2/conf-available (>= Apache 2.4) and put the following contents into it:

<ifmodule mod_fastcgi.c>
DirectoryIndex index.php index.html
AddHandler php5-fcgi .php
Action php5-fcgi /php5-fcgi
Alias /php5-fcgi /usr/lib/cgi-bin/php5-fcgi
FastCgiExternalServer /usr/lib/cgi-bin/php5-fcgi -socket /var/run/php5-fpm.sock  -idle-timeout 900 -pass-header Authorization

<directory /usr/lib/cgi-bin>
Options ExecCGI FollowSymLinks
SetHandler fastcgi-script
Order allow,deny
allow from all

Apache 2.4 only: In the above config, replace the older config lines "Order allow,deny" and "allow from all" with the newer config "Require all granted". Once this has been done, you should enable the use of this new config file by typing: sudo a2enconf php-fastcgi.conf

Then ensure that the actions module is enabled:
sudo a2enmod actions
and finally restart apache
sudo service apache2 restart

 Disable mod-php

The installation will automatically enable fastcgi but you need to disable mod-php:

sudo a2dismod php5

Install FPM

The fast cgi process manager is a "nice to have" when coupled with fast-cgi. I am including it here because it is part of the instructions I have!

sudo apt-get install php5-fpm 

Now edit the file /etc/php5/fpm/pool.d/www.conf and add the following lines:

user = <your username>
group = <your username>
listen = /var/run/php5-fpm.sock
listen.owner = <your username> = www-data
listen.mode = 0660
pm = dynamic
pm.max_children = 10
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
chdir = /
php_admin_value[error_log] = /var/log/fpm-php.www.log
php_admin_flag[log_errors] = on

Now restart php5-fpm and apache2:

sudo service php5-fpm restart && sudo service apache2 restart

Change Directory Permissions

Once this has all been done, you need to set the correct permissions on the files in the web root.

sudo chown yourusername:yourusername -R /path/to/webroot
sudo chown yourusername:www-data /path/to/webroot
sudo chmod 710 /path/to/webroot

Thursday, 10 October 2013

How Startups should design solutions to scale

Scale, the dirty S word for companies who lack experience designing such systems. A word which can strike fear into the hearts of many and which can be done badly at either extreme. You can either not consider scale enough, in which case you might fail very early (or at least have a lot of rework to do later) but at the other extreme you can be so heart-set on scale that you waste time and money trying to make a perfect scalable system. I believe from experience that the ideal is somewhere in between these extremes and I want to lay out some principles below about how to decide on your design when it comes to scalability.

  1. The first and overriding principle is that you should be aiming to scale but not too much in one go. If you were buying a new house, you might get one with some extra bedrooms for any children you might be planning on but you wouldn't usually buy a house with 8 spare bedrooms in case you end up with 8 children. Why? Because firstly, you don't really know whether you will end up with 8 children and more importantly, there is an expense with buying too much scaling room and after all, you can upgrade your house later if you need to. I think this works in software. Having some breathing room and knowing that you can cater for the next 6 months to a year of hoped-for growth is great but you cannot predict the future.
  2. Technology moves on. You don't know what technology might readily suit your system in a year or 2 years time. NoSQL databases, new languages, special hardware, new caching systems, all can have a massive effect on your system performance whereas if you try and build in 50 years of scalability, you will base it on today's technology and spend millions building your house which will look out of date in 5 years time!
  3. You don't know how much your system will need to scale. We are producing a system that could ultimately be used by millions of people around the world but if I plan the system around that, I will be paying for a lot of redundancy that will not be needed either for a few years or perhaps ever. By imaging a very good case 6 months/12 months, I can plan for, say, 100,000 users and base my design on that. I don't need to squeeze every millisecond out of my database queries or multi-thread every single part of my system. At the moment, I don't even use memory cache because on hosted servers, memory is expensive and we wouldn't be able to cache very much of any use anyway.
  4. If you succeed, you will rebuild. As I read the other day on an article, Twitter, Facebook, Google have all had to re-factor their technology to suit their scale. Languages have changed, back-ends have changed, parts have been moved around to try and make the bottlenecks occur at places that are easy to increase like web servers. None of these people could have realistically built their original systems in the languages they now use. This might be because the new tech didn't exist back then but it might be that the overhead of the development work required just wouldn't have provided payback when the user base was small, ironically, it might have cause them to be failures instead of successes.
  5. Your design will change! We have a system with relatively few pages, few use cases and not many routes through but we have already changed our design in about 4 major ways inside a year. This has had knock-on effects on the parts of the system that are doing work but if I had spent ages designing a super scalable system in the early days, I might already have had to tear that down and start again with the new system.
  6. If you end up being successful, you can afford to rework it later. Rather than assuming you need to get the Rolls Royce before you are viable, buy an Audi and prove that you are a good driver. Once you succeed, take on more developers and start to improve things that need improving.
  7. Development cycles are much shorter than they used to be. Our system is relatively simple but if I had to pull out SQL Server and put in MySQL, it wouldn't actually take very long, perhaps a few days or weeks. We shouldn't fear rework and replacement systems - this is part of what we employ developers for.
  8. Try and identify areas that whose performance will decrease linearly and others that might have an avalanche effect - monitor all of these. A web server will roughly slow down proportionally to the number of connections made which will relate generally to the number of users. At the point that the performance becomes unacceptable, I can usually add another web server and this is usually easy enough. Other parts of the system are potentially more error prone. What happens if you exceed your service provider's bandwidth allowance? Do you get throttled and cause a massive drop in performance, caused by that one small request over and above the limit? You need to know about these hard limits because if the performance drops massively, people might start to leave your service.
  9. Learn what is easy to scale and what isn't. I recommend all web apps are designed to work in a server farm. This is either automatic with many cloud services (PaaS) but even if you have to create the farm yourself with 2 web servers and the farm server, this then allows you to increase web connections very easily. Databases are hard to scale so keep the database as slick and quick as possible to avoid this issue early on. Try not to perform any CPU intensive operations on the database server. There are ways to split and shard databases but these are best avoided since there are all kinds of dragons there.
  10. Don't worry. Stick with what you know, employ people for bits you don't know and learn from your mistakes. It is more important that your company deals with issues in a timely fashion much more than it is to never make mistakes. Learning from your mistakes should be done by asking why a mistake happened and what can be done to avoid or reduce it happening again (test cycles, checklists, 3rd-party verification, whatever...)

Tuesday, 8 October 2013

Drupal Hybridauth Social Login plugin stuck in loop when logging in

We have been trying to create various PHP plugins for the PixelPin single-sign-on solution and one of these was for Drupal. I assumed it would be easy since we had already written the PixelPin plugin for HybridAuth for the wordpress social login and it is the same library.

We altered the Drupal plugin and added the PixelPin files yet when trying to login with PixelPin, the site got stuck in a redirect loop and didn't seem to log any errors apart from random ones appearing on the front page saying, "An error has occurred".

It took a while and lots of debugging code to realise that I had misunderstood the configuration of the providers. In the file, all providers start with a secret, a key and an id. Since we don't use application ids, I added PixelPin to the array on line 444 which unsets the id - I was left with a key and a secret. However, the HybridAuth library requires OAuth2 providers to use id and secret, not key and secret. If these are not set, an exception is thrown but this is somehow swallowed by the framework and leads to the redirect loop.

I changed it to remove the "key" instead of the "id" from the config for PixelPin and it was all OK again!

Monday, 7 October 2013

Calling .Net code from SQL Server


How do you call .Net code from SQL Server and why would you want to? There are various reasons why you might want to but they all come down to a simple answer, "doing something that is easy to do in .Net but driven from a database".

In my case, I want to trigger from one database and if certain changes are made, to log these and then call onto a web service to update a dependent system (that does not run on the same SQL server). Obviously the trigger is easy in SQL but logging and calling web services is much easier in .Net - it is also easier to debug from Visual Studio.

This is how to do it....

Create a Visual Studio Project

Firstly, create yourself a database project in Visual Studio. I believe some of these have changed names but in Visual Studio 2012, there is only one database project called, "SQL Server Database Project". I think the older versions had several projects with example files in them, in which case, choose the "CLR User Defined Function" project.

Once this is created, you might or might not have any code but if not, choose "Add New Item" on the project and look under SQL CLR C# for the item called "SQL CLR C# User Defined Function". Give it a name and add it.

Once you see this file, it looks very similar to normal C# but with a special attribute (Microsoft.SqlServer.Server.SqlFunction) that will let it be called from SQL Server. You will also notice that the types live in the System.Data.SqlTypes namespace which ensures they are correctly marshalled between .Net and SQL Server. Otherwise, it is all pretty normal stuff.

Set the Project Properties

Right-click the project in the solution explorer and choose "properties". Here, you can set the names for your assembly (if different from the project name) and also change the target framework to 3.5 if it needs to work on SQL Server 2005/8.

You can, and should, also set the Assembly properties so you can more easily keep track of your code. Pressing this button creates an AssemblyInfo file.

If your assembly does anything outside of itself like file IO or network access, it will need permission to do so. You specify this by setting the Permission Level (details are here). If you have chosen anything other than "SAFE", you will need to sign your library. Do this by pressing the Signing button and choosing to sign the assembly, if you do not have a strong key already, you can create one here in the dialog.

Add Login and User

A CLR function needs to be owned by a user (this code will become a database) so you will need to add a user without login to your project and then set this name also in the Project properties against the "Assembly owner" on the first page. If you try and create a login in the project, it will fail deployment later.

Build Project

Build and deploy the project, you should get no errors. You might optionally add additional functions or other tables etc. This build should produce a .dacpac file as well as the assembly dll.

Prepare the SQL Server

The SQL Server will not allow the CLR object to install or run by default.

Firstly, you will need to enable CLR integration for the SQL server. Run the following query against the master database:

sp_configure 'clr enabled', 1

Note this does not require a restart.

Secondly, you need to create a login linked to the key that you used to sign your assembly with. The easiest way is to create an asymmetric key from the assembly file like this:

    FROM EXECUTABLE FILE = 'C:\Users\Luke\Documents\Visual Studio 2012\Projects\MyProject\bin\Release\MyAssembly.dll'  

No password is required in this statement.

If you get an error here, it might be because the directory your assembly lives in does not give the SQL server user permission to read the contents of the directory, in  which case, just give "Users" permission to read e.g. ...\Myproject\Bin\Release.

Next, create a server login (I used the user interface) and point it to the key you just created in the "mapped to asymmetric key" dropdown list. Once you create this login, you need to give the login external access permission like this:

USE master

Import the Data-Tier Project

Right-click the databases tab on your server explorer and choose "Deploy data-tier application". The options are quite easy to understand, point it at your dacpac file and press go. What happens during this import is that the server will determine if it is happy to give your CLR code the permissions that were specified in the properties. For instance, if the code requires external access, it will use the assembly signing key to associate the code with a login (the one you just created) which is linked to the same key. This is how the server establishes the trust relationship since only a system admin can create server logins.

Try it out

It works as any other database function does. For instance SELECT DataTierApp.dbo.MyFunc( Param1, Param2)

Simple TraceListener example

One VERY common thing that we need to do in live sites is to "log" or "trace" information from the application so that we can debug things such as timings, return values or whatever else we need to look at. Often, these are not easy or possible to obtain in test systems which don't usually have the same deployment model, the same loads and the same number of users.

In .Net, they have a system that is based on something called Trace Listeners. It is very powerful and flexible in as much as you can trace different sources and redirect these sources to the same or different "log locations" such as flat file, XML, database or even Azure and other customer event logs.

The problem is that the basic examples are either too complicated or non existent so here is the basic setup required to get tracing working.

The main types you need are in the System.Diagnostics namespace which is in System.dll which is useful because virtually all applications will reference this and it will be available on all .Net deployments.

As with anything that is complicated/powerful, I HIGHLY recommend that you start with the example here and then change parts of it to suit your application. I have a habit of using a basic example and changing everything while I implement it which leads to painful debugging!

Firstly, add the correct configuration to your web/app.config so that anything that is traced is logged somewhere. I usually put this right at the bottom of the config so it is easy to find (it lives under the top-level Configuration element). One of the easiest ways to test this example is to log to an xml file as below:

      <add name="Default" value="Info"/>
    <trace autoflush="true" indentsize="4">
        <add name="xmlListener"
          initializeData="c:\logs\admintrace.xml" traceOutputOptions="ProcessId, DateTime" />
        <remove name="Default" />

Note that the Value of the switch must match one of the predefined values Error, Info, Off, Verbose, Warning. I think it can be a comma-separated list of more than one of these but don't take my word for it! The location of the file is the value of initializeData. Also, autoflush=true will reduce the performance slightly but will make your messages get written to file immediately. On a live system, you might want to leave this set to false and have it move data out in efficient size blocks.

Then, all you need to do is call static methods on the Trace class in your code like this:

Trace.TraceInformation("GetUserData duration(ms) " + stopWatch.ElapsedMilliseconds.ToString());

That's it! There are other methods like TraceError and TraceWarning which you can use but note that these will need to match the switch value (or be a higher level that the switch value) or otherwise they will not appear in the log. For instance, if the switch is set to Error, then warnings and info won't be logged. If you set it to Info, then info, errors and warnings will be logged.

Verbose is the most information traced but this might cause too much noise in your application in which case you will need to look at Trace sources and separating the trace calls into separate log files (this is done by namespaces). Also, note that as with all IO, this adds a load and fills up disk space in your application so try and avoid tracing too much information, you can always switch it to "Off" at run-time and it will stop logging.

Friday, 4 October 2013

Network performance from Android app using the Apache Library

Recently, the issue of performance came up with our mobile app and it reminded me of the first rule of optimisation: don't.

The thing is, we often make assumptions about what we think will be slow or fast and will perhaps attempt to optimise these areas rather than worrying about things that are slow and ignoring things that we assume will be slow.

For instance, our app calls a PHP web service which in one case, also has to call onto another SOAP web service written in .Net. Many naysayers would claim this must be very slow and .Net is slow right? Wrong. This is the fastest call into the web service.

Secondly, sending small amounts of data across the 3G network (in the UK) would not be noticeably slower than sending it over wifi/cable. Wrong. Despite most of these calls contains less than 1K of data, the speed difference is noticeable and in some places, very poor. The main issue is that performance is very erratic.

Thirdly, SSL adds a noticeable overhead? Again, wrong. Perhaps at large volumes it might all add up but there was no noticeable difference in speed between the two, the variations were much more related to other parts of the network.

What about DNS lookups? We know these can be slow so I compared a call to http://mydns... to http://10.1..... any difference? Nope.

In fact the two biggest factors that affected the performance (perhaps unsurprisingly) were the mobile network being pretty poor (roughly half the speed - more on that later) and the CPU or IO intensive operations actually carried out by the various web service methods. In my case, the two slowest operations (apart from uploading an image) were 1) sending emails inside the web service call and 2) Generating a pseudo-random code in PHP. These findings will lead me to optimise both of these, I will probably generate random codes in a background thread and dump them into a database table for later retrieval and for the email, I can either put it into another thread or otherwise use something like a database table and some kind of queue.

Talking about speed, it might seem obvious to say that our wired network is twice as fast as the mobile network but if we take one method in particular, it sends about 200 characters of data to the web service and the wired internet can carry this out in less than 0.5 seconds end-to-end. So, let us assume the wired internet is perfect and has zero latency, this means the method itself takes 0.5 seconds to execute on the web service. The mobile network took between 2 and 6.5 seconds to perform the same method meaning it is adding at least 1.5 seconds of latency to the app - this is when sending a mere 200 characters (and retrieving about 20 characters in response) over a H network with full reception - which is a maximum of 10Mb/second but even if it was only about 1Mb/second, the network is very poor with data transfer. Taking the DNS out of the equation means it must be good old-fashioned latency and poor quality equipment.

The moral? Well, I need to make my app work as fast as possible but if even the smallest network transaction are taking 2 seconds on the mobile network, I have to either come up with something cleverer or try and cheat the user by doing these things in parallel with the normal use of the app, something that is quite difficult the way it is currently designed.

Wednesday, 2 October 2013

There is a problem with the resource you are looking for, and it cannot be displayed

Internal server 500 and this is the only text I could get back from PHP running on IIS.

It was being caused by a disabled SOAP php extension and I (eventually) tracked it down firstly by ensuring the error_log was specified in PHP.ini and pointed to a directory that was accessible by PHP. I then ran the troublesome piece of code and got "SoapClient" is an unknown type in this error log.

I don't know why this wasn't picked up at application level and sent back to the client in the form of a useful error. My try/catch blocks and everything were bypassed, I guess because this was right down in PHP.

I added in the extension, re-started php-cgi.exe to reload the PHP.ini and it was all well again...

(This all took me 2 hours but should have taken me 10 minutes!)

Tuesday, 1 October 2013

How to add a loading or waiting page to an android activity

When you start using mobile apps that make network calls, you have to start using threads. In fact, the Apache network library requires it and throws an exception if you call the network on a UI thread. It makes sense, the network is an unknown quantity and you don't want the app locking up but threading can be a real pain, especially if you add it in afterwards instead of working out some design patterns.

One common type of pattern, however, is how to display a loading screen while waiting for something to happen. In my case, I have a "retry" button which calls a web service which sends another email. I need something to stop the user pressing anything else or assuming the system is locked up and I don't really want to invoke a whole other activity to handle this. There is an easy way:

This describes using something called a ViewSwitcher which allows you to define the please wait screen in the same layout file as your main view and then just use showPrevious() and showNext() to flip between the two. In my case, pressing the "retry" button calls showNext() which displays the please wait screen and once the email task calls back to say it's complete, I call showPrevious() to restore the main layout.

You can call setContentView but be warned, if you do this, the event handlers are not wired back up in onCreate so your buttons will stop working if you simply do that.