Monday, 27 October 2014

Deserialize object trees from json in Java

The worst kind of problems in software are those that are hard to search for in Google, in my case everything I tried was some variation of "object", "java" and "json".

I am trying to implement a spec in an Android app that needs to deserialize some json. Unfortunately, I do not know exactly what this json will look like since that is not in the spec and getting an answer will take too long. I need to start something now, even if I need to change it later.

The issue with the json is this: I have a request object that is serialized, but one of its fields, called "args" is an Object. That is because it can be one of a number of different classes depending on the value of another field "requestType". In other words, if requestType is Commit, the args will be of type CommitIn, if requestType is Register, args will be RegisterIn etc.

I started using gson by Google and although it is easy to deserialize the request itself, clearly the gson builder class can not know what type to deserialize for args. To make it worse, the request classes do not all inherit from the same class (although I could make it so that they could do if that would help).

I've read about RuntimeTypeConverter but this seems to imply that the serialized classes must all inherit from the same class and more importantly, that they have some kind of "type" field that the converter can use to know what concrete type to deserialize into. I can't find out whether this type information will be present.

I've come up with a variety of possible solutions but I'm not sure which is the best going forwards:

  1. Use the raw JSON, find the requestType and then modify the JSON to inject a type parameter which can then be used by the json deserializer. Crude but would work.
  2. Deserialize the request into a parent request object to work out the requestType and then deserialize again into a specialized subclass once the requestType is know, the type converter will then have a target concrete type. The messages are fairly small so the overhead should not be too bad?
  3. Extend the json classes so that it is possible to map the value of requestType into the conversion of the args object - ideally done in a generic kind of way something like ConvertTypeByProperty(targetField, propertyName, "Commit" CommitIn.class()).
  4. Create a type convertor for the top level object and do everything in a customised kind of way (although this kind of defeats the point of using a library I think).
I'm not sure how to proceed, any suggestions?


I ended up doing #4 but I didn't have to write my own converter or anything like that. By making each request a separate class with the correct type for "args" and each inheriting the other fields from a parent request class, what I could then do was use the built-in RuntimeTypeAdaptor class (actually it isn't included in the gson library for some reason but the source is downloadable from Google) and then all I have to do is point the type adaptor to the requestType to identify the type of request that is coming in, all the deserialisation then works correctly.

ASMRequest is the parent class and ASMCommitRequest is one of its subclasses

RuntimeTypeAdapter requestAdapter = RuntimeTypeAdapter.create(ASMRequest.class, "requestType");
requestAdapter.registerSubtype(ASMCommitRequest.class, "Commit");   // 1 example of specific request class if requestType equals "Commit"

 Gson gson = new GsonBuilder()
                    .registerTypeAdapter(ASMRequest.class, requestAdapter)

Android SSL/TLS/HTTPS - IOException: No peer certificate

I love it when I think I have deployed my app and it is all ready to test and the first thing that happens when someone tests it is a nice big fat error!

I got the device, put some logging in and found that an IOException was occurring when it was contacting a web service over SSL and the message was, "No peer certificate".

This is one of those errors that can be related to a whole number of problems with SSL certificate types and chains but fortunately, I knew it was basically working because my phone worked fine.

The problem was that the phone that was failing was not on a phone network so the "automatic date and time" was not working and the date was therefore set to January 1980. What this means is that phone date is not within the validity of the certificate being obtained from the web service. This effectively makes the certificate invalid and causes the error!

I guess I need either some kind of date/time check performed over non-ssl before attempting the SSL handshake or I need to modify the error message to prompt the user to check their date/time....

Thursday, 23 October 2014

Yii2 and the spaghetti maze of configuration files!

Warning: Don't modify ANY Yii 2 configs until you finish this article - you might lose your changes!

Configuration files can be a pain. Most frameworks have only 1 and that means that settings local to the developer need to go into it (and not be checked into source control), as well as usernames and passwords etc. It is very easy for temporary configuration to accidentally get added to a production environment and potentially cause all manner of problems.

Yii 2 have solved this in a clever but slightly complicated way! You might notice, especially if you chose the "advanced template", that there are about 20 configuration files. Which ones are you supposed to modify?

There are really two types of config (ignoring parameters, which work in a similar way), local and non-local. They are both merged at runtime to produce the final configuration so you can change either and the result will be the same but.....

...local settings should NOT be checked into source control. In other words, any config with "-local" on the end should only be for local settings and not for anything that needs to be deployed or kept in source control. Examples of this include things you are just testing for now or perhaps access to Gii, which you can easily add locally and not need to deploy to the live site.

So once you have decided whether your config needs to be local or non-local, which file do you actually change? It depends on the template but let us consider the advanced one, since it is the most complicated.

The simplest answer is that if the config applies to the back AND front end, you change /common/config/main.php or main-local.php. If it only applies to the front end, you change frontend/config/main.php (or main-local.php) and likewise for back end. This, again, is useful if the backend is hosted privately and might need a higher privileged database connection string than the one deployed to the front end for instance.

OK so far? There is one more complication. It is still possible that something needs to be set in config but only in development, not in production. If you want to properly test the production deployment, you need to easily remove your local configuration settings and restore them again when you are developing again. You don't want to do this manually, otherwise you are likely to get something wrong.

If you have noticed the environments folder, this is what that's for. There are more configuration files under e.g. environments/dev/common/config and these are only for local config, that is config that is only used for you and no-one else but since there are separate folders for each environment, you can decide that some of your local settings only apply to a particular environment and only put them into one main-local file.

Why did I say not to change anything till you read all of this? If you do NOT use the environments capability of Yii 2, then you might as well just change the configs in /common, /frontend and /backend (it's quicker and easier) BUT if you DO use environments, the top level local files will be overwritten by the files underneath the environment variables when you run init. In other words, if you only changed /common/config/main-local.php and then use "init --env=Dev" on the command line, you will lose your change as it is overwritten by /enviroments/dev/config/common/main-local.php. To be sure, you should probably always change the main-local files under each environment and then run "init --env=whatever" to copy them into the relevant place!

A bit more hassle, and I'm yet to be convinced that this hassle is worth it for the times when configuration trips us over but it's fairly powerful so fill your boots!

Wednesday, 22 October 2014

Schools should work more with industry

Recently in the UK, they have started a new 'programming' initiative for schools. The basic idea is that programming should become something that all school students study since computers and IT are so common now, that most of us might well benefit from this.

It's all great but in the great tradition of governments doing a $50 job for $5M, they have decided that loads of teachers need to be trained accordingly and this requires however many hundred specialists to teach the teachers. Clearly, the teachers were not employed for their programming skills and the idea that they will be able to adequately teach programming to school children after a few days of training is laughable. It's like expecting them to teach medical science after a week in a hospital!

What seems to be really lacking is schools capitalising on the skills of local industry and/or parents of children at the school who are experts in a whole range of subjects, not limited to programming. When was the last time you heard of a school inviting a bank employee in to explain mortgages to students or an engineer to talk about how power generation works. These would be really interesting and more importantly, they would provide the vast expertise of industry directly into schools, allowing children to ask all manner of questions and get honest informed answers, something they could clearly not expect from a teacher teaching something second-hand. Most of these people would do it for free, some would even be able to commit to a regular weekly slot as part of an ongoing programme.

I'm sure some schools do this but it doesn't seem to be very common. Perhaps it is just easier to not get people in from outside, perhaps teachers have an ego thing where a non-teacher couldn't possibly be allowed to teach a subject to a class. Whatever the reason, with all the social-networking that occurs nowadays, it should take 10 seconds for a school to find a relevant expert to teach whatever they need teaching and make it happen. Make it happen!

Tuesday, 21 October 2014

Forget Eclipse, it's all about Android Studio

I finally got so annoyed with using Eclipse for Android development that I decided to take the plunge and use Android Studio instead. It is based on Intelli-J Idea and feels so solid. Using Eclipse is like driving a very old car, you feel nervous because do something slightly wrong and it breaks. Despite numerous updates and goodness knows how much effort by lots of developers, it is virtually unusable in a development team since errors are so numerous and the IDE so flaky.

Copy and paste starts acting weird if you switch windows in the wrong way (the original window stays as the target). There are all kinds of issues with memory usage, I started getting out of memory errors when trying to debug even though I hadn't changed anything! Even navigating the IDE was basically terrible. The idea of separate "perspectives" is weird, especially since menu items change, meaning you can only do certain things from a certain "perspective". Debugging is flaky at best, sometimes hovering over a variable shows its value, other times it doesn't. The debugger doesn't seem to stop at the place where an exception is triggered, it breaks in some other thread and you have to use the call stack dumped out to try and navigate, sometimes clicking on the lines doesn't take you to a file.

Anyway, Android Studio docs told me to "export from Eclipse" to generate the correct Gradle build files required by Android Studio whereas the Eclipse export dialog told me to "import into Android Studio". Great! The decision was made when Eclipse crashed again while attempting to generate the Gradle build files.

The first thing I did was find and delete all the myriad of Android SDK locations that had been created by various managers/IDEs and then installed Android Studio. This installs the sdk under its own directory which is nice and neat! I then ran the Android SDK manager, downloaded everything again (which took ages) and ran up the studio.

"Import project", chose the Eclipse project and it did a wonderful thing. It copied the project into a new location, leaving the old rubbish where it was. It moved the files into a slightly different structure (for Gradle reasons) and more importantly, it no longer required dependent projects to all live in the same workspace, they were automatically converted to Gradle dependencies.

It pretty much all worked as expected, I just had to learn some slightly different terminology and, of course, change SHA-1 to SHA1*

* See previous blog post

SHA1 or SHA-1?

I moved an Android project from Eclipse to Android Studio and the only other thing I think I did was to update the build target from Android 18 to Android 21 (v5). After doing this, though, the code crashed because it couldn't create an RSA cipher class using the following:

 final Cipher cipher = Cipher.getInstance("RSA/ECB/OAEPWithSHA-1AndMGF1Padding");

It produced the familiar, "javax.crypto.NoSuchPaddingException: OAEPWithSHA-1AndMGF1Padding unavailable with RSA."

What was weird was that this code has been in place for months and working fine in our app, published and everything. I had a look at a few examples on the web and they were all missing the dash in the "SHA-1" part of the string: "SHA1".

I changed it to remove the dash and it all worked again.

I don't know why, I think it is because the old project was somehow using the spongy castle dependency whereas the new project in Android studio is somehow using the standard bouncy castle but I'm not sure where this is setup.

What is also interesting is that according to PKCS#1, the correct string IS "SHA-1" and NOT "SHA1" but somehow this works in one and not the other!

Thursday, 16 October 2014

Yii 2.0 on Windows under IIS

It is possible to install and run Yii 2.0 on Windows. I did it today and learned some important lessons!

Firstly, you need PHP 5.4 installed because Yii 2.0 uses some features of 5.4. In my case, I had 5.3 and 5.4 installed, 5.4 was already setup in IIS but the path pointed to 5.3 (just to confuse me!).

You will need to use the console as well as IIS, so typing php -v in a console will confirm firstly whether PHP is visible in your path (it doesn't have to be, because you can call it with the full path) and secondly, what version it is.

If you haven't set up PHP in IIS, you will need to do that before you can run the site. Mine uses FastCGI and php_cgi.exe from my php installation. Instructions for that are here: Installing PHP in IIS

Yii recommends using Composer to install Yii 2.0 I already had that installed but if you don't, follow the instructions here: Installing Composer on Windows

If you don't use Composer very often, run \path\to\php composer.phar self-update to make sure it is up to date.

You can run the first command,
php composer.phar global require "fxp/composer-asset-plugin:1.0.0-beta3", 
to require the Yii plugin, in any directory (I ran it from the composer directory because it was easiest). Before copying and pasting the code above, please check with the Yii site that this is the latest composer package.

Now, you are ready to create your first Yii application.

You run the command and point it to where you want your new app to live:
php composer.phar create-project yiisoft/yii2-app-advanced \path\to\app_name 2.0.0
The part in bold is where you point to the new directory you want to create with your app in. When you run this, you will probably be asked for a Github username and password - you will need to create an account if you don't have one. It says it is something to do with rate-limiting but I don't really understand why this is necessary.

 Before you can run the app, you need to do a couple of things. Firstly, as per in the root of your application, run the init script in the application root directory. There is a Windows batch file which might not work correctly depending on whether your php.exe in the path is the correct version. If not, simply run \path\to\php.exe init and it will work. This script will create some relevant configuration files.

Secondly, you then need to create and configure your database. Create a blank database and preferably a user who only has access to that database and grant them all privileges. Edit \common\config\main-local.php and set the database connection details to match. Then, run yii in the same way as you ran init, either using the batch file or using \path\to\php.exe yii migrate, which  should find a single migration script which will be applied to the database.

Create IIS sites for your frontend (and optional back-end), which point to \approot\frontend\web and \approot\backend\web

 Set the permissions on the entire app folder and children to allow access to the relevant Application Pool user for each of the sites. By default, IIS 7 creates an application pool per site and each application pool has a user created for it. To set permissions, right-click the approot folder in Windows Explorer and choose Properties. Select the Security tab and click the Edit button. In the security dialog, click Add and in the names text box, type IIS AppPool\<app pool name> and press OK, if you have it correct, the name will appear as <app pool name> in the security list and you can give it read & execute access (usually the default).

If you then visit your site, you should see the nice vanilla Yii app, looking all Bootstrappy and ready to modify.



Friday, 10 October 2014

The Sorrows of Offline-First Mobile Apps

Open-Closed Software Model

Bertrand Meyer the inventor of the Eiffel programming language is a massive supporter of the open-closed model of software development. Once released, code is closed to modification and open to extension. If you want different functionality, you extend an existing class, perhaps create a sibling of the class you want to modify and make it do what you want.

That always sounds great but my experience is that with the best will in the world, most software gets to the point where it needs fairly major rework, this will not be in the form of the mythical "rewrite" that we all pine for and we are not planning on keeping the old classes around so we do what we should not do, we modify the code that we have.

It's OK To Modify Software If...

Actually I think this is OK if we have a definition of software at two levels. There is the architectural level, which defines top level program flows and use cases and then there is the detail level that dictates what happens in the fine detail. So the structural level might dictate an authentication sequence whereas the detail level says how this data is presented to the user. Clearly, making changes that only affect details is less likely to cause bugs. For a start, with encapsulation, the effect of a single method change should be very small. Also, clearly, making changes that affect the structure are much more likely to cause bugs. A recent article I read talked about how often we throw a method call in somewhere that works the first time but then we restructure something slightly, miss the badly placed method call and cause a bug.

The PixelPin App

Anyway, we have an app at PixelPin that started life as a fairly simple beast that did two main things. You either logged in directly to the app or you could log into a 3rd-party app using the PixelPin app, which did slightly different things to ensure the trust relationship was intact. Both of these flows became calls to a web service and it all pretty much did what it needed to.

Except it was slooooowwwwwwwwww

It turns out that despite our mobile providers quoting very high MB/s speeds on their data networks, the reality is that the connection setup is very slow so to perform, say, 4 calls to a web service would add about 5-10 seconds EACH! to the time taken to register on the app.

Simple Optimisations First

Initially, I decided to improve this quite simply by not calling lots of web service methods but saving all the data up and calling the method once at the end of registration, it would no longer upload an image (PixelPin is an image-based authentication system), since registration restricted the user to a gallery image. All good so far.

But it was still too slow when logging in.

Good is Sometimes Not Good Enough

Most people expect login to be fairly instant but, again, the slowness of the data networks meant that even a single call with a small payload of data might take 5 seconds or longer and sometimes timeout. Although this is understandable, it is not acceptable and doesn't make for a sellable product.

What we needed was offline login!

Offline-First Sounds Simple Right?

One of the things we had to accept from that outset was that mobile devices do not have very mature security controls. You can save data onto the device but ultimately, especially on a rooted phone, that data is accessible. We have encrypted the data but the key, by necessity, is also stored on the device. That meant we had to include a range of measures to both make it hard to extract this information and also to limit the damage if this data was somehow obtained.

That said, the ideal was that login always occurs offline for speed reasons and the data is updated in the background by a Push Service to ensure it is always up-to-date. We can still make the data stale after a period of time to force the App to check with the server whether the data is still correct but this provides the basis of offline-first functionality.

The problem was not with the theory, the problem was I already had an app that I had spent quite a lot of time designing, analysing from a security perspective and implementing, how was I going to retro-fit the offline and avoid the rewrite?

Retro-fit, What Could Possibly Go Wrong?

I made a mistake, I assumed that the use-cases were basically the same but with an "offline module" in place of the calls to the web service. Ignoring the push functionality (which I would defer for now and replace with a single initial web service call), this should be simple right.

I added the offline data, encryption mechanism with key generator and got a basic login to work offline - great eh?

But as with many a developer, the joy quickly wore off when my team were testing it and reported a whole host of problems. Problems after logging in, particularly when selecting your own image to use and other seemingly random errors.

Two of these were nothing really to do with offline. One was related to understanding the Android lifecycle and knowing that you need to store important data for when the system kills off an activity on a device with limited resources and the other was simply an error when creating one of the web service methods that was modified for the new registration system. I felt I was reaching the summit and was almost ready for release when....

The last 1%

I've heard this saying in a number of ways but it is something like the last 5% of the work takes 95% of the time. We get the bulk of the app up and running very quickly, as we should with our modern tools and debugging suites (although I'm not sure I count Eclipse as modern!) but it's when we get to the last little bits that we learn that Android doesn't support our kind of encryption, without adding a library (including all the joys of trying to include Java libraries into the build AND the deployment) but a specific problem reared its head and I don't know why I didn't think of it before.

The original design used a session system on the server. Once you logged in, you had a session, which meant when you called subsequent methods like UpdatePasspoints, you passed the session id, which was both a useful identifier but it also ensured that the given user was authenticated - which would avoid somebody simply calling the web API with a random user and changing their passpoints.

What happens when you login offline? The server is not in the circuit and doesn't create a session. I have logged in locally but how do I convince the server that a given user has actually logged in and that it is not just a fake app pretending to have logged someone in?

I initially considered simply creating a session if the user wants to do something like change picture or passpoints, which would require an online connection (no big deal) but they are already logged in, I can't ask them to login again with the online mechanism. Perhaps I could always login to the server when using the app direct, on the basis that the only reason you would do that is to change your picture or passpoints but then we're back to the original problem with slowness.


I basically have to go back to the drawing board and redesign how this works with offline-first. There is a really handy mechanism that Google provide for Android which is a way of authenticating the app with the server to prove it isn't a fake app and this is really useful for part of the job. The other part is how to prove to the server that the user has logged in locally without exposing any private data or storing additional data on the device. Perhaps some challenge/response mechanism would work but that is for another day.....

Wednesday, 8 October 2014

Publish-AzureServiceProject - ConflictError: The specified DNS name is already taken

I think I might have posted this before but when using the Azure Powershell Cmdlets, when deploying a new project, you might see this error.

It occurs because every cloud service on Azure is given the DNS name and since this is the same for all users, your cloud service name must be globally unique on Azure.

In my case, a test project was simply called PHPAzureTest, clearly something that someone else has used. You will need to rename the project in the ServiceDefinition.csdef and ServiceConfiguration.Cloud.cscfg (but you don't need to rename the directory that the project is in) and then try again....

Tuesday, 7 October 2014

Cannot deploy PHP cloud service onto Azure for Server 2012

I was pleasantly surprised when Microsoft announced PHP functionality to work directly on the Azure Platform-as-a-service (PaaS) system. A colleague who was also saying how PHP was so amazing and suggested I use it for a web service was most pleased that I was finally able to do this and most importantly, it was pretty much fully automated so I didn't go backwards to IaaS and have to manually set up and maintain the system.

I got the system up and running and deployed it to Azure with no real problems but then noticed that my other cloud services were running in Server 2012 and the PHP web service was still running on 2008. Not a problem, I assumed I just needed to deploy it fresh and specify Server 2012, which I did, but it didn't work. I got the error, "The SDK package you are trying to deploy is not supported by the operating system you have chosen". It then says, "The operating system you are trying to deploy to is: Windows Server 2012 R2"

The reason I wanted to update was that the security defaults are better on Server 2012. Some older SSL ciphers are removed and the support for newer SSL ciphers has also been improved. This didn't make much sense to me, I knew I was trying to deploy to Server 2012 so what was the SDK package that didn't work?

Summary: It was caused by an old version of the Azure SDK setting .Net Framework 3.5 as a required feature, which doesn't work on Server 2012, which requires V4.0 or higher.

Details: After many exchanges with MS technical support, their support engineer pointed out that the package (.cspkg) contained a reference to .Net Framework 3.5. Why? No idea. I couldn't find it referred to anywhere and I even created a brand-new empty PHP project and the same thing was happening.

I downloaded the source code for the latest azure-sdk-tools, which includes the source code for the PowerShell cmdlets that are used to create and package the project for deployment. This immediately started ringing alarm bells since it reminded me of the very messy history of the Azure SDK, the multiple renamings, sometimes moving classes to different namespaces and changing their functionality, the moving of the code and the changing of the project names and the plethora of NuGet packages for all kinds of random versions of things.

By picking through the source code, I eventually worked out that the only thing likely to have put the required feature into the package was the program cspack so I worked out the command arguments that were being passed to it and ran the command manually in the console. The command was:

.\cspack "c:\projects\phpazuretest\ServiceDefinition.csdef" /out:"c:\projects\phpazuretest\cloud_package.cspkg" /role:"WebRole1;c:\projects\phpazuretest\WebRole1" /sites:"WebRole1;Web;c:\projects\phpazuretest\WebRole1"

I immediately saw my big clue in the console: "No TargetFrameworkVersion specified for role WebRole1. Using .NET framework v4.0 for packaging". That makes sense. cspack is adding the required feature because the PHP Azure Powershell Script doesn't specify it. Why not? Don't know but I immediately thought that if cspack has a default, perhaps the Powershell script for Publish-AzureServiceProject is calling an old version of cspack somehow?

I had the source code for the newest Powershell scripts, although I knew they weren't installed but they implied that they looked in the registry for Azure SDK versions, found the highest version and used that version of cspack. That sounded OK until I realised that the min and max versions were both the same (2.4 in the new code), which made me think the older code might do the same thing and find the highest version between 1.8 and 1.8 or whatever. To prove this, I simply renamed the two older versions of the Azure SDK I had installed, leaving the one that I knew worked correctly and surprise, the script generated a "File not found error".

OK so the next question was, "what versions of PHP Azure Powershell were being used"? This took ages to find on Google but I eventually found a Powershell script which displays all the versions of Azure libraries that are installed and where they come from:

(Get-Module -ListAvailable | Where-Object{ $_.Name -eq 'Azure' }) | Select Version, Name, Author | Format-List;

Which showed me that the Azure Powershell scripts being called were versions 0.6 or something! What on earth?

The next steps were, sadly, very standard. Find all references to Azure SDK or Powershell in the Control Panel and in the file system and remove it all. Then open the web installer and install the latest version of Azure Powershell. I think part of the confusion is that the Azure Powershell Scripts and the SDK are kind of the same and kind of separate so the dependencies are a bit weak (as I found out). Clearly I had installed newer SDKs but somehow they hadn't updated the Powershell script references from the old SDK!

Anyway, a spring clean of the Azure SDKs and Powershell Scripts and it was all up and running again. Only about 2 days work!

Although it will now deploy to server 2012, it doesn't run up properly. The scripts that are executed by Azure to install the PHP runtime are not running or are terminating so PHP doesn't get installed. I fed all this back to MS. I'm sure it could be worked out and then the scripts could be modified manually on your local file system before deployment but alas, not something that I have the time for right now.