Computer Student

IIS, host headers and the IIS screen of confusion

2012-01-15T08:29:00.000-08:00

If you ever use hosts file re-directs and multiple sites hosted on the same IIS server, you may at one point have teared your hair out when you see the IIS logo screen rather than the site you expect. You might think everything is setup correctly and then go and play with loads of stuff until it eventually works.
The reason is fortunately quite straight-forward but not always obvious so here we go:
Firstly, when you host a site in IIS, you can either host it under the default web site or you can host it under its own site. Of course you can't have more than one site occupying the same place otherwise when you type in http://www.myserver.com/ how would the system know what site to return? Generally, the default web site is located at the root of the web server and any sites underneath are reached by using their directory name (and therefore the index.html/html or default.aspx that lives in that directory) such as http://www.myserver.com/somesiteunderdefault/. Many times, however, we would consider this messy and unprofessional and would want each site to have its own url such as www.myserver.com and www.myothersite.com both pointing to the same web server. If we do this without setting up IIS, all that would happen is they would both hit the default web site. In order to make them hit different sites, we have to set up bindings.
To set up a binding, we click on the web site in question and (in IIS7) click on Bindings on the right-hand side. This allows up to setup multiple bindings. A binding is simply a protocol (http or https), a URL and an optional port number (80 being the default). So on one of our sites we say that when a request comes in for www.myserver.com, we want this site to be served back to the user. We repeat the process for our second site, setting up a binding to www.myothersite.com and we're good to go. What happens now is that when we request a site from the web server, the server will first check any bindings and match these to the Host Header, port, protocol passed in from the client (in other words, what is the url that was typed in to request this site) and if a match is found, it is passed to the relevant site to handle. If no match is found and there is no site in the default web site area (as is the case by default), you get the IIS splash graphic.
One of the confusing things is when using urls that are set in your hosts file, rather than set in your system/internet DNS, you have to be really careful since IIS is looking for the url you typed in. For instance, if rather than typing in www.myserver.com, you typed in either an ipaddress that pointed to the same machine or perhaps another url that has no binding set up, IIS will not match it and you will get the IIS screen again.
If this all gets a bit much for you, an easy way to find out what is going on is to install Fiddler and request the site from your browser, see what URL is looked up and then ping that server to check it is accessing the correct IP address for the web server, then double-check that the binding for that web site is correct in IIS. Be extra careful of additional sites that might be used invisibly to the user (although they will appear in Fiddler) and ensure they also have host entries pointing to the correct place and if relevant bindings in IIS.

Security by obscurity and permutations

2011-12-29T08:59:00.000-08:00

There are two methods of security in IT that are not usually wise. The first is security by obscurity, in other words, if we don't tell people how we implement security, they are unlikely to break it. The problem with that view is that it isn't true! DVD encryption was broken, various sites have been cracked by trying various attack vectors. In reality, it is better to be open about your security if you can ensure you have a suitably helpful audience who can then advise on areas that might be weak. Oyster Cards and even Chip and Pin have exhibited weaknesses that could possibly have been found by peer review before they were implemented but both were 'closed source' and both have been compromised.
The second method of security is by permutations. My system is safe because you would need to try X combinations before you can crack it and X is a very large number. Of course, as we know, X is never a large number for more than a few years as newer faster computers and networks allow higher bandwidth for hacking attempts. In reality, your passwords should be 16 characters or more with punctuation, upper and lower case etc to provide any decent level of brute force defence but remember again that just because something is unlikely, doesn't mean it won't happen. You can mitigate however against brute force by slowing down the responses either progressively or even by a fixed amount of time (say 10 seconds) which thwarts faster hacking equipment. You can also block suspicious activity (such as 5 failed login attempts) either for a period of time like an hour or until 'manually' unlocked. In any case, you should be open with people who use your systems what the limits of your security are with statements such as, "WPS requires an 8 digit pin which would require X days of brute force to get past" which allows someone to consider this statement, perhaps challenge it and then allow the manufacturers to re-design or upgrade the equipment to increase the security.
Sadly, we rely on hidden decisions made by people who may or may not be competent in their job and whose decision may or may not still be valid and which may or may not even be visible, being lost in the mists of time.
Anyway, hire competent and trained people and ensure they are keeping up-to-date with current threats otherwise your company will look stupid!

Another botched large IT project

2011-12-29T05:00:00.000-08:00

http://www.theregister.co.uk/2011/12/29/csc_nhs_it/

The NHS electronic patient records system was like a lot of large IT projects going to deliver large benefits to patient care and why shouldn't it? Currently your GP and hospital records are mostly on paper or stored on independent systems which cannot communicate so you could be admitted to hospital and the doctors might not know that you visited your GP a month earlier with similar symptoms. However, also as with most large IT projects it ended up running late and being much over budget until the Government had little choice except to shelve it and cut their losses.
We use so much IT nowadays that we still haven't realised that we are mostly rubbish at running IT projects. In fact, we are pretty bad at running many projects but IT seems especially poor. What seems unreasonable to me is that the risks of IT projects are well known but yet they are not managed or removed to provide the ability to produce a useful albeit large IT system of some kind.
There are various problems and I have outlined a few below. We need to understand these problems, many related to human nature, and then decide what we will do about them.

Price must not be the main factor when choosing a supplier. While there is so much pressure on price, people are inclined to avoid factors that are seen as low benefit/cost such as quality and process improvements. When things are running a bit late, nothing other than pure functionality is going to be given any space. With a more open relationship with our customers, we can show what we are charging for what and therefore negotiate if unforeseen things happen. A customer will pay a 40% margin if they know anything about business, we shouldn't need to keep this a secret.
Time must not be unrealistically short. Customers always seem to want 6 months of work in 3 months and we are inclined to agree because we want the work. We perhaps think we can just about manage but of course we can't. People leave, you can't always recruit new people in time etc. Part of quality assurance should require that a supplier will only agree to timescales related to the design and planning and NOT by an arbitrary deadline imposed by the customer unless this is specifically factored in. There needs to be protection against unscrupulous suppliers who will say whatever the customer wants to hear in order to win the work. "Prove it or lose it"
Someone once said, "the customer is always right" but that is nonsense when applied to all business scenarios. Specifically the customers is sometimes wrong and sometimes very wrong about many things. This balance needs re-addressing and as experts, we need to improve our profile so a customer will trust us when we say something cannot be done or not in the timescale required. I don't argue with my doctor over medical issues so why is it OK for a customer to argue with an IT supplier over a system?
The more people you involve, the more complex it gets. In order to counter this, you either involve less people (which risks alienating your users) or you make the system simpler as far as possible so there is something for everyone to use. For instance, an electronic patient record system could be very complicated but even if it was a multi-page document like a filing card system, that would be of enormous benefit just to record stuff like "administered 100mg of XYZ". I was once asked to write a database system for a Bariatric clinic even though there was a European-wide database. The Ero one was so complicated, it was useless to any one person.
Requirements capture needs to be much better understood and only people of a relevant level of understanding in the customer organisation should be allowed to have formal input into the process. Asking every man and his dog for their input even though they might not understand that complexity is cost and that a system is more robust if it is coherent is clearly nonsense. Ideally, you should have a single point of contact with a customer who can have the arguments with various stakeholders on your behalf from a position of understanding. We have some large customers who have 20 people on the authorisation list for functional documents. Can you imagine how long they take for approval?
We need to understand and manage change better. If we are taking 2 or 3 years to deliver a system, its requirements will change over time. It is futile to pretend we will set requirements in stone. This was done for electronic MOTs and we ended up with dial-up connections required for all garages even though the internet and broadband was in full swing. We need to accept that change will happen and at least consider some things that might happen (even if their details are not fully known). We need to communicate the cost of change to a customer so they don't change things for the sake of it and we need to try and get a system out of the door sooner, even if not all the functionality exists, so at least we can achieve milestones and get paid rather than having a continuous moving goalpost. We should obviously be writing code that allows for change as well and this needs to be taken seriously. All those short-cuts and "quick hacks" should simply not be allowed at all.
I think there needs to be better scrutiny of suppliers who are getting things wrong and even potentially legal action against people who might pretend to be doing it all right but are covering up incompetence and deliveries of systems that are basically rubbish as soon as they are released. If someone releases something under a quality assurance banner that cannot be maintained to any reasonable degree then they should have to pay some of the costs of re-designing it.
Quality needs to underpin the processes at every level and sadly this is still an after-thought or box-ticking exercise for many organisations. If they could understand what quality really means. The systems we deliver would be so much better.

Better Use of Email

2011-12-28T06:23:00.000-08:00

For some reason, email hasn't changed fundamentally since it's original inception back in the computer dark ages of un-powerful PCs running over slow networks. It is basically a text-based protocol with no security whatsoever. You can spoof who the email is from and read any of them with a network sniffing tool. Everything since then has been tagged on the top like PGP for security, IMAP for mail servers, Exchange Server (whatever that adds). Basically I don't like email because it has few useful features and it is extremely over-used for a variety of tasks that it is definitely not suitable for.
In terms of functionality, I would like a feature to be added to email which is simply an "expiry" date. Many emails that are sent are only relevant for a period of time (long or short) and the ability to allow the email systems to delete expired mail would clear logs of unnecessary backlogs. For instance, using an email to tell you that a backup succeeded or failed is mostly irrelevant after the next backup occurs so if you get 10 of these over the weekend, probably 9 of them can be discarded. However, I do not have the influence to make that happen but I do have an influence over how I use email and require others to use it.
My old manager had something like 5000 emails in his inbox. He was an extreme example of badly used email. What is the point of having that many when you can only handle a few at a time? Here are some suggestions as to why he had too many and what you can do to avoid the same mistake:

Self-importance! If you think you are too important, you want to be copied in on everything. This betrays major issues either with trust or just how badly the organisation is run.
Incorrect use of email. Having emails for things like successful backups is perhaps acceptable if you only have one server and it backs up once per day but really, there are proper tools that monitor these things and which don't need to spit out tonnes of emails into your inbox. A monitoring system that has to send emails constantly to convince you it is running is not really much good!
Incorrect use of email too. Lots of people claim they write emails to leave a 'paper trail' of decision making to cover their backs later on. This is tosh. Whatever you have written in an email is likely to be mis-construed which is a defence later on when someone decides that you didn't tell them to do X in the email. Pick up the phone if that's what is required. You can communicate 10 times more efficiently with your voice (and even more in person) than you ever can on email. Why take 10 minutes to try and formulate what you could describe and discuss in 30 seconds? If someone doesn't understand, they can ask you straight away rather than writing you a 10 minute reply!
Make correct use of importance and expiry. Although expiry is not part of the email standard per-se some providers give you an expiry option which allows you to mark an email like "you have left your headlights on" to expire after 4 hours and not to be looked at by everyone who has been on holiday for 4 weeks.
Use correct subjects. Your subjects should be very concise, especially on emails going to more than one person since some of those people might very much not be interested. An email with a subject like "Does anyone have a set of jump leads" is more useful than one with a subject of "request" and a body that contains the question.
Consider alternatives. Sometimes you could write your own application to do something that you currently use email for (such as monitoring backups, lift sharing or general enquiries). You could use a separate system (like Yammer) to allow people to ask non-work related questions to keep your inbox down. Also, an email like "Potatoes are ready" being sent every day at lunchtime can be annoying. How about telling people that the potatoes will always be ready at 12 unless there is an email to the contrary?

I heard about the head of a car company who had banned his employees from using email for the fact it is really inefficient. Will be interesting to see whether they can break the email mindset!

Why we must write clean code

2011-12-22T04:05:00.000-08:00

Engineers are often (rightly) accused of being pedantic. We insist that we use an integer rather than a short or that we should or should not use a pattern. The sales type people don't understand this and will tell us that as long as the customer sees a system that does what they want, they couldn't give two hoots (or 2 dollars) for how it is implemented. This might seem like a pragmatic approach since in a sense that is true but sadly there are a few assumptions in that statement that need to be exposed and which reinforce the need for strict coding practices.Firstly, the life of software is usually much longer than it is assumed. Look at Windows XP for instance, when it was first released in 2001, MS would have a plan that in 5 years, it would be pretty much replaced by whatever their next OS was going to be (Vista or Windows 7 perhaps). In other words, don't get crazy with it, if a bug will survive 5 years, we can leave it in and then fix it later. In fact, XP still holds around 33% of internet connected machines 11 years later. It is still being used and the bugs that might have laid unlikely for 5 years are much more likely to surface over 11 years. In a similar way, do we really design software for 10 or 20 years? If we designed a car to survive that long, we would have to be very careful about all components yet in software we can still be quite glib about the level of quality that is acceptable.Secondly, it is quite glib to say something like, "as long as the customer gets what he wants", both because over time this will prove less true and also because it is not as black-and-white as that. MS Word pretty much OK but there are things I don't like about it and I live with them - it is not either perfect or useless. The level to which we can achieve this 'perfection' however is important in people trusting our software. Also, this acceptance is based on functionality as much as reliability and there is every chance after a while that a customer will want to change something, a good test of how well the software was originally written.Another reason why we should strive for quality is to ease maintenance. We have staff turnover and we have to change and fix things. There is nothing worse than being presented with an obvious defect but being totally overwhelmed by the code we have to look through to find the cause of the defect. This is why we should keep our objects classified specifically and deliberately. Data objects contain properties to describe their data and little else. Control objects provide methods to do stuff and should not generally be persisted. Interface objects translate between specific systems and can be added or removed as the boundaries change.We have probably all been in the situation where we notice an issue in code and are told it is so low a priority that we should not fix it (partly this is because of the build time overhead - something we should consider) but as with rust, what starts with a small bit can grow and grow until a full rewrite or re-factor is required. We must be careful that we don't break code that might be 'wrong' but works because something else takes account of its wrongness but at the same time, we should feel at ease making code more readable and obvious. Of course, the best way to do this is to get the code correct in the first place so it doesn't need to be changed and this again requires company buy-in because it involves training and reviews. It requires keen eyes that can spot where mistakes are creeping in and finding ways to mitigate the risks. Even subtle coding styles can introduce errors so sometimes we might require very high-level training about separation of concerns, correct use of comments and ensuring that we are using tools that make this easy to do so the time required doesn't become unacceptable.We need to treat our software with more respect and we need processes and people who understand that so that what we produce is still running as expected in 10 or 20 years time, hopefully with a lot of repeat work!

Software Patents Mostly Make my Blood Boil

2011-12-19T03:59:00.000-08:00

I think I am probably of the opinion of many others that protecting your own ideas is fair enough (although I could be convinced that people should help each other) but let us say for instance that someone spends millions developing a working fusion reactor, it would be unfair if everyone just copied that idea and capitalised on someone else's investment but sadly for most software patents, they involve, in my opinion, virtually no 'intellectual property' but which are basically money making schemes that make lawyers rich and everyone else poorer. Let us start with something like MP3. Presumably this took time and effort to develop and is a protected technology, which although you might disagree with the marketing model, few people would argue with. Let us however look at some other patents like 'pinch to resize' and 'using clever routing to improve connection speeds'. Firstly, these require no intellectual prowess to conceive. In the same vein, I could conceive of flying cars and teleporters but without developing these I should not have the right to the intellectual property for these just because I can run fastest to the patent office. More significantly however, the idea itself does not provide a competitor with much. They still have to code it/implement it and this surely is where the cleverness comes in.It seems that most of these patent cases appear in the USA, the most materialistic and capitalistic country in the world? A country that panders to the dollar more than to the community or to progress in general. Even Google who are famously open and generous with their offerings seem to get stuck in this area since the old-fashioned companies attack them for sharing this amazing knowledge with the world at large rather than selling it for the money that only marketers dream of.Shame on you MS, BT, Apple, Google etc. Get over yourselves and start being proactive rather than reactive. Make stuff that people want to buy and don't pretend that you are unfairly losing market because someone made something that was the same colour as yours!

I need a new phrase...

2011-12-12T02:50:00.001-08:00

I need suggestions for a new word. In software engineering, we get taught about things that are important like code clarity and maintenance but when we start work, we soon realise that only functionality that can be seen by the customer has any real worth. Things that have been done messy is resigned to being, "yeah, we'll fix that later" and we all know it won't. We all know that companies do not have capacity for people to tidy up (and potentially break) existing code.It's a bit like functionality is the bourgeois and the clarity and neatness is simply the illegitimate or maybe the working class of software engineering. Yeah, we care, just not that much. The sad fact is that this needs to be done correctly at the start or it never will be, which is sad because it causes so much pain and lack of quality issues further down the line.What word for this sadly neglected area of software engineering? For this butler/chauffeur of the software community? For this important but under-valued foundation?I reckon something like UnderCode?

A code quality test

2011-12-09T03:31:00.001-08:00

Looks at the following code (which compiles) and see what is wrong with it:

var myDate = new DateTime(DateTime.Today.Year, DateTime.Today.Month + 1, 1 );

Spot it? I didn't! The obvious intention is to create a date which is the 1st of the next month, in general it works as expected but when the month is December, month + 1 = 13 which will understandably cause the constructor to throw an exception. The compiler won't catch this because the types are correct.What point am I making? Well, quality is about learning from mistakes. What can we learn from the above code? Obviously we could simply attribute it to human error and say a code review might pick it up (but possibly/probably not) but what else can we decide? Well, one thing about strongly typed languages is that we should aim to get the compiler to test as much as possible on our behalf to catch these sorts of errors so the real issue is that there should NOT be a DateTime constructor that takes 3 integers for arguments. It sounds reasonable on paper but it means that for 99% of combinations of parameters, the constructor will throw an exception. What we should have instead is a constructor that takes enumerated values and then we could make use of the AddMonths method (which we could have used in the above code) to get something more like this:

var myDate = new DateTime(DateTime.Today.Year, DateTime.Today.Month, DayOfMonth.First ).AddMonths(1);

Which would give a much more solid representation of what we are doing and would make it harder to fudge. That way, we would have to cast a number to an enum if we wanted to hack the code and this would be much more obvious in a code review:

var myDate = new DateTime(DateTime.Today.Year, (MonthOfYear)((int)DateTime.Today.Month + 1), DayOfMonth.First );

So you can get quality even in subtle ways - now all I have to do is update the code review process and get MS to drop the silly 3 x integer constructor.

Understanding Mocks

2011-12-08T02:38:00.001-08:00

When I first started using mocks, I must admit finding them slightly confusing. I knew what they were supposed to do but some of the language was a bit confusing so then I didn't know if I actually understood them at all. I wanted to write a very basic introduction to Mocks to get you in the mood for using them.Firstly, a mock object (and I will be referring to Rhino mocks) is designed to act in place of an actual piece of software whether a service or even one of your own sub-systems. This is good for two reasons. Firstly, you might not have access to the real-life object (perhaps a web service that you cannot call from your test environment) but secondly, they allow you to test only a part of your system without risking the introduction of defects from other parts of the system (which might be outside of your control or not finished yet). For example, suppose you are writing a user interface to a banking system and you want to test your user interface. If you connect it to the actual back-end, you might see loads of errors that are nothing to do with your code and which make it hard to know whether your code is correct or not. By "mocking" the back-end, you can tell the mock how to behave and therefore test whether your code does what it says without worrying about someone else's defects.To create a mock object you implement whatever interface you want to mock and then it is usual to encapsulate a generic mock property like so:

public class MyMockClass : IInterfaceIWantToMock
{
    private IInterfaceIWantToMockmock = MockRepository.GenerateMock();
}

If your mock is for a service, you should also implement IDisposable and setup your end point replacement for web config files. The mock could use any of the service bindings, in this case I use a named pipe:

public class MyMockClass : IInterfaceIWantToMock, IDisposable
{
    private IInterfaceIWantToMock mock = MockRepository.GenerateMock();
    private readonly Uri uri = new Uri("net.pipe://localhost/TDD.IInterfaceIWantToMock");
    private readonly ServiceHost host;

    public MyMockClass()
    {
         host = new ServiceHost(this, uri);
         host.AddServiceEndpoint(typeof(IInterfaceIWantToMock), new NetNamedPipeBinding(), uri);
         host.Open();
    }

    // Optional property to access mock
    public IInterfaceIWantToMock Mock { get { return mock; } }

    public void Dispose()
    {
        host.Close();
    }
}

You would then need to implement any members of your interface and in most cases will simply forward the call onto the mock object


    public void IInterfaceIWantToMock.SomeMethod()
    {
        mock.SomeMethod();
    }

Now the bit that can seem quite tricky, the "expectations". The reason it is tricky is because although it is called an expectation, you do not have to "require" the call to be made and you can either be very specific about expectations or very general. It is better to be specific but in some cases this might mean lots of extra code. You can also specify things like how many times to expect the call to be made and what to return when the method is called in this way. It is probably neater to specify expectations inside the Mock class itself but you could expose the mock in a property and set them up elsewhere (just a choice of how tidy you want it). Let us specify a very simple expectation on our mock:


    // In the Mock class
    public void ExpectSomeMethod()
    {
        mock.Expect(proxy => proxy.SomeMethod()).Repeat.Once();
    }

When this method is called, it tells the mock that if someone calls the mock method SomeMethod, you should allow it once but if it was called a second time, the mock would throw a null reference exception. You can specify a specific number of times or even Any() but don't be lazy with Any(), hopefully you will know how many times the method should be called. We will now look at a method which takes an argument and returns something to see what these look like. Bear in mind that lambda expressions are used extensively to make the syntax neater.


    // In the Mock class
    public void ExpectSomeOtherMethod(Guid id)
    {
        // Note that in this example, SomeOtherMethod takes a MethodRequest (which includes a Guid id) as a parameter and returns a MethodResponse
        mock.Expect(proxy => proxy.SomeOtherMethod(Arg.Matches(a => a.id == id))
            .Return(new MethodResponse { success = true })
            .Repeat.Times(2);
    }

If we dissect this, we need to remember that this method is NOT what is going to be called during our testing, that would be the method called SomeOtherMethod(MethodRequest req). This is merely saying that when we do call that method with the id given in the call to the expect method, to return a response which includes a flag called success. This time we have said this will be called twice. We have specified that the arguments to the mock are important by matching id with the subsequent call to SomeOtherMethod() (we could setup the expectation to not care about some or all of the arguments but again that is in danger of being lazy). One of the things that this implies is that we could call this expectation method more than once with different Guids and the mock would then expect SomeOtherMethod() to be called twice for each different Guid passed in. This allows us to be very specific about what we expect and what we don't expect. Suppose for instance we are dealing with 2 different Guids in a test, we want to make sure the correct one is passed to a certain mock method, therefore we ensure that the call to the mock matches the parameter to one we told it to expect (like we did about in ExpectSomeOtherMethod()).There are of course two sides to this story, one is that we want to tell the mock what we expect to pass to the mock but also we must specify what we would then expect the mock to return. This can cause us a risk because we might have to assume what the real-life object will return and we might get it wrong. Therefore, ideally, we would have input from the people who are writing the real object to ensure our mock expectations (specifically what they return for our arguments) are correct.The consumer of the mock then has to create the mock object, setup expectations, optionally verify that all expectations were met and then dispose the mock if required. If using something like NUnit, it is easiest to setup, reset and dispose the mocks in the [Setup], [TearDown], [TestFixtureSetup] and [TestFixtureTearDown] methods to ensure they are always reset correctly (otherwise you might have a previous expectation hanging over from a previous test). An example of doing it all in a single method is shown below:


    // In your test class
    [Test]
    public void SomeTest()
    {
        var mock = new MyMockClass();
        var id = Guid.NewGuid();
        mock.ExpectSomeMethod();
        mock.ExpectSomeOtherMethod(id);

        DoSomethingWhichWillCallOntoTheMock(id);

        mock.Mock.VerifyAllExpectations();        // Could also be done as a method on MyMockClass
        mock.Dispose();                           // Only if implements IDisposable (i.e. creates a service host)
    }

The assumption in this test is that DoSomethingWhichWillCallOntoTheMock() will do whatever it does including calling onto our mocked service. From our examples, it will call SomeMethod() once and SomeOtherMethod() twice with the given id. The call to VerifyAllExpectations is optional and when called will ensure that every single expectation was received. This would fail if you only called SomeOtherMethod() once for example or if you called it with the wrong arguments and it therefore hit a different expectation.Hopefully you get the idea now...

Exceptions for Exceptional Things Only!

2011-12-06T03:49:00.001-08:00

Exceptions are a strange beast. They are actually worded correctly and are supposed to relate to things that shouldn't happen such as malformed values being parsed and calling services with incorrect data. However, the line has been blurred between what is genuinely exceptional and what is merely incorrect. For example, if you have a service that is called by a web application and a certain method on that service takes a string which represents an integer, it would be reasonable to throw an exception if the string is null or not parsable as an integer. It would be part of the contract for that method (and could be enforced other ways as it happens) but represents a genuine "exception", a problem in application logic or design. Consider a similar example where a user inputs an integer into a text field on a web app and submits it to the code behind. In this case, an exception should NOT be used since this is not exceptional at all but completely expected (people type things in wrongly all the time).Why not use exceptions in this case? Well they seem pretty useful so I guess we could but there are two reasons why we shouldn't. Firstly it confuses our minds as to the correct use of exceptions, we start blurring the line between logic/business/coding errors and simple error handling. The second reason is performance. When an exception is thrown, the framework has to load up a debugging engine which loads the call stack data and which takes a significant amount of resource. This resource is closed once the exception is handled and then opened again if another exception occurs. Why? Because it is designed to be for exceptional conditions and not for normal logic flow. As an example, I ran a test app to compare Int.Parse with Int.TryParse while parsing a bad string. I caught the exception from Parse locally and ignored it. It took 4.24 seconds to run 100,000 exceptions and only 2mS to run 100,000 Tries (2000 times slower to allow exceptions to be part of normal logic).That last line gives a clue to how we might avoid exceptions. We can use alternative methods and operations. It should only be at boundaries that we need to check formats (user interface and interfaces with other systems) anything internally to our application should behave. If we are therefore testing text fields for correctness then we can either use the TryParse family of methods or use a RegEx to specify exactly what we want something to look like. That way we can call IsMatch as a boolean check rather than allow a parsing operation to throw. We can similarly check other fields and types using simple if ( ... ) return false or if ( ... ) DisplayError() bearing in mind that our checks are only useful if we can inform the user or a system admin that something has been entered correctly.As an acid test, you should generally never see an exception caught and ignored. Catching, logging and then continuing might be valid (as long as you have a way of fixing the underlying problem) but if you ignore the exception, you have made it unexceptional.

object does not match target type

2011-11-29T02:27:00.001-08:00

Another of MSs suitably abstract errors which don't really tell you what is going on. They have gone to the effort of catching a problem so why not describe it better?
Anyway, I got this while calling onto a Rhino mock class which I had supposedly setup an expectation for. It was unusual since my understanding would be if the expectation matched, I would get a response and if it didn't then I would get null back and it would fall over elsewhere. There is obviously a middle ground.

The code in question was this:

internal void EstablishDetect(DetectRequest detectRequest)
{
    mock.Expect(proxy => proxy.Detect(detectRequest)).Constraints(Property.AllPropertiesMatch(detectRequest));
}

A fairly simple use of the expectation. The problem appeared to be the fact that one of the request properties was a class, not just a number or string etc. and in the code I was using, effectively, I was using this for the expectation:

req = new DetectRequest { Id = Id, Stage = new ClassA() };

but calling this on the mock:

req = new DetectRequest { Id = Id, Stage = new ClassB() };

The different class was intentional and I would have expected if these didn't match, the AllPropertiesMatch() would fail and therefore the mock would return null. In fact, it generates the above error, "object does not match target type". All I had to do was to set-up the expectation with the same class I was passing into the actual call and it was fine.

Windows XP Activation - evil

2011-11-23T01:22:00.001-08:00

The problem with any reasonably secure activation system is a) it will cause a lot of problems for certain people/hardware and b) it will probably be hackable anyway. I had just endured the pain of losing my linux files due to the terrible way in which windows installations simply overwrite the master boot record ignoring any existing systems in my case actually overwriting it with a new type of boot record (GPT) which was incompatible with Windows XP 32 which then couldn't install. I thought my pain was over, all I needed to do was boot XP, install any missing drivers and be done with it. Oh no.
Activation. I have a legit copy of XP (although I had lost the key) so I expected to at least be able to get 30 days to find my key so I could activate it. In order to get past the installation, I used some key I found on the internet (expecting it to be blacklisted for activation).
I booted up to the login screen and was immediately told I had to activate before I could log in and surely enough, my only options were to activate or it would log me back off. This didn't seem right, I had never come across this before.
Surprise, surprise, there is a bug in the way activation works. The simple cause was that I had no network installed because neither of my drivers (wired and wireless) were installed by XP. Firstly, this causes activation to get confused and think you have done something bad which requires immediate activation (which is understandable if it worked) but the major problem is then you cannot activate over the network (no big deal) but if you click "telephone", it displays a page with your hardware id which is - BLANK. Why? Because part of this key comes from the network card which is not installed yet. I called Microsoft but they were mostly useless and told me to ring tech support.
What I had to do was reboot into safe mode (press F8 when starting up) but NOT safe mode with networking, that won't work. In safe mode, I had to get the network drivers (via CD because the network wasn't working) and attempt to install them along with the video drivers. The wi-fi installer wouldn't work in safe mode but the lan looked like it did. I also tried this hack to disable the activation check by changing a registry key.
Rebooted into normal and it still said I needed to activate but when I pressed "Activate Now", it told me I was already activated (presumably because of what I changed in the registry) so I changed it back. Rebooted again and this time it told me to activate and brought up the screen to activate. This time, the lan was enabled so I brought up the internet activation window and it told me my key was invalid (as I expected). Since I couldn't find my key, I found a key gen on t'internet and it generated me a key that worked! I then had the joy of installing service pack 1 and then 3 although after 3 was installed, the screen started working properly and the whole thing seemed better. I then obtained and installed the remainder of the drivers from the Samsung site (they are hard to find!) so that there were no more little yellow icons in device manager.
Now all I have to do is reinstall Ubuntu which will take MUCH less time (and no activation).

The pain of Windows XP installations! MBR/GPT

2011-11-23T01:00:00.001-08:00

I have spent about 10 hours of time trying to install Windows XP as a dual boot on my Linux laptop and all the problems I had and the fact I lost my Linux partitions are all related to Microsoft's selfish installer that overwrites the master boot record rather than installing alongside whatever is already there but it is worse than that.
I have two Windows XP CDs a full 64 bit one and an upgrade 32 bit one and the reason I am installing it, although I am more than happy with Ubuntu is because, as you might expect, there is a piece of software I want to use that only runs on Windows and Mac.
Simple, I thought, I've done it before. You partition your disk from the Linux boot CD using GParted and then run the windows installer which overwrites the MBR. You then re-run the grub install from the bLinux boot CD and you're back in business. Except I wasn't.
The problem I had related to something I didn't know existed. Guid Partition Table (GPT) which is a more modern and better use of the master boot record that doesn't require a limit on primary partitions and the funny use of extended and logical partitions - fair enough. What I didn't realise was that Windows XP 64 bit (and presumably other new Windows) installs this without telling you that it is totally incompatible with older systems like Windows XP 32. This wouldn't have even been noticed since grub can handle GPT except for one problem. I installed Windows XP 64 bit and most of the drivers for my hardware weren't there (or even on the net, they were probably never written). So I had got a working dual boot system with XP 64 but which was mostly useless.
No problem, I thought, I'll just run the XP 32 bit installation and choose the XP 64 bit partition as the destination and it can overwrite everything. Right? Nope. Run up the XP installation CD and all it can see is 131Gb of an unknown partition type (it's 160Gb disk but more on that later). I was confused. What was happening? Ah. Maybe I need to re-run grub to reset the MBR before the installation will read it correctly. Nope. Still the same. I couldn't work out what was happening, even from Google so the helpful people at ubuntuforums suggested the problem was related to my new GPT boot record which XP 32 didn't understand. They were right so now I was a bit stuffed, how can I overwrite the GPT with MBR so that XP 32 will understand?
This is where I cocked up a bit. I ran up GParted again and found out there was an option to re-write the boot record (with the MBR format) which I chose. It warned about data loss but I presumed it meant there was a risk. What actually happens is that it wipes all the current partition information and then writes the MBR. I tried disktool to get them back but somewhere, they must have experienced a quick format and lost the data. I wasn't too bothered. I have a backup of most of my stuff and use Google quite heavily, at least.
So I then spent the next hours trying various combinations of GParted and diskpart (the windows equivalent) as well as fdisk to try and get XP 32 to read the 160Gb disk. It was like nothing made any difference. Was I actually resetting the MBR at all?
This was when I learned something else. Various combinations of HDD, 32 bit OS and BIOS settings means that the installation would only be able to 'see' the first 131Gb of the disk. What I ended up doing was what is suggested for a clean dual boot install. Do Windows XP first. I removed all partitions, created a 50Gb partition for Windows to use and then ran the installation. Did the pain end there? Nope, see my next post.

Internationalisation - ooo, err

2011-11-15T14:30:00.001-08:00

Perhaps it is Internationalization? We live in a global market now with a web site in Nepal appearing in the Google results of Brazil and one of the knock-on effects of this is to make sure your web sites are foreign speaker friendly.
Unfortunately, this whole area is very complex because languages are complex, alphabets vary and you might have a Chinese person living in an English speaking country - all of this adds up to a headache which for most people is simply avoided. It needn't kill your brain however, I thought I would guide you through some of the pitfalls and give you some pointers to how to make your web sites work for more than just English people.

Use a pre-built application framework where possible, one that already includes functionality for multi-locale functionality. It is much easier if you use something pre-built even if it is slightly different than you might have designed it.
Learn what the words mean. A locale is a means of formatting dates and times. It is generally related to a single country or groups of neighbouring countries and to a single language. For instance, you can get a locale for Spanish (Spain) and a different locale for Spanish (Mexico).
A language is a complex idea since some languages are really just dialects of each other. For the most part we are concerned about languages that have different written forms rather than spoken forms since our websites are predominantly written down.
For practical reasons, if we make our site multi-lingual, we should restrict the languages we support to a minimum. For some of us, we might be happy with English, Spanish, Portuguese, Russian and Chinese as covering most of the globe. For others we might have Mandarin Chinese and Yue Chinese (Cantonese). Of course, if we are a Chinese site, we might have 12 Chinese languages. We would probably also decide that Scots English is close enough to English English to not need a separate translation (although again in some specialist areas this might be required).
It is easier to de-normalise databases for language work and make each row a single language selection. This row might have a country name, locale, language name (possibly translated into the other language e.g. Deutsch instead of German). Some of these values might be duplicated but it makes for easier administration.
If you want your site to be highly multi-lingual, consider adding a system whereby volunteers can translate the words for you rather than you having to pay people to do it.
Use as few words as possible on your site and use pictures where these are clearer. A picture of an exit door saves you umpteen translations of the word "Exit".
Try and auto-detect the language from the browser post variables for convenience.
It should be easy to get to the language you want. If you have, for instance, logged onto a site in China but are an English speaker, your site might have detected Chinese as the browser language and left you staring at something you cannot possibly understand. By using something like a flag in the top corner of all pages and translating the language names into those languages, it should always be easy to find the language you want.
Try and monitor the translation status of languages on your site and if you feel they are not adequately finished, include a mechanism to disable the language. It would look pretty lame if your site was only 10% Tajiki when switched to that language.
Default all messages to English if there is no suitable translation (don't put blank strings there instead!)
Do not re-invent the wheel. Even if you are building your site from scratch (which you hopefully are not!) the data for countries and locales is all out there for free, do not type in the names of 6400 languages.
Unlucky for some

Find in Files - Linux Style

2011-11-11T12:54:00.000-08:00

Lots of people ask this in forums and there are various complex answers that come back - enough to make the most hardened GNU cohorts shiver. Anyway, dead easy, just use grep from the command line.

grep -r 'search term in here' *

-r is to recurse sub directories and the * says look in everything. The result is written to the console but you could always redirect it to a file using "> something" at the end (without the quotes of course).

Cannot cast enum from type A to type A

2011-11-11T05:44:00.000-08:00

This is a common error that doesn't seem to make sense. How can you not cast an enum from a type to itself? Makes more sense when we realise that types are often shared by copying libraries between projects. We might create a service contract version 1 and then add an enum entry and create version 2. We update the sender with version 2 but not the receiver and then we get the error listed above.
Trying to get another piece of code working was taking much longer than it should have and I thought that the change was pretty easy. Unfortunately, the fix I was making was in a shared library which then had to be copied into another project which would run the code (and fail!) and without any easy way to get the debugger to work without project changes. What I realised was that there were various combinations of things I was trying but I wasn't recording them. Use update panel or not? use declarative event handler or code one? Deploy to deployment folder or bin folder? Am I testing the code that I've changed or not?
All I needed to do was actually write down the combinations I had tried and then be careful to point the web site I was testing to the correct place (we effectively have two web sites in the project). Once I did this, I realised that I had missed a combination, one that worked, and that the code was correct, I just kept breaking one thing as soon as I fixed something else.
Be methodical people!

Web site referencing wrong versions of DLLs

2011-11-10T02:37:00.000-08:00

Classic problem here. Visual Studio web site project that includes various projects and a number of MS and third party dlls that are referenced. A specific project was referencing Microsoft.Practices.Unity version 2.0 whereas when we built the project, we got the error "Thirdpartylibrary.dll references Microsoft.Practices.Unity version 2.0... which is higher than the referenced version 1.2".
Checking into it and the only version we reference is version 2.0 and I couldn't see where the version 1.2 was possibly copied from (into the website bin directory). Eventually I resorted to modifying a Powershell script which listed every dll in our project that was referencing Unity and looked for one which was version 1.2. It turned out to be EnterpriseLibrary.Logging, something we no longer used (and something which wouldn't obviously reference unity). Removed all references to that and it was fine.

There was an error reading from the pipe: Unrecognized error 109 (0x6d).

2011-11-03T03:23:00.000-07:00

It's annoying when you see an error that lots of people report but you can't find out any useful answers! I had this today when calling a Mock via a named pipe. It was a case of having changed so little and not seeing what might have caused something that looked so serious.
Anyway, after following advice from another post and enabling WCF logging in the app config for the unit tests, the log DOES include the error and its in red so nice and easy to find. The problem? I was returning an object from my mock expectation which had about 20 properties, only 5 of which I wanted to consume and therefore which I set. It turned out that one of the properties I hadn't set was an enum that defaulted to 0 although 0 was not a valid enum entry so the serializer was throwing the error.

Windows Installer not uninstalling files

2011-10-31T04:46:00.000-07:00

I was testing a windows installer change on a box and found that uninstalling did not remove all the files, even if I manually deleted the folder and installed from clean. In reality there is no such thing as a clean box! I had assumed that there was no reason this service had been installed on the box before so how could it all have been screwed up? I searched for various guids in the registry and didn't find them until I had the genius idea of searching for the filenames that weren't being uninstalled (some were uninstalled but most weren't). I found them under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Installer\UserData where the paths are linked to product ids so that files that are shared are not uninstalled if only one of the products is uninstalled. I manually removed all of these that pointed to my files (don't bother with MSs windows installer cleanup - does bugger all useful stuff) and worked out it was all fine and then had to go and look at what had happened.

Firstly, I had made an assumption that this box was clean but actually it had been cloned from another virtual machine and had remnants of a failed install/uninstall of this service in the registry. What I thought was a problem with my changes was a registry problem. Secondly, we need to understand how this can come about. Well, Windows installer adds the stuff correctly to the registry (I assume this is hard to muck up!) but there are two ways in which this can be corrupted. Firstly you can have problems with the uninstall process which leaves stuff in the registry or more commonly, people manually delete program folders leaving orphaned registry keys.

Execute Deployment Script In Single-User Mode

2011-10-24T06:44:00.000-07:00

This is an option in Visual Studio database projects under the sqldeployment file. The intention is simple enough. If there is a danger that a database might be accessed by other entities while deploying, you tick this box and after the database is created, it is set to single user mode while the script carries on adding schema objects etc. For some reason, at the end, it is not put back into multi-user mode - which I guess would have to be done manually.
Anyway, the problem I had was that I upgraded a database project that had been built in Visual Studio 2008 to Visual Studio 2010. I did this so I could target the SQL Server 2008 schema (compatibility level 100). There was a problem after deployment in that the database I had deployed (not surprisingly) was in single-user mode which meant it didn't work on our web application. I was confused because I had not changed anything in the database projects, just rebuilt them in 2010. It was also confusing because only 1 database was affected. The second point was simply answered by the fact that only 1 database had this option ticked but why had the problem only just surfaced.
When I checked the database.sql that had been deployed from the project, it was only after upgrading to 2010 that the single user mode had been added to the sql. In other words, it appears that a bug in VS2008 was ignoring this tick box and this had been fixed in VS 2010.
Great stuff eh? I can't find any other specific hits in Google for this so maybe no-one noticed (like we didn't).

The problem with 'global' variables

2011-09-23T07:42:00.000-07:00

Those of us who used C and C++ were warned that global variables are not good. We mostly agreed but would sneak things in if we could! C# might not have globals per-se but actually it has static variables which can exhibit several similar problems. Consider the following method in an asp web app:

private static void OnApplicationError(object sender, EventArgs e)
        {
            HttpApplication context = HttpContext.Current.ApplicationInstance;
            
            Exception ex = context.Server.GetLastError();
            context.Server.ClearError();

            if (HttpContext.Current.Handler is IRequiresSessionState)
            {                
                context.Session["UnhandledException"] = FormatError(ex);
            }
            
            EventLogWriter.WriteEvent(Constants.EventLogSource_PoSSite, (long) Events.UnhandledException,
                                      (int) Categories.UnhandledException, EventLogEntryType.Error, ex);

            context.Response.Redirect("Error.aspx");
        }

All pretty boilerplate stuff. The session is used in this case to pass the exception detail into the error page to display if required.
In a live system, this method was crashing but we did not know it was crashing, all we knew was the system was intermittently crawling to a halt. I was looking into this method because the event log messages were NOT being logged even though this was the event handler.
The problem, as you might have guessed was caused by a 'global'. What was happening was that underneath the call to WriteEvent, a call was made to obtain the 'global' Context to add certain information automatically to the event. What had happened was that the Context logic was broken and it was in some scenarios attempting to access the OperationContext from a service which didn't exist which threw an exception before the event was written. This in term caused the app to automatically forward to the error page (this page!) which then tried to write the new exception and caused exactly the same thing to happen, falling into a tight loop. The way it should have been written was:

private static void OnApplicationError(object sender, EventArgs e)
        {
            // snip
            EventLogWriter.WriteEvent(GetCurrentContext(), Constants.EventLogSource_PoSSite, (long) Events.UnhandledException,
                                      (int) Categories.UnhandledException, EventLogEntryType.Error, ex);

            // snip
        }

where it would have been obvious that WriteEvent consumes the context and which would have given us a clue as to what might have not been set. As it was, the 'global' access was hiding this fact (the WriteEvent was in an external library so not obviously available).
In short, don't use globals!

Designing for Database Replication

2011-08-26T02:53:00.000-07:00

If you are writing a web application by yourself or in a small team, the temptation is usually to get it finished as soon as possible because even small sites don't take 5 minutes and once it is finished, there is a sense of satisfaction. The problem with this is that many things aren't really considered early on. Sites like Twitter didn't expect so many people to sign up so quickly and if the site wasn't able to scale, people would have got fed up and left or gone to a rival site. We should be thinking about scalability early on in the design and production cycle.
You might decide that due to the nature of the system, it is unlikely ever to exceed a certain size. Perhaps it is an internal company site and would never require more than, say, 1000 users doing simple things. If this is the case, fine. However, any public site runs the risk that it will become popular and an increase in user numbers means an increase in database, memory, network and processor load. If this is the case, you should design in the ability to scale as early as possible, even if your site will not be scaled initially.
One of the realistic possibilities is that you will need multiple database servers and if you need the ability to write to more than one of these at the same time, you will need some way to replicate the data. The difficulty with this relates to the actual design and might not be too bad.
For example, take the most basic replication issue of multiple inserts into the same table. If you have an index column (which you should) and you create user John Smith with ID 1000 in DB1 and then someone else creates user Michael Jones with ID 1000 in DB2, what happens when these databases replicate? In this case, although we might presume that the different names are different people, it would be possible for 2 John Smiths to create an account at the same time (in the gap between replication cycles). In this scenario, the best idea is to use a GUID for the id instead of an integer. Since these are globally unique, you would never have 2 users with ID 1000 on 2 different DBs. This also applies to foreign keys which would also use the GUID and would not be in danger of incorrect linking when copied from one table to its counterpart in the second database.
Updates can be more tricky. If someone sets a piece of data in the row for John Smith in DB1 at the same time as someone changes it in DB2, replication will do 1 of 2 things. If different fields have changed and this can be detected, it will simply take both changes and merge them. If the same field has changed or there is no way in your DB to detect which field has changed, you have the following options:

Flag it to a user who needs to manually decide what to do
Accept the low risk of this happening and simply take the latest change as the correct one
Create a manual system which records the changes so they can be automatically or manually merged
Apply a design which avoids the chance of this happening.

Really, to design out of these scenarios is always best because it is less maintenance. In our example, suppose we run a site like Facebook. One way we could avoid the problem is only to allow writes on a single database server, the others are read-only. The second option would be to ensure that updating user "John Smith" is only permitted on a specific 'home' database in which case multiple changes are applied sequentially and are replicated to other databases correctly.

Really the options depends heavily on the type of application you are creating. You might need to perform some load testing on your environment to find out where it is stressed under high-load. Don't spend time improving things that are working OK.

Disaster Recovery

2011-08-23T01:21:00.000-07:00

It's quite fashionable nowadays to talk about DR (disaster recovery) or what happens when something goes wrong. For lots of people though, they can only think about backups and the money to buy a new server if it breaks but DR has to encompass much more than this. It is all very well having backups but do you ever restore them to make sure they work? There is too much at stake to assume it will be alright. What happens after you make an upgrade to your database servers or web servers? Do you restore the backups again to make sure they still work? You might have UPSs on your servers but what happens when the power goes off? Do you assume it will not be off for more than 5 minutes? Do your UPSs cause the servers to shutdown so they are left clean or do you just wait for the UPS to run out and the servers to die? What would happen if your offices burned down? Would you be able to continue business or would all of your important information be lost?
Fortunately, it isn't rocket science, you simply need to perform a fault analysis of every element in your network (not every workstation but a workstation), servers, power supplies, UPSs, backups, internet etc. and then consider what would happen, how you would know and what you would do about it.

Finding code faults

2011-08-23T01:10:00.000-07:00

I read a great computer science book once (whose name I can't remember!) which talked about having certain 'rules' when we write software and one of them was, "when a defect occurs, it should be obvious where the fault is and it should only need fixing in one place". This rule implies encapsulation and more importantly, well defined and thought out fault handling.
Ignoring the debate of exceptions vs other error handling scenarios, it is obvious that most software still does not have a very well thought out system of fault handling. I have to support a live system and many defects that occur are related to specific scenarios and are extremely difficult to track down. Some of them we assume are environmental because we cannot see any path that would make sense yet all of these should be obvious. If something encounters an unexpected scenario, it should log that or display an error (or both). I think the underlying problem is that we might review code for neatness and main logic being correct but we rarely review code to look for what might go wrong. How often have you asked what errors a service call might throw (does the developer even know?) how are you going to handle that.
We can save so many hours or even days with a simple error like "X failed because the Y object was not populated" but we need a process to ensure this happens. Leave it to people and they will forget or not bother because of the pressures of implementing functionality.
The other big saver is trying to make a method either a list of calls to other methods or some sort of functional work and not both. This allows sequences to be easily reviewed visually and for functional methods to be small and concise and easier to re-factor.

Time outs and Application Integrity

2011-08-14T00:54:00.000-07:00

When we design software systems, we tend to think of perfect real world scenarios where all connections succeed, where services are always available and where the network is always available and running quickly. Sadly, many of us don't really consider what happens if some of these are not true at any given point. Perhaps in our system, it is acceptable to simply allow an error or timeout and for the user to hit refresh (as long as this actually works!) but for some systems, especially ones that require many screens of information to be navigated and saved into the session, this might not be adequate.
The company I work for writes a system to apply for mortgages. Various screens of information are captured and various service calls are made along the way. We had a defect raised the other day which appears to have occurred because something took slightly too long and the page displayed an error. The problem was, the system was then left in an undetermined state. The user interface was happy to try the screens again but the underlying system had completed what it was supposed to do and therefore would not return a 'success' code. There was no way to easily fix the problem, it would have required some direct hacking of a database to attempt to force the case through. In the end, the customer simply re-keyed the whole case but this is not always easy and is not popular if it happens too frequently.
This was a case where we hadn't properly thought through the possible error conditions. In our case, the risk is increased due to the numbers of people using the web app so it is not surprising that these things happen. Not only does it not look too good but it takes time and money to investigate the problem even if we end up not being to fix it. It is also not great as an engineer guessing as to the cause in the absence of any other discernible problem.
The solution as with all good design is to consider all aspects of the work flow that are outside of our control. This means the server(s) that is hosting the various parts of the system, the network and any third-party calls to services or other apps. We need to consider not only the timeouts, which in our case we do, but also what happens if something times out at roughly the same time the underlying system has completed successfully. This is especially true if you are using multi-threaded applications (which in many cases for web apps you will be). Ideally, your system should be able to go back in time to a known-good state from all pages but certainly those which are higher risk (which call onto third-party services etc) which should cancel and delete all current activity and take you back to a place that you know the system can recover from. This might be slightly annoying to the user but much less so than a full-on crash. You can also mitigate this risk by displaying a user friendly message such as, "the third-party service was unavailable. Please press retry to re-key the details and try again".

Managing Quality in the Workplace

2011-06-17T14:10:00.000-07:00

Consider the scenario, you have a software project which is dependent on 2 dlls from the same project built elsewhere. Simple scenario but already you have several things that can go wrong:

You might forget to update the dlls when they are changed
You might only update one dll when they are changed
You might deploy the application in a way that means the defect is not noticed until runtime

What can you do? You have several approaches you can take:

Hope it never happens (the most common and least effective measure)
You can write a process which reminds people to update dlls - not bad but people don't always read or understand the implications of processes
You can write a tool that updates things automatically - good idea but can require investment in the development for a simple scenario. Also how do we monitor whether it is working?
Write unit tests that can test for the presence of the correct items - a good idea generally if possible but how to ensure they are kept up-to-date and are testing all the right places?
You can design projects to not require such a dependency - best idea if it is possible without any other knock-on disadvantages or..
A combination of the above dependent on the exact requirements.

As I have said before, anything is better than nothing. There is no single solution because all decisions in life have pros and cons which need to be considered objectively rather than dismissing out of hand. How much damage could occur if the defect is injected. Charge that at £1000/$1500 per day and then how much to develop or create the solution? Even a minor bug can have knock-on effects, both causing other defects and also damage to reputation before you have even thought about fixing the defect itself. The fix is then another potential defect injection since systems are rarely tested to the same degree after a bug fix than they are initially. Basically, even a small defect fix, if you are going to fix it, costs thousands in real terms, money which you are NOT going to get from the customer. Get it right the first time since you will not have time to do it right a second time if you didn't have time to do it right in the first place!

MVC hacking for foreign keys

2011-06-16T05:49:00.000-07:00

I have an unusual scenario for MVC entity framework 4. Firstly I have a table with a primary key (not used for foreign keys) and a unique constraint (which is used for foreign keys). Entity framework does not recognise unique constraints and therefore does not let you use them for foreign keys. The fix is to open the crm.edmx in an xml editor (you might be able to do it from the designer!) and then change the key to point to your unique field and not to your primary key. This allows EF4 to point foreign keys to your field. If you had already created the keys and then re-generated, you might have found that the associations have lost their referential settings which you will need to restore. Obviously if you regenerate, this change might be undone.
The second issue is that although one of my table columns has a theoretical foreign key to another table, the key does not exist which allows us to test the scenario where an item is orphaned (the foreign key does not exist on the customer system), therefore EF does not provide all the wiring up for a drop-down list on my view to edit this 'foreign' field. The answer is to use a ViewModel which is simply a class that contains your entity and any other supporting data, in my case a list of SelectItem which is used in the DropDown list. This is populated in the constructor. Then rather than creating the view against the entity, you specify the view model instead and link fields to Model.EntityPropertyName.Name rather than just Model.Name. This also means you can simply link your drop down to the list of SelectItem in the view model which does not exist in the entity itself:


 public class AccountDetailViewModel
    {
        public AccountDetailViewModel()
        {
            AccountDetail = new AccountDetail();
        }

        public AccountDetailViewModel(AccountDetail source)
        {
            AccountDetail = source;
            // Get list of 'Account' entities from db and add number and rowid to the list
            var accounts = new Entities().Accounts;
            Accounts = new List();
            foreach (var acc in accounts)
            {
                Accounts.Add(new SelectListItem{ Text= acc.AccountNumber, Value= acc.RowId.ToString()});
            }
        }

        public AccountDetail AccountDetail { get; set;}
        public List Accounts { get; set; }
    }

And then in the view:


<%: Html.DropDownListFor(model => model.AccountDetail.PrimeAccountInternalRefNo, Model.Accounts, Strings.DropDownListSelectItem)%>
<%: Html.ValidationMessageFor(model => model.AccountDetail.PrimeAccountInternalRefNo)%>

Who is Responsible for Web Security?

2011-06-06T12:28:00.001-07:00

Web security for many people is quite simply one of the most important responsibilities that IT departments have. Even a simple forum site, if cracked, can expose passwords and email addresses which can then be used to access other sites - since many people use the same passwords for all their logins.
It is a simple but important question: Who is responsible for application security? Perhaps it's not you because you are not paid? Rubbish. Not you because you are the manager and it is a developer job? Nonsense. Sadly, my experience is that there is a lot of assumption about whose responsibility it is but few people who will stand up and take responsibility when things go wrong. What is worse is that in many ways it is too late when a site has been hacked. Whoever is sorry becomes irrelevant.
Think about some of the risks you have as an individual or company. The integrity of the firewall, the integrity of the network, the applications, the personnel management of people who might cause damage from the inside. Have you even thought about them and done a formal risk assessment? I doubt it. Most people have a very poor and non-methodical approach to security and then try and blame others when it goes wrong. You use an off-the-shelf product but do you keep it up-to-date? You use a lot of networking but are your staff really qualified to keep it secure? Can someone simply plug into your network switches and immediately gain access to your network layer?
So many risks, so little control. Is it any wonder why even high-profile companies suffer from hacking? It is time we started taking this thing seriously. We need auditing, expertise and responsibility. We need people to own up to where their expertise is lacking and the compulsion from management to pay in order to put things right.
I'm not going to hold my breath though. Contact me if you need any consulting on the security risks faced by electronic commerce.

Holistic Approach to Security

2011-06-06T12:26:00.000-07:00

Great article outlining the ways in which we need to consider application security: Software Mag

OO Design - do it properly

2011-06-06T12:19:00.001-07:00

If there is one thing I have noticed about software, it is that people tend to do something well up to a point and then let it all fall down with a "quick hack". In fact, there is no such thing because a quick hack immediately creates technical debt which invariably takes much longer to fix later on, if it is fixed at all. Once you have compromised in one area, it is then hard to try and keep the OO design pure in other areas and before you know it, your code has become a great ball of mud.
For instance, at work we have a service which is used by more than one customer so it has a base part and then a customer-specific part. However, somehow, the base project has ended up with an id of type Guid which is actually customer specific but which goes through the design as if it was customer agnostic. The result? Well, now we are not clear where these Guids should live because another customer simply uses an integer for a customer ID and not a Guid. The design is polluted and other decisions have been made which are not good.
One of the problems is that people are still taught in very structured, low level ways, to think about guids and ints and strings even though normal people don't understand such concepts. All they understand is a "product ID" or "text".
When we design in OO, we must be merciless with abstraction. Even if our id is a Guid, we should make it an abstract type since the fact it is a guid is irrelevant, it is a unique key and could be abstracted into UniqueKey which can inherit or contain whatever it needs to. This way, the abstraction holds true when another customer uses text for a unique key because IT IS STILL A UNIQUE KEY!
The thing is, we only learn these things after a mistake is made which is why it is important to train, to code-review and to retain people who are good at their job by recognising and rewarding them otherwise they will think the grass is greener on the other side and jump ship, leaving us with the average people who are not good enough to move jobs or who are not interested enough in work to care!

VSTS Errors - TSD03006, TSD04111

2011-05-23T04:54:00.000-07:00

Error TSD03006: has an unresolved reference to object.
This pain in the neck had occurred after creating a new VSTS project and adding what I thought was perfect SQL. The error pointed to Database.sqlpermissions and said the role I had granted a permission for did not exist even though it did!
Well there were two things wrong in my project and one assumption which made it hard to track down. The fundamental problem was I had spelt authorisation with an S instead of a Z in my role SQL which made it invalid but my assumption was that errors are printed in dependency order (i.e. fix the first one to fix the rest). This assumption was wrong and the error for the role was the last one listed!
The other problem was that even if your database properties say that your database is case-insensitive, the build environment will only recognise references that match case. For example if you reference a column called id with [Id], the reference will fail.

DRY and Coding for the Unknown

2011-05-18T04:51:00.000-07:00

Do you know what DRY means and practice it? How do you code for the unknown? These two facts equate to the highest possible quality yield in software production but are often misused or not done at all. If things are left to individuals, they will be variable at best and absent at worst. If management however can insist on processes that include these, they will see high quality software produced.

A friend of mine once told me about a software test that consists of writing a function that takes 3 integers and determines whether the 3 values could form a valid triangle. The test is how many of the checks could you think of? How many do you think there are? One of them for instance is that the values cannot be negative but there are around 16 tests all of which provide complete coverage of the permutations possible. The point here is that coding for things we know about and can think of is easy, coding for those we can't forsee is impossible by definition.

So how do we code to cope with the unknown? Well Unit tests can be helpful but not in all scenarios because you have to set up the unit tests to assume a certain set of inputs for an expected output, proving just one or two cases from millions of possibilities is weak to say the least.

Experience and documentation can be useful. For instance, you know the range of values that an INT can take although in most cases negative, zero and positive are enough to test a method. What about nullable fields? Do you test for null? What about the max values? What happens if you pass Int32.MaxValue into a method which adds 1 to it, do you know what does/should happen? By writing down standard choice values for certain types, you can then build up permutations to use in tests. On this topic, it is also good to not allow fields to be null in an object if they are never allowed to be null in normal use. No need to test something that is illegal (unless it is to prove that the validation works).

The other way that works really well for coding for the unknown is keeping methods in small and manageable chunks. You can then put very simple constraints on it and 'know' that your code is bombproof since it will fail in an expected way if called with illegal data rather than simply crash. Imagine having something like

private void SomeMethod(int i, int j)
{
  if ( i < 0 ) throw new ArgumentException("i", "i cannot be less than zero");     
if ( j < 0 ) throw new ArgumentException("j", "j cannot be less than zero");     // Now we can do functionality with values we know are valid
}

Another way in which you can help is by using intelligent types. If an integer must always be greater than 0, you could either use an unsigned type or even create you own struct called IntGreaterThanZero or similar which agains throws an exception if you ever try to assign an illegal value to it. In this case, you catch the error much earlier on in development and your method could become:

private void SomeMethod(IntGreaterThanZero i, IntGreaterThanZero j)
{
  // Now we know that our method will succeed because the constraint will already be correct
}

This brings me to the other point and that is DRY (don't repeat yourself)
which says if you have to do the same thing more than once, you should probably re-factor. I don't mean that you cannot test for something being equal to true in more than one place but if you are doing carbon copies of the same code (or similar) then you are asking for trouble. Consider the following code:

private SomeType ServiceCallMethod()
{
 return CallService();
}

private void SomeMethod()
{
 // Doing other stuff
 SomeType t1 = ServiceCallMethod();
 if ( t1 != null )
     DoSomething(t1);
}

private void SomeOtherMethod()
{
 // Doing other stuff
 SomeType t1 = ServiceCallMethod();
 if ( t1 != null )
     DoSomethingElse(t1);
}

How many things do you think are wrong with this code? Really the difference between poor code and good code comes down purely to your ability to code for known issues and to assume others that are not obvious.
Firstly, the point here is that we have two methods that do different things but end up calling the same method and then both have to check for null and call something else. The problem here? If someone else calls the ServiceCallMethod, how can we insist that they check for null in the return value? If they don't, the problem could manifest much later in the code and might takes several minutes, hours or even days to track back to a bad service call. The point here being that we can push the null check down into the ServiceCallMethod and it becomes impossible to call the method without checking for null (this assumes someone doesn't just call the service directly somewhere else but that still comes back to DRY). This is not the whole story though. What else is missing?

Consider the bigger picture. Just from what you can see/understand from the code, there is one big unknown and that is what will happen when you call the service. It is not just a case of succeed or fail but it could throw an exception, it could return null or some other populated or semi-populated object, it could return a whole host of errors which might be related to network, security or the service itself. Currently none of these are checked or catered for. You might say that these situations should simply allow the program to die but that is naive and actually a wrong assumption. Leaving exceptions to fly up the stack to the program can lead to either other behaviour actually failing and leading you to fix the wrong area or otherwise be masked by a catch statement which might re-throw or not. Where exceptions are possible, they should be caught and logged properly, it is up to you whether you then rethrow it or perhaps retry, show a helpful message to the user, "Sorry, the remote service cannot be contacted" or something else. You cannot always tell what exceptions are thrown by a method call so by catching those as soon as possible, it allows the people calling the method to know what to expect and to deal with it.

Defence-in-depth and the Sony debacle

2011-05-03T13:45:00.000-07:00

Well I say debacle but to be fair probably only because you expect someone as large as Sony to have all the best security experts but it again underlines something I have mentioned many times before. The security issues are well known and documented and have been for a long time, the problem is that most companies do not have the processes or quality control to actually audit these things. People make software updates, buy-in third-party software, contract out parts of the system to others who they cannot guarantee and all sorts of things, these are day-to-day realities of IT companies but yet so many organisations are simply too lax with security. At least the data that was stolen (or potentially stolen?) from Sony was encrypted so it is perhaps unlikely to be cracked since cracking encryption is extremely difficult if you don't know what encryption method has been used. The problem with credit card numbers is that you know the answer is either numeric or numeric with dashes so you have a crib to the solution. On the other hand, if you ensure your columns are not called CreditCard and ExpiryDate but something esoteric like CX25 and EX54 and possibly even padded to a fixed length then people are unlikely to be able to deduce what the data is that they have stolen!

Even basics like locking out accounts that have been accessed too many times or with multiple incorrect passwords makes it so much harder to brute-force attack systems.

So many options, so much documentation, so little motivation!

Workflow, workflow, workflow

2011-05-03T13:37:00.000-07:00

I've been using Windows Workflow this year and to be honest, I really like it. Well, I like it in theory but there are a few things that are buggy (I am using version 3.5) and which could easily cause me to not use it if I had a choice:

The projects take an age to load and compile. VS2005 and about 100 projects but still, it is virtually unusable and I would imagine that many organisations have projects larger than this. I have a quad core with 8Gb so I would expect it to fly.
If you make the name of an activity the same as the class name, it will not tell you that it is invalid. What actually happens is that it will create a new member variable with the new name but keep the old name as it is so it looks unchanged. The newly created member will be unreferenced. Not sure why the names can't match.
I have quite a few problems related to VS2005 crashing and shutting down and this coupled with the time it takes to load the project is impossible!
The Toolbox takes about 2 minutes to populate (open the first time) with all the activities in my solution. I tried deleting a load of them from the toolbox so I can just have the ones I need but for some reason, they are automatically all added in again when you build.

For state-machine type systems however, Workflow is definitely worth the hassle because it reduces the amount of code you otherwise have to write to produce a pseudo state-machine in normal classes. The designer is pretty usable too. Just make sure you understand dependency properties and the correct levelling of activities otherwise you will end up with a hotch-potch of non-reusable components that all require code activities everywhere to tie them together.

Good Programming Practice - avoiding the if statement

2011-04-26T14:26:00.001-07:00

Those of us who have programmed for a while are fairly familiar with and comfortable using the if statement in code but the if statement if mis-used can be both confusing and a source of latent defects, that is defects which are not realised for a while and then jump out and bite you. What is the distinction between a good and a bad if?
A good if statement should always be valid whatever happens to the system so a method which creates a type of class to manage a port might look like:


if ( typeToCreate == "TCP" ) port = new TCPPort();
else if ( typeToCreate == "HTTP" ) port = new HTTPPort();

If you think about it, although we might decide to use a different port in the future, the logic always holds true that if the type is TCP, we will create a TCP port. If we need to expand the functionality of the port by extending its interface, we can do so without 1) touching the if statement and 2) invalidating the logic of the if statement. Ideally, we would then have an else statement which might throw an exception to ensure we are using a valid port type.
Now consider the logic below in a classic if statement:


LoadMessage();
if ( message.SubString(0,1) == "S" ) DoSomethingForStringMessage();
// Carry on with processing

Can you see what the problem is here? There is an assumption buried deep beneath the code that says a message which is a string ALWAYS has an S as the first character and a message which does NOT have an S is NEVER a string. This might seem perfectly reasonable and correct but what happens when somebody decides in 3 years time that X represents an extended string. These new messages will fail the logic test but possibly in a way that is not obvious. This is a very easy to create defect which might well pop its head up at the customer site!
I am not a Puritan but I believe in making code as reliable as possible. We would not accept an aircraft designer thinking something is a quick hack but yet we allow it in software because perhaps we don't give it the respect and effort that should be given to it. Of course, it would be unfair to bring this issue up without telling you about alternatives. Ultimately there are three things that can really help you 1) The compiler is your friend - use constructs that the compiler can check 2) OO design allows you to embed logic into classes and 3) Do not repeat yourself. If you need to ask the same question in multiple places, you are asking for one or more of them to fall out of sync.
Here is an example of using the compiler and OO to help you code logic strongly:


LoadMessage();
message.PreProcess();
// Carry on

Here we introduce a class to represent the message (it might only be one class to begin with) and we create a method that is fairly generic and allows us to preprocess the message. In our case, we probably want a class to represent a normal message and one that represents string messages (which could inherit from the normal one). In this case, the string message PreProcess allows us to do whatever it is we needed to do inside this message with NO logical branching. Adding new messages with inheritance allows us to force people to consider the various methods that are available by making them abstract/pure virtual.
This leads to the 3rd principle which says in one place, and one place only, I will use some sort of data loader to decide which class to load for any new messages I create, this follows the safe method that the logic is always true since by definition, if I add a new message, I either have to extend or modify the logic in a single place to load my new handler class and this will remove the chance of a bug:


if ( message.SubString(0,1) == "S" ) CreateStringMessage();
else if (message.SubString(0,1) == "N" ) CreateNumericMessage();
else throw new NotImplementedException("No handler for message type");

If I add a new message which might also begin with S, I might need to do this:


if ( message.SubString(0,1) == "ST" ) CreateStringMessage();
else if (message.SubString(0,1) == "N" ) CreateNumericMessage();
else if ( message.SubString(0,1) == "S" ) CreateNewMessage();
else throw new NotImplementedException("No handler for message type");

The important thing being that the logic is in a single place and not buried somewhere down in a method. Using the exception mechanism for defensive programming is then an easy and quick way to fall over during testing if something has been forgotten.

Installing Driver for EdiMax 7711ln

2011-04-26T14:10:00.000-07:00

Click here for a great article about how to get your 7711ln Wireless LAN network (PCI) card working in Linux. I am using Kubuntu 10.10 (kernel 2.6.35-28) and installed the card as normal not really expecting it to not work! Fortunately, EdiMax provide the driver on their website which is in fact a Realtek driver for the 3562 chipset (the driver actually builds for various flavours including this).
Ignore what it says in the readme file which is very complicated and confusing when confronted with all the options and simply follow the instructions on the link. You can ignore any errors about the lack of the /tftpboot directory.
In the line which says

sudo ifconfig ra0 inet up

, replace ra0 with the interface name for your wireless card which might be different. Type ifconfig by itself on the command line and you will see a list, probably including eth0, lo and either ra0 or wlan0/1. ra0 or wlan0/1 is your wireless card.
If you have a newer Linux, once you follow the instructions and reboot, you might find that the card is still not seen by the Network Manager in the system tray (you won't see any wireless networks) and that is probably because by default Linux will load the rt2800pci (old built-in) driver and this will take priority over the one you just built. To find out, type lspci -v at a command line and see what is written next to the entry for your wireless network card. It will start with something like:


05:01.0 Network controller: RaLink Device 3060
etc..
    Kernel driver in use: rt2860
    Kernel modules: rt3562sta, rt2800pci

If you have the problem, the driver in use will show rt2800pci, in my case it has loaded the driver rt2860 from the driver I built myself contained inside rt3562sta.
In order to force it to use your newly built driver, you need to blacklist the built-in driver. Simply open /etc/modprobe.d/blacklist.conf in a text editor (as sudo) and then add the following to the end (with a comment to help remember why you did it!):

blacklist rt2800pci

Reboot and the job's a good un'

Project DLL not copied into bin directory of web site

2011-04-11T12:43:00.000-07:00

Simple situation. I reference a dll in the web.config of a site. I clicked "Add reference" and browsed to the directory where the dll was and added it with no problems. When it came to checking into source control and building, I got "Could not find type ..." which was muchos confusing.
It turns out that because the library I was referencing was the same version as one that I had installed in the GAC, it told the project to look in the GAC for the library rather than the location I had referenced. All very clever and very confusing.
In my case I could remove the library from the GAC and reference it again. This time I got a refresh file (which tells the project where to start looking for the library) which I checked in for safe keeping and all was good in the world. Nice!

Rhino Mocks - Get broken after running test with WCF

2011-03-21T04:11:00.000-07:00

I discovered something that might explain why tests sometimes seem to fail after an individual test has failed for no obvious reason.
Often in NUnit, you setup various mocks in [TestFixtureSetup] and then run one or more tests. In [TestFixtureTearDown], these mocks are disposed of. What is important to know is that if an exception is thrown during TestFixtureSetup, the TestFixtureTearDown is NOT called.
What this means is that suppose your mock #1 redirects an endpoint to a locally hosted mock, if you setup mock #1 but while setting up mock #2 an exception is thrown, then mock #1 will not be disposed and this means one of two things might happen. Firstly, when you try and setup the mock again, it might fail on the basis that something is already "listening" at the endpoint it is trying to use. Secondly, if you don't dispose the mock, the endpoint is not reset back to what it should be. In this case, if another test does not use a mock but DOES use the endpoint for something then it will have been left pointing to your local mock site but with nothing listening on it. Your test might then fail because it gets no response to its request.
To avoid all of this, add a catch to your TestFixtureSetup which will call Dispose on all mocks if they are not null in the case of an exception.

CJuiAutoComplete Custom Listing Example

2011-03-14T13:37:00.000-07:00

The CJuiAutoComplete in Yii is great but because it uses the standard interface, there is one thing that cannot be done directly with the Yii control, that is customising the list that is displayed when you type something in. In my last example was a simple list that just displayed the label and selected this when you clicked it (and stored an id number in a hidden field) but in my case, I am searching for church names and there would be lots of duplicates so the name of the church by itself is not enough.
There is an 'undocumented' feature of the jQuery autocomplete (which the Yii control uses) which is mentioned on various forums and which can be used to override the default behaviour. It is called _renderItem and which by default only prints the labels. This needs to be 'manually' linked to the jQuery object since it cannot be joined directly to the CJuiAutoComplete and this uses the Yii function registerScript(). Anyway, this is the code I used (see my earlier example for what all the other controls are):


Yii::app()->clientScript->registerScript('custom_church_search','
    $("#churchac").data( "autocomplete" )._renderItem = function( ul, item ) {
 return $( "<li></li>" )
 .data( "item.autocomplete", item )
 .append( "<a>" + item.label + ", " + item.street + ", " + item.city + "</a>" )
 .appendTo( ul );
};');

Most of these names and code I left as per the example because I wasn't totally sure what they were. The only things I changed were the name of the script which is used as a dictionary lookup, the jQuery selector (#churchac) and the line which appends the <a> reference. This by default prints only item.label. In my case, all I did was add some commas and the street and city names. You should retain the <a> tags if you want the arrow keys and such like to work.

Windows 7 Gripes

2011-03-10T01:23:00.001-08:00

I am generally pleased with Windows 7. It seems to perform well and is generally usable but there are some things that are annoying about it. At the end of the day, an operating system is just a platform to run productivity applications, it should not be any real beast in its own right and should not get in the way.
My biggest complaints are changes. For instance, trying to get into the various options screens of Windows Explorer is horrifically complicated when you are used to the older way of Tools-Options, a fairly universal concept that millions of people were used to. Now they are spread across various unintuitive icons and menus which look like they were 'designed' by a child.
Another thing which I come across a lot is the renaming of standard shortcuts so you can't find them. Most of us were used "Add or Remove Programs" which has now been renamed to "Programs and Features" which doesn't really say much differently except it is now in a different place in the control panel because it starts with P instead of A. Why are people allowed to change these? If you want to mention features, call it "Add or Remove Programs/Feature". When you test these things, you would think somebody would notice and complain.

metadata file xyz could not be found

2011-03-03T05:32:00.000-08:00

I got loads of errors like this in a VS2008 project the other day. It seemed really confusing since these errors related to files I had not even touched.
It turned out that a dependent project had not been built - which had not stopped the solution building - which meant the project references of other projects had nothing to reference and therefore could not find the metadata.
Looking up the output window to the first error showed what actually caused the problem and fixing this, got rid of the cascaded errors.

Why more memory can speed your computer up

2011-02-28T01:18:00.000-08:00

Firstly, we need to be clear. The memory I am talking about is called RAM, not the 'memory' which exists on your hard disk to store programs and files in. They are measured in the same way but RAM is more expensive and usually smaller, for instance 4 gigabytes (Gb - approx 4 billion bytes) whereas a hard disk might be 240Gb or larger.
The point here is that a hard disk although cheaper is much slower because it contains moving parts, literally solid disks of data being read by a moving head. RAM is electronic and very fast.
When you run programs on your PC, they will require a certain amount of program memory and this will be allocated from your PC RAM. However, if you do not have enough RAM (if you are already running too many programs and/or you don't have much RAM in the first place) then the computer will still allocate the memory but it will put it on the hard disk instead - a bit like putting your possessions into storage when you run out of room in your house.
The data can be moved into and out of this 'virtual memory' but like the storage unit, it takes much longer to send it and get it back. This makes the computer slower because programs wait for the memory contents to come back before they can carry on. You can tell this is happening because the hard disk will spend lots of time spinning.
As time goes on, programs require more and more memory, partly because they do more things and partly because people tend not to worry about memory requirements when writing software.
The easy answer when this happens is simply to buy more memory for your PC. You can either search a site like crucial.com for what memory your PC takes or take the model number and make to a shop where they can check for you. Most memory is fairly standard and you should be able to get at least 4 Gb depending on whether your computer can access this much (after a while, there are too many spare rooms in the house and you have to go back to using storage). For most people using non-specialised software, this is more than enough memory to get good performance.
1Gb will cost around £15/$25 in the UK so it is not much money to improve your PC.
If your PC is ancient, you are probably better off buying a newer basic model for a couple of hundred pounds then all the components will be improved.

CJuiAutoComplete Example

2011-02-12T12:38:00.000-08:00

I've started using Yii and am very impressed with it from an ease-of-use and installation as well as certain areas that have been thought about properly and implemented well. I wanted an auto complete box on a view and had a quick look at the documentation on the Yii site but it had no worked examples and I was new to Yii and didn't understand how it connected up HTML, jQuery and the php code. I have now got something working which I would like to share for others use. Firstly, here is how I declare my Yii php code for the control:


widget('zii.widgets.jui.CJuiAutoComplete', array(
     'name'=>'churchac',
     'source'=>CController::createUrl('user/churchsearch'),
     'options'=>array(
         'minLength'=>'2',
         'focus'=>'js:function( event, ui ) {
          $( "#churchac" ).val( ui.item.label );
   return false;
  }',
         'select'=>'js:function( event, ui ) {$("#churchac").val( ui.item.label );$("#User_church").val( ui.item.value ); return false; }',
     ),
     'htmlOptions'=>array(
         'style'=>'height:20px;'
     ),
)); ?>

A few points to note here. Firstly I have added function handlers for select and focus to override the default behaviour. This is because I want to display a church name to the user but store the database id in a hidden field which is linked to the model for when this view is saved. In my case I have simply updated the autocomplete box (#churchac) and the hidden field (#User_church). Also note the "js:" at the start of the strings for these handlers which stops Yii from escaping the quotes around the jQuery selector names on the HTML.
Next up is the controller function which looks like this:


public function actionchurchsearch($term)
{
    $sql = 'SELECT columns FROM tables LEFT JOIN etc..';
    $keywords = explode(',',$term);
    $sql = $sql.' WHERE name_1 LIKE \'%'.trim($keywords[0]).'%\'';    // Must be at least 1
    if ( count($keywords) >= 2 )
    {
        $sql = $sql.' AND city LIKE \'%'.trim($keywords[1]).'%\'';
    }
    if ( count($keywords) >= 3 )
    {
        $sql = $sql.' AND name_2 LIKE \'%'.trim($keywords[2]).'%\'';
    }
    $sql = $sql.' LIMIT 10';
    $schema=Yii::app()->db->schema;
    $builder=$schema->commandBuilder;
    $command = $builder->createSqlCommand($sql);
    echo json_encode( $command->queryAll());
}

A few more notes here. The function needs the word "action" added to the front as with other actions. You must also allow access to it in the accessRules() function of your controller. It takes a single parameter which is the search text that the user has typed in. In my case, I allow a multi-part search so I split the string up into separate elements and build a standard SQL statement to find the results. I have used the more basic CDbCommandBuilder class since the CDbCriteria is very much geared around retrieving data to populate models with but in my case, I just want simple search data to use in the autocomplete. Also note, you should provide an alias for the columns that you want to display (called label) and to select (called value). You can add other fields but you would need to explicitly use these if required. Note that I don't return the data but echo it to the response since this is an Ajax function. I use a built-in php helper function called json_encode and this ensures that the data is in a format that is usable by the auto-complete control (obviously you can't just echo a php array and expect javascript to handle it).
As a side note, remember this function can be called many times over a very short period so you might need to ensure that it works efficiently and caches search terms if appropriate to avoid frequent database queries.Also, see my newer post on customising the return data:

Perforce, Depots and Workspaces

2011-01-31T01:12:00.001-08:00

I have started using Perforce Source Control recently and having previously used Visual Sourcesafe and Subversion (with Tortoise in Windows), I thought it would be a pretty straight swap but being new to depots and workspaces is very confusing so I thought I would provide an idiots guide from someone who doesn't understand enough of it to make it confusing (yet!).

The depot is the repository on the server but filtered to only show the section from your CURRENT workspace
The workspace shows ALL of your workspace files from ALL of your workspaces including files/folders that are not present in the depot.
You should create an "ALL" workspace to include the entire depot so that you can always browse to something you haven't got and "Get latest Revision"

For example, suppose you have a depot with folders named A, B, C and you want one workspace for working on A and one for B. You would need to create, initially, an "ALL" workspace including everything. If you click on the depot tab with this workspace selected, you will see A, B and C. On the workspace tab - at this point - you will have nothing because you haven't taken any files.

Imagine you now create and select a new workspace pointing only at A, on the depot tab now, you will only see A and after right-clicking and "Get Latest Revision", your workspace tab will now show A even if you change workspace.

Do the same for B and you will now see A and B in your workspace and B in the depot if that new workspace is selected:

Workspace selected	Depot	Workspace
"ALL"	A,B,C	A,B
A	A	A,B
B	B	A,B

It is a little confusing but hey, if you have to use it, you have to use it!

I don't like Moodle

2011-01-28T01:48:00.000-08:00

I'm sure this will draw the usual comments but I am not a big fan of Moodle. It promises much but has so many weaknesses with basic things that I think the project oversight is not good enough.

For a start, the screen layout is clumsy, both in quality/professionalism but also layout. On some pages there is a left-hand menu, on others is a different left-hand menu that is much more fully styled with icons, some have no menu. Logout is sometimes in the top right, sometimes right at the bottom. Too much use of centre justify means that lists with different e.g. lengths of names, moves the grid and therefore the mouse clicks. It really does look like 100 people have written stuff independently and stuck it together.

Permissions appear to be far too fine-grained which gives a massive list to look through for various permissions, most of which you can only work out by looking at the php source and finding out what is what.

Documentation is astonishingly hard to find for library functions and general usage even though the Moodle site is large and has a lot of information on it. I am not convinced the search is working correctly.

We have been using it for a small distance-learning university and have found many assumptions that cannot be applied to our organisation even though it is a "learning organisation" - not limited to the course structure and the way assignments are handled.

I have actually modified some of the code to suit and added a new assignment type (one of the few areas that looks decent) and here is my next gripe: the code is mostly horrific.

One of the foundations of community projects is well structured and moderated code changes. Things like if...else with two large blocks of mostly similar SQL inline with everything else on a page leading to effectively a single 1000 line long function is not maintainable, quite simply it is asking to break (something I did often but fortunately in new pages I had copied from existing ones).

Look on the other hand at Yii. I know it is a framework rather than an application but actually many of the things are much better thought out. The user interface looks neat and consistent by default, there is a good use of OO for common things like output encoding etc, built in support for translation. It's just a shame that these projects start small and don't consider "big issues" until it is too late to do a fundamental change.

Developers - Make your life easier

2011-01-19T07:06:00.000-08:00

There are various things that make life easier and where a small investment could yield larger returns (or savings).

Today, we spent a few hours trying to get my login working into a test system. After about an hour, we found out that rather than being defined in an xml file, they were now in a database and we needed a test app to update the database.

When this change was made, all the developer had to do was to delete/move/rename the XML files in our source control and we would have immediately asked the question as to who and why it was changed. 5 minutes saved by the developer cost 2 man-hours of money to the company.

Also, very simple things like having some basic wiki articles on how to do certain things can save the hassle of asking busy people questions and waiting around for answers. This is especially true of anything that might have subtle non-obvious steps like changing permissions or adding test data to databases.

The first time these things happen, learn from them and adjust. They should not happen a second time!

Pains and Permissions on Web Folders

2011-01-16T07:31:00.001-08:00

When you develop web sites in Linux and haven't used it much before, you might find certain things that don't work properly, files don't write or can't be edited or at the other end of the scale, you might have lots of file access open which shouldn't be. Basically, the Linux security mechanism is very simple, everything being treated as a file and it describes permissions for 3 levels of user: the owning User, the Group and (Other) users. Each of these levels can have permission to read and/or write and/or execute the given file. For instance, if you open a console somewhere and type ls -l, you might see something like this:

drwxr-xr-x 6 root root 12288 2011-01-15 12:47 Downloads

The letters at the beginning of the line describe firstly that this is a directory and then the read/write/eXecute permissions for each of Owner, Group and Others. For instance, in this instance the owner can read, write and execute whereas everyone else can only read or execute it. In the case of a directory, execute doesn't mean anything.
This is important because depending on the user and group of the file, you might find that you cannot write something even if you are an admin user. For instance, in the above example, root is the owner and group for this file so if I attempt as user "luke" to write this file, I will not be permitted to do it. If you have unpacked an archive (for instance a PHP website framework) into the /var/www directory, you will probably have used sudo and therefore the files will be root:root by default. The first thing you should do afterwards therefore is change these to be owned by the web server account which is called www-data by default in the Apache web server. We do this by changing ownership (the chown command) like this:

sudo chown -R www-data:www-data ./MyNewSite

The sudo ensures you will have permission to carry out the change and the -R applies the change recursively into the directory you specified. In this case we are assigning the owner to be the USER www-data and the group to be the GROUP www-data (same name, different things). Obviously you could assign different owners or groups than www-data but I find this is easiest in my web directory.
You will now probably face another problem in that even if you are a member of the www-data group (if not, add yourself to it) that the group permission on most files by default is read-only which means as "luke" I still cannot modify these files without using sudo or kdesu to invoke the program. You could change the ownership of the files to "luke" but the easiest thing is to modify the files again but rather than changing ownership, you want to change access modifiers (using chmod) so that the members of the www-data group can also modify these files. We do this:

sudo chmod -R g+w ./MyNewSite

Again, the sudo is to ensure you are allowed to do it and the -R is to recurse directories. Be careful since Linux treats r and R differently and using the wrong r might do something different than expected.

Assuming now that Luke is in the www-data group, I will be able to read AND write any files in my new site but they are still owned by the web server which can access them as required.

Once you have a finished site, it is worth locking things down again and perhaps setting the permissions to a+r-wx which will mean the files are all read only. This will not work if some of the files need to be written by the web server (i.e. log files etc) so be careful what you change. At least make all php files (and any configuration such as connection strings if not php) as readonly.

Hiding MVC Controller Methods

2011-01-14T07:40:00.000-08:00

In ASP.net MVC, all public methods are callable by default in a Controller class. This could obviously not be what you want although I would question why you have public methods that are not callable from the web! Anyway, if you want to hide the method from being treated as an Action, simply mark the method with the [NonAction] attribute (System.Web.Mvc) and a call to the method will return a resource not found error.

Great Quote about Code Maintenance

2011-01-14T02:23:00.000-08:00

This is from ASP.Net MVC in Action by Palermo et al and pertains to decisions we make to do something the quick and easy way. The problem with this is described:

"The best practice still stands to avoid putting data access in your presentation layer; any data access concern in a controller action creates technical debt that will put a tax on maintenance for the life of the application"

Well said!

OWASP Update - ASVS

2011-01-06T05:43:00.000-08:00

Something I am a firm believer in is the community learning something and everyone else benefiting. For instance, we don't all have to discover electricity because others have come before us and we all benefit.

In the world of software applications, particularly web applications, the security weaknesses that we all face are very similar and are known - people have done the work for us and we know where vulnerabilities lie so why do we not design these protections in from the start of our systems?

Well there are of course several reasons. Maybe we didn't write the system, maybe it is open source, maybe we are not very well trained or experienced, maybe we mistakenly believe that our site would not be the target of any type of attack.

Whatever the reasons, OWASP has made great progress in defining an industry standard specification which describes fixed levels of web security and provides an objective means to measure both the security of our own applications (meaning that we can prove government compliance for critical web apps) but also a way of measuring the quality of a security tool that we might use to test our code.

The link is here: Application Security Verification Standard Project and as they say, even if you can't follow all of it, anything is better than nothing. Having an incomplete code-review checklist is better than no checklist.

It is worth spending time reading through the materials (owasp also have the free esapi security framework for use in applictions) and if you are a large organisation, probably worth investing in some training. The principles are not difficult but they are also not all obvious and having someone to analyse your processes and point out areas of weakness is usually easier than trying to learn ASVS and analyse your own processes.

Let's hope that one day these kinds of things just become the de-facto standard and that by standardising and learning from each other, we can kiss goodbye to most web-based attacks.

Your approach to query optimising

2010-12-30T04:52:00.001-08:00

Optimising a query in SQL Server is a bit like trying to pimp your car. You might think logically that putting in an engine with more horsepower will increase the performance of your car when in reality there are other factors that might limit this or at least that might break! In the same way, using the execution plan, you can concentrate on changing a table scan to an index seek and feel victorious only to discover it has made little difference on the bottom line. This can be for a number of reasons:

If you have many joins in the execution plan, then one of these might only be 0.5% and decreasing this to 0.25% although a lot for the one object adds up to little in the overall scheme of it. Consider simplifying the query or creating a smaller query to use in a specific place.
If the number of rows or columns that you are querying is small, then there might not be a great deal of difference in performance between various indexes or the table itself.
If you join on text/varchar/etc columns, you can incur a large overhead which includes not just the memory/IO itself required for more bytes of comparison but also the fact that you cannot include large text columns in 'index columns' if it would push the total size of key to more than 900 bytes. If you simply add it to the 'included columns' which is permitted, the index will not be used when joining on the text column. Try and use integer ids wherever possible.
You might see "Index Seek" and think this is the best you can get but actually if this Index Seek is carried out on a large index, it might be much slower than it needs to be. You can try this by creating the bear minimum index for what you need and running the query again.
Use the cost % information to decide where your bottlenecks are. If you have hundreds of joins all of less than 1%, then you are probably only able to achieve small improvements, whereas if one or more costs is significant (>10%), especially if it is under a table or clustered index scan, you can probably achieve large time savings by optimising these.
As a last resort (or a first resort if you can!), there are two things you can do. Design things differently so you can simplify the query (reduce the number of joins, rows or columns) or otherwise run the query from a job and dump the results into a cache table which can be used in certain areas that can avoid running the query and can live with data that is a number of minutes or hours out of date.

Which SQL indexes are faster?

2010-12-29T08:59:00.000-08:00

Even wondered what sort of index lookups are the best for your queries? What is the preferred speed order. Well, of course, it depends. In some cases there is little difference and it usually depends on whether the query optimiser thinks it will only need a subset of data from an index and also how much data is in the index compared to the main table. Bear in mind that just because you might get an Index Seek/Scan, it doesn't mean that's the fastest you can go. Using a large index for an index seek might be 10s of times slower than creating a smaller index with the required columns. Anyway, in order fastest to slowest, this is the best I can come up with:
Index Seek - A subset of an index. It's speed relates to how few columns you can include to get what you need and you will need some sort of 'where' clause for the optimiser to reduce rows as it joins. Make sure you include your WHERE clause columns and any JOIN columns in the 'index columns' you can then include other columns you want to select in the 'additional columns' area.
Index Scan - this scans the whole index and might be fast if it contains few columns and/or rows otherwise it is little better than a clustered index scan or table scan. If you have indexes that contain most columns from the underlying table, you are probably better off removing the index or redesigning your queries to not require so many columns. Also, ensure that you include where clauses as early as possible in your queries (i.e. not all at the outer level) which might promote an Index Scan to an Index Seek.
Key/RID Lookup - this might be very slow since it requires an index seek/scan followed by an index/clustered index/table lookup in which case it is not much better than a clustered index scan. It might otherwise be somewhere closer to the speed of an Index Scan if it comes from an Index Seek for a small subset of data. Generally best avoided by modifying/creating other indexes.
Clustered Index Scan - this is usually not much faster than a table scan, if at all, since the only difference is the clustered index is ordered by something which may or may not be useful for a particular query to use. The clustered index itself is however very helpful to speed up joins in general so don't delete the clustered index just because it isn't much better than a table scan. In the case of a small table with few columns/rows this index is probably as fast as it will get since more specific indexes will not be significantly smaller than the underlying table/clustered index. In this case, do not seek to replace Clustered Index Scans with Index Scans.
Table Scan - the base table is scanned, every row. The slowest and best avoided since it not only affects performance by itself but can also slow joins to other tables/views. It increases the amount of IO required even if you don't need all of the columns returned. This is because the data is paged into memory as it is being queried. A smaller set of data in an index takes up less memory and therefore less pages = faster.
I have also seen Clustered Index Seek and Table Seek. These are better than their Scan cousins but still best avoided if possible using an index. The only time clustered or table seeks might be the fastest you would get, like clustered and table scans, is if the underlying table is small or basic.

Making your queries faster

2010-12-29T08:13:00.000-08:00

If you run SQL Server management studio, you can include the execution plan with your query (Ctrl-M), it displays a network of how the tables and views join and more importantly, what types of scans are used to get the data. It also shows lines that are proportional to the number of rows being operated on. If you can see any big lines, then possibly your join condition is too generous (and you are using the WHERE clause incorrectly to fix the join) or maybe you do actually want to select tons of data in which case big lines might be acceptable.
As far as table and index scans are concerned, there are really two levels of understanding, one of which will help massively and the other is perhaps when you want to squeeze the last drops of performance out of the query once you have made the major changes.
Let's first consider the easier deductions from the execution plan. If you see any "table scan" elements on your diagram, that is generally bad and unless you have a good reason, you need to create an index on the table that can be used (even if your table is small - it can decrease join performance noticeably). What a table scan means is that the query has to scan all columns in the whole table to find and match rows. Virtually every table should contain a primary key column(s) and this creates an implicit clustered index so at very least you should be scanning the clustered index. If you do not have a primary key (suspicious!), you can still create indexes and it is these that will remove table scans as well as removing clustered index scans which are slightly better than table scans but include all columns still and can be slow also. By creating an index you should be able to include a few key columns which will dictate the sort order (which can help with join performance) but also (SQL Server 2005) you can include additional columns which the query can use but which will not unduly slow the creation and update of the index since they do not dictate ordering. If you have an index on TABLE with, e.g. columns A and B and your query says something like select A from TABLE where B = 2 will use the index directly. You will get an index seek ideally (a subset of rows from the full index, less than about 15%), which is very fast, or perhaps an index scan which has to read all rows but is preferable to table scans since the amount of data being used is a subset of the table data.
The second level of performance comes when your indexes are not quite right and you get strange items like "Bookmark Lookup" or "RID Lookup" which are both similar and caused by the same basic problem. Imagine in your example above you did select A, C from TABLE where B = 2. Although the index can be used to find the matching rows which is fast, in order to return column C, the query needs to find the primary key in the index (which is included by default) and then using this it needs to scan the clustered index or the base table if there is no clustered index in order to find the data for column C. This is a bookmark lookup (or an RID lookup if the base table has no clustered index - it is a 'heap'). These are also slow but possibly faster than a clustered index scan or a table scan. You might find sometimes that the query optimiser decides to not bother using the index and assumes you might as well go straight to the base table. To fix this issue, you would need to include column C into your index or create a new index with A, B and C in it.
Bear in mind that there is always a trade-off between the number of indexes and the insert/update/delete performance of the table as well as the amount of memory required depending on the size of the index which is related to the size of the underlying table. Generally, it is better to add a column into an existing index rather than create a whole new index which is slightly different. If one index has the same columns as another (even if the other index has a few more) then it is redundant e.g. Index1 = A, B, C; Index2 = A, B means that Index2 is redundant since all of the information is already available in Index1.
Also, the actual performance gain depends on the indexed columns relative to the base table columns. If your index had 5 of 6 available columns from the table then there will be little difference between an index scan and a table scan. Also, if you are selecting a text field from a table but using a WHERE clause which is NOT based on the text field, you might find it quicker to split the index into one without the text field (which reduces the size of the index and speeds it up) and then create a second index with just the text field in which the query should be able to use to get the resulting text field for each matching row.

Why Knowledge and Design is important for Software

2010-12-17T07:41:00.000-08:00

My company bought a new company in the last year and we have created a new web application to provide some functionality based on an existing system but created from scratch so we could use as many best-pratices as we could from the start.
Currently the new company only uses a small amount of the existing functionality but plans are afoot to increase this. One of the things I have been working on is in the quotation part of the system where the business selects the equipment for a quotation, adds certain services and then produces tender documentation. Our current system has around 22,000 quotes (some of which are test quotes) and one of the luxuries this provides is a set of genuine test data to try out any designs we create for the new company to see whether it scales well. Let's be honest, how many times have we implemented something by putting in a few rows of test data and then deployed it into a system that might end up with 10s or 100s of thousands of rows?
Anyway, there have been two things that have been obvious with our major re-design of the table layout and which have produced an increase in performance of around 5 times for the same data set running on a virtual test server with much less memory and CPU than the live server - pretty impressive. The first thing is that you need knowledge to write good software! This might seem obvious but someone might understand databases at a basic level but without understanding things like indexes, foreign keys and referential integrity, at best you will write something average, at worst it will simply not perform or scale and you won't understand why. Foreign and primary keys are not just for integrity but allow certain queries to perform much faster because of assumptions the system can make about the number of rows that will join in a query. The second thing is that we spent time in design. To be honest it was a Visio diagram that was sent round and which we all looked at. We considered the various objects in the current system and how they would work in the new system, which led to additional columns and the like but a large reduction in the number of tables required and therefore a reduction in joins and a performance increase in queries. Why? Well although you can partially think and design while you implement, in reality, you will never make large changes and see the big picture while adding tables into a database. If we had dived straight in, we would probably have something much closer to the old system whereas the new design is radically different and more efficient in the process. The few days of time we spent checking out the design has saved some time in implementation, a lot of time in potential rework and loads of performance. It seems counter-intuitive to be designing things and not just getting on and doing it but to be honest nowadays, implementation only needs to take a few days once the design has been completed.
Knowledge and Design - don't take shortcuts.

Poor passwords

2010-12-15T07:42:00.000-08:00

A news story today warned of thousands of internet users being asked to change passwords after a hack exposed login details. What was worrying was how many people used passwords such as 123456 or "password" which are clearly not secure in the slightest. What was interesting was that it took a hack to make various providers ask their users to change their passwords rather than simply not allowing them in the first place.
Quite clearly a brute force attempt at someones password will start with around 20 common words, phrases or number combinations, it would appear that thousands of people would fall foul of that.
What do we do? Firstly, if we allow people to choose their own passwords, we simply either blacklist certain words like "password" and/or we use a minimum password strength such as one letter, one number, one capital etc (although this can annoy people if taken to an extreme).
The other thing we should do is run a tool against our current users' passwords and work out which ones are not secure and then reset those accounts, sending an email or a page messages saying that your password is too insecure.
The other thing that seems sadly lacking is simple intrusion detection so that, e.g. 3 attempts at a password locks out the account for a period or permanently until reset. Even if you use a simple password, if the tool only gets 3 chances to crack it, it is unlikely to succeed.
Come on people, let's be proactive and not be tomorrow's headline.

Don't just secure your apps, manage their security

2010-12-14T02:52:00.001-08:00

Assuming you are one of the few people who genuinely secure their web apps using all the best known security practices like validation, authentication, authorisation etc, there is a good chance that you have ignored the common but important factor called Time.
Securing an app is fine but over time 2 things happen. Things change and things are forgotten. Why is this important in a web app particularly? Well if you used SHA-0 to hash your passwords a few years back because you knew it was the latest and greatest hashing algorithm, what are you going to do now it has been found to be less secure? Sure, you might have heard about the weaknesses and you might know that your system needs updating, but how does your organisation know that this issue will be picked up and addressed in the future?
Most organisations rely on people just knowing these things. They rely on the experience of developers or architects who have been around a while but this is simply not good enough. It is not good enough to be secure now when you consider how things can change.
In a similar way, people forget things like the fact that I used login X to access database Y for reason Z so that when someone else comes across it, they might re-use the login for something else, elevate its priviledges for some reason and weaken the overall security. How should they know to leave it alone?
A management system is required, any management system to begin with, but something which is adapted over time to be useful and useable. Make it unuseable and guess what? people won't use it. This needs to record information about the system including but not limited to security information. You might list the logins, their purpose and what access they should or shouldn't be allowed to have. It could list securables and secrets on the file system and what security they have been setup with. It should be expressly prohibited for ANYONE to modify a security setting without a review and without recording it in the management system.
It honestly could be setup on a wiki or anything as simple as that which allows regular review and work arising to be allocated. You can even flag things which we might know to be issues for the future such as encryption systems and hashing mechanisms, all of which become weaker over time.
If you don't manage security in your apps, your app is at best good for now but could break down rapidly as new exploits are discovered and no mechanisms exist to trace these to your own systems.

KDE4 Network Manager fails after upgrade

2010-12-07T12:36:00.000-08:00

Not sure exactly when but I did an upgrade to KDE4 and the kernel and in the process my networking stopped working, wireless and wired. It worked fine in recovery mode and I did another update in case there was a fix but no dice.
Fortunately I have a friend who knows the score, he told me to:

sudo nano /etc/NetworkManager/nm-system-settings.conf

and set managed=true instead of false. I then had to:

sudo nano /var/lib/NetworkManager/NetworkManager.state

and set NetworkingEnabled=true. I then had to restart the service:

sudo service network-manager restart

and it was all good again. Well done whoever made that happen in a package upgrade! Most people, including me, would not easily find out what has happened and would conclude that Windows works better even if we hate M$.

Working on a live database

2010-12-07T03:11:00.000-08:00

Even if you have a test system to try out some new functionality, at some point it needs to be deployed to live. If you are lucky, this is done by copying the up-to-date test system over the top of live but in reality, most of us have to modify the live database directly, perhaps during a small time-slot and then it is expected to work.
There are a number of problems that are associated with live databases, specifically SQL server in my case, that are worth knowing before you go and break something.

If you can, avoid altering or removing any tables/columns in the first place. If you can avoid this and perhaps leave a few dead things in place, you are less likely to break something.
If you can use some tools like Red Gate to search for references to your objects, it can help you track down what might break. Red Gate also make a tool for moving objects from a test to a live database in a GUI rather than manually.
For any objects that are accessed by something external to the database, mark these with schemas so you know that there might be knock on effects to other systems. e.g. webapp.procUserFetch
Avoid selecting * from anything since this hides whether a specific column is referenced or not.
If you are adding a new column, make sure it is nullable, even if it ultimately won't be. This avoids any existing inserts failing due to the lack of a value for the new column.
Try not to rename columns if you really don't need to. You can always alias them in a select statement.
Adding new tables, procs and views is safe
If you change a table structure, you must use sp_refreshview on the views that reference it otherwise their cached query plan can be out-of-date and you won't know until some funny data is returned in the select.
If you are deleting objects, rename them first or move them to a dead schema so that they can quickly be restored if there are any problems. You can put the rename date in their new name and after a week or so, delete them permanently.
If you can, script the new or changed objects and store them in a repository so you can track and blame changes that are made.
Use the format procNameVerb when creating stored procedures so that like procs are listed together e.g. procUserInsert rather than procInsertUser.
Have a backup of the database if you are making major changes so that you have a last-resort way of falling back to a known-working copy.

Writing good software systems

2010-11-25T03:43:00.001-08:00

Writing good software systems cannot be reduced to a simple set of rules, it is a complex and wide-reaching subject where some problems have more than one solution, one which might or might not be better than the others. It is full of compromises and various technologies. However, there are certain fundamentals to good software that are not negotiable, certain frameworks that must be in place if you do not want to survive on luck alone or on the fact that one of your engineers might happen to be very good at their job and you get quality by default rather than by design.

Training - If you employ people who are not trained/qualified/experienced, you cannot expect good software. You would not employ an unqualified doctor and software is not something so easy that a kid can do it. You can write bad software easily but knowledge is required to write good software. You must consider ongoing training as well, partly to adapt to new technologies and partly to challenge the way we get used to working.

Specifications - Coders tend to enjoy coding much more than they enjoy writing specs. This is understandable but it is essential that functionality is specified either by the programmer or in larger outfits by people specifically employed for this purpose. Every hour spent on a spec can save 50 hours of wasted development. Get used to really grilling the functionality in the spec, think of details such as security, validation, error (both expected and unexpected) although much of this can be pulled out into common specs that apply to multiple systems. Get the specs signed off by your customer, even if they are internal. If someone then turns around and says something is wrong in the spec, it becomes a development item (low priority) not a defect to fix immediately. It also makes people take their analysis to the correct stage. "No spec" means we've considered the easy 90% and not the hard 10%, a spec should force these harder details to be considered and potentially to affect the things we thought were easy to do.

Quality control - People make mistakes and are sometimes lazy, if you expect them not to be then you should not be a manager. The trick here is to put things in place that prevent mistakes (automating tasks) and pick them up when they are made (such as code reviews). Although this is a learning exercise and might be slightly different depending on the type of work being carried out, how many people etc, the key here is to start with the basics and learn from yours and others mistakes. Something goes wrong, ask yourself why, how it could be avoided, whether it is worth the extra cost/time compared to the liklihood of the mistake being made and then bring in a process/modify a process to learn from your mistake. This really is not rocket science and it will help you to become Quality Accredited if you seek this in the future.

Regular Reviews - Do not think that quality is a one-person thing. Have regular reviews with your teams and value their feedback. People close to the coalface know what is happening and do not appreciate having all decisions made by people they do not feel understand the problems. Create an objective atmosphere in these meetings so that you can measure the value of suggestions e.g. "your idea will take another hour per day but for this kind of defect, which we've seen once, I would suggest it is not worth it." This avoids arguments between two people who both know what the right answer is.

Get the foundations right - It might often be the case that timescales are tight and engineers aren't given enough time to do things properly so getting the foundations right is essential. Having a solid security model, for instance, is much easier to build in from the start and extremely difficult to fit later on. This comes back to having enough input from your team so that they buy into this foundation.

Do not forecast the future - You cannot pre-judge every change that might be made to a system in advance so do not attempt to build in everything at the beginning. If your code is written well then extending it later should not be too much of a challenge. There are however certain things that are reasonable to assume such as a web site might want its look and feel changed in the future. It is reasonable to factor these into the design. Also, people are not paying for all the future possibilities, if you keep the system simple, it will be more reliable and quicker to build. If the customer then turns round and wants a load more functionality, they can pay for the work arising from this.

Using static in .Net

2010-11-19T07:39:00.000-08:00

How often do you use the static keyword? Do you disagree with it from a puritan viewpoint that it demonstrates the hiding of functionality that belongs in a separate class? Do you even know what it means?
A static item essentially means there is only one copy. In C#, a static class cannot be instantiated (like a Singleton) and can only contain static members. A normal class can have static members (methods or fields) but these cannot be called on instances of the class, only on the class itself.
As with many things in life, static can be used well and it can be misused. However, I want to suggest that its use, if correct, is both good for performance and is also good for reliability! How do I know? Well look at the following code:


public class Car
{
    private static int carCount = 0;
    public static Car BuildCar()
    {
        ++carCount;
        // build car and return it
    }
}

This is an example of lazy or misinformed programming. Why? Because why should a car 'know' about the number of cars that have been built? In reality, you should probably create a factory class that looks like the above and then it wouldn't have to be static right?


public class Car
{
    // etc
}

public class CarFactory
{
    private int carCount = 0;
    public Car BuildCar()
    {
        ++carCount;
        // build car and return it
    }
}

We could then keep a single instance of this class somewhere and access it to make our cars. This is OK but it misses a trick when it comes to performance. In simple systems, to be fair, it is unlikely to be noticeable but in the above example, although we are only intending for there to be 1 CarFactory instance, the compiler doesn't know this and therefore every call to one of the instance methods has to resolve which instance (i.e. which block of memory) is being accessed. By using static, we tell the compiler that we only want one 'instance' and therefore the class is loaded when the assembly/program is loaded and is instantly available - bit like a direct pointer to methods/fields. This is good!
The same basic truth also concerns methods on classes that can be static i.e. any methods that do not access instance variables in the class. So although they do not have to be static, by making them so, we avoid the resolution of instance. Look at the following:


public class CarFactory
{
    public void PaintCar(ref Car car, Color colour)
    {
        car.Colour = colour;
    }
}

The method does not have to be marked static but in this case, it only operates on the values passed into the method and therefore should be marked static for performance reasons. Since, like all good programmers, we never modify methods after they are released, we would never need to change it to non-static, we would simply add another non-static method to do the work differently! In some cases, classes have static methods that operate on passed in values and instance methods that do the same thing on the instance itself:


public class Car
{
    public static void PaintCar(ref Car car, Color colour)
    {
        car.Colour = colour;
    }
    public void PaintCar(Color colour)
    {
        this.Colour = colour;
    }
}

The second and not to be ignored benefit of using static is that the compiler will enforce the fact that a static method is not allowed to access instance fields and that a static class cannot be instantiated. Why is this important? Because this improves reliability. Take the code snippet above. If our Car class had 20 member fields (instance, not static) and we wanted to test the two functions above, how many combinations would we need to test? In reality, we might look at a function and see that it doesn't access member fields but what if it called other methods which called other methods? Ideally we would need to consider every possible combination of instance fields as well as the parameters passed in. For the static method? We know that we cannot accidentally read or write any instance fields so we simply test for all valid permutations of the input (we might test in our case for a null Car, a null Color, a valid Car and a valid Color to ensure our code is robust).
Of course we always ignore reliability because we think we are good engineers and won't make stupid mistakes. To be perfectly honest, we make plenty and need as much help as we can to avoid them. That's why you should use static wherever possible!

What do you use Service Broker for?

2010-11-19T07:17:00.001-08:00

SQL Server has a system called Service Broker which is a queued, asynchronous system enabling calls to be made to procs or simply messages posted onto a queue which can be removed at will and processed. I have seen various people ask when you would use service broker, with its complexity, over a simple call to a stored proc.
The two key words here are queue and asynchronous. A queue ensures that things are processed in order so if, for example, you are running an important financial system, presumably various things must happen in order even if there are thousands happening all the time. In other words if you have 100s of people making changes, you might want to ensure that they are placed in a queue as a single point of entry and which will ensure that the first-come is first-served. You have an amount of control over the queue and whether messages are retained, the way in which the conversation is carried out etc but it is a way of controlling parallel environments.
The second and perhaps most important is the asynchronous nature of the service broker. Why wait for something that you do not need to wait for? If you are sending a letter to someone, you don't post it and then wait at the post box until the recipient tells you they have received it. You either don't care or if you do, you set up a converation with the recipient who can tell you if/when it arrives. Many scenarios exist like this in computing. If I want to cancel my credit card account, once I have requested the close, the Customer Services agent should not have to wait for the whole process to execute, simply to start the ball rolling and then assume it has worked unless the other end fails and reports back. This is especially important if the sequence is fired from a user interface which is waiting for the code to complete before updating the screen.
SQL Server allows you to invoke a stored proc on receipt of a message into the queue which might be useful but you can also leave the queue to fill and service it on a job every so many hours which might help avoid server load or meet various Quality of Service (QOS) requirements (such as all accounts are always updated at 6pm).
My general rule of thumb for software is to keep it simple, although you can sometimes guess how intensive an operation is and go straight for something like service broker, our assumptions can be wrong (a view with 10000 rows runs in a few seconds whereas another seemingly more simple view takes 5 minutes) so best to approach it with a 'if it aint broke, don't fix it' since it is reasonably easy to retrofit Service Broker in place of a stored proc at a later date as needed.

Service Broker Queue not calling Activation Procedure

2010-11-19T04:14:00.000-08:00

More fun with the Service Broker today on SQL Server 2005. I had got everything working on a test database and wanted to move the changes into the live database, both on the same server. I scripted all the stuff up and then realise service broker wasn't running. I found this because when I selected from sys.transmission_queue (which should be empty if the message is delivered to a queue), the exception column told me this. I tried to alter database to enable service broker but it wouldn't work with users logged in and I didn't want to boot all few hundred of them off! I wrote a SQL server job that ran the enable code at 5:50am with the following code:


CREATE PROC [dbo].[procEnableBroker]
AS
BEGIN TRY
 ALTER DATABASE MyDb SET ENABLE_BROKER WITH ROLLBACK IMMEDIATE ;
END TRY
BEGIN CATCH
 ALTER DATABASE MyDb SET NEW_BROKER WITH ROLLBACK IMMEDIATE ;
END CATCH

I needed the second bit in the catch which was some weird thing about needing a new service broker ID? The rollback boots out any uncommitted transactions and does the alter.
This all seemed OK so the messages were now being put on the queue but the queue was not calling the activation proc. If I ran the proc manually, it worked OK so I ended up thinking it was something to do with permissions. I followed some MS advice which as usual is buried amongst a lot of smoke and mirrors and did the following:


EXECUTE AS USER='dbo';
EXEC [dbo].[procQueueActiviationProc]

in order to match the user that it would normally run as (EXECUTE AS OWNER on the queue) and then I got the old, "the remote server could not be accessed because the current security context is not trusted". All I had to do then was to call:


ALTER DATABASE MyDb SET TRUSTWORTHY ON

And then after a little while and poking the queue again, it all came into life. I quite like the idea of Asynchronous calls for some things, this is a proc that updates a load of readonly fields and which takes up to about 10 seconds, something that we needn't wait for.

Cannot retrieve message from Service Broker Queue

2010-11-12T03:22:00.001-08:00

I thought I had followed the examples to the letter and yet when I tried to pass a simple message, it seemed to send it but wasn't received by the queue. Eventually I found a view called sys.transmission_queue which holds all messages until they are posted onto a queue (I thought it was just a log so wasn't suspicious that it was still full of messages!). In the transmission_status column for all my messages was "Error 15517 State 1: Cannot execute as the database principal because the principal 'dbo' does not exist, this type of principal cannot be impersonated, or you do not have permission."
It was caused because I was working on a database backup where the security doesn't really map back up properly since I am sysadmin on the server. Anyway, thanks to http://btburnett.com/2008/05/sql-server-2005-service-broker-error-15517.html I realised that all I had to do was call

ALTER AUTHORIZATION ON DATABASE::[dbname] TO [SA]

which I presume sets the database owner to sa and then I was able to make it work. All the previous messages were re-serviced and all ended up on my receiving queue.

Why web security is simply not good enough

2010-11-08T05:29:00.000-08:00

The Royal Navy with egg on their face today as some hacker grabs private information from one of their sites and releases it to the public. The trouble here is that what will happen is the site will be secured more tightly and we will "ensure it doesn't happen again" but these things never address the underlying issue.
There are organisations like OWASP who are trying to push for a common framework for web application security but everyone needs to be onboard if it is to implemented. We are talking about service providers, software vendors, the public and governments, if only a small measure of these are involved, it won't happen.
Do people really understand the problems though? Although the common weaknesses in web sites are well known, it would appear that many people are either unaware of these weaknesses or otherwise they are unable or unwilling to do anything about it. Just to give you a heads up, here are lots of common problems which lead to insecure web application processes:

There are simply not enough skilled people for the volume of web sites

The web allows people to experiment and build their own sites which are not easy to distinguish from good sites. The end-user trusts each of these to more-or-less the same degree even though the amateur sites are unlikely to follow any good practices.

You can buy or obtain off-the-shelf products to build your own sites and simply assume they are secure by default. These systems are not necessarily built well but you have to rely on experience and a wide user base to find and/or fix any insecure areas.

There is no way to physically stop people from either not implementing security or making a mistake when they code a site.

Many sites are fixed and updated regularly so it is only likely that at some point, unless there is a secure deployment regime, that a security hole is going to be created.

People tend to go with the easy approach rather than the secure approach. For instance, people use the database admin login to access the database meaning a hacked site is as good as an open door to all your data

Even when using secure practices, it is possible to use them incorrectly (such as using a flawed regular expression when testing for valid user input). Some of these might have inherent bugs or incomplete functionality.

People who write and deploy web applications do not require formal training of any sort to at least demonstrate they have been taught about secure practices, even if they cannot be forced to use them.

Although frameworks like ASP.Net and PHP have various secure controls available, they are easy to ignore, whereas these frameworks could actually insist on secure programming even if some people don't like it. For instance, by default, a text box control should not allow unusual punctuation, you should have to manually allow it on a per-item basis if required. Safety by default

Whoever has written the site, there is no worldwide accepted certification that ensures that a site is secure. There are mechanisms to prove who the site belongs to and various specific bodies like the Payment Card Industry who have their own audit procedures but nothing required for the general population.

By default, once someone has connected their computer to the internet they are both a potential menace and a potential weakness, it is like a load of criminals being moved into a residential area without telling anyone and then being surprised that the crime rate has increased.

Obviously any discussion about what could be done would be met with disapproval and accusations of censorship or the like but in reality, the system does not need everybody's approval since it could be done as an "exclusive club". You would need a way to achieve and prove certification including proving that your certification is valid (which could use digital certificates) and this would involve an audit of procedures/processes and functionality - possibly against a specific release and possibly in general for your organisation or as an individual? This certificate then qualifies you for the 'green flag club' which indicates to end users that a site is about as good as it could be and which would then ultimately allow people to choose to run in ultra-protected mode (for their safety/benefit) which would then cause people who write sites to get their site certified so that they can then access the club (and provide the benefit of a more secure site). The un-certified sites can live in the badlands!
Right, when do we start?

Devil in the detail - customising the ASP BulletedList

2010-11-02T02:48:00.000-07:00

When you do standard easy programming, you can cover a lot of ground very quickly. Modern tools and class libraries are very useful and do most things very well. When it comes to more specific programming, the devil is very much in the detail. For example, the other day I wanted to replace some hard-coded HTML generation, which made a tabbed divider, with a more standard ASP control. Since our old tab control was based on a <ul>, I found the closest control was an ASP .Net BulletedList.
It was easy at first since by setting various options on the list, I could get it to generate a <ul> with items and using the link button mode, it all looked pretty good. I applied the original CSS styles and I thought I was 99% there.
Oh, I thought, I really need to distinguish the selected item in the list since they represent tabs, I want to style the selected one differently. Since the bulleted list is supposed to be, well, a bulleted list, all the item selection from the base class ListControl has been overridden with NotImplementedExceptions and even if an item is selected by getting the individual ListItem and setting Selected = true, the item is not rendered any differently.
Well, this wasn't a problem since OO allows us to inherit and extend the functionality, I implemented my own version of RenderContents() in a new class, inherited from BulletedList which renders an attribute for class in the LI if it is selected, this was all great - OO is amazing! I then wanted it to remember the selected item between postbacks and to fire the SelectedIndexChanged ListControl event when the tab had changed. This is where it all started kicking off. For a start, since BulletedList has overridden the ListControl SelectedIndex and SelectedValue setters, I could not call the original versions which were fine, I had to use the Red Gate reflector tool to find the original code and duplicate it in my class overriding the overridings! I then found various references to internal and private functions which, again, I needed to duplicate in my code in order to use the code I had copied from the internal workings of the ListControl (I only wanted to make small changes!). I then realised that since the BulletedList already handle the click event and did not fire the SelectedIndexChanged even from ListControl, I would not easily be able to do what I wanted.
In the end, I decided the easiest thing was to inherit directly from ListControl which would give me all the Item selection code and then add in anything from the BulletList class that I needed (fortunately not all of it!) and then I could make my own class handle the postback event when it was clicked and fire the OnSelectedIndexChanged event from ListControl.
I removed any of the display modes and bullet types that I was not using to make it neater and got rid of the start bullet number. I also got rid of the various local variables used to "cache" values for loops. It seems that MS did not consider very well how their code would be specialised in sub-classes which is why there is a hotchpotch of public, protected and private/internal functions meaning that trying to base your code on the original does not work without lots of duplication (unless there are other public utlities to do the same things).
Anyway, if you're interested in the code:


/// 
/// A specialisation of the list control which is based on link buttons and tab styles
/// 

public class SelectableTabControl : ListControl, IPostBackEventHandler
{
 protected override void OnInit(EventArgs e)
 {
  base.OnInit(e);
  this.CssClass = "tablist";
 }

 /// 
 /// Handle the data being bound
 /// 

 /// 
 /// Ensure a default item is selected
 protected override void OnDataBound(EventArgs e)
 {
  base.OnDataBound(e);
  if (SelectedIndex == -1 && Items.Count > 0)
  {
   SelectedIndex = 0;
  }
 }

 /// 
 /// Add attributes of this control to the HTML
 /// 

 /// The HTML output stream
 protected override void AddAttributesToRender(HtmlTextWriter writer)
 {
  string uniqueID = this.UniqueID;
  if (uniqueID != null)
  {
   writer.AddAttribute(HtmlTextWriterAttribute.Name, uniqueID);
  }
  base.AddAttributesToRender(writer);
 }

 /// 
 /// Render the HTML for this control
 /// 

 /// The HTML output stream
 protected override void Render(HtmlTextWriter writer)
 {
  if (this.Items.Count != 0)
  {
   base.Render(writer);
  }
 }

 /// 
 /// Prevent this control from being given child controls
 /// 

 /// 
 protected override ControlCollection CreateControlCollection()
 {
  return new EmptyControlCollection(this);
 }

 /// 
 /// Render the individual elements for this list
 /// 

 /// 
 /// The default implementation does not render a class for selected items
 protected override void RenderContents(HtmlTextWriter writer)
 {
  for (int i = 0; i < this.Items.Count; i++)
  {
   if (this.Items[i].Attributes != null)
   {
    this.Items[i].Attributes.AddAttributes(writer);
   }
   writer.AddAttribute(HtmlTextWriterAttribute.Class, this.Items[i].Selected ? "selecteditem" : String.Empty );
   writer.RenderBeginTag(HtmlTextWriterTag.Li);
   this.RenderBulletText(this.Items[i], i, writer);
   writer.RenderEndTag();
  }
 }

 /// 
 /// Return the tag to use for the overall control
 /// 

 protected override HtmlTextWriterTag TagKey
 {
  get { return HtmlTextWriterTag.Ul; }
 }


 /// 
 /// Render the individual items from the list
 /// 

 /// 
 /// 
 /// 
 /// Replacement for base class function which references inaccessible values. This only includes the LinkButton style
 protected void RenderBulletText(ListItem item, int index, HtmlTextWriter writer)
 {
  if (!this.Enabled || !item.Enabled)
  {
   writer.AddAttribute(HtmlTextWriterAttribute.Disabled, "disabled");
  }
  else
  {
   writer.AddAttribute(HtmlTextWriterAttribute.Href, this.GetPostBackEventReference(index.ToString(CultureInfo.InvariantCulture)));
  }
  if (AccessKey.Length != 0)
  {
   writer.AddAttribute(HtmlTextWriterAttribute.Accesskey, AccessKey);
  }
  writer.RenderBeginTag(HtmlTextWriterTag.A);
  HttpUtility.HtmlEncode(item.Text, writer);
  writer.RenderEndTag();

 }

 #region Code duplicated from internal .Net classes
 /// 
 /// From System.Web.UI.WebControls.BulletedList
 /// 

 /// 
 /// 
 private string GetPostBackEventReference(string eventArgument)
 {
  if (this.CausesValidation && (this.Page.GetValidators(this.ValidationGroup).Count > 0))
  {
   return ("javascript:" + GetClientValidatedPostback(this, this.ValidationGroup, eventArgument));
  }
  return this.Page.ClientScript.GetPostBackClientHyperlink(this, eventArgument, true);
 }

 /// 
 /// From System.Web.UI.Util
 /// 

 /// 
 /// 
 /// 
 /// 
 private static string GetClientValidatedPostback(Control control, string validationGroup, string argument)
 {
  string str = control.Page.ClientScript.GetPostBackEventReference(control, argument, true);
  return (GetClientValidateEvent(validationGroup) + str);
 }

 /// 
 /// From System.Web.UI.Util
 /// 

 /// 
 /// 
 private static string GetClientValidateEvent(string validationGroup)
 {
  if (validationGroup == null)
  {
   validationGroup = string.Empty;
  }
  return ("if (typeof(Page_ClientValidate) == 'function') Page_ClientValidate('" + validationGroup + "'); ");
 }

 /// 
 /// From System.Web.UI.Control but minus supports event validation which is far too esoteric (and inaccessible)
 /// 

 /// 
 /// 
 internal void ValidateEvent(string uniqueID, string eventArgument)
 {
  if ((this.Page != null))
  {
   this.Page.ClientScript.ValidateEvent(uniqueID, eventArgument);
  }
 }
 #endregion

 /// 
 /// Gets whether the All tab is selected (which is always the last one)
 /// 

 public bool AllTabSelected
 {
  get { return SelectedIndex == Items.Count; }
 }

 #region IPostBackEventHandler Members
 /// 
 /// Handle a postback event occuring.
 /// 

 /// The value of the event argument
 /// This event is raised when the page is posted back and if the postback was caused by this control then the eventArgument is the index of the tab that was clicked
 public void RaisePostBackEvent(string eventArgument)
 {
  ValidateEvent(this.UniqueID, eventArgument);
  if (this.CausesValidation)
  {
   this.Page.Validate(this.ValidationGroup);
  }
  SetPostDataSelection(Convert.ToInt32(eventArgument));
  this.OnSelectedIndexChanged(EventArgs.Empty);
 }

 #endregion
}

wget doesn't get all levels

2010-10-24T03:37:00.000-07:00

I had a problem trying to grab a whole directory structure of files from an ftp site using wget. When I tried to get the files directly they were fine but when trying to get the whole site, it didn't get certain files which definitely existed.

It turns out that wget defaults to 5 levels of depth, even when downloading physical files from ftp (rather than following symlinks). I changed it so it so that it uses 10 levels instead with -l10 and it was fine. The number of levels make sense when you see the default output structure of servername/dri1/dir2/dir3/dir4 even though your files might appear to be level 4.

I'm not sure if rsync is easier to use for this purpose but the wget command is easy enough:


wget -r -N -l10 ftp://username:password@servername/dir

Where -r is recursion, -N only gets items that have been modified and -l10 increases the number of levels to 10 (That is 'L' 10, not 110!)

Document Generation with Word 2007 and ASP.NET - part 2

2010-10-19T08:48:00.000-07:00

In the first part, we looked at how to build a document ready to accept dynamic data which can be used to generate dynamic reports. Whether we are planning to use a blank template, individual read/write documents or merging documents together, the basic idea is the same:

Get data from data source

Create XML part for this data

Insert this data into the document

Return the result to the client

Getting data from the data source is beyond the scope of this document but in my case, I have an existing database access layer which I call to return a single row of data (a DataRow).
Creating the XML part is quite easy using the XmlWriter class as in the following function:


private void GetData(Stream stream, string quoteRef)
{
 // DataRow dr = etc..
 XmlWriter writer = XmlWriter.Create(stream);
 writer.WriteStartElement("Quote");
 writer.WriteAttributeString("Reference", quoteRef);
 writer.WriteElementString("QuoteName", "Test New Quote");
 writer.WriteElementString("TotalSellingPrice", Convert.ToDecimal(dr["TotalSellingPrice"]).ToString("N2"));
 writer.WriteElementString("TotalCost", Convert.ToDecimal(dr["TotalCost"]).ToString("N2"));
 writer.WriteEndElement();
 writer.Close();
}

Since the XML is text, it is convenient to format numbers at this point rather than playing with it in the document (but you can if you need to).
To insert this into the document, we need to use the System.IO.Packaging classes which allow you to work on the zip file (which is the docx). You might need to reference WindowsBase.dll if you haven't already to get these classes.
My function is then:


private void InsertCustomXml(MemoryStream memoryStream)
{
 // Open the document in the stream and replace the custom XML part
 Package pkgFile = Package.Open(memoryStream, FileMode.Open, FileAccess.ReadWrite);
 PackageRelationshipCollection pkgrcOfficeDocument = pkgFile.GetRelationshipsByType(strRelRoot);
 foreach (PackageRelationship pkgr in pkgrcOfficeDocument)
 {
  if (pkgr.SourceUri.OriginalString == "/")
  {
   // Add a custom XML part to the package
   Uri uriData = new Uri("/customXML/item1.xml", UriKind.Relative);
   if (pkgFile.PartExists(uriData))
   {
    // Delete template "/customXML/item1.xml" part
    pkgFile.DeletePart(uriData);
   }
   // Load the custom XML data
   PackagePart pkgprtData = pkgFile.CreatePart(uriData, "application/xml");
   GetData(pkgprtData.GetStream(), "QUO-016952");
  }
 }
 // Close the file
 pkgFile.Close();
}

This function takes a stream which represents a docx document and adds the custom XML in, there is nothing in this function which you would change (except the string parameter passed to GetData) you need a const defined which matches the correct namespace:


private const string strRelRoot = "http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument";

The reason I use streams is because I use the system in different ways. To retrieve a single document:


public MemoryStream RetrieveDocument(string fileName, bool addCustomXml)
{
 // Read the file into memory - default constructor is expandable
 MemoryStream memoryStream = new MemoryStream();
 byte[] buffer = File.ReadAllBytes(fileName);
 memoryStream.Write(buffer,0,buffer.Length);
 // If we want to add in the XML, do it here otherwise we might want to add it at top level
 if (addCustomXml)
 {
  InsertCustomXml(memoryStream);
 }

 return memoryStream;   
}

NOTE the comment that you need to use the default constructor for the memory stream otherwise it will not be expandable when you insert your custom XML.
If I am merging several documents (in my case using the Aspose Words dlls from Aspose) I merge the docs and THEN add the custom XML to the main document:


public Document MergeDocuments(List docs, string templateDoc, string outputPath)
{
 Document dstDoc = new Document(templateDoc);

 foreach (MemoryStream Doc in docs)
 {
  Document srcDoc = new Document(Doc);
  dstDoc.AppendDocument(srcDoc, ImportFormatMode.UseDestinationStyles);
  Doc.Close();
 }
 // Add in the custom XML via the memory stream
 MemoryStream ms = new MemoryStream();
 dstDoc.Save(ms,SaveFormat.Docx);
 InsertCustomXml(ms);
 dstDoc = new Document(ms);
 return dstDoc;
}

Aspose merges actual docs whereas I need a memory stream for my functions so I use the Aspose doc class to append documents and then convert it to memory stream to add the custom XML, once it is done, I return a new Aspose doc so it can be saved appropriately. One of the things that is easier in docx is that to merge docs, you can simply merge the content and keep a single set of styles, custom xml and the rest of it but it does mean that any custom XML/styles in individual docs is lost when the documents are merged.
In my web layer then, I call these functions and return the result to the browser using content-disposition to hint that it can be saved instead of viewed inline:


protected void Page_Load(object sender, EventArgs e)
{
 TenderGenerator gen = new TenderGenerator();
 const string TemplateFile = @"~/App_Data/QuoteTemplate.docx";
 List myList = new List();
 myList.Add(gen.RetrieveDocument(@"c:\work\QuoteDocuments\QUO-016952\Documents\UniqueReport.docx", false));
 myList.Add(gen.RetrieveDocument(@"c:\work\QuoteDocuments\QUO-016952\Documents\Financial Summary.docx", false));
 Document doc = gen.MergeDocuments(myList, Server.MapPath(TemplateFile), "");
 doc.Save(Response, "CustomerDocument.docx", ContentDisposition.Attachment, SaveOptions.CreateSaveOptions(SaveFormat.Docx));
 Response.End();
}

In this case, I have a test page so the doc paths are hard-coded and the whole thing is driven from the page load event, in real life, this will be driven from a button press and database driven list of docs. Note that I use the template file stored in App_Data which already has the custom XML linked to it so that the top level document can have the custom XML replaced (in the previous post I mentioned that for some reason, if the custom XML is not present, the new data is not added in its place). Hopefully this is all easy enough to understand.

Document Generation with Word 2007 and ASP.NET

2010-10-19T08:12:00.000-07:00

I had been asked to look at this for a project and spent many an hour trawling the interwebs to find out about solutions. The thing that makes it hard is that MS have a tendency to change technologies frequently (although generally for the better) and their MSDN articles are galaxian in size so trawling the various guides, APIs, white papers, support and other pages takes time.
Anyway, I've found some helpful articles and thought I would write a very easy to follow guide to getting the basics working.
If you want to generate a document on-the-fly, chances are that you want to populate the document with dynamic data from some form of database or other data source (the actual source is not important). Currently, this has to be done with various horrible COM technologies or otherwise by directly hacking Word 2003 XML files.
Word 2007 has the ability to link data controls on the page to XML data in the document, which can be generated dynamically. A Word 2007 document is actually a ZIP file (rename it and see!) which contains the various related XML files and which form the overall document, the XML which we will use to generate docs is custom XML and can follow any or no schema. In my case it is a very simple XML file with nothing more than the XML element and a single entity with 3 elements. Because this custom XML is separate from the rest of the document, I can change it at my leisure without affecting any formatting.
Now when adding these data controls, know as Content Controls, there are certain things that are not that obvious. Firstly, the controls can only be linked easily to the XML using code. There is a whole schema thing where you can define a schema and tell Word about it but I went another route by creating a Ribbon Control add-in for Word and using the C# code to add a correctly linked Content Control to a document. These can be added without the XML being present (they just won't have any data until the XML appears at some point). If you create a Word add-in project in Visual Studio and create a Ribbon(XML) component, which is fairly easy, you can then add the code which will look something like this (note the public modifier on the button handler):


// In the handler for a certain button - Ribbon1.cs
public void OnSellingPriceButton(Office.IRibbonControl control)
{
 Globals.ThisAddIn.CreateBoundDataItem("Total Sale", "TotalSellingPrice");
}

the reason this calls a function via the AddIn is because the AddIn has access to the underlying document whereas the ribbon doesn't (although I think it is possible to do something weird to gain access). The add-in code in my case is then:


public void CreateBoundDataItem(string theTitle, string theProperty)
{
 Microsoft.Office.Tools.Word.Document doc = Globals.ThisAddIn.Application.ActiveDocument.GetVstoObject();
 Tools.PlainTextContentControl plainTextControl1 = doc.Controls.AddPlainTextContentControl("plainTextControl" + Count.ToString());
 plainTextControl1.Title = theTitle;
 plainTextControl1.XMLMapping.SetMapping("/Quote/" + theProperty, null, null);
 ++Count;
}

Special notes here! You CANNOT bind XML to the rich text control so use a plain text control. Also, you need to get the vsto object in order to get the correctly typed document to access the controls collection. Also, when adding the control, you have to provide a unique name so I use a static int variable in this class to generate a unique name.
Obviously the SetMapping code (XPath) will be unique to your XML format, mine, as I said, is very simple. Also, the title property is what appears on the control tab when it is inserted into the document (as an aide-memoire). You can set other properties here like whether the data is readonly and whether the control can be deleted.
The cool thing about the add-in is that once it is built, you simply copy the output from the bin/relase directory into your AddIns directory in AppData (search for AddIns) and Word will automatically show the ribbon.
Once the ribbon is installed, it will create content controls pointing to whatever part of data you need. At this point there may or may not be any xml in the custom parts of the document but if you are using the Packaging classes in C# (which we use in the server end), it appears that it can replace the XML but not create it (might be some permissions thing?) so you will need to make sure that the document starts with a correctly formed XML custom part, even if the element data in it is blank or invalid. You can do this by running some VB in Word 2007 with your document open (you will need to enable the developer tab in Word Options), in the immediate window, type each of these lines and press 'enter' after them to invoke them:


ActiveDocument.CustomXMLParts.Add
ActiveDocument.CustomXMLParts(4).Load ("c:\CustomerData.xml")

You need to use the index 4 (the doc already has 3 built-in ones) but your path to the XML will obviously be wherever it is. The system will create an item called item1.xml in the customxml directory inside your docx zip structure and the fields will immediately be available to your content controls.
That is the document part done. You have a couple of alternatives now, depending on what you are doing but worth work in the same way. You can either keep a single document with the custom XML present but no content, and use this as a basis for any generated documents or you can save a bunch of different documents, each with the custom XML in place and then work on these individually or together. It depends on whether your generation is one-way or whether it will be loaded and saved.
The next post will be the server side which will look at the various options.

Embedding Excel cells into Word 2007

2010-10-14T02:28:00.000-07:00

I'm trying to write something that generates Word documents from a database which can then be modified by a user to include optional fields - things that will change and need updating in a document.
I'm having a bit of a struggle working out how to do it but I thought about old-school DDE and embedding LINK fields into Word. It is not ideal since it requires an absolute file path but I thought I would try it out. I could then create building blocks based on these LINKs and allow users to add them in. I attempted to copy the example in the Word help and surprise, surprise, it didn't work at all. I kept getting various errors all amounting to the fact that the field was invalid. I tried all sorts of options and almost gave up when someone mentioned the easier way to do it!
Copy the cell(s) in Excel that you want to use and then use Paste Special in Word to paste as a link (it's an option on the left-hand side of the dialog). It will automatically create the correct format for the LINK field (in my case some slightly different switches) and should all be good.

When a SQL JOIN is slower than it should be

2010-10-01T07:47:00.001-07:00

I've experienced the situation a few times where select * from viewA runs very fast, select * from viewB is very fast but when joined together, they are very slow (like 10 times the execution time).
I also had today the scenario where joining an old view was fine but joining a new one, which was not only virtually identical but with less columns and one less join, took 30 minutes instead of 10 seconds!
I have experienced minor improvements by changing join orders and trying to use more efficient WHERE clauses, avoiding SELECT in the WHERE clause etc but this was a whole other league of performance issue. When you only join two views, you don't have many options but I tried simplifying the ON clause (no different), removed the GROUP BY (no difference) remove the column which was summing an amount (no difference) and then I realised the only thing remaining was the JOIN. I changed this from INNER JOIN to LEFT OUTER JOIN and all of a sudden, I was back up and running.
As it happens, there are no rows in view 1 which do not appear in view 2 so the INNER JOIN isn't actually required here but I was still surprised. I looked into a few web posts and the best answer was, "an inner join is an implicit outer join + where column is not null" but I don't actually believe this and still wouldn't expect the problem in my 500 row dataset but obviously somehow it is a problem (although this is by FAR the worst case of it I have ever seen).

Process is more important than Software

2010-09-20T06:38:00.000-07:00

I have worked in various places over my nearly 20 years of working life and what surprises me is that it is often the basics that let us down, not subtleties or unusual scenarios. How many times do utility companies mess up a change of name or address? How many times does an airline lose luggage or mess up reservations?
I think most of us understand an extraordinary situation causing problems for any company but how do so many let the basics make them look like idiots?
One of the basics is the idea of a process. If you work out how to make the perfect cookie, do you try and guess the ingredients the next time you make some? Of course not, it goes without saying that you write down the recipe and follow it on the basis that if it worked the first time, it will work the second, all things being equal. I think it is the last part that seems to make many businesses consider processes either ineffective or maybe not even essential for business except maybe on a production line.
Consider my humble industry of Software Development. The first time you write a piece of software and get bitten by, for instance, somebody typing in dodgy input which makes the system fall over, you would think, "Ah, if I add that to a checklist for code reviews then it won't happen again". OK, we have to accept that we won't/can't necessarily retro-fix all the other places in code (although we should certainly consider it) but at least all NEW code will be OK. Although weaknesses do get found all the time, the total number in a given scenario is not massive so we can get to a place that only the newest exploits can affect our code.
This also works at a service level. Think of a support centre, it doesn't take much to look at the number of calls, type of calls and amount of time people aren't doing anything and work out how to improve the situation and free up resources. You might decide that you need a Frequently-asked-questions page or a current status web page or phone message that avoids calls from lots of people about something you already know about. The improvements to process and the lessons you learn don't change every day so over time you can hopefully employee less people to do more.
The title alludes to people who want to buy-in software to replace a current manual system but without thought to the process itself and what could be improved. Since computers and people are good at different things, it is safe to assume that a computer dominated process is going to be different from how things are done manually (a computer can run air-traffic control with less people because it can constantly detect and even correct conflicting air movements, something that is much more complicated to do by hand).
Anyway, consider your processes, what you are actually trying to gain as a result and you might find a whole load of people who are employed simply to support an unecessary business practice. You can then redeploy them to places that need extra resource at no extra cost to the company or you can let people leave and not need to replace them! Wonderful.

The 'Beta' ASP Calendar Control

2010-09-20T05:43:00.000-07:00

We are having problems on our web site with the jQuery datepicker when used inline to look like a calendar. The problem only exists in IE8 (surprise, surprise) and it is fine in FF but anyway, I was recommended the asp calendar which although a server control might do everything we needed it to.
In terms of functionality, I think it provides all the things we need, you can say whether you are allowed to select months, weeks and/or days and you can also style various aspects. I must admit there were a few things which made it look a bit amateur though.
The color for the day table cells is harded coded to black so you cannot override this inthe styles. Also, you cannot set the styles for the selected day because although it is theoretically possible to do, the styles are not used and are not reported back to the code behind either apparently (very sloppy Microsoft, you do have testing procedures?). The other thing that you can't do is apply a margin to the day cells (which I was trying to do to match the JQuery calendar). This is presumably because the day cells also have a hard-coded width of 14% which takes precedence over the request for a margin which would require wider cells.
My conclusion? Well, to be pragmatic, this control works where the jQuery one doesn't despite being a server control which is less than ideal. Also, because it is a server control, things like date selection all work with the correct types and I don't have to cast between strings and dates in the code-behind. I guess I'll use it.

Error 401: Unauthorised when calling web service

2010-09-08T07:15:00.000-07:00

I was calling a web service from code in ASP.net and got the dreaded 401 error. The web service was fine from a browser (but I couldn't execute anything since the test page only works on the local server and this remote). I used the Visual Studio "Add Web Reference" and logged into the HTTPS web service, it all seemed fine.
In the end, I ran up Fiddler (the web browser tool) and accessed the web service in the browser, the header came back telling me that the web service was expecting "Negotiate/NTLM" authentication and I was attempting to authenticate using basic. It now works and here is the code I had to use:


MyWebService service = new MyWebService();
CredentialCache myCredentials = new CredentialCache();
myCredentials.Add(new Uri(service.Url), "Negotiate", new NetworkCredential("MyUserName", "MyPassword"));
service.Credentials = myCredentials;
service.PreAuthenticate = true;
// Call functions etc

Note, I did not require any special certificate code which some people have suggested and this works on an HTTPS connection. I also did not need to specify the domain in the Add() function, just the username and password.

How to speed up SQL queries

2010-09-03T04:18:00.001-07:00

One of my pet hates is people going to a computer forum and asking something like, "I have a query that takes 50 seconds, how can I reduce it to 10" without a copy of their code and with all the telltale signs that they are nowhere near qualified or experienced enough and who expect people to simply do their work for them. Anyway, I want to partially mitigate this by giving you some beginners advice on improving the performance of queries. After reading these, you should write good queries in the first place, it is definitely easy than rewriting something and hoping you haven't changed anything.

Structural Improvements e.g. always do this

Avoid carrying out functions like LEFT, LEN etc in queries. These probably betray a poor design (like joining on varchar columns) but might be sometimes unavoidable

Use WHERE and ON correctly. People sometimes get these confused. Your ON clause should ensure that the rows that get joined are logically matched and then a WHERE clause is to optionally reduce the result set. If you have an ON which is not complete enough, you will multiply the join significantly and then have to reduce it after joining using your WHERE clause.

Do NOT use CURSORS, in most/all? cases you can use JOIN to avoid them.

Do NOT use selects in a where clause. It means every row that is matched needs to call another select (300,000 rows = 300,000 selects)

Test your query on large datasets before deciding it performs acceptably.

Only select columns you need, not *

Design your DB tables to link using ID numbers and not text/varchar columns

Design Improvements e.g. you might need to do this

Use the query analyser to suggest index creation. Understand that indexes usually result in faster selects but slower insert/update/delete so depending on how often a table is read compared to how often it is updated, this may or may not be acceptable. The alternative is an indexed cached table (updated every X minutes)

Select required data from tables to join into temp tables or table variables, especially if the dataset is large, the table is in another database or you need to join on it multiple times. You might also want to create indexes on these tables. Consider joining a table with 500,000 rows and 80 columns that takes up 180Mb. Creating a temp table with 8 of these columns and perhaps using a where clause to reduce it to 100,000 rows will obviously perform much faster.

Use cached data if acceptable rather than joining raw tables. If the data you need does not need to be up-to-the-second correct, you might write a proc that updates a cached table every 15 minutes locally and then use this in your join

Why are specs important?

2010-08-17T08:08:00.000-07:00

When I used to work at my previous company, I always felt that specs were annoying and inconvenient and didn't really serve much purpose other than for ticking boxes for quality auditors or whatever. Then I came to my new company who have almost zero procedures or specifications and I now miss them. For those of you out there who are not convinced or those faux experts who claim "at the end of the day you have to start coding and see what happens", here are reasons why paperwork is important:

It is much quicker to define and review a concept or system on paper than it is to write the code. It is almost impossible to review a completed system in code because of the volume of information and any changes are generally impossible (at least to achieve in a robust way)

A document is easier to team review including with non-technical people/managers who can understand basic system info but not the detail

It gives visibility of intent to the customer, especially if there is more than one customer (this includes product managers and stakeholders). It is easier to argue and agree functionality when it is a single paragraph of text than to rip up a load of work later on in code and attempt to rewrite it without breaking things.

It allows changes to be measured against what was originally agreed which means its easier to say, "no we're not changing it because it was agreed" and equally easier for a customer to say, "this is not working compared to what was agreed, please fix it".

Changes to the functionality can be formally agreed and recorded electronically so no-one is in a position to easily argue that the system is "wrong" which usually means they personally think it should be different.

Often people will disagree about the intent or content of certain functionality perhaps as a balance between management requirements and the additional workload required but since these issues are agreed up front, either the manager can accept a reduction in functionality to appease the worker or they will insist that the worker does what the system requires of them even if they don't personally agree.

Various interworkings and calculations are much easier to write and review on paper before the sometimes significant amount of work required to actually implement those calculations.

By having processes that govern the production of specifications, if something bites you in the released product or perhaps in test, a note can go into the process that might say something like, "ensure you have considered any additional security that might be required" or "Consider the fact that you will interact with new data and it will probably require validation to be defined"

SQL Server Security - Roles, Logins, Users, Schemas and Permissions

2010-06-23T06:05:00.000-07:00

When you first approach the area of security in SQL Server 2005, it can seem very complicated. If you search on the internet you will find thousands of sites who all seem to assume that you mostly know what you're doing but if you're like me, it is hard to know where to start.
Well here is a brief and hopefully newbie-friendly guide to SQL Server security.
Firstly the 'why'. Many programmers appear to take the lazy route when setting up security, you take the path of least resistance and highest privilege so that your application simply works and doesn't hassle you with security errors. What is wrong with that approach? Well if you are the only person who works on the system and it is on a private network, that might be the most practical solution causing the least work but what if your system starts being used by more people? What if you want to expose it to the www? You can retro-fit security, but it is much more time consuming than thinking about it at the start. The 'why' in the bigger sense comes down to the following:

You do not want to accidentally change data in the DB

You do not want somebody to do more (i.e. higher privilege) than they are supposed to do either accidentally or deliberately

You do not want anauthorised people to do anything in your system

For 1. you might have some configuration data for your system stored in a table. Although you are strictly allowed to change it, you generally do not want to and do not want to accidentally change it. You could set the security to be 'select' only. For 2. you might have a table of 'items' which must only be updated by an elevated user and selected by normal users, you would need to set 'select' for a normal user and perhaps 'select, update, delete, insert' for an elevated user. For 3. you need to ensure that everyone who accesses the database is authenticated and no-one who is not can therefore do anything. This might include people who work for your company but are not allowed into this specific database.

Schemas
The first design you need to undertake is to decide what schemas you want to create. A schema is a logical way to group objects (tables, views, procs) so that you can set permissions on the whole group rather than on individual items. The definition of a schema is a group that does or might need different access permissions depending on the user/user role of the currently logged in user. Do not create too many since this can make maintenance messy. I have used schemas like admin (select for everyone, CRUD for admin users), writeonly (for error logs and the like, only allow select and insert), general (anyone can have CRUD access), readonly (items pulled in from elsewhere and which are not updated in this DB also for views) locked (admin tables that are important and unlikely to have rows deleted so have no delete permission) and cache (tables that are dropped and recreated for caching things). If you notice, each of this has a unique set of permissions applied both for system resilience and for security.

Once I have designed my set of schemas, I need to create them for the database under /security/schemas and then I either need to transfer over all the objects I foolishly created in the dbo schema or create new objects into these new schemas.

Roles
I now have to consider the sort of database roles for my users. This is NOT an authorisation system for a web application so I don't need to distinguish at too low a level (e.g. user A can access table A and B but user B can only access table B) otherwise I end up with a lot of maintenance for no benefit, the authorisation at page level should be done in the web app. What I might have however are roles for admin, general, readonly and perhaps a couple of specials like finance who might be able to read stuff that most people can't. These are created under /security/roles/database roles. Once you have created the roles, you can if you want select the schemas and add permissions for the newly created roles. For instance you might given the schema cache full permissions for everyone whereas the admin schema gives full access to the admin role and only 'select' to general and readonly users.

Logins
Everybody who logs into the database needs a login. You can have a single login and get your application to login as this single user. The problem is then you lose the ability to make the most of roles and permissions because everybody appears as one 'person' to the DB which will then need access to everything for all users and which defeats the purpose. Ideally you should use windows integrated security if you can and remove the anonymous login ability on the web server. You will then need to create a login for each user on the database server under /security/logins that includes the domain name e.g. DOMAINNAME\username which will connect into active directory and which avoids the need to manage user security in your app. Failing the ability to use windows, you can use SQL server logins which will still need creating and which you will then need to manage in your web app including a secure login page to ensure the login details cannot be 'sniffed'. You create the login in the same place but select "SQL Server Authentication" rather than "Windows Authentication". You would then need to set the password policy information which again is important if you have opted for SQL Server logins.

Users
In order to use these logins in your roles, you need to create a user IN EACH DATABASE to link to the login. The database user can then be added to a role when it is created. Under /security/users, right-click and select "New User". This will bring up a simple dialog which allows you to give the user a name (related to the database, it does not have to match the login), tell it what login this user related to e.g. DOMAINNAME\username and then tick what roles this user is in at the bottom. In more complex setups you might assign the user to own certain schemas, giving them the ability to control and alter these schemas but for now ignore that. You do not have to add the user to a role at this point.

So for user BrinerL you might have:

server login: MYDOMAIN\brinerl
database user: brinerl -> MYDOMAIN\brinerl
belongs to role: admin
admin has permissions: Insert/Update/Select/Delete/Execute on all objects.

It's quite straight-forward really. You can automate the adding of the logins with a stored proc if required so that perhaps the first time somebody logs into the web site, it creates them a login and user and gives them the basic readonly role. If they want more then a DB admin would need to either move their roles in Management Studio or log in to a page that can carry out the necessary work from some sort of front end.

Ajax, javascript and postbacks

2010-05-12T03:45:00.000-07:00

The number of times I have this problem! You create a page that uses updatepanels and MS Ajax which, to be fair, is pretty good and works with little understanding of what's going on. Then, however, we also use javascript libraries for added coolness and to carry out work on the client and need to manually call a postback on the server after carrying out work in javascript. You manually call __doPostBack and you might get a postback but it doesn't call the event handler, you might get a premature postback or maybe no postback at all. To make things worse, the name and id of the control have been mangled by the placeholders so the btnExit id might now be ctl00_MyContentPlaceHolder_btnExit. It all gets confusing. Well here's what you do:

Create your button as usual and allow its names to be mangled (not sure if you have any choice!). Then add a click handler either using onclientclick or we use jQuery.click(function(e) { }) functionality. If using jQuery, call e.preventDefault() on the passed in event which will ensure you don't get the auto-postback which you get with submit buttons and then carry out your javascript functionality, possibly with the result deciding whether to postback. When you're done, call __doPostBack() with the NAME of the button control (the one with dollar signs usually) and a blank string for the arguments. You should have no problems. e.g.

__doPostBack('ctl00$BodyContentPH$ButtonAddLocation', '');

MediaWiki and Blank Edit Tab

2010-04-15T09:54:00.000-07:00

I installed MediaWiki at work not long ago and can't remember anything being amiss which is why I can't really remember it. Anyway, I installed it at home and the installation all appeared to be smooth, the database was created correctly and the page told me it was all successful but when I clicked on the Edit tab, the page was blank. No HTML at all, no nothing.
I couldn't find any specific issue on the web which surprised me but I assumed it must be my configuration options which it didn't like. I eventually found a post talking about upping the memory limit otherwise certain operations would fail.
It is slightly confusing because there is an entry in /etc/php5/apache/php.ini which I had already changed to 32M and which hadn't worked but there is also an entry in the LocalSettings.php of the wiki site itself and this also needs to be upped. I changed it, didn't have to restart anything, refreshed the page and voila - it worked.
A bit surprising that the setting in the download was not even sufficient to edit the main page which has hardly anything on it, it would have been more of an issue editing some massive page perhaps.
Well, never mind, the fix was easy after all that. Note that there have been issues reported about international character handling causing the same symptoms.

Not another "security breach"

2010-04-14T01:35:00.001-07:00

I was reading today about hackers who compromised Apache.orgs servers and obtained the passwords of various users. An interesting part of the article is this: "The breach, the second to hit Apache.org in eight months, also exposed a much larger list of passwords belonging to people who accessed the site's bug-tracking section. While the databases used a one-way hash to disguise the passwords, two of the lists are vulnerable to dictionary attacks because Atlassian, the maker of issue-tracking software used by Apache, failed to add "random salt" to them."

Now in short, a hash is supposed to change some text, say a password, into something that doesn't bear any obvious relationship to the original password. When someone logs in, the password they type is hashed in the same way and compared to the stored hash without ever having to actually know the password. This is all well and good except that hashing algorithms are publically available which means it is possible for a hacker to run loads of dictionary words through the hash and store the results in a database table. Then if you know the hash, you can do a lookup in your database table and assuming it is a normal enough word, you will probably find the original password. Couple this with people who use the same passwords for lots of systems and disaster is afoot (assuming you can gain access to the hashed passwords). So hashing is not in itself a silver bullet security system. HOWEVER, it is possible to make it more secure with something called salting which I have covered in an earlier post.

Again, the main point is not that there was a failure but all these vulnerabilities are already known about yet they still occur, even in high-profile organisations like Apache. At what point is something like OWASP going to become a mandatory badge for web software so that it includes tickboxes like "I have added at least 8 bytes of salt to passwords before hashing". Whether the badge is voluntarily added or carried out by an auditer, at least when failures occur, companies can be prosecuted for not adhering to the checks. It seems so obvious but sadly I can't see it happening for a while.

Website images that suck!

2010-03-23T15:04:00.000-07:00

I don't know how much longer I can take this. Why can people not learn to use the correct image formats for the type of picture they are displaying on a web site (or any other media for that matter)?
I just visited a web site with a solid coloured logo all covered in strange artifacts because it was saved as a jpeg - it looked terrible and even the uninformed should know that it is wrong. It is then a matter of finding out why and fixing it. I also saw an advert for, amongst other things, a printing company whose graphics were messy and again covered in jpeg artifacts and when I had some business cards printed, they managed to make the small font look messy because obviously they converted the image I sent them to a jpeg and made the nice sharp card look stupid.
Right, here it is, a simple introduction to what image formats you should use and why.
It all comes down to compression or making an image take up less space by cleverly reducing the amount of information stored on your computer. Why? Well, although your computer might have loads of space, an uncompressed image (called a bitmap) can be, lets say of the order of 5 to 10 megabytes in size. While this is not massive, when this is being downloaded onto a web browser, it can be very slow and doesn't need to be this large.
How do we compress? There are two ways. The best way is called lossless compression and involves looking at areas of the same colour and rather than storing every dot (or pixel), we simply save something like '20 dots of black' (obviously not in readable text like that but in 1s and 0s), the reason this is lossless is that when we uncompress the image to print it on the screen, we get back exactly the same image we started with. Sort of like deflating a football and pumping it back up again. Logos with big solid colours will compress really well using lossless compression. Unless you have other reasons not to, use the PNG format. This handles millions of colours and can even store transparency so your image can go over something else and you can see through it!
So what else is there then? If this all sounds too good to be true, unfortunately it is when it comes to photographs. Quite simply there are not enough of the same colour pixels to compress it well using lossless compression e.g. PNG (it will save but won't be very small). For this reason, a Joint Photographic Experts Group worked out how you could compress photographs even though they have millions of dots. The result is the famous jpeg format. This format is lossy. In other words, if I compress using jpeg and then uncompress, I do NOT get the original picture but something that is acceptably close - it is a compromise. In fact, you can set the quality level so that poorer quality makes for a smaller image. This might be used for thumbnails. The main thing to remember here is that jpeg is for photographs NOT solid colour logos. If you save a solid image using jpeg, when it is uncompressed, you get those horrible feathered edges and dotty artifacts in the empty areas - something that wouldn't be noticeable on photos.
You now have no excuses:
photos = jpeg/jpg and solid colour images = png

Unable to start debugging the web server 500.19

2010-03-22T05:01:00.000-07:00

My debugging in Visual Studio (VS2008) stopped working this morning, I think it was as a result of Windows Update on Windows 7 adding some "security updates". I run out-of-the-box VS2008 (SP1) and IIS7 built-in to Windows 7 and I didn't even get the browser, just some non-resizable window with what looked like the html source for a browser error and 500.19. After being annoyed, I decided to try it in a browser with no debugger and got some more useful information but still didn't understand what had changed (Invalid config data because I can't read it type error).
Anyway, eventually I added the IIS_IUSRS group to my code directories with read access (only) and it all started working again.
Not sure whether this permission was not required before or implied or whether one of the updates removed its permissions from everywhere but it was a bit of a pain and not obvious which 'user' the system was trying to access the data with. I assumed the debugger was running as me and I had full permission but obviously somewhere it gets changed from me to IIS_USRS!

Prevention vs Detection

2010-03-18T02:06:00.000-07:00

I read another story about government (or in this case the Police) adding audit trail functionality to the Police National Computer. This is being done to help detect abuse of the system.
Now audit trails are great and should be used in many databases to know actually who/what/when, however, what the government still fail to grasp is that prevention is better than detection. For instance, if you were running a banking system, you wouldn't rely on an audit trail to find out who transferred £10 million out of the bank to someone elses account, you would have a system that requires authorisation to make the transaction in the first place. Simply knowing who committed the crime will not help in the majority of cases (the accused has fled, money already spent, information already used to their benefit). This is as well as the fact that catching someone then requires legal action that may or may not work and will cost even more money.
This is a crucial principle and it is what needs to be recognised for databases that contain identity information. Saying that "we have training", or "we audit all access to the system" is quite bluntly not good enough. Once that information is stolen and sold (usually) it cannot be recovered, the damage cannot be undone, someone is left to pick up the pieces or a guilty person mascarades as an innocent person and can commit all manner of crimes.
Dear Government, this is Computer Security 101. Prevention is better than detection. If you can't prevent, don't deploy the system!

Why security is important on Intranet Apps

2010-03-15T02:13:00.000-07:00

I have heard people say that security is not important when dealing with internal web applications, that is ones that exist behind a firewall and which are only used by employees, all of which will have a user login. The argument is that we trust our employees by and large and that the firewall means we do not have to concerned with annoymous hackers who might attack apps exposed to the www. Unfortunately this logic is ill-conceived and simply wrong. Here are some reasons why this approach is bad.

Just because an app is behind a firewall doesn't make it unreachable from the net. Firewalls can sometimes be hacked, it might be disabled accidentally long enough to get to your web app at which point damage can be caused. It is like having a castle whose only defence is an outside wall. However hard it might be to penetrate, once it is, the castle falls.
Although there is a general trust of employees in most companies, it is not correct to decide security policy based on this. Even employees can act maliciously or curiously and can do anything from deliberate damage to accidental damage so the system must treat them as untrusted, only giving them what they need. If something was to be changed then at least it can be narrowed down to the few people who have access to a certain part of the app. If a certain area has common access, audit all important operations so that you can tell quickly and easily who has done something. Maybe they acted ignorantly and need training, maybe they were trying to damage the system.
Adding the security code to a reasonable degree is not massively difficult. Most of the information exists on the web and lots of code has already been written. Why leave it out and then potentially in the future, when exposing it to the www, would you try and retro-fit it with all the work and risk that involves. Why not put it in at the beginning. There are several reasons why an app might be exposed to the www so why base policy on the fact that it never will?
Effective defence means defence-in-depth. It means you rely on several measures to protect a resource, like several walls in your castle, so that even if the outer wall is broken, you have time to counter the attack before the inner walls are broken - this is just common sense. Relying on a single defence and assuming that there is little chance of an attack is small consolation after such an attack occurs. This is worse if what you are protecting is valuable or covered by statute and where you might have to excuse your policy before a judge or enquiry.

Just think about it, read about it and do it!

Factory Method - Why use it? What is it for?

2010-03-10T02:56:00.001-08:00

The factory method is a creation pattern used in software engineering. Like a lot of patterns, it is sometimes hard to understand why to use it, since it involves a little more code and doesn't seem to provide a great benefit. Here is a real-world example and a software example.
Consider the Bank of England, they produce money and distribute it to other banks/people who can then spend it at which point it will eventually end up back at a bank. Now suppose that we all had to make our own money. We would need the tools but even if it was easy, what would happen if the bank wanted to change a coin or introduce a new one? Everyone who produced coin would have to get the new tooling/instructions and this would be a headache. It would be hard and some people might carry on making the old coins and would only have a problem when they were spent. We don't do that. Money both originates from and ends up back at the banks. It is a factory method. If the bank wants to change a coin, it changes it and the new ones appear automatically. In this case old coins take time to be used up but it is relatively easy to do.
Now consider a software problem. A classic: database access. I want to call a function that calls a database function with parameters, how do I do this? The most trivial way is this:

DataAccess.CallFunction("nameOfMyProc", new SqlParameter("@Name", Value));

What is the problem with this? It assumes that we are using MS Sql Server and this will never change. It assumes that the SqlParameter constructor takes exactly 2 parameters and possibly these are valid assumptions but do not promote either re-use in other systems that might not use Sql Server and does not allow for using a different database without loads of code changes. If the SqlParameter constructor changed which is not unheard of, everything breaks. A little effort now and the factory pattern can give us the same functionality with none of these problems. We create a function in our data access class called CreateParameter and this can return an abstract base class of parameter (in my case DbParameter) so consider:

DataAccess.CallFunction("nameOfMyProc", DataAccess.CreateParam("@Name", Value));

Nothing here assumes Sql Server usage; if the 'shape' of the SqlParameter which we are actually returning from the CreateParam function changes, we can possibly keep this change hidden inside the data access layer (depending on what the new argument might be) and we can literally unplug a Sql Server layer and plugin e.g. a MySql layer instead with NO code changes outside of the DataAccess.
A lot of people would say, "we will always use Sql Server, it's not an issue" but there are many reasons why you might want or need to change in the future. Imagine someone releases a free and fully compatible database that gets rave reviews - save a few quid! Imagine for some reason that you have new database servers which are not compatible with Sql Server. Imagine you get bought out by a larger company who insists you use MySql. It would look really professional if you simply wrote another data layer and knew that it would all work!

Windows 7 - First Impressions

2010-02-26T01:53:00.001-08:00

I have the privilege to now have a Windows 7 desktop at work. I want to find out whether it runs all my development software etc. Anyway, a guy here installed Windows 7 Professional on this new Dell Optiplex 780 (Core 2 duo 3GHz 3.25 Gb usable RAM) and installed Visual Studio 2005, 2008 and SQL Server 2005. He setup the domain configuration etc and then gave it to me to finish off. So far I have tried to copy files from my old PC to my new one via a network share, install MS Office and carry out Windows Update. Here are my results (not great I'm afraid).
I have had lots of problems with the network. This machine has an Intel Gb card connected to a 100Mb hub on my desk into the network. It basically works but it seems when I am doing a lot of network intensive stuff (windows update and copying files) the network fails, the copying fails and the only thing I can do is reboot to fix it. I tried logging off and on and then the system froze on login.
A second problem was related to using VMWare Client which for whatever reason doesn't currently support Windows 7. I found a workaround that required modifying a config file under program files. I have admin access to the whole PC so I clicked Open With on the file, opened it in Wordpad and then tried to save it. "Access Denied". I tried tacking ownership and all sorts of things. "Access Denied". I have full access to the directory and can create/delete/copy things but cannot modify the file. There is no option to "Open with .. as administrator". Another x for MS, they still haven't got this security thing worked out. Anyway, I worked around it by copying the config into Documents, editing it and then copying it back. How ironic that I cannot save the file but simply move it elsewhere and do the same thing (this is not obvious). After doing this, I created a batch file as per the original workaround and tried it out, no probs. I then rebooted and tried to link my desktop shortcut (to the original exe) to the new batch file and lo and behold, it had been removed by the system. Again, a poorly thought out security system that simply doesn't work. What am I supposed to do? Some horrible hack? Turn off the security to make it work? Spend hours trying to fix it? NONE of this is intuitive.
The other thing and it exists on other software is the Knight Rider progress bars. Windows Update has one. The point of a progress bar is...to show progress, not to swish from side to side which means nothing whatsoever (it doesn't mean anything is actually happening, just that Windows hasn't locked up). Come on MS, a download progress bar is easy.....

Ajax Control Toolkit AutoCompleteExtender problems

2010-02-17T01:57:00.000-08:00

I have used this autocomplete control and to be honest it seemed to work pretty much as expected. However, I had originally coded it into a specific page and wanted to make it into an asp.net user control so I could use it all over the place. I had problems though so here are the things I had to do to make it work:

Move the service method from the class to a web service (I already had one so I added it to that). Make sure that the web service is marked with [ScriptService] and the method with [ScriptMethod] I also marked it as [WebMethod] just for fun.
This method when in a web service MUST be non-static and although it can have any name, it MUST return a string[] or List and MUST have two parameters that are of the type AND NAME: string prefixText, int count since the reflection system uses the names to look these up (messy but that is how it is!)
For some reason I couldn't use a server tag to insert the unique ID of my text box control into the TargetControlID of the autocomplete control since it would be mangled inside a master page. I therefore added the code: autoCompleteExtender1.TargetControlID = enterSearchNumber.UniqueID; into the Page_Load method inside my user control. This is NOT inside if (!IsPostBack) so that it links every time.
I created an event in the user control that is fired when the user presses a "Load" button using the normal EventArgs type technique and which returns the selected object in the event args.
I set the ServicePath property of the autocompleteextender inside my UserControl to (in my case) servicepath="~/Quotation/BusinessLogicAccess.asmx" and also obviously set the servicemethod property to the name of the function.
Then since it is used inside an UpdatePanel for Ajax, I had to add the following inside my ScriptManager:

Other than that, it was the normal UserControl stuff like using @Register in the control to link to the AjaxControlToolkit.dll and using it as normal.

Text boxes and Text areas not posting back

2010-02-15T04:18:00.001-08:00

I have a page with some text boxes that the user can change in the rows of a datagrid. Because however there is potentially a lot of text, the datagrid row only displays a small box with about 30 characters and an edit hyperlink brings up a JQuery dialog to edit the text.
The problem I had was that when I updated the row, even though the text was displayed in the small textarea, it was seemingly not posted back and not updated.
I thought it was some funny datagrid thing or viewstate but then I noticed that the values for quantity and item (a select) were posted back so something was wrong.
Simply problem, because the text areas were set to enabled="false" the browser (or the standard) wrongly assume that they can't be modified and don't bother posting them back in the querystring. Instead I had to use readonly="true" which works but doesn't look the same as a disabled text box. Potentially quite a hazrd for someone who literally loads an item, sets all the fields from what he thinks the datagrid is displaying and then overwrites it all!
UPDATE: Setting read-only to true does NOT work. It causes the value to be passed back in the querystring but ASP still ignores the changed value. I think what i will do is leave the box enabled but handle the onclick handler to display the full-size edit dialog! Nice. I could also cover the text boxes with transparent divs but that is one of those things I don't really want to do!

Event Handler not called from DropDownList

2010-02-03T01:43:00.000-08:00

Had a really annoying problem with an asp.net site, one of those things that turns out to be really simple yet takes ages to track down. I have a master page with a script manager and update panel and then a contentplaceholder inside the update panel so that the content area of all new pages on the site can have ajax for free. People talk about large overheads etc but quite simply the updates are much snappier with the update panel and you get a free 'waiting' dropsheet if it takes a while to do.
Anyway, in my page, I had a dropdownlist with a list of categories in it and an event handler connected to onselectedindexchanged, I set autopostback to true but although i could get the postback, the page would not call the event handler. I assumed it was a combination of Ajax and master pages but it was much more simple.
The data I had bound my list to was a collection that expects an int (for datavaluefield) and a display string for the datatextfield. Since my data did not have unique ids, I returned zero for each value. What happened then? Well since the value didn't change when I changed the dropdownlist, the system didn't detect that the selected index had changed and didn't call the event handler. It would have been more obvious if the event was called onselectedvaluechanged since the form only posts back the selected value and not the index but anyway, hope this helps someone else.

You MUST backup Linux PCs

2010-01-12T03:41:00.001-08:00

Lots of people know that when you delete a file on a computer, you do not remove the contents of a file, you simply remove its "index" entry. Certain programs can forensically scan a disk and recover this data. However, as I found out the other day, this process is VERY slow and mostly not 100% effective. There are a few chances with the journalling system but that is hit and miss.
Why? Well for a few reasons. Firstly and most obviously, the file data on the disk might now be overwritten by a new file - one reason to shutdown as soon as you cock something up. You might want to kill the power since shutting down can cause log writing etc to overwrite the disk. Onto more details however, small files are often stored in adjacent areas on the disk, in addition to this, certain files have known markers (i.e. a jpeg starts with a specific byte and finishes with another) so small images can often be recovered easily. Large files however are often broken into chunks and put anywhere on disk, although within a certain boundary. When you delete a file in ext3 or 4, the information about where these chunks are is lost (unless you are fortunate enough to have updated the file recently and you can find the info in the journal).
At best you know that the file exists somewhere in un-allocated disk space so you can dump all this to a single file. You can then hope that the blocks are contiguous which to be honest is likely for small files and unlikely for large ones. You can then scrape this data to find beginning and end markers which again only works for certain documents (although if you know some of the contents of your file, you can use this to find it). If the blocks are broken up then best case is you get other parts of files inside your recovered file which might or might not be fixable and worst case, you cannot put your file back together (i.e. you will not be able to manually scrape binary files with unreadable content).
Another issue is that the scraping can lead to hundreds of thousands of recovered files (anything you have ever deleted, you might even have moved something and end up with several copies).
The Police might find this useful because they would only need to recover several images or documents to incriminate someone but for specific files, it is more of a miss than a hit.
The moral, make sure you back up all your important files. I don't know how mine were deleted (not exactly anyway) so it can and does happen!

CacheItemRemovedCallback not firing

2010-01-12T03:33:00.000-08:00

I was getting a bit annoyed using HttpRuntime.Cache in an asp.net app. I really needed the cache since I was generating reports and to do these each time can be very slow (~30secs). The problem was that the cache never seemed to expire.
When caching, I used the Add() function to add a placeholder of the string "caching" and then started a thread to do the work, this was to prevent caching the item more than once at a time. Inside the thread function, after generating the report, I then used the indexer [] brackets to update the "caching" placeholder to point to the generated report. This worked in as much as it was caching but it never expired. I added a CacheItemRemovedCallback function just to check and it was never called.
After trying loads of things, I found that calling the Insert function via the indexer [] form does NOT re-use the expiry settings of the original Add function and possibly also not the dependency and removed callbacks - it sets the item to no expiry.
All I had to do was call the Insert function directly and give it the expiry I wanted and all was well again. Another major omission from the MSDN docs but at least now you know!

A New IDE Would be Nice

2010-01-11T13:56:00.000-08:00

Most things are like church. You carry on doing things the same way you always have until someone asks why and no-one really knows. You just have.
Take the trusty Integrated Development Environment (IDE) I use Visual Studio mostly and to be honest it isn't bad from a usability point of view. However, I have been reminded recently how easy it is to introduce bugs into code because at the end of the day these IDEs are generally little more than glorified text editors. OK, we all know in theory that our design tools generate code for us so we can't make mistakes but in reality we do, we change the name of something and the best we get is an auto-prompt asking what word we want. If we change a name then maybe the compiler will find any dependencies and show us errors. At best this is time consuming but at worst if you are using late-bound technologies/reflection etc then your program can fail at run-time.
Rather than a text editor, it would be nice to see a tool that I guess is more like a design tool but treats fields and functions as symbolic objects that can be linked symbolically rather than by name. This way they can be renamed willy-nilly and can have properties attached (meta-data) such as pictures, descriptions etc that can form something much more rich and useful than plain text and hopefully, eventually, something more robust that is harder to break when we modify and refactor, something that even, maybe, can prevent us making errors in code since it will not permit broken symbolic links, will not permit us to leave pointers un-initialised etc and will lead to high quality software that doesn't have the usual umpteen thousand bugs when it goes into system test for the first time.
Maybe...

Hacked Email Accounts, Phishing etc

2009-12-18T02:40:00.000-08:00

A colleague of mine was concerned today because she noticed funny stuff happening related to her Hotmail account. She was not tech savvy but knew something was wrong because she thinks her emails have been deleted from her inbox and also she had a reply from a friend thanking her for an email she did not send. I decided I would write a guide to email safety for people who have little or no technical knowledge.
The first important point is that email is generally extremely insecure. Although it is possible to use techniques mechanisms to make it better, in general email is sent in plain readable text over public networks. On top of that, the way in which email is sent is archaic and easy to spoof.
For instance, when you receive an email that says: from: markysmith@hotmail.com, you assume that the email is from that person. You assign an amount of trust to the information in it depending on how well you know Marky Smith so you might click on links to various sites, "check this out" or go to a product page at a shop. The problem is that you have no guarantee that the email is from the real Marky Smith even if it is the correct email address.
Firstly, the from address can be set to anything so I can send an email and change the from address to "bill.gates@microsoft.com". You must not trust the from address by itself.
Secondly, people often obscure information in emails to make them look trustworthy. They might pull actual images from, say, a bank website and create an email that looks like it comes from a bank. If it is sent to a million people then chances are some of those will believe it to be genuine. The sender might use web links that look real such as ebbay.com so that a quick look and all appears to be correct. Also, it is possible to write a link that looks like www.microsoft.com but make it direct you to a totally different site (the idea is to hide horrible looking links and make them friendly like "click here" but it can be used the other way round).
Thirdly, these various problems can be used by people who write viruses, especially ones that can read address books, and they can send email pretending to be from person a to person b while actually coming from the computer of person c so that Mr Evil can lure you into a sale or a site that will infect your PC.
What can you do?
1) Make sure you have an up-to-date virus scanner if you use Windows or Mac. I cannot stress enough that your PC can cause massive harm if it connects to the internet and gets infected by something. Harm to you, obtaining your personal data or launching some sort of attack against someone else. This will prevent many situations where you might accidentally get a virus.
2) Use a site like opendns or software like Net Nanny to prevent accidental or deliberate navigation to dodgy web sites, many of which could harbour viruses with the promise of "free nude piccies". You might need someone to set it up for you but talk to your local PC shop (not PC World!) about what options are available.
3) Never trust emails. No official body ever needs to send links in emails but if they do, try and navigate to it by opening a browser and going to the home page directly (don't click the link). Only click email links if you are expecting the email, such as just having registered for something and having to confirm it. Use a browser like Firefox (free download) which will warn you if the link you are clicking is not the same as the text displayed (i.e. it is dodgy). If you send emails with links in to your friends, get in the habit of typing something in the email that proves you are who you say you. Phrases like "click here" could be genuine but could easily be bogus whereas "Hi Sallykins saw this vides, reminded me of our night in town" is more likely to be genuine.
4) Educate your friends and family. Even youngsters are not getting taught very good web etticate at school. The more people who wise up, the harder it will be for people to get their way.
5) Change passwords every few months and try and keep them different. It would be better to write them down somewhere at home suitably obscured (I sometimes hide them in addresses in address books) than to never change them for fear of forgetting. Also use strong passwords with numbers and characters, you can make simple words doing this, such as @lbatr0ss

Be safe!

Password handling and hashing

2009-10-20T01:47:00.000-07:00

A lot of people take a very simplistic or optimistic view on security. You assume that no-one will ever hack your system (perhaps there is no obvious reason to do so), you trust everyone who works for you or you simply don't really think about it. One of the most serious problems is the handling of passwords. Here are some facts that you might not know about password best practice.

People often use the same passwords for more than one site so if you allow access to human-readable passwords from your site, it migth well compromise something of higher value like ebay, hotmail, banking etc.
You must use defence in-depth. Just because you think that your database is secure doesn't mean you can stored passwords in plain text. Using this approach, if somebody does hack in, they then have an open door to do whatever they want.
Although it is hard to hack a hashed password, it is not impossible. (hashing is obscuring it in a way that makes it very difficult to find the original value). A hacker only needs to find a word with the same hash regardless of whether it is the actual password. Some people run machines 24-7 building a list of word-hash pairs so it can reverse lookup hashes. For this reason, you should salt your hashes. This means if somebody chooses a password of, say, "PrettyBoy", before hashing it, you change it in some known but secret way like add the characters "123" to the end (or preferably something more obscure) before hashing. Suppose then the hash of "PrettyBoyab145" is ABC123, even if a hacker discovers this (which would require knowing which hashing algorithm you use) and does a reverse lookup for the password, he finds "PrettyBoy123", types it and the password fails because checking the password "PrettyBoy123" adds the salt "123" to get "PrettyBoy123123" which when hashed (e.g. XYZ456) and compared to the stored value of ABC123 will not match. Of course with a simple salt, the hacker might guess the actual password but if you add a number to the bytes of the password then this will be much harder to decipher since reverse lookups against standard hashing algorithms will probably produce garbage which won't look like a real password.
You NEVER need to store plain text passwords. If someone forgets the password, use a reset mechanism. Ask them a question or two, use their registered email address and allow them to set a new password but make sure the system cannot even theretically allow a hacker to call the reset function against someone elses account. e.g. the stored proc that resets the password should look something like: procResetPassword(varchar @account, varchar @email, varchar @answer1, varchar @answer2, varchar @newpassword, varchar @newpasswordconfirm). A hacker cannot know all of these, if he does then he can simply use the normal "forgot password" page and wouldn't need to hack in the first place.
You need to stay up to date with the latest knowledge concerning hashing algorithms. As technology gets faster, older algorithms tend to become insecure because it takes less time to reverse them by brute force. You don't need to update the minute somebody suggests a weakness but once a year checking and upgrading can be useful. Bear in mind, this will affect existing passwords so you would either need to know which hash to use for a given user or have to get all users to reset their passwords.
Even having a good backend system does not prevent somebody snooping on the transmission between the web page and back end when by default information is sent plain text. You should consider redirection to HTTPS (SSL) for login, even if you then redirect back to standard HTTP. This way, there is encryption between client and back end.
Even if the transmission is secure and you have a secure backend, a session is generally only a cookie so it is usually easy for somebody to jump straight onto a machine used by somebody else and gain access to the site. Cookies might also be stored and reused later. Ensure you understand sessions. Always provide an accessible logout button which wipes the session and try and use existing libraries instead of rolling your own. Allow others' mistakes to improve your code.

Play safe!

Creating QT Widget in 4.5

2009-08-05T01:31:00.000-07:00

I have started writing an app in QT for controlling my keyboards via MIDI. QT is the obvious choice because it works really well on Linux, can be ported across platform (although I am using a Linux MIDI library) and the tools and help that come with it are superb. Anyway had a little fun creating a widget.
I wanted a widget for a double list box with the two buttons to move items from the left into the right list and vice-versa. Assuming that I might well use it again, I wanted to make it visible to the designer so I found an article on the Trolltech website which pretty much describes the process: Here but I had a few fun experiences along the way since I am not massively familiar with QT and the QTCreator IDE. So I created a new project using "New" and chose a widget as the project type. This doesn't set things up exactly as required (it inherits from some styleplugin thing and the output path is a style path) but it does some of the work like setting the "debug_and_release" compiler switch and "library" as the output type.
You need (assuming one custom widget in the plugin) a class for your actual widget which is literally as you would write any other widget - inherits from QWidget and houses whatever contents it needs. The other is the plugin class which you pretty much copy from the example and rename for whatever you want.
The first problem was seeing the target.path and CONFIG variables mentioned. I couldn't find these and it wouldn't build then I found out that the .pro file is actually text and can be opened into the IDE where some of these variables are. It is confusing because there are some project properties under the projects button but hey-ho.
The next problem I had was related to accessing the designer directory which would allow my plugin to be available to all other projects in the QTDesigner. This is what was set in target.path but by default the permissions are for root access only so this is what I did:
Navigate to the directory above designer (in my case /usr/var/qt4) and then run "sudo chmod 777 designer" which allows full access into the designer directory for everyone. I tried read/write only but then when it tried to "stat" a file it failed so had to give it the works.
It now all builds fine, just make sure you follow the tutorial.

Securing your web apps

2009-06-18T04:51:00.000-07:00

Web apps are so commonplace today that we totally take them for granted. If you write and maintain sites, how much do you think about their security? How much do you know about what dangers and vulnerabilities exist in your software/web server/database server and in what way can you protect yourselves?

Why protect?

You don't want people to gain access to private/sensitive information. Once it is out, it is out!

If your site is infected with malware, it will affect your reputation and can result in your site being blocked to everyone (including customers!)

You don't want people damaging or deleting your work in what might be a totally malicious attack. People with no backup policy have lost immense amounts of information which is mostly related to someone's time and therefore money.

Just because your site is on an intranet doesn't mean someone won't get through your firewall one day (for whatever reason) and also if your app was to go public, you DON'T want to retro fit security

You might think you trust everyone in your company but this is a false assumption and bad security can expose things accidentally as well as deliberately and there is possibly a lot of data to expose that you would rather stayed hidden.
It is good discipline to get you used to thinking security

If people get passwords from your users and something like an email address, there is a good chance they will be able to log into common sites like EBay, Facebook and Hotmail since people regularly use the same passwords for more than one site.

By more people securing their sites, malware has less effect and will reduce the current efficiency of people trying to spread chaos/SPAM via infected machines.

How do we approach security?

A weeks security course would be money well spent

Assume that all data from outside your control zone could be malformed. The browser is not the only way for information to be passed to your web app so any client code/javascript/error checking can be totally bypassed.

Rely on a chain of protection at an appropriate standard in every level of your app. A key checker on the web page is OK but doesn't stop bad code being injected so you also need sanity checking in the code-behind files and in your business objects and in your database. Don't think that one layer will prevent all attacks

Use stored procedures, NEVER use SQL directly to query databases from your business logic since you open up so many avenues for SQL injection. Parameters in stored procedures escape any text that might contain SQL code so it can not do anything useful in the database.

Never reflect untrusted user input back to the web page without sanity checking it. It is easy to inject cross site scripting code into an unchecked input field which then gets parsed by the browser on reflection as valid HTML which can be used for all manner of bad things. Using regular expressions takes some learning but they are fast and very powerful. You might, for instance, not allow usernames to contain anything other than a-z, A-Z, 0-9 and the @ symbol.

Do not go outside of standard software and techniques unless you are properly qualified and able to do so. Standard software gets a lot of exposure and bugs are usually seen and fixed quickly.

ALWAYS hash passwords so they can never be seen even after a successful hack. You do NOT need to be able to tell your users what their password was. If they forget it, have a system to reset it/set it to something else in a secure way.

Encrypt ALL data that has either intrinsic value (bank account details) or that cannot be recovered/changed esily if lost like names, dates of birth, mothers maiden names etc. Again you cannot guarantee that a hack is impossible so better let people steal a load of data they can't read. Have a secure system for protecting encryption keys.

Become involved in owasp, a not-for-profit organisation that does a lot of work in the area of web security and who have many resources and articles about safe coding and safe processes.

Always use intrusion detection, setup to balance the security with the extra server load/cost. For instance, always lock out accounts that have 3 invalid logins and which require a phone call or secure reset method. This prevents brute-forcing of password.

Don't be blaze about it, you don't want to be the next person to lose 100,000 patient records!

Good OO Practice

2009-06-16T02:42:00.000-07:00

OO is more like art than science. There is usually more than one way to achieve certain functionality but here are some general suggestions for good code.

Encapsulate stuff. In other words, make fields private and other functions private, protected or public - make the security as restrictive as you can. The more secure, the less that the function can be called which reduces code use and the chance for problems as well as making it easier to know how to test it. It also prevents people calling functions that seem to be right when they are supposed to be called internally to a class. "Internal" is useful for keeping stuff visible inside a single assembly (dll). Think about the testing required for a dll with 100 public functions rather than one with 10!

Only pass into functions what you need to use. Do not pass a parent object and then extract some property of it, if it needs the child, pass the child in. This again makes the intent of the function clearer and allows irrelevent changes (such as changing the parent type) to not affect functions that don't care.

Try and make functions either call a list of other functions OR do something technical and low-level. The functions that call functions are easy to understand and check, the others should be small and easy to unit test. Mixing them makes them hard to understand.

Try and work out whether it is better to replace structured constructs like switch statements and if/then/else with classes. Putting logic into structure is so much more robust and easier to check. Consider a statemachine in a switch, what is easier, to ensure that all the other case statements change the state variable correctly or in an OO state machine to have a class named MovingState that has specific transitions and guards coded into one function in one class? Again encapsulation is your friend. You can't affect anyone elses variables or functionality because you cannot see them.

Try and consider a class as either data, control or boundary and don't mix them. Data classes are persistent (usually) and have functions related only to their data. Control classes can be created and destroyed as they are used and manipulate data or send information to the system boundary. Boundary classes translate from one system to another (including the user interface, ports, the network etc) and can be considered pluggable so that, for instance, a web service can be plugged in where a user interface was previously since they can both send data that is conceptually the same even if one has no actual user interface.

Use the safest types for the data you have. If you are doing money stuff, a decimal is almost always preferred to a double or float because the maths functions work properly. If your data will be integer, use integers for similar reasons. If you have pairs of data that are related, either use something like a Map/Hashtable to link them or otherwise create a type to encapsulate both, don't assume that you can keep any related things up to date with each other without a setter function or constructor as the only means to do so.

Calling Web Services from Reporting Services

2009-06-10T03:51:00.000-07:00

I wanted to do this because I didn't like the idea that we needed SQL procs in a database for reports that have to do the same thing as business logic dlls did for the web app pages. This duplication not only made things harder (being done twice) but allowed the possibility of breaking one when the other was changed and the fact that SQL is so different from code that one was often much easier to change than the other. I found out that you can call web service functions from reports so that the same code can populate reports AND web pages. It is not massively easy to work it out and my example uses Reporting Services 2005 and .Net 2.0. It also uses the Reporting Services XML Data Provider for the dataset (since the web service returns xml in the form of soap).

Create your web service as per a normal web service. In order to be generic and not need changing or growing for every different sort of function I would need to call, I made some generic functions that use reflection to call whatever function I want. You can probably work out what I did if I show you the code for the generic function:
```
[WebMethod()]
public ArrayList ExecuteArrayMethod(String typeName, String method, object[] ParamArray)
{
 ArrayList returnObject = null;
 Type type = AccessType(typeName);

 try
 {
  if (type != null && type.GetMethod(method).GetCustomAttributes(typeof(ExportedMethod), false).Length > 0)
  {
   returnObject = (ArrayList)type.InvokeMember(method, BindingFlags.Default | BindingFlags.InvokeMethod, null, null, ParamArray);
  }
 }
 catch (Exception)
 {
  throw;
 }
 return returnObject;
}
```
There are a couple of points of note. The ExportedMethod attribute was something I added to control what functions were allowed to be exported to the reports and what couldn't, it is optional. Also, the arraylist allows collections of any type to be exported via the function. I have created a dataset flavour which I haven't tested yet and which I don't think I'll need.

Obviously the functions that actually generate your report data must be callable from the parameters you have available in your web service. If the function needs a load more information like connection strings, arguments etc, then either pass them in via the web service function or if they are private, create a parent function that can obtain these values and pass them to the worker function. The function you call from the web service must be static (I think!).

You will need to declare the types of any exported classes (i.e. types put into the ArrayList) at the top above the web service class definition even though you never need to reference them by name in the report. This forces the webservice to include the property data in the xml schema used by callers of the functions. You do this:
```
[XmlInclude(typeof(ClassName1)), XmlInclude(typeof(ClassName2))]
```

Build and deploy the web service to a url somewhere, such as http://myhost/webservices/myservice.asmx

Navigate to the asmx file in your browser and click on the function that you will be calling from your report. You should get an invoke section at the top which you can use to ensure that your code is returning the expected data. Do not go straight to the report without checking this, the report syntax is hard enough without the underlying data actually not working! Also note the RESPONSE syntax under the soap 1.2 section (the second of the two brown boxes). It will contain some soap bits (ignore these) and then a response tag, a result tag and then data depending on what you are returning. These are the important bits.

Open up your reporting services project (or create a new one) and add a datasource of type XML. In the connection string field, put the path to the asmx web service including the http and the asmx filename itself. Under credentials, you will need to put whatever is needed to call functions on this web service. In my case, it is a private intranet so I use windows credentials.

Create/edit a report and add a new dataset that uses this XML datasource, set the type to text and this is where the fun begins. You need to use a query syntax similar to XPath but not quite (search the web for details, there are various MSDN articles about it). This names the method, the parameters and the expected return values (and optional types). I will show mine so hopefully it makes more sense:


<Query>
  <Method Name="ExecuteArrayMethod" Namespace="Namespace.Of.The.Webservice">
      <Parameters>
       <Parameter Name="typeName">
       <Parameter Name="method">
       <Parameter Name="QuoteReference">
   </Parameters>
  </Method> 
   <ElementPath IgnoreNamespaces="true">
           ExecuteArrayMethodResponse{}/ExecuteArrayMethodResult{}/anyType{Location,Description,Schedule,Item_Code,Quantity(Integer),OriginalSale(Decimal),RevisedSale(Decimal)}
</ElementPath>
</Query>

Note that the anyType is the type that is returned in my soap response because I used an arraylist, it might be something else depending on your data. I have also typed some of the return values so that they can be treated as numbers rather than the string defaults. This XPath relates to the following web service xml response:


<soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
  <soap12:Body>
    <ExecuteArrayMethodResponse xmlns="Namespace.Of.The.Webservice">
      <ExecuteArrayMethodResult>
        <anyType />
        <anyType />
      </ExecuteArrayMethodResult>
    </ExecuteArrayMethodResponse>
  </soap12:Body>
</soap12:Envelope>

Every time you save the xml, it will parse it and give you an error if there is a syntax problem, it does not understand line breaks so you might get a line1, character 300 error or similar, unfortunately, you have to try and build up the element path if your syntax is wrong and hope for the best.

If your web method requires parameters (as mine does), click on the ellipses next to the dataset (...) and click the parameters tab. You need to define any parameters passed to the method here so either a) use static values (in my case the method parameter is static for any given report) or to use a previously created report parameter if it will be different for each instance of the report (like my QuoteReference is).

Exit the edit pages and click the ! button to call the method. You might be prompted for parameters and you should see your expected table of data or you might get an error. If you get data but it is incorrect, check your element path. If you get no data, your element path might be wrong or there might be no data being returned, remove the element path and leave it blank to see if there is actually data present, if there is, your element path is still wrong. if you get an error, click the little red X icon and make the dialog bigger, the errors are often hidden but usually useful "no permission", "could not instantitate type" etc

Once you get data in the right format, your dataset on the left should now show the field names which can be used on the report in the normal way.

One of the big benefits for us is that the workload is shifted from the database server which also seems to be chugging away to the web server which is fairly lightly loaded since the work is now mostly done in a web service not a view/proc.

When you deploy your report to another server, you might get other errors and these are sometimes related to the permissions with which the report server is attempting to access your web service with. When you go to the web service in your browser, you are usually logging in as you and you might have access, the report server however logs in to the webservice as the report server (and not the person running the report). I got round this by modifying the datasource on the live server to use specific login credentials rather than windows credentials.

Why have a software process?

2009-06-09T02:29:00.000-07:00

There is always lots of hype about software processes for quality/productivity etc but like a lot of people, I was often unconvinced about their value. At the end of the day I enjoy coding, not writing documents and so I would pay lip service to the process of design and specifications but only the minimum amount of work and then dive into code. I was attracted to the idea of things like eXtreme programming which promised quality without the overhead of design and specification but sadly this is a lot of gas imho, quality is not just related to the lines of code but about a whole lot more which is why the process is important. Now that I have less process and less design and specification to work to than my previous employer, I find myself uncomfortable and pine for documents to solidify the quality of my work. So why have a process including design, specification etc?
1) The earlier on the process, the easier/quicker/cheaper it is to fix things. For example, I had a spec review which revealed that a whole chunk of functionality was not required. A minute making a decision saved me potentially days of code (however good it was)
2) It requires that your customers (even internal ones) think about what they want. It is easy to spend 10 seconds saying, "We want a page to do x" with all the cost associated with it, whereas for them to spend a few hours defining it means they will think about whether it is really worth it and allows holes in their assumptions to come out.
3) It enables everyone to have buy-in and sign-off on functionality including everyone agreeing formulas/workflows etc and also allowing budget holders to question the value of the work compared to the proposed benefits.
4) It reduces the likelihood of rework at a later date where the original system implements assumed functionality and then the customer realises it is wrong and needs it to be changed. EVERY change is a risk and takes time/costs money. These risks are in productivity and reputation and why bother reworking? Sometimes it is unavoidable but reduction is better than nothing.
5) It allows systems to be seen in a larger context by people who care about more than one area of the system. With one spec next to another on the desk, a manager can ask questions related to the interaction or consistency of systems/sub-systems with each other. For instance, they might notice duplicated functionality and allow something to be removed in one sub-system since it will be present in another.
6) It permits a quality system to be implemented. If you notice problems in code that are actually related to design weaknesses, how can these be fed back into the quality cycle unless you have design and specification stages to modify?
7) You can have code of 100% reliability/quality from the programmers point of view but that lives in a function/class/system that is not actually of any business use. This is why quality and process CANNOT rely solely on a code view of quality such as eXtreme programming and other shortcut processes.

If you don't have a system, start one. The managers will often need convincing that the extra time is worthwhile but actually if implemented properly it will save time not increase it although there no doubt will be some short term overhead as you learn the way things work.

Viewstate, postbacks and pains in the neck

2009-05-28T08:47:00.000-07:00

It took me a long time to get my head round what was going on asp.net. Sometimes controls did not get their correct values and things didn't always get bound. Here is a simple guide that hopefully will help you know what is going on.
1) The web is not connection based by default. When you go to a web page, you request a page, you are given it (usually) and that is the end. Things like session etc have been cobbled on the top of this system to try and keep track of things but the server never knows for sure if you are still connected which causes all kinds of problems with cookies and secure sites and loggin out etc. If you remember this connectionless pattern, this will help you understand.
2) What happens when you FIRST request an aspx page? The server recognises that the request is a "GET" HTTP request so sets the property IsPostBack to false. It calls various event handlers in a prescribed order (you can find these on msdn) most importantly Page_Load which you would use to set the page up. You probably managed this.
3) What happens when you change something on the page after it is served? This depends on what control you change. By default buttons cause a post back but data fields like text boxes and radio buttons do NOT unless you set their autopostback attribute to true.
4) When you finally cause the postback, a POST request is passed to the server along with the values of any fields on the page (which is why you need to put all the aspx controls inside a form). The server sees that it is POST and assume this is a "post back". It unpacks all the form values into member variables and it also uses something called viewstate to remember any data that is not a current value for a web control. For instance, a text box has no viewstate because the only data that needs to be remembered between posts is the text in it, this is already passed in the form data with the post. Controls like TreeViews however want to remember whether they were expanded etc and because this is not part of their 'current value' it has to be specially stored in viewstate. The viewstate is a hidden field with a load of ascii characters which the system automatically packs and unpacks for you (although you can disable the viewstate to save bandwidth or customise it).
5) If you write a custom control then you might have to save data into the viewstate so that it is 'remembered' between posts (but only if it needs remembering otherwise you are wasting bandwidth). You do this with the Viewstate property which is an array of objects.
6) The area that confuses most people is when working with a data source and a postback needing to update something in a database and then change the screen somehow. Firstly you need to know that Page_Load is called BEFORE your event handler. If you need to access data in the event handler, it must be set up first in Page_Load. Secondly, you will need to change your screen data in code otherwise it will remember its last values.
7) There are some times when you do not want to update from your datasource during postbacks. This is for two reasons. Firstly, because controls remember their values, once they are setup, if they are not going to change while the page is being used, there is no point keeping updating them, secondly you might want a load of controls to come up as disabled when you first open the page and then to become enabled when you click a button. However if you disable them in Page_Load, they will disable every time you cause a postback (such as clicking the button to unlock them!). Use the IsPostBack property to only do things on the first time in Page_Load.

Mixing Forms and Windows Authentication

2009-05-22T07:28:00.000-07:00

My friend works on an ASP.net web application at work and uses windows authentication to access the various pages with database backed roles to provide authorisation. He then asked me how to further lock down certain parts of it to require a password. This would be so that certain orders could only be viewed by people who knew the password even if they were generally allowed access to the page.
Of course, the general authentication schemes are designed for a per-page basis and it seemed like rolling your own flavour was quite involved as well as using something like ESAPI from OWASP. He didn't want to spend weeks developing it so I had a play around.
I initially investigated mixing windows and forms authentication so that for the most part you would use windows but for a more protected page, it would re-direct you to a login. I thought this would be best because a lot of authentication stuff is already built in to the ASP libraries. However I shortly realised my mistake. The problem is that the authentication is the first thing to occur when the page request is made. At this point, you do not know whether the page is one that needs a password or not so you have to allow it to authenticate. If you then read the database and know you need a login, you can then force a redirect to the login page but it gets really messy because then you have to keep track of authentication state so it allows the user in the first part but then blocks and then if a correct password is entered, it needs to allow you in until you close the browser. After many hours, I opted for a simple redirect to a login page which gets passed the return url in the querystring and a logout button in the order that flushes the cache and then goes to a neutral page instead. Simple is good.