Password handling and hashing

A lot of people take a very simplistic or optimistic view on security. You assume that no-one will ever hack your system (perhaps there is no obvious reason to do so), you trust everyone who works for you or you simply don't really think about it. One of the most serious problems is the handling of passwords. Here are some facts that you might not know about password best practice.

1. People often use the same passwords for more than one site so if you allow access to human-readable passwords from your site, it migth well compromise something of higher value like ebay, hotmail, banking etc.
2. You must use defence in-depth. Just because you think that your database is secure doesn't mean you can stored passwords in plain text. Using this approach, if somebody does hack in, they then have an open door to do whatever they want.
3. Although it is hard to hack a hashed password, it is not impossible. (hashing is obscuring it in a way that makes it very difficult to find the original value). A hacker only needs to find a word with the same hash regardless of whether it is the actual password. Some people run machines 24-7 building a list of word-hash pairs so it can reverse lookup hashes. For this reason, you should salt your hashes. This means if somebody chooses a password of, say, "PrettyBoy", before hashing it, you change it in some known but secret way like add the characters "123" to the end (or preferably something more obscure) before hashing. Suppose then the hash of "PrettyBoyab145" is ABC123, even if a hacker discovers this (which would require knowing which hashing algorithm you use) and does a reverse lookup for the password, he finds "PrettyBoy123", types it and the password fails because checking the password "PrettyBoy123" adds the salt "123" to get "PrettyBoy123123" which when hashed (e.g. XYZ456) and compared to the stored value of ABC123 will not match. Of course with a simple salt, the hacker might guess the actual password but if you add a number to the bytes of the password then this will be much harder to decipher since reverse lookups against standard hashing algorithms will probably produce garbage which won't look like a real password.
4. You NEVER need to store plain text passwords. If someone forgets the password, use a reset mechanism. Ask them a question or two, use their registered email address and allow them to set a new password but make sure the system cannot even theretically allow a hacker to call the reset function against someone elses account. e.g. the stored proc that resets the password should look something like: procResetPassword(varchar @account, varchar @email, varchar @answer1, varchar @answer2, varchar @newpassword, varchar @newpasswordconfirm). A hacker cannot know all of these, if he does then he can simply use the normal "forgot password" page and wouldn't need to hack in the first place.
5. You need to stay up to date with the latest knowledge concerning hashing algorithms. As technology gets faster, older algorithms tend to become insecure because it takes less time to reverse them by brute force. You don't need to update the minute somebody suggests a weakness but once a year checking and upgrading can be useful. Bear in mind, this will affect existing passwords so you would either need to know which hash to use for a given user or have to get all users to reset their passwords.
6. Even having a good backend system does not prevent somebody snooping on the transmission between the web page and back end when by default information is sent plain text. You should consider redirection to HTTPS (SSL) for login, even if you then redirect back to standard HTTP. This way, there is encryption between client and back end.
7. Even if the transmission is secure and you have a secure backend, a session is generally only a cookie so it is usually easy for somebody to jump straight onto a machine used by somebody else and gain access to the site. Cookies might also be stored and reused later. Ensure you understand sessions. Always provide an accessible logout button which wipes the session and try and use existing libraries instead of rolling your own. Allow others' mistakes to improve your code.
   

Play safe!

Creating QT Widget in 4.5

I have started writing an app in QT for controlling my keyboards via MIDI. QT is the obvious choice because it works really well on Linux, can be ported across platform (although I am using a Linux MIDI library) and the tools and help that come with it are superb. Anyway had a little fun creating a widget.
I wanted a widget for a double list box with the two buttons to move items from the left into the right list and vice-versa. Assuming that I might well use it again, I wanted to make it visible to the designer so I found an article on the Trolltech website which pretty much describes the process: Here but I had a few fun experiences along the way since I am not massively familiar with QT and the QTCreator IDE. So I created a new project using "New" and chose a widget as the project type. This doesn't set things up exactly as required (it inherits from some styleplugin thing and the output path is a style path) but it does some of the work like setting the "debug_and_release" compiler switch and "library" as the output type.
You need (assuming one custom widget in the plugin) a class for your actual widget which is literally as you would write any other widget - inherits from QWidget and houses whatever contents it needs. The other is the plugin class which you pretty much copy from the example and rename for whatever you want.
The first problem was seeing the target.path and CONFIG variables mentioned. I couldn't find these and it wouldn't build then I found out that the .pro file is actually text and can be opened into the IDE where some of these variables are. It is confusing because there are some project properties under the projects button but hey-ho.
The next problem I had was related to accessing the designer directory which would allow my plugin to be available to all other projects in the QTDesigner. This is what was set in target.path but by default the permissions are for root access only so this is what I did:
Navigate to the directory above designer (in my case /usr/var/qt4) and then run "sudo chmod 777 designer" which allows full access into the designer directory for everyone. I tried read/write only but then when it tried to "stat" a file it failed so had to give it the works.
It now all builds fine, just make sure you follow the tutorial.

Securing your web apps

Web apps are so commonplace today that we totally take them for granted. If you write and maintain sites, how much do you think about their security? How much do you know about what dangers and vulnerabilities exist in your software/web server/database server and in what way can you protect yourselves?

Why protect?

1. You don't want people to gain access to private/sensitive information. Once it is out, it is out!


2. If your site is infected with malware, it will affect your reputation and can result in your site being blocked to everyone (including customers!)


3. You don't people damaging or deleting your work in what might be a totally malicious attack. People with no backup policy have lost immense amounts of information which is mostly related to someone's time and therefore money.


4. Just because your site is on an intranet doesn't mean someone won't get through your firewall one day (for whatever reason) and also if your app was to go public, you DON'T want to retro fit security


5. You might think you trust everyone in your company but this is a false assumption and bad security can expose things accidentally as well as deliberately and there is possibly a lot of data to expose that you would rather stayed hidden.
   
6. It is good discipline to get you used to thinking security


7. If people get passwords from your users and something like an email address, there is a good chance they will be able to log into common sites like EBay, Facebook and Hotmail since people regularly use the same passwords for more than one site.


8. By more people securing their sites, malware has less effect and will reduce the current efficiency of people trying to spread chaos/SPAM via infected machines.

How do we approach security?

1. A weeks security course would be money well spent


2. Assume that all data from outside your control zone could be malformed. The browser is not the only way for information to be passed to your web app so any client code/javascript/error checking can be totally bypassed.


3. Rely on a chain of protection at an appropriate standard in every level of your app. A key checker on the web page is OK but doesn't stop bad code being injected so you also need sanity checking in the code-behind files and in your business objects and in your database. Don't think that one layer will prevent all attacks


4. Use stored procedures, NEVER use SQL directly to query databases from your business logic since you open up so many avenues for SQL injection. Parameters in stored procedures escape any text that might contain SQL code so it can not do anything useful in the database.


5. Never reflect untrusted user input back to the web page without sanity checking it. It is easy to inject cross site scripting code into an unchecked input field which then gets parsed by the browser on reflection as valid HTML which can be used for all manner of bad things. Using regular expressions takes some learning but they are fast and very powerful. You might, for instance, not allow usernames to contain anything other than a-z, A-Z, 0-9 and the @ symbol.


6. Do not go outside of standard software and techniques unless you are properly qualified and able to do so. Standard software gets a lot of exposure and bugs are usually seen and fixed quickly.


7. ALWAYS hash passwords so they can never be seen even after a successful hack. You do NOT need to be able to tell your users what their password was. If they forget it, have a system to reset it/set it to something else in a secure way.


8. Encrypt ALL data that has either intrinsic value (bank account details) or that cannot be recovered/changed esily if lost like names, dates of birth, mothers maiden names etc. Again you cannot guarantee that a hack is impossible so better let people steal a load of data they can't read. Have a secure system for protecting encryption keys.


9. Become involved in owasp, a not-for-profit organisation that does a lot of work in the area of web security and who have many resources and articles about safe coding and safe processes.

Don't be blaze about it, you don't want to be the next person to lose 100,000 patient records!

Good OO Practice

OO is more like art than science. There is usually more than one way to achieve certain functionality but here are some general suggestions for good code.

1. Encapsulate stuff. In other words, make fields private and other functions private, protected or public - make the security as restrictive as you can. The more secure, the less that the function can be called which reduces code use and the chance for problems as well as making it easier to know how to test it. It also prevents people calling functions that seem to be right when they are supposed to be called internally to a class. "Internal" is useful for keeping stuff visible inside a single assembly (dll). Think about the testing required for a dll with 100 public functions rather than one with 10!


2. Only pass into functions what you need to use. Do not pass a parent object and then extract some property of it, if it needs the child, pass the child in. This again makes the intent of the function clearer and allows irrelevent changes (such as changing the parent type) to not affect functions that don't care.


3. Try and make functions either call a list of other functions OR do something technical and low-level. The functions that call functions are easy to understand and check, the others should be small and easy to unit test. Mixing them makes them hard to understand.


4. Try and work out whether it is better to replace structured constructs like switch statements and if/then/else with classes. Putting logic into structure is so much more robust and easier to check. Consider a statemachine in a switch, what is easier, to ensure that all the other case statements change the state variable correctly or in an OO state machine to have a class named MovingState that has specific transitions and guards coded into one function in one class? Again encapsulation is your friend. You can't affect anyone elses variables or functionality because you cannot see them.


5. Try and consider a class as either data, control or boundary and don't mix them. Data classes are persistent (usually) and have functions related only to their data. Control classes can be created and destroyed as they are used and manipulate data or send information to the system boundary. Boundary classes translate from one system to another (including the user interface, ports, the network etc) and can be considered pluggable so that, for instance, a web service can be plugged in where a user interface was previously since they can both send data that is conceptually the same even if one has no actual user interface.


6. Use the safest types for the data you have. If you are doing money stuff, a decimal is almost always preferred to a double or float because the maths functions work properly. If your data will be integer, use integers for similar reasons. If you have pairs of data that are related, either use something like a Map/Hashtable to link them or otherwise create a type to encapsulate both, don't assume that you can keep any related things up to date with each other without a setter function or constructor as the only means to do so.

Calling Web Services from Reporting Services

I wanted to do this because I didn't like the idea that we needed SQL procs in a database for reports that have to do the same thing as business logic dlls did for the web app pages. This duplication not only made things harder (being done twice) but allowed the possibility of breaking one when the other was changed and the fact that SQL is so different from code that one was often much easier to change than the other. I found out that you can call web service functions from reports so that the same code can populate reports AND web pages. It is not massively easy to work it out and my example uses Reporting Services 2005 and .Net 2.0. It also uses the Reporting Services XML Data Provider for the dataset (since the web service returns xml in the form of soap).

1. Create your web service as per a normal web service. In order to be generic and not need changing or growing for every different sort of function I would need to call, I made some generic functions that use reflection to call whatever function I want. You can probably work out what I did if I show you the code for the generic function:
   

   
   [WebMethod()]
   public ArrayList ExecuteArrayMethod(String typeName, String method, object[] ParamArray)
   {
    ArrayList returnObject = null;
    Type type = AccessType(typeName);
   
    try
    {
     if (type != null && type.GetMethod(method).GetCustomAttributes(typeof(ExportedMethod), false).Length > 0)
     {
      returnObject = (ArrayList)type.InvokeMember(method, BindingFlags.Default | BindingFlags.InvokeMethod, null, null, ParamArray);
     }
    }
    catch (Exception)
    {
     throw;
    }
    return returnObject;
   }
   

   
   There are a couple of points of note. The ExportedMethod attribute was something I added to control what functions were allowed to be exported to the reports and what couldn't, it is optional. Also, the arraylist allows collections of any type to be exported via the function. I have created a dataset flavour which I haven't tested yet and which I don't think I'll need.


2. Obviously the functions that actually generate your report data must be callable from the parameters you have available in your web service. If the function needs a load more information like connection strings, arguments etc, then either pass them in via the web service function or if they are private, create a parent function that can obtain these values and pass them to the worker function. The function you call from the web service must be static (I think!).


3. You will need to declare the types of any exported classes (i.e. types put into the ArrayList) at the top above the web service class definition even though you never need to reference them by name in the report. This forces the webservice to include the property data in the xml schema used by callers of the functions. You do this:
   

   [XmlInclude(typeof(ClassName1)), XmlInclude(typeof(ClassName2))]


You must ensure that these exported classes have default constructors and public getters AND setters for all class properties that you want to access from the report. This allows the system to serialize and de-serialize the objects across the network. The service will build fine but you will get an error when you try and execute the function in reporting services.

4. Build and deploy the web service to a url somewhere, such as http://myhost/webservices/myservice.asmx


5. Navigate to the asmx file in your browser and click on the function that you will be calling from your report. You should get an invoke section at the top which you can use to ensure that your code is returning the expected data. Do not go straight to the report without checking this, the report syntax is hard enough without the underlying data actually not working! Also note the RESPONSE syntax under the soap 1.2 section (the second of the two brown boxes). It will contain some soap bits (ignore these) and then a response tag, a result tag and then data depending on what you are returning. These are the important bits.


6. Open up your reporting services project (or create a new one) and add a datasource of type XML. In the connection string field, put the path to the asmx web service including the http and the asmx filename itself. Under credentials, you will need to put whatever is needed to call functions on this web service. In my case, it is a private intranet so I use windows credentials.


7. Create/edit a report and add a new dataset that uses this XML datasource, set the type to text and this is where the fun begins. You need to use a query syntax similar to XPath but not quite (search the web for details, there are various MSDN articles about it). This names the method, the parameters and the expected return values (and optional types). I will show mine so hopefully it makes more sense:
   

   
   <Query>
     <Method Name="ExecuteArrayMethod" Namespace="Namespace.Of.The.Webservice">
         <Parameters>
          <Parameter Name="typeName">
          <Parameter Name="method">
          <Parameter Name="QuoteReference">
      </Parameters>
     </Method> 
      <ElementPath IgnoreNamespaces="true">
              ExecuteArrayMethodResponse{}/ExecuteArrayMethodResult{}/anyType{Location,Description,Schedule,Item_Code,Quantity(Integer),OriginalSale(Decimal),RevisedSale(Decimal)}
   </ElementPath>
   </Query>
   

   
   Note that the anyType is the type that is returned in my soap response because I used an arraylist, it might be something else depending on your data. I have also typed some of the return values so that they can be treated as numbers rather than the string defaults. This XPath relates to the following web service xml response:
   

   
   <soap12:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap12="http://www.w3.org/2003/05/soap-envelope">
     <soap12:Body>
       <ExecuteArrayMethodResponse xmlns="Namespace.Of.The.Webservice">
         <ExecuteArrayMethodResult>
           <anyType />
           <anyType />
         </ExecuteArrayMethodResult>
       </ExecuteArrayMethodResponse>
     </soap12:Body>
   </soap12:Envelope>
   

   
   


8. Every time you save the xml, it will parse it and give you an error if there is a syntax problem, it does not understand line breaks so you might get a line1, character 300 error or similar, unfortunately, you have to try and build up the element path if your syntax is wrong and hope for the best.


9. If your web method requires parameters (as mine does), click on the ellipses next to the dataset (...) and click the parameters tab. You need to define any parameters passed to the method here so either a) use static values (in my case the method parameter is static for any given report) or to use a previously created report parameter if it will be different for each instance of the report (like my QuoteReference is).


10. Exit the edit pages and click the ! button to call the method. You might be prompted for parameters and you should see your expected table of data or you might get an error. If you get data but it is incorrect, check your element path. If you get no data, your element path might be wrong or there might be no data being returned, remove the element path and leave it blank to see if there is actually data present, if there is, your element path is still wrong. if you get an error, click the little red X icon and make the dialog bigger, the errors are often hidden but usually useful "no permission", "could not instantitate type" etc


11. Once you get data in the right format, your dataset on the left should now show the field names which can be used on the report in the normal way.


12. One of the big benefits for us is that the workload is shifted from the database server which also seems to be chugging away to the web server which is fairly lightly loaded since the work is now mostly done in a web service not a view/proc.


13. When you deploy your report to another server, you might get other errors and these are sometimes related to the permissions with which the report server is attempting to access your web service with. When you go to the web service in your browser, you are usually logging in as you and you might have access, the report server however logs in to the webservice as the report server (and not the person running the report). I got round this by modifying the datasource on the live server to use specific login credentials rather than windows credentials.

Why have a software process?

There is always lots of hype about software processes for quality/productivity etc but like a lot of people, I was often unconvinced about their value. At the end of the day I enjoy coding, not writing documents and so I would pay lip service to the process of design and specifications but only the minimum amount of work and then dive into code. I was attracted to the idea of things like eXtreme programming which promised quality without the overhead of design and specification but sadly this is a lot of gas imho, quality is not just related to the lines of code but about a whole lot more which is why the process is important. Now that I have less process and less design and specification to work to than my previous employer, I find myself uncomfortable and pine for documents to solidify the quality of my work. So why have a process including design, specification etc?
1) The earlier on the process, the easier/quicker/cheaper it is to fix things. For example, I had a spec review which revealed that a whole chunk of functionality was not required. A minute making a decision saved me potentially days of code (however good it was)
2) It requires that your customers (even internal ones) think about what they want. It is easy to spend 10 seconds saying, "We want a page to do x" with all the cost associated with it, whereas for them to spend a few hours defining it means they will think about whether it is really worth it and allows holes in their assumptions to come out.
3) It enables everyone to have buy-in and sign-off on functionality including everyone agreeing formulas/workflows etc and also allowing budget holders to question the value of the work compared to the proposed benefits.
4) It reduces the likelihood of rework at a later date where the original system implements assumed functionality and then the customer realises it is wrong and needs it to be changed. EVERY change is a risk and takes time/costs money. These risks are in productivity and reputation and why bother reworking? Sometimes it is unavoidable but reduction is better than nothing.
5) It allows systems to be seen in a larger context by people who care about more than one area of the system. With one spec next to another on the desk, a manager can ask questions related to the interaction or consistency of systems/sub-systems with each other. For instance, they might notice duplicated functionality and allow something to be removed in one sub-system since it will be present in another.
6) It permits a quality system to be implemented. If you notice problems in code that are actually related to design weaknesses, how can these be fed back into the quality cycle unless you have design and specification stages to modify?
7) You can have code of 100% reliability/quality from the programmers point of view but that lives in a function/class/system that is not actually of any business use. This is why quality and process CANNOT rely solely on a code view of quality such as eXtreme programming and other shortcut processes.

If you don't have a system, start one. The managers will often need convincing that the extra time is worthwhile but actually if implemented properly it will save time not increase it although there no doubt will be some short term overhead as you learn the way things work.

Viewstate, postbacks and pains in the neck

It took me a long time to get my head round what was going on asp.net. Sometimes controls did not get their correct values and things didn't always get bound. Here is a simple guide that hopefully will help you know what is going on.
1) The web is not connection based by default. When you go to a web page, you request a page, you are given it (usually) and that is the end. Things like session etc have been cobbled on the top of this system to try and keep track of things but the server never knows for sure if you are still connected which causes all kinds of problems with cookies and secure sites and loggin out etc. If you remember this connectionless pattern, this will help you understand.
2) What happens when you FIRST request an aspx page? The server recognises that the request is a "GET" HTTP request so sets the property IsPostBack to false. It calls various event handlers in a prescribed order (you can find these on msdn) most importantly Page_Load which you would use to set the page up. You probably managed this.
3) What happens when you change something on the page after it is served? This depends on what control you change. By default buttons cause a post back but data fields like text boxes and radio buttons do NOT unless you set their autopostback attribute to true.
4) When you finally cause the postback, a POST request is passed to the server along with the values of any fields on the page (which is why you need to put all the aspx controls inside a form). The server sees that it is POST and assume this is a "post back". It unpacks all the form values into member variables and it also uses something called viewstate to remember any data that is not a current value for a web control. For instance, a text box has no viewstate because the only data that needs to be remembered between posts is the text in it, this is already passed in the form data with the post. Controls like TreeViews however want to remember whether they were expanded etc and because this is not part of their 'current value' it has to be specially stored in viewstate. The viewstate is a hidden field with a load of ascii characters which the system automatically packs and unpacks for you (although you can disable the viewstate to save bandwidth or customise it).
5) If you write a custom control then you might have to save data into the viewstate so that it is 'remembered' between posts (but only if it needs remembering otherwise you are wasting bandwidth). You do this with the Viewstate property which is an array of objects.
6) The area that confuses most people is when working with a data source and a postback needing to update something in a database and then change the screen somehow. Firstly you need to know that Page_Load is called BEFORE your event handler. If you need to access data in the event handler, it must be set up first in Page_Load. Secondly, you will need to change your screen data in code otherwise it will remember its last values.
7) There are some times when you do not want to update from your datasource during postbacks. This is for two reasons. Firstly, because controls remember their values, once they are setup, if they are not going to change while the page is being used, there is no point keeping updating them, secondly you might want a load of controls to come up as disabled when you first open the page and then to become enabled when you click a button. However if you disable them in Page_Load, they will disable every time you cause a postback (such as clicking the button to unlock them!). Use the IsPostBack property to only do things on the first time in Page_Load.