James Crisp

Software dev, tech, mind hacks and the occasional personal bit

Page 2 of 18

Automated Testing and the Test Pyramid

Why Do Automated Testing?
Before digging into a testing approach, lets talk about key reasons to do automated testing:

  • Rapid regression testing to allow systems/applications to continue to change and improve over time without long “testing” phases at the end of each development cycle
  • Finding defects and problems earlier and faster especially when tests can be run on developer machines, and as part of a build on a CI server
  • Ensure external integration points are working and continue to work as expected
  • Ensure the user can interact with the system as expected
  • Help debugging / writing / designing code
  • Help specify the behaviour of the system

Overall, with automated testing we are aiming for increased project delivery speed with built in quality.

Levels of Automated Tests
Automated tests come in a variety of different flavours, with different costs and benefits. They are sometimes called by different names by different people. For the purposes of clarity, let’s define the levels as follows:

Acceptance Tests
Highest level tests which treat the application as a black box. For an application with a user interface, they are tests which end up clicking buttons and entering text in fields. These tests are brittle (easily broken by intended user interface changes), give the most false negative breaks, expensive to write and expensive to maintain. But they can be written at a level of detail that is useful for business stakeholders to read, and are highly effective at ensuring the system is working from an end user perspective.

Integration Tests
Code which tests integration points for the system. They ensure that integration points are up and running, and that communication formats and channels are working as expected. Integration tests are generally fairly easy to write but maintenance can be time consuming when integration points are shared or under active development themselves. However, they are vital for ensuring that dependencies of the system you are building continue to work as you expect. These tests rarely give false negatives, run faster than acceptance tests but much slower than unit tests.

Unit Tests
Tests which are fine grained, extremely fast and test behaviour of a single class (or perhaps just a few related classes). They do not hit any external integration points. [This is not the most pure definition of unit tests, but for the purposes of this post, we do not need to break this category down further]. As unit tests are fast to write, easy to maintain (when they focus on behaviour rather than implementation) and run very quickly, they are effective for testing boundary conditions and all possible conditions and code branches.

Automated Test Strategy
An important part of test strategy is to decide the focus of each type of test and the testing mix. I’d generally recommend a testing mix with a majority of unit tests, some integration tests, and a small number of acceptance tests. In the test pyramid visualisation, I’ve included percentages of the number of tests, but this is just to give an idea of rough test mix breakdown.

So, why choose this sort of mix of tests? This mix allows you to cover alternative code paths and boundary conditions in tests that run very fast at the unit level. There are many combinations of these and they need to be easily maintained. Hence unit tests fit the bill nicely.

Integration tests are a slower and harder to maintain, so it’s better to have less of them, and target them specifically to cover off the risks of system integration points. It would be inefficient to test your system logic with integration tests, since they would be slow, and you would not know if the failure was from your code or from the integration point.

Finally, acceptance tests are the most expensive tests to write, run and maintain. This is why it is important to minimise the number of these, and push down as much testing as possible to lower levels. They give a great view of “Is the System working?”, but they are very inefficient for testing boundary conditions, all code paths etc. Generally, they are best for testing scenarios. For example, a scenario might be: Student logs in to the portal, chooses subjects for the semester in a multi-step wizard, then logs out. It would be best to avoid fine grained acceptance tests – they would cost more than the value they would add – ie, it would be better they had never been written. An example of such a test could be: Student chooses a subject and it is removed from the list of subjects available to be chosen. Or student enters an invalid subject code and is shown the error “Please choose a valid subject code”. Both of these would be best pushed down to a unit test level. If this is not possible, it would be best to include them in a larger scenario to avoid the set up overhead (eg, for the error message test set up, it may be necessary to log in and enter data to get to step 3 of wizard before you can check if an invalid subject code error message is displayed).

With acceptance tests, I’d recommend writing the minimum up front, just covering the happy paths and a few major unhappy paths. If defects come to light from exploratory testing, then discover how they slipped through the testing net. Why weren’t they covered by unit and integration tests, and could they be? If you have tried everything you can think of to improve lower levels of testing, and are still having too many defects creeping through, then consider upping your acceptance coverage. However, keep the cost/benefit analysis in mind. You want your tests to make your team go faster, and an overzealous acceptance suite can eat into the team’s time significantly with maintenance and much increased cost of change. Finally, keep in mind there is a manual testing option. Manual testing needs to be minimised, but if it is 3 minutes to test something manually (eg, checking something shows up in a minor external system) or a week to automate the test, you’re probably going to be better off keeping a manual test or two around.

Team Roles and Tests
Ideally it would be great to have developers who were interested in testing and the business, testers who knew how to code and the business context, and BAs who were interested in testing and coding too. Ie, a team of people who could do each role in a pinch, but were more specialised in a particular area. Unfortunately this is pretty rare outside of very small companies with small teams. In a highly differentiated team, with dedicated Developers, BAs and QAs, this generally ends up with developers writing and maintaining unit tests, doing a lot of integration tests and helping out with acceptance tests. QAs generally write some integration tests and look after writing and maintaining acceptance tests with help from developers. BAs are sometimes involved in the text side of writing acceptance tests.

English Language Text Files & BDD
There are many tools that bill themselves as Behaviour Driven Development (BDD) testing tools which are based on having features written in formulaic English, backed by regular expressions matching chunks of text then linked to procedural steps. Often using such a tools is considered BDD and a good thing in a blanket way.

Lets take a step back. BDD is about specifying behaviour, rather than implementation and encouraging collaboration between roles. All good things. BDD can be done in NUnit, JUnit, RSpec etc. There is no requirement in BDD that tests are written in formulaic English with a Given-When-Then format. The converse is also true – if you write tests which are about implementation using an English language BDD framework, you are not doing BDD.

Using an English language layer on top of normal code is expensive. It is slower to write tests (you need to edit two files every test, and get your matching regexes right), harder to debug, refactor and maintain (tracing test execution between feature text, regex and steps is time consuming and there’s less IDE support), and less scalable as suites grow large (most of these frameworks use steps which are isolated procedures with global variables for communication and no object oriented abstraction or packaging). Also for many people who are used to reading code, the English language layer is less concise and slower to read than code for specifying behaviour.

What do you get from an English language layer? It means that less technical people (eg, business sponsor and BAs) can easily read tests, and maybe just possibly, even write the English language half of tests.

It is worth carefully weighing up the costs and benefits in your situation before deciding if you want to fork out the extra development and maintenance cost for an English language layer. Unit tests and integration tests are not likely to pay dividends having an English language layer – non-technical people would not be reading or writing these tests. Acceptance tests are potentially the sweet spot. If your team’s business representative(s) or BA(s) are keen to write the English language side of the tests, while keeping in mind the granularity of a scenario style approach, you could be on to a winner. Writing these tests could really help bring the different roles on the team together and come to a common understanding.

On the other hand, if your team’s business sponsor(s) are too busy or not interested in sitting down writing tests in text files, and the BAs are not interested or have their hands full elsewhere, then there are few real benefits in having the English language layer and the same costs apply.

A middle ground is to generate readable reports from code based test frameworks. With this approach you get a lot of the benefits with much less cost. Business people and BAs cannot write tests unaided, but they can read what tests ran (a specification of the system) in clear English.

Traceability from Requirements
A concept that most commonly comes up at companies doing highly traditional SDLCs is traceability from requirements all the way to code and tests. Clearcase and some of the TFS tools are the children of this idea. Acceptance testing tools are often misused to automate far too many fine grained tests to attempt to prove that there is traceability from requirements.

A friend was explaining his area of expertise around proof of program correctness for safety critical systems where many peoples’ lives depend on a system. I totally agree that in this small subset of systems, it is vital to specify the system exactly and mathematically and prove it is correct in all cases.

However, the majority of business systems do not need to be specified in such detail or with such accuracy. Most business systems which people attempt to specify up front are written in natural language which is notorious for inexactitude and differing interpretations. As the system is written, many of these so called requirements change as the business changes, technical constraints are realised, and as people see the system working, they realise it should work differently.

Is traceability back to outdated requirements useful? Should people spend time updating these “requirements” to keep them in sync with the real system development? I would say a resounding NO. There is no point in this. If the system is already built, these artefacts have realised their value and their time is over. If documentation is required for other purposes such as supporting the system, then it is worth writing targeted documents with that audience and purpose in mind.

On the testing front, this traceability drive can lead to vast numbers of fine grained acceptance tests (eg, one for each requirement) being written in an English-language BDD test framework. This approach is naive and leads to the problems we have covered earlier.

Recent Experiences
On a project a couple of years ago which had a test square rather than a pyramid (hundreds of acceptance tests that ran in a browser), it took 2 people full time (usually a dev and a QA) to keep the acceptance suite up and running. Developers generally left updating the acceptance tests to the dedicated QA/Dev, as it took around an hour to run the full test suite, and had many false negative failures to chase up (was the failure due to an intended screen change, an integration point being down, or a real issue in the software?). The test suite was in Fitnesse for .NET, which was hard to debug and use, and had its own little wiki style version control that didn’t fit in well with the source version control system. The acceptance test suite was red the majority of the time due to false negatives, so it was hard to know if a build was really working or not. To get a green build would take luck and several tries nursing the build through. I would count this as a prime example of an anti-pattern in automated test strategy, where there were far too many fine grained acceptance tests which took far too much effort to write and maintain.

On my last 3 projects, taking the test pyramid approach, we had a small number of acceptance tests which were scenario based, covered the happy paths and a few unhappy paths. On these projects, tests ran in just a few minutes, so could be run on developer machines before check in. The maintenance overhead of updating tests could be included in the development of a story rather than requiring full time people dedicated to resolving issues. Acceptance builds had few false negatives and hence were much more reliable indicators. We also chose tools which were code/text-file based, and could be checked into version control with the rest of the source code. The resulting systems in production had low defect rates.

Conclusion
Some of the ideas in this post go against the prevailing testing fashions of the moment. However, rather than following fashion, let’s look at the costs and benefits of the ideas and tools we employ. The aim of automated testing is to increase throughput on the project (during growth and maintenance) while building in sufficient quality. Anything that costs more to build and maintain than the value it provides should be changed or dropped.

Rails Refactor is now a Gem!

Rails Refactor, a small command line tool to make rails refactoring more fun, is now available as a gem. To use:
gem install rails_refactor

More info available on github.

VIM is Sydney Rails Devs’ Favourite Editor

Outstanding news! As part of the rails refactor talk at RORO (Sydney Rails Group) tonight (great evening by the way!), I asked for a show of hands on people’s favoured editors. I was amazed to discover the vim has edged out TextMate with just over half of the people at the group using it as their editor of choice! As an aside, Netbeans had one supporter, RubyMine and Emacs had zero. The groundswell of support for vim (and the cheering) was impressive!

PS – this is a very nerdy post, but as a long time vim fan, I had to report on it 🙂

Short Talk on rails_refactor at Rails group

I’ll be giving a short talk with Ryan Bigg on Rails Refactor at the next Sydney Rails Group (RORO) meet (Tuesday, Feb 8 from 7pm) . We’ll be talking about Rails Refactor’s birth at a hack night last year, what it can do for you right now, and its bright future as your refactoring tool of choice. Hope to see you there.

Short Talk: Starting Android Development

I’ll be giving a short talk on Starting Android Development on Tuesday at the Sydney ALT.NET group.

We’ll be covering:

  • the platform
  • app design and abstractions
  • Java and IDEs for Android Dev
  • Emulator
  • Code walk through of a simple application I’m writing

Richard Banks (@rbanks54) will also be giving a talk on .NET bdd tools.

More info and RSVP on the ALT.NET blog.

See you there!

Migrating from Palm OS to Android

Palm and its Demise
I’ve been using Palm organisers and smart phones since the year 2000. I enjoyed developing for them, writing several medical applications, and using them extensively for calendaring, contacts and memos (PIM). The Treo smart phones were visionary at the time, providing integrated phone and PIM functionality, plus push email and basic web browsing.

I was at the lavish developer conference in Sydney, when Palm was the market leader, and announced they were splitting into two separate businesses – software and hardware, and developing a new OS (Cobalt), which never saw the light of day. After that, Palm slowly lost its lead. I would have been interested to try out Palm’s last throw – the Pre and WebOS – but it never made it to Australia. Now Palm has been purchased by HP, and it is abundantly clear that Palm has had its day, and it’s time to move on. Goodbye Treo 650 and Palm OS Garnet!

Where to next?
Well, major contenders for smart phones at this point are iOS and Android (sorry Windows Phone 7, maybe next release :)). The iPhone is a nicely crafted piece of consumer electronics, and it’s the obvious choice for many people. Personally, I like the polish, but find the limitations of the OS and clumsy notification system, vendor lock in and tightly controlled environment does not appeal. Android, especially with 2.2, is pretty smooth. It requires a lot more tweaking than an iPhone to get it to a good state, but once set up, it’s a really nice experience and lets you do quite a lot of stuff you can’t do on an iPhone. I chose a HTC Desire HD and Android 2.2.

Android Migration
What I particularly want to share is how I migrated my data across from Palm OS to Google services and Android 2.2, and what applications I chose to replace the beautifully crafted Palm PIM system. There’s still some Palm users out there hanging on, and I’d encourage you to take the leap and move over to Android.

Aims

  • Powerful calendar app on the Phone with hour by hour day view, easy and quick to add/change events, and ability to include additional public calendars. Synchronisation with a desktop application.
  • Contacts synced with Google mail / Google contacts and a desktop application.
  • Memos/Notes synced only with a desktop application (not cloud)
  • Push email

Apps & Architecture
After quite a bit of research and trial, I decided to go with:

  • Google services for Calendar and Contact storage in the cloud
  • Business Calendar for android calendar app. Uses Google services, day by day view and supports multiple calendars. It works pretty well though still in Beta, and has frequent updates and improvements.
  • Built in contact app from HTC. It syncs with Google contacts (links to facebook too) and works well with the phone app. It’s meant to sync with twitter too but HTC apps for twitter don’t seem to have been updated for new Twitter authentication system.
  • Outlook 2007 for the desktop PIM application (I prefer Palm desktop, but yes, no future there)
  • gSync to synchronise Outlook with Google services for Calendar and Contacts (this works pretty well, though not 100% reliably for things like deleting one occurrence of repeating events). I also set the synchronisation for the calendar to only be 100 days in the past and future as this made it a lot faster to sync)
  • B-Folders Android app and wireless sync to desktop for memos/notes (B-Folders works ok for this but is a bit clunky for editing notes on the phone and requires you to enter a password frequently)
  • Built in Gmail app works well for email and I use the Gmail web client with offline sync on my PC

Data Migration

  • First, sync data with Outlook. Only outlook <= 2003 is supported. I installed Outlook 2000 for the sync. To change from Palm desktop to Outlook for HotSync, on Windows, run PalmOne > PIM Conduit Sync > Sync with Outlook from the start menu. If you don’t have this app, you can download the latest version of Palm Desktop from the Palm site and it will include it.
  • I had a lot of errors during sync but managed through largely retrying to get a clean sync to happen.
  • After a clean sync, I upgraded my outlook to 2007 as this has a better user interface and works with gSync.
  • Next, I used gSync to sync about a year of past calendar data and all contacts into Google services. It works pretty well. Some calendar events seemed to get duplicated but not enough to be a major hassle. I did try syncing more years of history in Calendar with the cloud, but it seemed to slow down my Business Calendar start up time significantly, so I cleared out everything and only synced a much smaller length of time – about a year. I then changed the sync to only 100 days in the past and future to make it run faster (takes about 3 minutes). I currently have the sync run a few times a day automatically but sometimes kick it off manually too.
  • Use ‘Google contacts > More actions > Find & Merge Duplicates’ to clean up and combine your contacts. I had a lot of email addresses in Google contacts which also had contact records from the Palm. This command did a good job combining them.
  • I exported all memos using Palm Desktop into individual text files (one per category), and then imported them into B-Folders as per these instructions. I had to manually change line endings (\r\n to \n) to avoid double spacing. A few days later, a new version of B-Folders was released which can import all exported Palm memos from a single categorised file. I haven’t tried this but feature, but it sounds like a time saver! The B-folders sync between phone and desktop app is manual and initiated from the phone. It has worked well so far.

Conclusion
In conclusion, I now have my data and quite workable PIM functionality on my Android phone. Business Calendar has a great multi-day view that my old palm didn’t, but it is a bit slower to add new events. The HTC Contacts app gets pictures from Facebook which is pretty cool, and syncs with my gmail so I don’t have to maintain email addresses in two places. It is a bit more clunky to edit and add contacts though. On the memo/note front, B-folders encrypts notes which is cool… but a bit irritating to need to enter your password every time you launch the program. Also it is more clunky to edit notes and does not save your last open note, and position in the note between launches. The rest of the phone functionality is great though and a huge step forward from the Palm. It’s really for a separate post to talk about these, but good web browser, GPS with maps, train timetables, movie times near you, twitter client, etc make it an amazing device and well worth the upgrade.

TRON and NORT

On the ultra-geeky front, I watched the original TRON last night, kindly leant to me by my buddy Doctor Dray. Having never seen it before, but heard a lot about it, I was keen to watch it at last. The core idea of computer programs personified is pretty cool, and the 80s rendering is interesting to watch (looks like stuff we did in computer graphics class at uni!). The plot does stretch belief a bit too thin at times though. To get an idea how far movie tech has come between the 80s and today, check out the original 80s trailer and the new Tron Legacy trailer.

Also, at high school, I and my fellow geeks spent quite a bit of time writing games in C like the TRON light cycle game, cunningly avoiding copyright violation by calling them NORT. After writing the 2 player version, we moved on to writing simple AIs so that you could play against the computer. Recently going through an old computer’s hard disk, I found the code for these. Thanks to the backward comparability features of Windows they still run, although they were written in Borland C/C++ for DOS! Amazing blast from the past.. here’s a picture of AI NORT in action:

nRake Microsoft Case Study

nRake is now the subject of a Microsoft case study. Check it out here:

UPDATE: Now on the Microsoft Case Study site.

Rails Refactor & Hack Night

During the RORO hack night last Wednesday, Ryan Bigg (@ryanbigg) and I worked on a Rails Refactor, something I’ve been meaning to get going for a long time.

How often have you wasted time doing renames in rails? Sure it’s hard to automate everything without understanding the code, but there sure are a lot of mechanical steps that you can easily automate. Ryan and I took on controller renames and got a fair way in the few hours we spent on the night.

Code is on github

To rename a controller:
$ rails_refactor.rb rename OldController NewController

  • renames controller file & class name in file
  • renames controller spec file & class name in file
  • renames view directory
  • renames helper file & module name in file
  • updates routes

To rename a controller action:
$ rails_refactor.rb rename DummyController.old_action new_action

  • renames controller action in controller class file
  • renames view files for all formats

Looking to extend it with model renames, and then more complex refactoring.
If you like it, please fork and contribute 🙂

Odd Date and Time Comparisons in Rails & Hack night

While comparing Dates and Times in Rails (both 2.3 and 3), I came across an odd behaviour:

>> Time.parse("Mon, 26 Jul 2010 9:59") == Date.new(2010, 7, 26)
=> false
>> Time.parse("Mon, 26 Jul 2010 10:00") == Date.new(2010, 7, 26)
=> true
>> Time.parse("Mon, 26 Jul 2010 10:01") == Date.new(2010, 7, 26)
=> false

Also

>> Date.new(2010, 7, 26) == Time.parse("Mon, 26 Jul 2010 10:00")
=> false (Rails 2.3)
=> nil (Rails 3)
>> Date.new(2010, 7, 26) == Time.parse("Mon, 26 Jul 2010 0:00")
=> false (Rails 2.3)
=> nil (Rails 3)

Tonight, we’ll be having the RORO hack night in the ThoughtWorks Sydney office, with a focus on open source projects (your own or contributing). A patch for this date/time behaviour might be an interesting area to pursue.

Page 2 of 18

Powered by WordPress & Theme by Anders Norén