Mobile software development - Functional testing

Simon Judge's blog made me think, again. He wrote about Mob4Hire, a company offering people for mobile application testing. Testers get paid via PayPal after bidding on projects (i.e. mobile applications/solutions) and developers get testers at a (hopefully) reasonable price. Finding testers might be especially useful if your geographic area is not the one you'd like your software to be tested in.

You know, lots of developers (dare to say: most?) do not recognize the importance of testing. This is the least pleasant part of development, I must admit, yet one of the most important. There are various kinds of testing including (but not limited to) unit-, regression, load, "smoke", etc. testing. The one, Mob4Hire provides solution to is called functional testing, where one can test the whole solution end-to-end. You know, mobile handsets are very-very fragmented in terms of platforms: applications can be developed in many programming languages like Java, Symbian C++, Windows Mobile Win32/C#, iPhone Obj-C, Linux C/C++, etc. And even when sticking to the same platform and programming environment, JME for example, the supported features vary very much from device to device. This, along with the complexity of what operators allow 3rd-party programs to do, makes it very difficult for a new application to be thoroughly tested.

Nokia already provides a free service for developers wishing to test their mobile applications written for Nokia S60 platform: it's called Nokia RDA (short for Remote Device Access). It is an Internet-based solution, where you can remote control a real mobile phone. You can request, for example, that SIM- and/or memory card be inserted in the test device as well as more than one phone be reserved for your test session to test peer-to-peer communications.

DeviceAnywhere provides a similar solution to Nokia RDA, however, it's not limited to a particular platform, nor to only 1-2 network operators. According to their web site, their service is "a revolutionary online service that provides access to hundreds of real handsets, on live worldwide networks, remotely over the Internet". Unfortunately, it is not free of charge.

Note that you cannot test everything with these solutions. For example, applications that use camera, GPS, accelerometer are basically out of question as well as ones using external accessories.

Another option for functional testing is making use of the services of Test Houses. Professional testers verify the quality of your software (compared to Mob4Hire, for example, see Simon's opinion) so that you can be sure you get the most what you paid for. Sometimes it's even required for your application to pass certain tests in order to get certified by some authorities. However, you may need to pay a lot for this service, see the list & pricing of Test Houses that Symbian Signed accepts, for example.

Finally, let's talk about community-driven testing. Once your application is in such a shape that it is ready for external people to play with it, you can ask them to go and use it extensively. You can offer free copies of the software to them, for example, or they may do it just for their own gratification - it's the same. This way of testing works extremely well in solutions based on client-server architecture with a mobile front-end and a server back-end. It's quite common in these scenarios that the mobile-end is just a light-weight client software that can be freely distributed, thus it doesn't cause any inconvenience if software distrubition is not strictly controlled. The point is that you may get lots of people playing with your software, because it's their passion. And passion drives people to do their job well, simply because they enjoy it, they love your program and they'd like it to be even better. I'm really a great supporter of this kind of testing. :)

Can you recommend any other way for performing functional mobile software testing? Please let me (us) know!

Tote