The company in question, a high-tech start up, got its initial VC funding in 2000 to develop an Enterprise-class software product to address a major deficiency in the way that public companies report their financial status. This was made all the more urgent by the Sarbanes-Oxley requirements.
The resulting requirements called for a complex, multi-tier system, and it had to be developed quickly and without breaking the bank.
The management team wanted to outsource the QA function. They wanted a low ratio of development engineers to QA engineers (i.e., 2:1 or less). Mature products usually have a 3:1 or higher ratio, but this product was complex, written from scratch with POJO, by a newly assembled team. All told, there was a good chance that the early releases would have a high bug count and they had to be caught before the product made it to customers.
They also expected other benefits from outsourcing this function to a third party:
After some research, they outsourced QA to India. Going offshore met all their requirements, at a price they could afford. Or so it seemed.
The QA team consisted of six people, one in the US and five in India. A US-based QA Lead acted as the bridge between the US- and India-based teams. He also ran weekly bug reviews, generated weekly bug reports, and researched and selected SDLC tools.
The India-based team consisted of the following:
The main measure for this team were the number of regression tests generated, their ratio of automation and, of course, cost.
In addition to the usual cast, they used several tools that they had not used before.
For white-box, GUI-based testing they used QFtestJUI, an inexpensive tool written in Java that has equal support for Microsoft Windows and Unix/Linux platforms.
They used TestLink, a very useful test management and execution tool, particularly for remote,off-sync teams. This tool allowed them to create and maintain test specifications online from day one. In spite of a clunky User Interface, it made it very easy to see which test were automated and which were manual, as well as what tests were run for each build and their results.
Of course, they made extensive use of IM and VoIP to stay in touch as much as possible.
In the end, the offshore team created over 600 regression tests, and automated over 70% of them.
The company developed good processes to balance the mix of manual testing and test automation with a limited staff.
The team was able to keep up with a weekly build regime and they generated got good constructive reports. They were effective in catching regression bugs and in pointing out product-level problems, not just bugs.
Their number one problem was turnover. India’s outsourcing industry has boomed and the unfortunate side-effect is extremely high turnover and a shortage of experienced engineers.
About three months after the project started, the project lead quit the team. It turned out that he had been commuting three hours each way, every week and, understandably, it took a toll. It was a terrible blow and it took several months to find another lead.
Six months later, they lost their two most experienced white-box automation engineers. On the same day! They simply found jobs closer to their home state during a holiday visit. In less than 10 months, they had experienced 60% turnover and lost a lot of their investment plus the extra time it took to find and retrain the new people.
Finally, the replacement Lead had to be let go a few months after he joined.
Look for a locale that does not suffer from high turnover. If you can find the right team, outsource to a nearshore or onshore locale. In many cases, their Total Cost of Engagement (TCE) is the same or less than offshore.
Make sure that at least one team member in the remote locale, preferably the lead, has experience working in the US or Western Europe. Make sure that you have competent and committed local management.
Outsourcing is here to stay, because of its many benefits. The key is to learn from past mistakes and do better next time.
A version of this paper is published as an article entitled “Outsourcing QA” in Software Development Magazine, January 2006, in the “Global Success Stories” special section on outsourcing [registration required]
All your info is kept private. We'll never spam you.
Send us your resume and we’ll guide you trough the process of landing your dream job.