Issues with Testing Automation Products

1. Lack of Comprehensive Test Automation Tools
The testing industry suffers from the inability of plug-and-play automation tools to deal with customized application components in real-time. Furthermore, such test automation tools are not comprehensive enough to address test needs, particularly across multiple platforms at the IS level during ongoing development efforts.

Consequently, tool-creating organizations often must develop unique and intricate solutions to make these automation tools work with their applications. Yet, this is rarely, if ever, mentioned in the literature or sales pitches for testing providers. In addition, while virtually all automation tools contain some scripting language that bypasses each tool? failings, testers typically neither have the development experience nor the training necessary to exploit these programming environments.

2. Tool Rigidity
When selecting a specific tool, many firms do not understand that nearly all testing tools currently on the market have limitations. As a result, the implementation or use of a tool can result in excessive exception handling and duplication of efforts.

For example, many testing teams are driven to reduce coding duplication and maintenance that occurs largely with captured test scripts. Consequently, they create compartmentalized test environments for each application and hence modularized libraries where re-usability is not permitted. Separate test teams then encounter challenges when they think beyond their individual projects.

Although each team sets up a partially reusable framework, each team is completely unique - even when common library functions are the same. This ultimately translates into duplicate development efforts, duplicate debugging and duplicate maintenance. As changes to various applications begin breaking automated tests, script maintenance and debugging become a significant challenge. Additionally, upgrades in the automation tools themselves cause significant and unexpected script failures. In some cases, the necessity to revert (downgrade) to older versions of the automation tools is inevitable.

Moreover, resource allocation for continued test development and test code maintenance becomes more difficult in these scenarios. Eventually most projects are put on hold, budgets are slashed and automation tools are rejected or shelved. The immediate costs of such efforts can be debilitating, notwithstanding the opportunity costs therein.

3. High Cost of Failures
Given extensive application diversity and the lack of comprehensive enterprise-wide testing frameworks and tools, organizations are forced to deploy a variety of automation tools. While the tools may be evaluated for their relevance to certain applications, the integration between multiple applications, along with testing scenarios that must be played out within the IS area, are often  missing. Instead, the focus rests heavily on end-user testing and the motivation to release new versions quickly. The unfortunate results for this approach include substantial testing failures and a large number of fixes and patches required in post-release production environments.