How to evaluate Testing Tool

Recently started testing of an enterprise grade web application developed in JAVA, that brought me round to that ever important question: How to evaluate testing tools?

After shooting this question to many senior candidates during interviews, I found out that the crowd out there has not yet given a good thought on that one. Here are the criteria for evaluating a tool/ language before you start developing automation frameworks.

1.       NEVER choose a tool/ language for developing you automation framework simply because that's the only thing you know or have worked on in the past. Like I say "Don't try to use a Hammer on a screw."

2.       Begin by determining what kind of user interfaces are there to test and what kind of software components your framework would have to interface with. This could range from GUI's like plain HTML,flex,win32 apps to components like Java/C API's, Databases, 3rd party services like ssh support etc etc. Ascertain whether your target tool/language can "talk" to these interfaces comfortably.

3.       Determine the amount of support available for development using these languages/ tools in the domain areas required. Official/ unofficial forums, extent of search engine indexing, active developer community out there being a few of them.

4.       Determine the ease of developing frameworks using these as against other choices. This is where a couple of POC's can come in handy.

5.       How maintainable and scalable is your framework developed on this. Since testing frameworks are highly dynamic pieces of code that work with changing product code bases.

6.       How lightweight is the tool, since you do NOT want a tool that is so heavy that impacts the performance of your test runs.

7.       What is the amount of support it has for reporting and collaboration. You might want to generate reports in various formats if you want it to range from being "mailable" to your stakeholders to interfacing with your defect management and test case management software.