How to evaluate Testing Tool

Recently started testing of an enterprise grade web application developed in JAVA, that brought me round to that ever important question: How to evaluate testing tools?

After shooting this question to many senior candidates during interviews, I found out that the crowd out there has not yet given a good thought on that one. Here are the criteria for evaluating a tool/ language before you start developing automation frameworks.

1.       NEVER choose a tool/ language for developing you automation framework simply because that's the only thing you know or have worked on in the past. Like I say "Don't try to use a Hammer on a screw."

2.       Begin by determining what kind of user interfaces are there to test and what kind of software components your framework would have to interface with. This could range from GUI's like plain HTML,flex,win32 apps to components like Java/C API's, Databases, 3rd party services like ssh support etc etc. Ascertain whether your target tool/language can "talk" to these interfaces comfortably.

3.       Determine the amount of support available for development using these languages/ tools in the domain areas required. Official/ unofficial forums, extent of search engine indexing, active developer community out there being a few of them.

4.       Determine the ease of developing frameworks using these as against other choices. This is where a couple of POC's can come in handy.

5.       How maintainable and scalable is your framework developed on this. Since testing frameworks are highly dynamic pieces of code that work with changing product code bases.

6.       How lightweight is the tool, since you do NOT want a tool that is so heavy that impacts the performance of your test runs.

7.       What is the amount of support it has for reporting and collaboration. You might want to generate reports in various formats if you want it to range from being "mailable" to your stakeholders to interfacing with your defect management and test case management software.

How to learn white box testing

Q- If I have been a QA tester, and don't have white box testing, where do you learn the skill?

 

A-     White box testing, also known as glass box or clear box testing, is testing that takes place where the testing had working knowledge of the code. In the AST BBST Foundations course, glass box testing concerns are illustrated with the question, "Does this code do what the programmer expects?" This is in contrast to the black box concern of, "Does this product fail to do what users -- human and software -- expect?"

From a learning perspective, this can mean a number of things. Testers working at this level are often comfortable with programming, hardware, networks, databases and application servers. Depending on the software, specialized knowledge of certain programming techniques or specific technologies may be required. Examples could include custom protocols, effective use of connection polling, language-specific test frameworks, among others.

Here's what I normally recommend for people getting started down this path:

• Learn the basics of computer science: You can do this through schooling or through some well selected books with a good self-study ethic. I recommend looking for entry-level courses and books on computer organization, networking, databases and file systems, data structures, assemblers/compilers/interpreters, algorithms, and discrete mathematics.
• Get comfortable working in a language: Today, I favor Ruby. But I taught myself Pascal as a teenager, learned C/C++ in college, and used Java in my first job as a programmer. It doesn't matter what language you learn -- just get fluent. You need to be comfortable enough that you can turn out simple programs rather easily and you need to be able understand and follow more complicated code -- even if you couldn't write it yourself.
• Practice writing unit tests, stubs, and harnesses: As you learn a language, you'll want to be sure you learn how to do unit testing in that language as you progress. The reason isn't so you can write better code yourself -- that's a happy side effect -- but so you get comfortable looking at and editing those types of tests. Similarly, as you start to practice testing, you're more than likely going to encounter situations where you'll want to mock out part of your test. This is a common pastime for white box testers, as they often use custom stubs and harness to mock out parts of the component their testing or to add more visibility to the results of their testing.
• Download and play with tools: This type of testing usually involves some tooling help if you plan on getting anywhere fast. Anything from your common runtime analysis tools (memory profilers, performance monitors, code coverage tools, etc.) to simple static analysis tools can help you learn the lingo and can get you thinking about common problems in white box testing. There are several good open source sites where you can find tools in this area; I currently favor opensourcetesting.org.
• Learn about security testing: Get to know your buddies at the Open Web Application Security Project (OWASP). You can think of security testing as a capstone project for white box testers. Not only is it a practical application of the skills and tactics that make white box testers successful, but it pulls together everything else you're learning. If you work through the OWASP materials, you see that you'll need to understand a lot of the computer science materials. You'll need to be able to read code. You'll need to use tools -- lots of tools -- and it will be helpful to be able to write your own simple scripts. I find that OWASP makes all of their material accessible to the beginner or near beginner). They also have local groups -- perhaps in your area -- where you can get some peer support for your learning.

Testers: Time to gear up for mobile software testing

The economy has many businesses retrenching or in a holding pattern -- but mobile applications designed to be accessed via smartphones or personal digital assistants (PDAs) are poised to be one of the next big things, according to many experts. If so, what impact will that have on enterprise quality assurance (QA) and testing organizations?

"Good testing practices apply regardless of the platform," said author and testing consultant Judy McKay, but she and other testing pros say mobile apps will also pose some unique challenges.

For starters, "the mobile phone is a frontier-based mentality," said William Coleman, vice president of business development at LogiGear Corp., a software testing services company in San Mateo, Calif. "There are four or five operating systems all competing for supremacy," he said, and "very loose standards … that many phone manufacturers and app developers circumvent."

But despite the lack of standards -- and the down economy -- it appears mobile app development is forging ahead. In a study released in January, Evans Data Corp. found that 94% of corporate developers expect the development of wireless enterprise applications to either increase (47.6%) or stay the same (46.4%) this year, with the Asia-Pacific region leading the growth.

For the business user, "we're seeing enterprise apps being developed on enterprise standard OSes like WinMo [Windows Mobile] and RIM, but we don't see a rush to the others just yet. Enterprises trust Microsoft and RIM," said Coleman. According to the Evans Data study, 40% more developers plan to target Windows Mobile than Apple iPhone, and 46% more plan to target .NET (compact framework) than Google's Android platform.

Not only are there multiple operating systems to take into account, but testers will also have to deal with multiple versions of an operating system, various hardware devices and form factors, and the strength of a carrier's network connections and services.

"The amount of permutations does create a significant problem for testing and time to market," said Doron Reuveni, CEO and co-founder of uTest, a Software as a Service (SaaS) marketplace for software application testing based in Southborough, Mass. "The issues in mobile apps are quite significant compared to testing Web and desktop apps."

Andrew Reshefsky is a uTest tester who has worked on mobile email encryption programs for corporate clients. A big challenge, he said, is that users are on different networks, and "you have to figure out whether the issue is because of the network or the software."

Another issue is the device itself. "Every phone has an OS; some phones can update the OS, some can't. If the problem is with the phone you have to figure that out, because software developers don't like to fix problems that don't exist [with the software]," he said.

Testers also face the challenge of shorter development cycles. "The time to market is extremely short, so usability testing is a big thing," McKay said. According to Evans Data, 40% of wireless development projects take three to six months to complete, and 60% are completed in less than six months.

McKay added that there will be "more emphasis on performance, connectivity and upgradeability."

"When you have a captive user, they're stuck with the performance you give them, but with mobile devices they expect very fast performance, but you don't have a guaranteed level of connectivity," she said. If users are unhappy with performance, "there's a higher risk that the app will be completely rejected."

uTest's Reuveni said their customers are looking for two levels of testing. First, "they want some level of certification that this works on this variety of carriers, networks, phones, etc., with this version of the software, so there's a degree of comfort when they release it," he said.

Second, he said, is to look for flaws. "In mobile there's a lot of ad hoc exploratory testing, partly due to the apps being developed so quickly, but also the customer wants real user behavior, so they're looking for a combination of flaws and feedback from real users that understand and have seen a lot of mobile apps."

Test automation will be key

To meet the challenges of testing mobile apps, automation will be key, McKay said. "Any time you do performance load you've got to use tools," she said. LogiGear's Coleman added, "As complexity [increases], the requirements for testing get more demanding. The only way to capture that is through automation."

LogiGear specializes in test automation, and Coleman said they are working in the mobile area. "The big hole in this space is automation for mobile phones," he said.

It's a nascent area, said Manish Mathuria, founder and chief technology officer of InfoStretch Corp., an outsourcer of quality assurance, test automation, software development and mobile testing services. "One reason is the form factors of the different devices in use vary so widely; for an automation tool to cover all of them reliably is not an easy task," he said.

In terms of adopting automation, Mathuria said, QA teams have a long way to go as well: "Enterprise QA teams are sometimes barely equipped to manage automation well on the desktop side."

Mathuria said that today the great majority of mobile apps his company sees are consumer-facing, with games being the biggest area, followed by utilities, but everyone interviewed for this article agreed that the enterprise is likely to follow. "The smartphone is the new computer abstraction," Coleman said. "Ten years from now we won't be using laptops."

Are testers prepared for the mobile revolution? "Testers tend to be behind the development curve," McKay said. "Developers are out experimenting with new things, but testers may be deeply involved in the last project."

Her advice? "Keep an eye on what developers are doing and thinking, and the ops people. [Testers] can no longer pretend they don't need to know about performance of load and connectivity. There is a lot of expertise that will have to be developed, and this will be the time to get better."

Cloud computing creates software testing challenges

The "cloud" promises to create new opportunities for enterprise developers as well as for suppliers offering services and tools for this new paradigm. For testing organizations, there will be both new challenges and new tools for answering what Soasta CEO Tom Lounibos calls the one key question: Can I go live?






The impact on testing is that the end-user experience is being influenced by the cloud provider and all other parties involved.
Vik Chaudhary
VP of product management and corporate development, Keynote Systems Inc.











"Testing all the layers — from your application to the cloud service provider — is something testers will have to become efficient in," said Vik Chaudhary, vice president of product management and corporate development at Keynote Systems Inc. in San Mateo, Calif.
According to market research firm IDC, spending on IT cloud services is expected to grow nearly threefold, to $42 billion by 2012. Cloud computing will also account for 25% of IT spending growth in 2012 and nearly a third of the IT spending growth in 2013, IDC projected.
IDC makes a distinction between "cloud services" and "cloud computing." Cloud services, according to the market research firm, are "consumer and business products, services, and solutions that are delivered and consumed in real-time over the Internet." In contrast, cloud computing as defined by IDC is the infrastructure or "stack" for development and deployment that enables the "real-time delivery of products, services, and solutions over the Internet."
Chaudhary explains the shift: "Enterprises like Schwab, Travelocity, etc. have been deploying their own data centers for years. The challenge was to manage highly scalable applications and how to ensure the best experience. Legions of people were employed by these companies to monitor/test/add servers, etc."
What's happening more recently with new cloud infrastructure like Google App Engine, he said, is that organizations can run their applications on Google's infrastructure.
"That means the bar to deploy applications in the cloud is so much lower. You don't have to have data centers or ops teams; you can focus on building the application and the functionality. It's a paradigm shift in application development," he said.
It's a shift for the tester, too. For example, Chaudhary said, "If you build an application and you use the BlackBerry to access a manufacturing application hosted by a cloud company like Salesforce, Salesforce does a certain amount of testing, to ensure the server is available, etc. But when it comes to the application itself, does it run on two phones or 50 phones? Do you have a long page to load?"
In addition, the cloud hosting company may use a third-party service to speed performance. "The impact on testing is that the end-user experience is being influenced by my company, by the cloud provider, and all other parties involved," he said.
Reducing testing costs
While Lounibos said Mountain View, Calif.-based Soasta Inc. has a growing group of customers that don't own servers and do everything in the cloud, "the majority are still more traditional; they use managed service providers and are dabbling in the cloud." However, he said, cloud-based testing is a way for organizations to learn the cloud and reduce the costs of testing at the same time.
"Traditional customers see testing as a money pit. They're looking for ways to reduce costs. The [main] argument for cloud computing for the enterprise is, is it reliable enough," he said. "This is not so for testing. Testing [in the cloud] just replicates the real world; it doesn't have the issues associated with production, but it has the benefits of cost reduction."
With cloud computing, Lounibos said, testers "have access, availability, and affordability to enormous amounts of computing power, which is what's needed in testing. The idea of being able to provision 125 servers in 5 to 8 minutes and only pay for the hours you test is so compelling. You no longer have to have huge test labs for Web applications."
Soasta's CloudTest, for example, is available as an on-demand virtual test lab in the cloud or as an appliance. It supports load, performance, functional, and Web UI/Ajax testing. According to Lounibos, "We were built on top of the cloud for the cloud."
For its part, Keynote offers KITE (Keynote Internet Testing Environment) for testing and analyzing the performance of Web applications across the Internet cloud. KITE offers instant testing from the desktop as well as from a variety of geographic locations.
For Internet applications in particular, Chaudhary said performance testing needs to move to the cloud.
"When it comes to performance, you're not depending just on the application but on all the providers [involved]. And do you [the user] have DSL or a dialup line, or a mobile device? Performance testing by nature is environmental," he said.
For mobile applications, Chaudhary said both performance and functional testing should move to the cloud.
"For mobile applications, the functional testing also depends on the providers. Say you've got a screen for login. The size of the page and the screen on phone and the provider can all affect if the application works," he said.
By testing in the cloud, Chaudhary added, organizations can more easily and cost-effectively test for hundreds of devices.
With applications that run on the cloud, "you need to test network performance, server performance, database performance, software performance on the application, and how it's cached in the client," said Dennis Drogseth, a vice president at market research company Enterprise Management Associates Inc., based in Boulder, Colo. "If you have a single application that runs in one place, you can test it geographically in one place. What you have with an Amazon or Facebook, for example, is all kinds of pieces coming in from different geographies, and you can't know ahead of time where they'll be. It's definitely more complicated than running a test script on a single server-based application."
The challenge is to run tests across all the diverse components and geographies to identify problems, he said, and organizations that develop an application "typically don't have access to those types of environments. So [a company like] Keynote is giving those testers a working environment where they leverage the Internet cloud and all the vagaries and look at real networks and desktops."
New testing tools needed
Drogseth said new types of testing tools will be needed. "You can't do cloud computing with application development testing tools for a LAN or a single server. You need tools to allow you to understand the network and desktop implications and all the pieces. You need to bring the network to the developer."
"I suspect over the next five years every testing vendor will try to come to the cloud. I think we will have a new generation of testing companies," said Lounibos. "This [cloud computing] is a big market coming down road; it's just the way we'll consume services."

Software Testing Advice for Novice Testers

Novice testers have many questions about software testing and the actual work that they are going to perform. As novice testers, you should be aware of certain facts in the software testing profession. The tips below will certainly help to advance you in your software-testing career. These ‘testing truths’ are applicable to and helpful for experienced testing professionals as well. Apply each and every testing truth mentioned below in your career and you will never regret what you do.
Know Your Application
Don’t start testing without understanding the requirements. If you test without knowledge of the requirements, you will not be able to determine if a program is functioning as designed and you will not be able to tell if required functionality is missing. Clear knowledge of requirements, before starting testing, is a must for any tester.
Know Your Domain
As I have said many times, you should acquire a thorough knowledge of the domain on which you are working. Knowing the domain will help you suggest good bug solutions. Your test manager will appreciate your suggestions, if you have valid points to make. Don’t stop by only logging the bug. Provide solutions as well. Good domain knowledge will also help you to design better test cases with maximum test coverage. For more guidance on acquiring domain knowledge.
No Assumptions In Testing
Don’t start testing with the assumption that there will be no errors. As a tester, you should always be looking for errors.
Learn New Technologies
No doubt, old testing techniques still play a vital role in day-to-day testing, but try to introduce new testing procedures that work for you. Don’t rely on book knowledge. Be practical. Your new testing ideas may work amazingly for you.
You Can’t Guarantee a Bug Free Application
No matter how much testing you perform, you can’t guarantee a 100% bug free application. There are some constraints that may force your team to advance a product to the next level, knowing some common or low priority issues remain. Try to explore as many bugs as you can, but prioritize your efforts on basic and crucial functions. Put your best efforts doing good work.
Think Like An End User
This is my top piece of advice. Don’t think only like a technical guy. Think like customers or end users. Also, always think beyond your end users. Test your application as an end user. Think how an end user will be using your application. Technical plus end user thinking will assure that your application is user friendly and will pass acceptance tests easily. This was the first advice to me from my test manager when I was a novice tester.
100% Test Coverage Is Not Possible
Don’t obsess about 100% test coverage. There are millions of inputs and test combinations that are simply impossible to cover. Use techniques like boundary value analysis and equivalence partitioning testing to limit your test cases to manageable sizes.
Build Good Relations With Developers
As a tester, you communicate with many other team members, especially developers. There are many situations where tester and developer may not agree on certain points. It will take your skill to handle such situations without harming a good relationship with the developer. If you are wrong, admit it. If you are right, be diplomatic. Don’t take it personally. After all, it is a profession, and you both want a good product.
Learn From Mistakes
As a novice, you will make mistakes. If you don’t make mistakes, you are not testing hard enough! You will learn things as you get experience. Use these mistakes as your learning experience. Try not to repeat the same mistakes. It hurts when the client files any bug in an application tested by you. It is definitely an embracing situation for you and cannot be avoided. However, don’t beat yourself up. Find the root cause of the failure. Try to find out why you didn’t find that bug, and avoid the same mistake in the future. If required, change some testing procedures you are following.
Don’t Underestimate Yourself if Some of Your bugs Are Not Fixed
Some testers have assumptions that all bugs logged by them should get fixed. It is a good point to a certain level but you must be flexible according to the situation. All bugs may or may not be fixed. Management can defer bugs to fix later as some bugs have low priority, low severity or no time to fix. Over time you will also learn which bugs can be deferred until the next release.

SQL Injection – How to Test Web Applications against SQL Injection Attacks

Security testing of web applications against SQL Injection, explained with simple examples - By Inder P Singh.

Many applications use some type of a database. An application under test might have a user interface that accepts user input that is used to perform the following tasks:

1. Show the relevant stored data to the user e.g. the application checks the credentials of the user using the log in information entered by the user and exposes only the relevant functionality and data to the user

2. Save the data entered by the user to the database e.g. once the user fills up a form and submits it, the application proceeds to save the data to the database; this data is then made available to the user in the same session as well as in subsequent sessions

Some of the user inputs might be used in framing SQL statements that are then executed by the application on the database. It is possible for an application NOT to handle the inputs given by the user properly. If this is the case, a malicious user could provide unexpected inputs to the application that are then used to frame and execute SQL statements on the database. This is called SQL injection. The consequences of such an action could be alarming.

The following things might result from SQL injection:

1. The user could log in to the application as another user, even as an administrator.

2. The user could view private information belonging to other users e.g. details of other users’ profiles, their transaction details etc.

3. The user could change application configuration information and the data of the other users.

4. The user could modify the structure of the database; even delete tables in the application database.

5. The user could take control of the database server and execute commands on it at will.

Since the consequences of allowing the SQL injection technique could be severe, it follows that SQL injection should be tested during the security testing of an application. Now with an overview of the SQL injection technique, let us understand a few practical examples of SQL injection.

Important: The SQL injection problem should be tested only in the test environment.

If the application has a log in page, it is possible that the application uses a dynamic SQL such as statement below. This statement is expected to return at least a single row with the user details from the Users table as the result set when there is a row with the user name and password entered in the SQL statement.

SELECT * FROM Users WHERE User_Name = ‘” & strUserName & “‘ AND Password = ‘” & strPassword & “’;”

If the tester would enter John as the strUserName (in the textbox for user name) and Smith as strPassword (in the textbox for password), the above SQL statement would become:

SELECT * FROM Users WHERE User_Name = ‘John’ AND Password = ‘Smith’;

If the tester would enter John’– as strUserName and no strPassword, the SQL statement would become:

SELECT * FROM Users WHERE User_Name = ‘John’– AND Password = ‘Smith’;

Note that the part of the SQL statement after John is turned into a comment. If there were any user with the user name of John in the Users table, the application could allow the tester to log in as the user John. The tester could now view the private information of the user John.

What if the tester does not know the name of any existing user of the application? In such a case, the tester could try common user names like admin, administrator and sysadmin. If none of these users exist in the database, the tester could enter John’ or ‘x’=’x as strUserName and Smith’ or ‘x’=’x as strPassword. This would cause the SQL statement to become like the one below.

SELECT * FROM Users WHERE User_Name = ‘John’ or ‘x’='x’ AND Password = ‘Smith’ or ‘x’=’x’;

Since ‘x’=’x’ condition is always true, the result set would consist of all the rows in the Users table. The application could allow the tester to log in as the first user in the Users table.

Important: The tester should request the database administrator or the developer to copy the table in question before attempting the following SQL injection.

If the tester would enter John’; DROP table users_details;’—as strUserName and anything as strPassword, the SQL statement would become like the one below.

SELECT * FROM Users WHERE User_Name = ‘John’; DROP table users_details;’ –‘ AND Password = ‘Smith’;

This statement could cause the table “users_details” to be permanently deleted from the database.

Though the above examples deal with using the SQL injection technique only the log in page, the tester should test this technique on all the pages of the application that accept user input in textual format e.g. search pages, feedback pages etc.

SQL injection might be possible in applications that use SSL. Even a firewall might not be able to protect the application against the SQL injection technique.

I have tried to explain the SQL injection technique in a simple form. I would like to re-iterate that SQL injection should be tested only in a test environment and not in the development environment, production environment or any other environment. Instead of manually testing whether the application is vulnerable to SQL injection or not, one could use a web vulnerability scanner that checks for SQL injection.