Cloud computing creates software testing challenges

The "cloud" promises to create new opportunities for enterprise developers as well as for suppliers offering services and tools for this new paradigm. For testing organizations, there will be both new challenges and new tools for answering what Soasta CEO Tom Lounibos calls the one key question: Can I go live?






The impact on testing is that the end-user experience is being influenced by the cloud provider and all other parties involved.
Vik Chaudhary
VP of product management and corporate development, Keynote Systems Inc.











"Testing all the layers — from your application to the cloud service provider — is something testers will have to become efficient in," said Vik Chaudhary, vice president of product management and corporate development at Keynote Systems Inc. in San Mateo, Calif.
According to market research firm IDC, spending on IT cloud services is expected to grow nearly threefold, to $42 billion by 2012. Cloud computing will also account for 25% of IT spending growth in 2012 and nearly a third of the IT spending growth in 2013, IDC projected.
IDC makes a distinction between "cloud services" and "cloud computing." Cloud services, according to the market research firm, are "consumer and business products, services, and solutions that are delivered and consumed in real-time over the Internet." In contrast, cloud computing as defined by IDC is the infrastructure or "stack" for development and deployment that enables the "real-time delivery of products, services, and solutions over the Internet."
Chaudhary explains the shift: "Enterprises like Schwab, Travelocity, etc. have been deploying their own data centers for years. The challenge was to manage highly scalable applications and how to ensure the best experience. Legions of people were employed by these companies to monitor/test/add servers, etc."
What's happening more recently with new cloud infrastructure like Google App Engine, he said, is that organizations can run their applications on Google's infrastructure.
"That means the bar to deploy applications in the cloud is so much lower. You don't have to have data centers or ops teams; you can focus on building the application and the functionality. It's a paradigm shift in application development," he said.
It's a shift for the tester, too. For example, Chaudhary said, "If you build an application and you use the BlackBerry to access a manufacturing application hosted by a cloud company like Salesforce, Salesforce does a certain amount of testing, to ensure the server is available, etc. But when it comes to the application itself, does it run on two phones or 50 phones? Do you have a long page to load?"
In addition, the cloud hosting company may use a third-party service to speed performance. "The impact on testing is that the end-user experience is being influenced by my company, by the cloud provider, and all other parties involved," he said.
Reducing testing costs
While Lounibos said Mountain View, Calif.-based Soasta Inc. has a growing group of customers that don't own servers and do everything in the cloud, "the majority are still more traditional; they use managed service providers and are dabbling in the cloud." However, he said, cloud-based testing is a way for organizations to learn the cloud and reduce the costs of testing at the same time.
"Traditional customers see testing as a money pit. They're looking for ways to reduce costs. The [main] argument for cloud computing for the enterprise is, is it reliable enough," he said. "This is not so for testing. Testing [in the cloud] just replicates the real world; it doesn't have the issues associated with production, but it has the benefits of cost reduction."
With cloud computing, Lounibos said, testers "have access, availability, and affordability to enormous amounts of computing power, which is what's needed in testing. The idea of being able to provision 125 servers in 5 to 8 minutes and only pay for the hours you test is so compelling. You no longer have to have huge test labs for Web applications."
Soasta's CloudTest, for example, is available as an on-demand virtual test lab in the cloud or as an appliance. It supports load, performance, functional, and Web UI/Ajax testing. According to Lounibos, "We were built on top of the cloud for the cloud."
For its part, Keynote offers KITE (Keynote Internet Testing Environment) for testing and analyzing the performance of Web applications across the Internet cloud. KITE offers instant testing from the desktop as well as from a variety of geographic locations.
For Internet applications in particular, Chaudhary said performance testing needs to move to the cloud.
"When it comes to performance, you're not depending just on the application but on all the providers [involved]. And do you [the user] have DSL or a dialup line, or a mobile device? Performance testing by nature is environmental," he said.
For mobile applications, Chaudhary said both performance and functional testing should move to the cloud.
"For mobile applications, the functional testing also depends on the providers. Say you've got a screen for login. The size of the page and the screen on phone and the provider can all affect if the application works," he said.
By testing in the cloud, Chaudhary added, organizations can more easily and cost-effectively test for hundreds of devices.
With applications that run on the cloud, "you need to test network performance, server performance, database performance, software performance on the application, and how it's cached in the client," said Dennis Drogseth, a vice president at market research company Enterprise Management Associates Inc., based in Boulder, Colo. "If you have a single application that runs in one place, you can test it geographically in one place. What you have with an Amazon or Facebook, for example, is all kinds of pieces coming in from different geographies, and you can't know ahead of time where they'll be. It's definitely more complicated than running a test script on a single server-based application."
The challenge is to run tests across all the diverse components and geographies to identify problems, he said, and organizations that develop an application "typically don't have access to those types of environments. So [a company like] Keynote is giving those testers a working environment where they leverage the Internet cloud and all the vagaries and look at real networks and desktops."
New testing tools needed
Drogseth said new types of testing tools will be needed. "You can't do cloud computing with application development testing tools for a LAN or a single server. You need tools to allow you to understand the network and desktop implications and all the pieces. You need to bring the network to the developer."
"I suspect over the next five years every testing vendor will try to come to the cloud. I think we will have a new generation of testing companies," said Lounibos. "This [cloud computing] is a big market coming down road; it's just the way we'll consume services."