Summary: Are you in the midst of an SAP R/3 upgrade or implementing SAP R/3 from scratch? Are you prepared to perform all the activities and tasks that are concomitant with the various SAP testing phases? The author identifies testing risks and shares lessons learned from testing SAP R/3. The author provides insights and recommendations for avoiding these testing risks that could halt SAP testing efforts and consequently delay the expected deployment of SAP into a live production environment. The insights range from documenting test cases to working with testing artifacts from the ASAP methodology.
Title: Lessons Learned from Testing SAP R/3
SAP R/3 is the market leader in ERP installations and ERP sales. SAP has thousands of tables, multiple industry specific solutions, thousands of transactions, and connectivity to an unlimited number of legacy systems. Furthermore SAP can be configured differently from one company to another which creates a myriad of permutations for executing an SAP transaction. Installing and customizing SAP is a daunting challenge. Testing SAP R/3 is in and of itself another intractable challenge.
Many projects fail to test SAP correctly and consequently suffer staggering financial losses after deploying SAP into a live environment. The key to maximizing the value and ROI of SAP is to install and customize SAP correctly based on the documented requirements and to test it extensively based on the documented test cases and end-to-end business scenarios. Below some lessons learned are offered and identified to help organizations test SAP.
1. Not following methodology or No methodology at all
Some companies implementing SAP adhere to the ASAP methodology. Other companies have ad-hoc or ASAP-like methodologies for implementing SAP.
Even companies that are supposedly implementing SAP based on the ASAP methodology are not very strict and stringent in adhering to all the activities, deliverables, and tasks associated with ASAP. Consequently, but not surprisingly these companies have much confusion, obfuscation, befuddlement when they attempt to implement SAP. Compounding this problem is the fact that many large companies implementing SAP hire two or more implementation partners and multiple subcontractors that have incompatible approaches, methodologies, and lessons learned for implementing SAP.
The project manager and the steering committee should specify within the project charter how SAP will be implemented and what deliverables will be produced based on either ASAP or some other proprietary methodology. The objective is to have defined, proven, and repeatable processes for implementing SAP and that the project members have the knowledge or know-how for adhering to the methodology. The creation of an audit team or standards team would be helpful in enforcing compliance with the chosen methodology.
2. Inadequate test tools
SAP R/3 comes with an internal recording tool known as CATT (eCATT). One of the advantages of CATT (eCATT) is that since it is part of the standard SAP system it’s free of charge. However CATT does have some limitations which impel many companies to procure other test tools.
There are many vendors offering commercial automated test tools and test management tools for testing SAP. Companies purchasing automated test tools expect and erroneously believe that the test tools will be the panacea to their entire SAP recording and testing needs. Unfortunately, this is not the case, since no two SAP implementations are exactly the same across two or more companies or at times even within different divisions of the same company. Consequently, a company implementing SAP might need to procure test tools from more than one vendor in addition to the CATT (eCATT) tool.
An SAP implementation could be implementing SAP add-ons such as BW (Business Warehouse), APO (Advanced Planning Optimization), SEM (Strategic Enterprise Management) or even modules such as PS (Project Systems) that generate graphs and charts that a recording tool does not recognize. Furthermore, a company may move its SAP GUI from the desktop (fat client) to running SAP as web-based (thin client) or through an emulated Citrix session which could render the existing test tools useless.
Companies that wish to move to an automated testing strategy should articulate and document what SAP modules and SAP add-ons they are installing in addition to any legacy applications integrating with SAP. This information should be provided to the vendors of automated test tools in order to determine what can actually be recorded and tested with the test tools. The company should further investigate with the vendor what additional benefits over CATT (eCATT) the automated test tools provide. The objective is to get the test tools that will maximize SAP recording.
3. Decentralized test teams
Some projects that I have consulted for, have a decentralized testing approach where each individual business process team, technical team (ABAP/4), and Basis (security) team conduct their own testing in the absence of a QA/testing team.
The decentralized method suffers from various drawbacks including redundancy, inconsistency, lack of standards, missing testing metrics, unreported test results, limited managerial visibility, limited test coverage, chaotic defect resolution, no independent verification of test results, etc.
Projects with a decentralized test approach perceive control as a significant advantage of decentralized testing. With decentralized testing each team has the independence and control as to how their particular area will be tested based on their own defined test procedures.
The perceived benefits from decentralized testing do not offset its drawbacks. A more robust approach for testing an ERP system is centralized testing where there is a QA team for defining the standards and procedures for the various testing cycles and documenting the test plan, lessons learned, UAT plan and the test strategy. The test procedures and test strategy include what test case templates will be used, how and what metrics will be reported, how peer reviews will be conducted, ensuring that all test requirements have been met, etc. In addition to the QA team under the centralized model the testing team provides expertise for executing test cases whether manually or with automated test tools. The testing team also provides independent verification for the test results since the testers were not associated with the configuration of the transactions, workflow development, creation of user roles, or the development of the RICE (Reports, Interfaces, Conversions, and Enhancements) objects, etc.
4. Working in Silos: Limited or no access to SMEs, Configuration Team
Since SAP is integrated and a change in one of its components can have a cascading effect on other aspects of the software it is often the case where the various SAP teams need to collaborate with one another.
This concept seems mundane and at first hand easy to grasp. But due to internal politics, deadlines, budget constraints, etc this concept is ignored and overlooked. When collaboration falls through the cracks the various SAP teams (configuration, RICE, Basis, testing, infrastructure, data, etc) start working independently in isolation or in a silo.
I have been in projects where the testing team members who arrived during the middle of the realization phase have been unable to get support from the SAP configuration team members and SMEs for drafting test cases, peer reviews, and sign offs.. In other projects the QA and testing team are almost considered a “bother”, “nuisance” to the other SAP teams.
The test manager and PM need to foster an environment of cooperation among the testing team and the various other SAP teams for the successful completion of testing artifacts and testing cycles.
5. Missing test bed
Within the SAP client landscape a dedicated QA environment should be allocated to the testing team for the various testing phases. The environment should be production sized, and contain production like data, have a stable baseline, and frozen configuration by the time the testing cycles commence.
What I have seen at many projects is a so-called “dedicated testing environment” that many SAP teams have access to for training, last minute configuration, etc during the middle of a testing cycle. This creates conflict among the teams and causes confusion over the ultimate ownership of the testing environment.
The test manager should insist on a test environment where the testing team has complete control (for instance control over the accounting periods, running payroll, etc) and where the environment is not shared with other entities in particular during the test execution phase.
6. No controls for promoting/transporting objects into production
In many SAP projects that I have consulted for, there are very little controls for promoting objects into a live production environment. Other projects are very lax with their rules for promoting objects and have no means for auditing the approval process for transporting objects or do not include all the necessary stakeholders in transporting objects.
ERP systems like SAP R/3 offer benefits such as tight integrations across its modules and sub-modules and access to real-time data, but these benefits can backfire or even become otiose when a poorly tested or not even tested object is transported into a production environment. A faulty transported object such as a configuration change or code for an interface can have a deleterious cascading effect on other SAP functionality in the production environment. For instance a change to the HR module within payroll can affect functionality in the FI-CO module.
To mitigate the risk of transporting objects it is necessary to include approvals and set up a system of workflow where stakeholders from the testing team, Basis team, and configuration leads have input into objects that will be transported into production. A project should set priorities for transporting objects based on their criticality, document and approve objects that get transported after having been tested, and establish the frequency in which objects will be transported (i.e. every Friday at 4:00 pm) to avoid impacting end users as much as possible.
The company Mercury Interactive (www.mercuryinteractive.com) offers the tool Kintana which can help companies to move objects into various SAP clients and instances while maintaining a history log for auditing purposes of the objects that were transported and the entities approving the transports. Alternatively, I was a project where the organization created a home grown repository within Lotus Notes that provided managerial visibility for moving and transporting SAP objects which included notifications and approvals.
Whether a commercial tool is purchased or a home grown application is developed the main objective is to place stringent controls for moving objects into production with the necessary approvals while maintaining a history log.
7. Flaws with BPPs
For those projects adhering to the ASAP SAP methodology BPPs (Business Process Procedures) are artifacts or outputs from the realization phase. BPPs are documented based on stand alone or for single SAP transactions. The BPP provides detailed information in the form of instructions with screen shot print outs for how to execute a given SAP transaction (i.e. SAP transaction MM01 – Create Material).
Admittedly, BPPs can assist a tester or end user on how to execute a given SAP transaction based on the project’s specific customizations. However, and this is often the case the SAP configuration team will document BPPs in a given environment such as the configuration environment with configuration data and then the SAP configuration team fails to update the BPPs to reflect the changes of the QA environment. The failure to update the BPPs diminishes their value for the testing team.
When BPPs are not updated they contain obsolete data, and may lack the necessary information to execute an SAP transaction for an SAP environment that has had customizations since the time the BPPs was initially created. Furthermore, many organizations do not perform any form of version control on BPPs which makes it difficult to discern whether a BPP is finalized, completed, peer-reviewed or still undergoing changes.
Poorly documented BPPs or outdated BPPs have a propagating effect on the creation of test cases and test scripts. Since BPPs are documented per stand alone SAP transaction, the testing team will need to link multiple BPPs for end to end SAP test cases (i.e. Hire–to–Fire test case) that involve multiple SAP transactions with pre and post conditions. A single poorly written BPP or outdated BPP can inhibit the creation of a test case containing several SAP transactions.
8. Missing peer reviews
Peer reviews help refine work-products and deliverables. Peer reviews also provide independent verification and give the end customers or SMEs an opportunity to provide feedback during the early stages of the SAP implementation. Peer reviews can occur at many junctures during the implementation of SAP for tasks such as filling out CI (Customer Input) templates, drafting test cases, creating functional design specs, development of business process flow diagrams, documenting BPPs, code walk-through, etc.
Inexplicably many SAP projects do not engage in the practice of peer reviews or have any templates or forms for documenting peer review feedback. The end result is that often times the end users have complaints about the quality of test cases during the UAT (User’s Acceptance Test) and about how a particular process was customized in SAP versus how the process was previously executed in the end user’s legacy systems.
Test managers should solicit the feedback, input and even the sign-off from the SMEs for the various testing artifacts as soon as possible or as the testing artifacts are produced. It is in the best interests of the test team to identify problems with the testing artifacts as soon as possible as opposed to weeks before the SAP cut-over or SAP go-live dates.
9. Problems obtaining valid test data
Arguably, the most prevalent risk to conducting an SAP test whether it is integration, functional, string, volume, smoke, or security test is obtaining valid SAP test data.
When dealing with SAP data invariable one or more of the questions below will arise before a testing cycle ensues:
Where will the test data come from? How will one produce new data for transactions that require unique data? How will the data be loaded into a new SAP client and instance? What happens when an interface or conversion needs to send data into core SAP R/3 from a legacy system and the interface or conversion is not executing properly or has yet to be developed? What happens when the transactional data has not been created?
Assuming that the testing team has a dedicated test bed or QA box the test manager will need to ensure that all the necessary data (master, transactional, test, etc) is properly loaded as part of the assessment for the test readiness review. Ideally an SAP test will be conducted within a test bed containing data that closely mirrors production data and where the testing environment is production-sized. The test manager should prioritize the test cases that will be executed and determine and document any work-around for test cases that cannot be executed due to missing test data.
10. Missing flow processes, diagrams
Diagramming or modeling how a business scenario will flow within SAP provides invaluable insights to the testing team and the configuration team. Diagrams and flow processes can illustrate the lifecycle, stages, sequences, activities, states and events associated with a particular business process. Unfortunately, many projects undergoing an SAP implementation or SAP upgrade fail to diagram their business processes leaving testers or newly hired resources in a bind to comprehend how legacy processes, or new customizations derived from gap analysis will be executed within SAP. SAP testers and end users participating in the user’s acceptance test should have access to diagrams depicting how business processes flow within SAP. The absence of such diagrams puts undue burden on the testing team and places the end users in a position of disadvantage.
The SAP business analysts in conjunction with the SAP configuration team, and possibly the SMEs (Subject Matter Experts) can develop diagrams in tools such as Visio, or the ARIS modeling tool that integrates and links with SAP’s Q&adb (Question and answer database) for those SAP implementations following the ASAP methodology. Alternatively, with Rational’s tools one can develop UML (Unified Modeling Language) diagrams to illustrate processes with interaction diagrams (sequence and activity diagrams).
11. No integration testing with external components
SAP integrates with many legacy systems via interfaces, IDOCs, conversions, BAPIs, connectors, or even middle ware such as Mercator. SAP can even be integrated with other commercial ERP systems, forecasting/planning systems, and CRM applications.
Although, the integration of SAP with other systems is critical to the successful implementation and deployment of SAP, many companies merely test during the integration test the interaction of SAP among its various modules and add-ons. Although the aforementioned type of integration testing is necessary it is also critically important to test the integration of SAP with non-SAP systems.
Some examples of integration testing that I have conducted between SAP and non-SAP systems are: SAP integrating via IDOCs to bar coding software, SAP properly sending outbound data to legacy system and the legacy system processing the data correctly, end to end financial reconciliations between SAP and data warehouses, integration between SAP and planning systems for MRP runs, etc.
12. Limited security access
I have seen SAP testing efforts come to a halt because the testers did not have the necessary privileges and permissions to execute certain SAP transactions or the necessary roles to submit electronic approvals. This problem is exacerbated when testers are executing test cases at odd hours or during weekend hours and there is no Basis support to augment the tester’s security role.
Successful execution of certain test cases in SAP whether manually or with automated test tools will require the test team members to have super user access. This is of utmost importance when conducting positive and negative testing for security roles. The testing team should coordinate and plan with the Basis team the necessary roles and permissions for developing test cases and for executing test cases in the designated QA environment or test bed.
Custom Search
Tips to software practitioners
1Software Testing Objectives
The Major Objectives of Software Testing:
- Uncover as many as errors (or bugs) as possible in a given timeline.
- Demonstrate a given software product matching its requirement specifications.
- Validate the quality of a software testing using the minimum cost and efforts.
- Generate high quality test cases, perform effective tests, and issue correct and helpful problem reports.
Major goals of Software Testing:
uncover the errors (defects) in the software, including errors in:
- requirements from requirement analysis
- design documented in design specifications
- coding (implementation)
- system resources and system environment
- hardware problems and their interfaces to software
2Software Quality Factors
Functionality (exterior quality)
- Correctness, reliability, usability, and integrity
Engineering (interior quality)
- Efficiency, testability, documentation, structure
Adaptability (future qualities)
- Flexibility, reusability, maintainability
3Software Testing Myths
- We can test a program completely. In other words, we test a program exhaustively.
- We can find all program errors as long as test engineers do a good job.
- We can test a program by trying all possible inputs and states of a program.
- A good test suite must include a great number of test cases.
- Good test cases always are complicated ones.
- Software test automation can replace test engineers to perform good software testing.
- Software testing is simple and easy. Anyone can do it. No training is needed.
4Software Testing Limits
- Due to the testing time limit, it is impossible to achieve total confidence.
- We can never be sure the specifications are 100% correct.
- We can never be certain that a testing system (or tool) is correct.
- No testing tools can copy with every software program.
- Tester engineers never be sure that they completely understand a software product.
- We never have enough resources to perform software testing.
- We can never be certain that we achieve 100% adequate software testing.
5Software Testing Principles
Principle #1: Complete testing is impossible.
Principle #2: Software testing is not simple.
Reasons:
Quality testing requires testers to understand a system/product completely
Quality testing needs adequate test set, and efficient testing methods
A very tight schedule and lack of test tools.
Principle #3: Testing is risk-based.
Principle #4: Testing must be planned.
Principle #5: Testing requires independence.
Principle #6: Quality software testing depends on:
Good understanding of software products and related domain application
Cost-effective testing methodology, coverage, test methods, and tools.
Good engineers with creativity, and solid software testing experience
6Software Testing Activities
Test Planning
Define a software test plan by specifying:
- a test schedule for a test process and its activities, as well as assignments
- test requirements and items
- test strategy and supporting tools
Test Design and Specification
- Conduct software design based well-defined test generation methods.
- Specify test cases to achieve a targeted test coverage.
Test Set up
- Testing Tools and Environment Set-up
- Test Suite Set-up
Test Operation and Execution
- Run test cases manually or automatically
Test Result Analysis and Reporting
- Report software testing results and conduct test result analysis
Problem Reporting
- Report program errors using a systematic solution.
Test Management and Measurement
- Manage software testing activities, control testing schedule, measure testing complexity and cost
Test Automation
- Define and develop software test tools
- Adopt and use software test tools
- Write software test scripts and facility
Test Configuration Management
- Manage and maintain different versions of software test suites, test environment and tools, and documents for various product versions.
7Bug Writing tips
12 Bug writing tips:
1.Be very specific when describing the bug. Don’t let there be any room for interpretation. More concise means less ambiguous, so less clarification will be needed later on.
2.Calling windows by their correct names (by the name displayed on the title bar) will eliminate some ambiguity.
3.Don’t be repetitive. Don’t repeat yourself. Also, don’t say things twice or three times.
4.Try to limit the number of steps to recreate the problem. A bug that is written with 7 or more steps can usually become hard to read. It is usually possible to shorten that list.
5.Start describing with where the bug begins, not before. For example, you don't have to describe how to load and launch the application if the application crashes on exit.
6.Proofreading the bug report is very important. Send it through a spell checker before submitting it.
7. Make sure that all step numbers are sequenced. (No missing step numbers and no duplicates.)
8.Please make sure that you use sentences. This is a sentence. This not sentence.
9.Don’t use a condescending or negative tone in your bug reports. Don’t say things like "It's still broken", or “It is completely wrong”.
10.Don’t use vague terms like “It doesn’t work” or “not working properly”
11.If there is an error message involved, be sure to include the exact wording of the text in the bug report. If there is a GPF (General Protection Fault) be sure to include the name of the module and address of the crash.
12.Once the text of the report is entered, you don’t know whose eyes will see it. You might think that it will go to your manager and the developer and that’s it, but it could show up in other documents that you are not aware of, such as reports to senior management or clients, to the company intranet, to future test scripts or test plans. The point is that the bug report is your work product, and you should take pride in your work.
The Major Objectives of Software Testing:
- Uncover as many as errors (or bugs) as possible in a given timeline.
- Demonstrate a given software product matching its requirement specifications.
- Validate the quality of a software testing using the minimum cost and efforts.
- Generate high quality test cases, perform effective tests, and issue correct and helpful problem reports.
Major goals of Software Testing:
uncover the errors (defects) in the software, including errors in:
- requirements from requirement analysis
- design documented in design specifications
- coding (implementation)
- system resources and system environment
- hardware problems and their interfaces to software
2Software Quality Factors
Functionality (exterior quality)
- Correctness, reliability, usability, and integrity
Engineering (interior quality)
- Efficiency, testability, documentation, structure
Adaptability (future qualities)
- Flexibility, reusability, maintainability
3Software Testing Myths
- We can test a program completely. In other words, we test a program exhaustively.
- We can find all program errors as long as test engineers do a good job.
- We can test a program by trying all possible inputs and states of a program.
- A good test suite must include a great number of test cases.
- Good test cases always are complicated ones.
- Software test automation can replace test engineers to perform good software testing.
- Software testing is simple and easy. Anyone can do it. No training is needed.
4Software Testing Limits
- Due to the testing time limit, it is impossible to achieve total confidence.
- We can never be sure the specifications are 100% correct.
- We can never be certain that a testing system (or tool) is correct.
- No testing tools can copy with every software program.
- Tester engineers never be sure that they completely understand a software product.
- We never have enough resources to perform software testing.
- We can never be certain that we achieve 100% adequate software testing.
5Software Testing Principles
Principle #1: Complete testing is impossible.
Principle #2: Software testing is not simple.
Reasons:
Quality testing requires testers to understand a system/product completely
Quality testing needs adequate test set, and efficient testing methods
A very tight schedule and lack of test tools.
Principle #3: Testing is risk-based.
Principle #4: Testing must be planned.
Principle #5: Testing requires independence.
Principle #6: Quality software testing depends on:
Good understanding of software products and related domain application
Cost-effective testing methodology, coverage, test methods, and tools.
Good engineers with creativity, and solid software testing experience
6Software Testing Activities
Test Planning
Define a software test plan by specifying:
- a test schedule for a test process and its activities, as well as assignments
- test requirements and items
- test strategy and supporting tools
Test Design and Specification
- Conduct software design based well-defined test generation methods.
- Specify test cases to achieve a targeted test coverage.
Test Set up
- Testing Tools and Environment Set-up
- Test Suite Set-up
Test Operation and Execution
- Run test cases manually or automatically
Test Result Analysis and Reporting
- Report software testing results and conduct test result analysis
Problem Reporting
- Report program errors using a systematic solution.
Test Management and Measurement
- Manage software testing activities, control testing schedule, measure testing complexity and cost
Test Automation
- Define and develop software test tools
- Adopt and use software test tools
- Write software test scripts and facility
Test Configuration Management
- Manage and maintain different versions of software test suites, test environment and tools, and documents for various product versions.
7Bug Writing tips
12 Bug writing tips:
1.Be very specific when describing the bug. Don’t let there be any room for interpretation. More concise means less ambiguous, so less clarification will be needed later on.
2.Calling windows by their correct names (by the name displayed on the title bar) will eliminate some ambiguity.
3.Don’t be repetitive. Don’t repeat yourself. Also, don’t say things twice or three times.
4.Try to limit the number of steps to recreate the problem. A bug that is written with 7 or more steps can usually become hard to read. It is usually possible to shorten that list.
5.Start describing with where the bug begins, not before. For example, you don't have to describe how to load and launch the application if the application crashes on exit.
6.Proofreading the bug report is very important. Send it through a spell checker before submitting it.
7. Make sure that all step numbers are sequenced. (No missing step numbers and no duplicates.)
8.Please make sure that you use sentences. This is a sentence. This not sentence.
9.Don’t use a condescending or negative tone in your bug reports. Don’t say things like "It's still broken", or “It is completely wrong”.
10.Don’t use vague terms like “It doesn’t work” or “not working properly”
11.If there is an error message involved, be sure to include the exact wording of the text in the bug report. If there is a GPF (General Protection Fault) be sure to include the name of the module and address of the crash.
12.Once the text of the report is entered, you don’t know whose eyes will see it. You might think that it will go to your manager and the developer and that’s it, but it could show up in other documents that you are not aware of, such as reports to senior management or clients, to the company intranet, to future test scripts or test plans. The point is that the bug report is your work product, and you should take pride in your work.
Be aware of SOA application security issues
Extensible Markup Language (XML), Web services, and service-oriented architecture (SOA) are the latest craze in the software development world. These buzzwords burn particularly bright in large enterprises with hundreds or thousands of systems that were developed independently. If these disparate systems can be made to work together using open standards, a tremendous amount of time, money, and frustration can be saved. Whether or not we are on the verge of a new era in software, the goal alone is enough to make security people cringe. It might be easy to glue System A and System B together, but will the combination be secure?
Today, most discussions about Web services security focus on standards: from WS-Security and SAMLdown to SOAP and even XML. While the standards are important, measuring the security of a Web service requires looking past the standards and considering all of the ways the system might fail. Attackers focus on a system's weak points, and a security assessment should do the same. Consider the following:
- Complex protocols might seem bloated or redundant. Skipping or changing steps might not create any problems for normal operation, but it could but jeopardize the security guarantees the protocol offers.
- Application platforms that support Web services are incredibly flexible. In practice, that means they have incredibly complex configurations. Mistakes in these configuration files become vulnerabilities in the service.
- Services are often built on top of legacy software that was not originally designed to operate as a service. Vulnerable code that's never been exposed to the network before offers attackers new opportunities.
· Eliminating problems like those is part of any comprehensive approach to SOA security. We'll take a look at an example of each and then talk about the business software assurance techniques you should use to prevent them.
· Most people know better than to make up their own cryptography, but a surprising number of programmers are still willing to try their hand at a home-brewed authentication protocol. Last month Google found out just what a bad idea that is. The single sign-on protocol they used for Google Applications was derived from SAML, but it left out a few seemingly unnecessary pieces of information from two protocol messages. The result was a severe security flaw that allowed a dishonest service provider to impersonate a user at another service provider. The moral to this story is if you adopt a standard, don't stop at "good enough." Subtle vulnerabilities creep in where standards compliance falls off.
· Web service containers such as WebSphere, WebLogic, or .NET WSE are amazingly versatile when it comes to Web services support, but what at first might seem a blessing quickly turns into a nightmare for the people who have to configure these systems. People stitch together configurations from sample projects or flail until somehow their service starts working. The result is service configurations that don't do exactly what the author intended. The example below shows an Apache Axis 2 Rampart client configuration that does not require inbound messages to be encrypted (since the
·
· ... ·
·
·Timestamp Signature
· ... · · · · When this configuration is buried in an enormous configuration file, the mistake is hard to catch. The result is a Web service that doesn't offer all of the security guarantees its author would.
But there are a tremendous constellation of security errors that aren't related to standards or to configuration. In fact, Web services don't introduce new types of security concerns as often as they provide new opportunities to make old mistakes. If a legacy system gets a brand new Web services interface, all sorts of problems that used to be buried deep in the system might now be able to find their way out. By exposing what used to be the internal workings of a program directly to the network, programmers may inadvertently bypass input validation or access control mechanisms, expose too much access to the internal workings of the program, or provide a new forum for making session management mistakes.
The situation is exacerbated by the fact that the security requirements for a Web service are at least somewhat ambiguous. Web services are supposed to be flexible so that they can be used by other programmers to assemble applications that the creator of the Web service may not have envisioned, but this makes it difficult for a Web service to understand the security needs of its callers. What sort of output validation should a Web service perform? If the Web service is intended to be used directly by a Web browser, then it should take precautions to prevent cross-site scripting (XSS). But if the author of a Web service doesn't know how it will be used, then it is hard to make the right security decisions.
Consider the code below. It's a method taken from DionySOA, a project that advertises itself as a Reseller/Broker service platform built using SOA and Web services. The method is exposed through a Web service. (You get a hint that it might be externally accessible when you see that it throws java.rmi.RemoteException. Knowing for sure requires looking at the application's configuration files.) The method contains a blatant SQL injection vulnerability. It takes its attacker-supplied parameter, concatenates it into a SQL query string, and executes the query. Although it is possible to make this kind of mistake without any help from Web services, we can't help but believe that the Web services setup made it easier to forget about input validation.
public supplier.model.SupplierProduct[ ]
searchName(java.lang.String in0) throws
java.rmi.RemoteException { System.out.println("searchName("+in0+")"); String query="SELECT * FROM products " +
" WHERE name like '"+in0+"' ";
return this.doSQL(query);
}
Similarly, if a newly exposed method used to rely on another part of the application to perform access control checks, the Web services interface might now bypass those checks, making it easy to lose track of the trust boundary.
There's nothing fundamentally wrong with making it easy to create a Web service, but creating a good Web service is really not so easy. The Web services frameworks we are aware of do not give a programmer any guidance about the security implications that might be involved in exposing the insides of a program.
We don't mean for these examples to discourage the use of Web services, XML, and SOA frameworks, but rather to make users aware of the missteps that can occur when deploying these technologies. These examples will hopefully be a catalyst for taking security into account as part of the software development process. Business software assurance requires building security into every step, rather than trying to tack it on at the end. Just as quality assurance is the operational solution for managing product quality, business software assurance is the operational solution for managing software risk. The following activities should be part of any software project:
- Think like a bad guy. How would you attack this system? What would you hope to gain?
- Automated vulnerability detection -– source code analysis and dynamic security testing -– to identify vulnerabilities before the code is deployed.
- Real-time monitoring and active protection to defend the code once it's in production.
As companies become increasingly dependent on software to run the business, and as new technologies enable old applications to find fresh new uses, companies must make security a priority. A more proactive and systematic approach towards software security will be -- in the end -- less costly and more predictable than an ad-hoc security strategy.
How to specialize in performance testing
How to specialize in performance testing
Q-I have been learning performance testing for the past eight months. However, I was not given any opportunities in performance testing in my present company even though I performed well in the internal interviews. I really want to become a good performance tester -- it is my dream to become a performance tester. Please guide me as to how I can make this happen.
Expert’s Response: I think this is a great question. It's specific, which makes it easy to start answering, but it's also general enough that anyone who is interested in specializing in something within software testing should be able to pull something from the answer. I think there are three overarching dynamics to your question:
· How can you best structure your future learning to support your goals?
· How can you best market your abilities to get the opportunities you want?
· How can you best structure the work you're currently doing to support your work and learning objectives?
Continuing to learning about performance testing
The activity that you have the most control over is your own learning. I've been studying and doing performance testing for eight years, and I honestly still learn something new about performance testing almost every week. It's a deep and rich specialization in software testing. There's a lot to performance testing that still needs to be formalized and written down. It's still a growing body of knowledge.
If your dream is performance testing, then you need to continue to learn. Reading articles, blogs, books and tool documentation is a good place to start. Attending conferences, training, workshops and local groups is a great place to meet others who have similar passions. If you don't have opportunities like those, then join one of the many online communities where performance testers have a presence. Depending on your learning style, dialog and debate can be as great a teacher as reading, if not greater.
Finally, no learning is complete without practice. I'm so passionate about the topic of practice that I wrote an entire article on it. Many of the materials you read will include exercises. Work through them. Many of the conferences, training, and workshops you attend will show examples. Repeat them. Going through the work on your own, even if you already know the outcome, provides a different kind of learning. Some people learn best when the experience is hands-on.
For performance testing, I think a great place to start practicing is in the open source community. Given the nature of performance testing, most tool knowledge is transferable to other performance testing tools. Learning multiple open source tools will also give you different ideas for how you can solve a performance testing problem. Many times, our available tools anchor our thinking about how to approach the problem. If you've practiced with multiple tools, you're more likely to have variety in your test approaches and solutions.
Once you know how to use a couple of performance testing tools, if you can't seem to get the project work you need at your current employer, and you're unwilling or unable to leave for another opportunity, then I recommend volunteering your time. There are a lot of online communities that help connect people who want to volunteer their technical talents to nonprofits or other community-minded organizations. Finding project work outside of your day job can be just as valuable as formal project work.
Start marketing your skills and abilities
If you're serious about performance testing as a career, I recommend you start pulling together some marketing material. A resume is the place most people focus their limited marketing skills. That could be a good place for you to start as well. What story does your resume tell a potential employer? Is it that you're a performance tester? How has each of your past experiences helped you develop a specific aspect of performance testing? Remember, one of the great challenges performance testing presents to practitioners is its variety. That makes it easy to relate a variety of experiences to the skills a performance tester needs.
Don't forget to include your training on your resume. I've had to remind several people of classes they've attended, workshops they participated in, or people who have been an active member of an online community for years and have not included that on their resume. If it helps you tell the story of your expertise, get it on there. Include anything that shows an employer that you're passionate about performance testing and you're continuously learning more about it.
Depending on the types of companies you want to work for, or the types of projects you might want, a certification might be appropriate. Certifications relevant to performance testing aren't just performance testing tool certifications. Appropriate certifications may also come in the form of programming languages (e.g., Java certification), networking (e.g., CCNA), application servers (e.g., WebSphere administrator certification), databases (e.g., Oracle certification), or even a certification in the context you want to work in (e.g., CPCU certification if you want to work in the Insurance industry). I'm not normally a big fan of certifications, but they are clear marketing products.
Finally, I think the best way to market yourself is to write. Start by being active in an online community. Answer questions on forums or debate ideas on mailing lists. As you learn, catalog your learning in a blog so others can benefit from your hard work. If you feel you're really starting to understand a specific aspect of performance testing, try writing an article or paper on it (for example, email your idea to an editor at SearchSoftwareQuality.com -- they'll point you in the right direction for help if you need it). Present your idea at a conference or workshop. The more of a public face you develop by writing, the more you learn. My experience has been that people are very vocal in their feedback on what you write. You should get to learn a lot. Even if you don't become the next Scott Barber, when a potential employer Googles your name, they'll quickly see that you know something about performance testing and have a passion for it.
Align your project work with performance testing activities
Even if you can't get performance testing projects at your current employer, you can still get project work that relates to performance testing. Does your team test Web services? See if you can get involved; it will get you experience with XML, various protocols and, often, specialized tools. Does your team test databases? See if you can get involved; it will get you experience with SQL and managing large datasets. Does your team write automated tests? See if you can get involved; it will get you experience programming and dealing with the problems of scheduled and distributed tests. Does your team do risk-based testing? See if you can get involved; it will get you experience modeling the risk of an application or feature and teach you how to make difficult choices about which tests to run. I could go on with more examples. Take your current opportunities and make them relevant for learning more about performance testing.
If you can't get your own performance testing project, ask if you can work with someone else. What if you volunteer some of your time? What if you work under someone else's supervision for a while? Work with your current manager to understand what factors are preventing them from giving you the opportunity. Perhaps they can't give you the opportunity for a number of reasons out of their direct control. Perhaps they can, they just haven't given it enough attention. After a conversation where you try to figure it out with them, you should have an idea of what opportunities are available at that company. Just recognize that sometimes you have to leave for different opportunities. If you do that, make sure you're clear with your new employer as to what your expectations are.
I hope that's helpful. Your question is a great one, and I feel like it covers a general concern software testers have. The general form of the answer is the same for people who might want to specialize in security testing, test automation, Web service testing, test management, or any other aspect of testing where there can be specialization. Stay focused on your learning and development, actively market your knowledge and abilities, and work to align your work with your goals -- even if that means taking projects outside of the specialization to help you develop a specific skill.
How to do integration testing
How to do integration testing
Q- How do testers do integration testing? What are top-down and bottom-up approaches in integration testing?
Expert’s response: Ironically, integration testing means completely different things to completely different companies. At Microsoft, we typically referred to integration testing as the testing that occurs at the end of a milestone and that "stabilizes" a product. Features from the new milestone are integration-tested with features from previous milestones. At Circuit City, however, we referred to integration testing as the testing done just after a developer checks in -- it's the stabilization testing that occurs when two developers check in code. I would call this feature testing, frankly…
But to answer your question, top-down vs. bottom-up testing is simply the way you look at things. Bottom-up testing is the testing of code that could almost be considered an extension of unit testing. It's very much focused on the feature being implemented and that feature's outbound dependencies, meaning how that feature impacts other areas of the product/project.
Top-down, on the other hand, is testing from a more systemic point of view. It's testing an overall product after a new feature is introduced and verifying that the features it interacts with are stable and that it "plays well"' with other features.
The key to testing here is that you are in the process of moving beyond the component level and testing as a system. Frankly, neither approach alone is sufficient. You need to test the parts with the perspective of the whole. One part of this testing is seeing how the system as a whole responds to the data (or states) generated by the new component. You want to verify that data being pushed out by the component are not only well-formatted (what you tested during component testing) but that other components are expecting and can handle that well-formatted data. You also need to validate that the data originating within the existing system are handled properly by the new component.
Real-world examples? Well, let's assume you are developing a large retail management system, and an inventory control component is ready for integration. Bottom-up testing would imply that you set up a fair amount of equivalence-classed data in the new component and introduced that new data into the system as a whole. How does the system respond? Are the inventory amounts updated correctly? If you have inventory-level triggers (e.g., if the total count of pink iPod Nanos falls below a certain threshold, generate an electronic order for more), does the order management system respond accordingly? This is bottom-up testing.
At the same time, you want to track how well the component consumes data from the rest of the system. Is it handling inventory changes coming in from the Web site? Does it integrate properly with the returns system? When an item's status is updated by the warehouse system, is it reflected in the new component?
We see constant change in the testing profession, with new methodologies being proposed all the time. This is good -- it's all part of moving from art to craft to science. But just as with anything else, we can't turn all of our testing to one methodology because one size doesn't fit all. Bottom-up and top-down testing are both critical components of an integration testing plan and both need considerable focus if the QA organization wants to maximize software quality.
Test coverage: Finding all the defects in your application
Test coverage: Finding all the defects in your application
Q-If trace matrix does not meet the requirement for test coverage, what would you suggest for the same? How can I assure the coverage of all functionalities by a team member as a team leader?
Expert Response: The trace matrix is a well-established test coverage tool. Let me offer a quick definition -- the purpose of the trace matrix is to map one or more than one test case to each system requirement, the trace matrix is usually formatted in a table. The fundamental premise is that if one or more than one test case has been mapped to each requirement, then all the requirements of the system must have been tested and therefore the trace matrix proves testing is complete.
I see flaws with this line of reasoning and here are my primary reservations on the over-reliance of the trace matrix:
- A completed trace matrix is only as valuable as the contents. If the requirements are not complete or clear than the test cases designed and executed might fulfill the requirements but the testing won't have provided what was needed. Conversely if the requirements are clear but the test cases are insufficient then a completed trace matrix still doesn't indicate the testing coverage and confidence that is being sought by a completed table.
- The trace matrix design relies too stringently on system requirements -- that is the primary design of the trace matrix -- to ensure all system requirements have been tested. But all sorts of defects can be found outside of the system requirements that are still relevant to the application providing a solution for the customer. By looking only at the system requirements and potentially not considering the customers' needs and real life product usage, essential testing could be overlooked. Testing only according to specified requirements may be too narrowly focused to be effective in real life usage -- unless the requirements are exceptionally robust.
Overall I feel the trace matrix might provide a clean high level view of testing but a checked-off list doesn't prove an application is ready to ship. The reason some people value the trace matrix is the matrix attempts to offer an orderly view of testing; but in my experience testing is rarely such a tidy task.
So how do you call the end of testing? And how can you assure test coverage?
- To be able to assure coverage at the end, I'd start with reviewing the beginning -- look at the test planning. Did your test planning include a risk analysis? A risk analysis at the start of a project can provide solid information for your test plan. Host a risk analysis either formally or informally, gather ideas by talking with multiple people. Get different points of view -- from your project stakeholders, talk to your DBAs, your developers, your network staff, and your business analysts. Plan testing based on your risk analysis.
- As a project continues, shift testing based on the defects found and the product and project as it evolves. Focus on high risk areas. Adapt testing based on you and your testing team's experience with the product. Be willing to adjust your test plan throughout the project.
- Throughout testing, watch the defects reported. Keep having conversations and debriefs with hands-on testers to understand not just what they've tested but how they feel about the application. Do they have defects they've seen but haven't been able to reproduce? What is their perception of the current state of the application?
In my view, there is no one tool including the trace matrix that signals testing is complete but the combination of knowing how testing was planned and adapted throughout the project, a thorough review of the defects reported and remaining, and the current state of the application according to you and your team's experience should provide you with an objective assessment of the product and the test coverage.
Subscribe to:
Posts (Atom)
