REMEMBER: Your homework #4 is due Thursday!!
This week we will work on:
The real purpose of software testing, of course, is to make sure that the software works. A systematic testing approach is required that will be planned, documented, and maintainable. Keep in mind Dijkstra's Law of Testing, which says, "program testing can be used to show the presence of bugs, but never their absence." Nevertheless, a good testing strategy will show up the maximum number of problems before an initial release, and will facilitate testing and finding problems in future releases.
A planned testing approach will permit the software being developed to include code that will facilitate its own testing and debugging.
A documented testing approach will allow test cases to be easily understood and will facilitate re-use; documenting tests also helps in determining their adequacy for the task at hand.
A maintainable testing approach will help with using the same test cases after any and every change to the software.
A good strategy for software testing joins software test case design methods together into a well-planned series of steps that result in the construction of software that operates in a manner consistent with its requirements, i.e., correct software. The test strategy provides a "road map" to describe the steps to be performed as part of testing, at what phase in the development process these steps are planned and then undertaken, and the amount of effort, time, and resources that will be required.
A number of testing strategies have been proposed in the literature. All of them provide the software developer with a template for testing, and all of them put forth the following general ideas:
Systematic testing uses two different pieces to provide a consistent approach: the Test Plan provides the strategy for selecting and executing the tests, and the Test Implementation implements (or instantiates) the Test Plan. There is a subtle implication in this fact, namely that the purpose or focus of software testing is NOT to show that a program is free of errors, rather it is to detect the program errors. Although this may seem a slight difference, it has an important consequence for designing tests, since it says the focus should be on positively working to make the program fail so as to expose bugs.
There are actually six steps involved in the software testing process:
The last three steps are often combined in automated testing so they happen while the test is executing. For example, the test harness may, during its run, show the input values and the corresponding output values on the screen, as well as logging them to a file. Since both the expected and actual output should be known at every step, the comparison may be made real-time, and the pass/fail result displayed. The result may be written to the same or to a different log file than the test output results. Post-processing is sometimes performed on the resulting file(s) to provide a formal report of the test completion and outcome.
To repeat, the focus of the testing should be on finding defects, actually "breaking" the software. It sounds harsh, but breaking the product while it is still in development, then fixing what is broken, is MUCH less expensive in the long run than releasing software with bugs and having your customer or users find the bugs for you. Thus, testing is the "last bastion" from which software quality can be assessed and errors may be uncovered.
There is an inherently incestuous relationship that occurs at the beginning of the testing process, since the developers who have worked hard to produce correct, defect-free software are now required to attempt to "break" their own product. The developers frequently take the position of "proving the software is correct", rather than "uncovering defects", which is the antithesis of the purpose of software testing. For many programmers, it is hard to switch gears from "proving it's right" to "finding what's wrong". In addition, programmers have an obligation to demonstrate their programs are error free, that they work according to the requirements, and that the programs will be completed within constraints of time and budget. Besides, who knows the program better than the person who wrote it? This attitude produces a subtle conflict of interest; further, the whole process can become psychologically demoralizing to a programmer who is personally involved with the code. The result is that the programmer my be inclined to "test lightly", designing tests that show proper program performance, but which don't uncover any program errors. This provides further rationale for the ITG or QA department to test the software.
One last set of definitions:
So, to work…
The test implementation puts the planned tests into action. The implementation consists of the code for the test harness and the test cases, as well as the inputs, the logging tool, the expected outputs, and the comparison or checking mechanism. When implementing the test strategy according to the test plan, it is important to remember the test philosophy of the plan. You must also remember to control the code and data in a CM system so that it may be used to test future releases. The test implementation may take on any of several forms, as described in the following paragraphs.
The bottom up approach to testing focuses first on the units or modules of the program as independent parts. Tests are written to exercise all the facets of the code. Ideally, all or nearly all lines of code will be tested automatically. This implies that every line of code in every "if" block, every iteration through a "while" or "for" loop, every exception that is thrown, should all be executed. This (obviously) requires many passes through the same set of code, each pass with differing inputs and different expected outputs.
The bottom up approach is the preferred method in modern programming projects, especially in the case of object-oriented programming. OOP lends itself very readily to unit testing, since a small "test" program can be built into the module code itself, in addition to having a test harness that provides external inputs and can log the outputs produced. For example, in Java, each class file might have a small "main" method at the bottom, which could take command line arguments that would enable different paths of execution through the code of the class. The test harness would call this main method over and over, with different sets of arguments in each call, and would log the results. By providing the proper set of input values as arguments, the class could be thoroughly (NOT exhaustively!) tested.
In top down testing the process starts by testing a top level module using "stub code" or just "stubs." These are small "stand-in" or "temporary" modules that perform the simulated actions of the real modules they take the place of. As testing progresses, the stubs are gradually replaced with the real modules as they are developed. In this manner, the program can be built up from the top level and working downward to increase the "real" functionality of the system.
Top down testing has its uses; for example, if a system is incompletely specified, so that the requirements for the lower level modules are not complete at the time coding starts, it might be useful to code the top level routines so that the customer may see what the initial design ideas look like. Another use might be in the case of spiral development of a system which is procedural. However, keep in mind that the use of stub code may mislead the test team into thinking that there is more functionality complete than is actually present. This can lead to missed requirements unless careful documentation is followed.
If the code to be tested is readily available, the test author may be able to analyze it and develop a set of tests that is based on the analysis. Knowing how the software is coded is a distinct advantage to the tester, both in writing the test code and test harness, and in determining the inputs and expected outputs for the test. The test can know what all the conditions are that occur during testing, what kinds of control structures are involved that require execution, and what inputs are required to ensure that all parts of the code are thoroughly exercised. This is known as white box testing.
White box testing is difficult to do in systems, particularly if they are large systems, due to its exhaustive nature. However, it can be used effectively at unit level, since the code base to be tested is smaller. In this way, white box testing can be combined with bottom up testing to produce a test suite that is very comprehensive, without necessarily being exhaustive.
In black box testing, on the other hand, the test author has no idea what the internal workings of the code may be. She knows only the inputs and expected outputs. In this case, what is needed are tests which show the actual and expected outputs match for all the inputs used for the tests.
To create black box tests, heavy reliance is made on the system requirement specification, along with the architectural and process model views of the system, to decide what the input to the tests will be, what the expected outputs should be, and how to interface properly to the modules being tested. Also, a great deal of inference may be needed to make some of the determinations.
Once the parts have been individually tested and have been proved to work correctly, the system as a whole must be tested together. This system testing must be done in a more realistic working environment than the unit level tests, since it will more accurately show the way the system will be used by the customer.
Functional tests are very similar to black box testing, except at the system level. Inputs and expected outputs are selected from the system requirements and other specifications, and from the results of the unit testing (which may be black or white box). By understanding how the system is supposed to work, proper test case selection may be made.
These system tests are based more on the white box testing philosophy. The internals of the system are used to select test cases, inputs, and expected outputs which will guarantee that every method or function of every class will be called at least once with every argument and output return value exercised (or at least the largest subset practicable). In this manner the methods can be shown to operate and communicate properly, robustly, and as expected.
If a module takes a certain discrete set of inputs and produces a discrete set of outputs, it is certainly feasible (especially in an automated test environment) to check all the inputs and to verify all the outputs produced. However, if a method is designed to accept input from an unbounded set, such as all real numbers or all positive integers, there is no way to test every possible input. The same ideas also apply to the outputs produced. If all inputs and possible outputs are tested, this is known as exhaustive testing, so named because it exhausts all the possible combinations, inputs, and outputs.
In many cases, however, it is impossible to be exhaustive. In these instances, it is still possible, by carefully selecting the test cases, to comprehensively test the module. To this end, pick several test cases that are in the middle of the range of expected inputs, as well as several that are on the edges of that range. Don't forget to include a couple of inputs that are out of range, to verify error handling capabilities.
Now that you know ALL about testing, it's time to figure out how to apply that knowledge.
In the era of software-driven systems (such as cell phones, automobiles, video devices, and pretty much
any and everything else you can think of) the number of interactions between the components of these
software systems creates a situation in which it would be difficult (if not impossible) to actually
have a person manually test the software, even at the lowest "unit" level. Imagine
having to manually unit test a module like JavaScript's Math object to verify the correct
operation of every function for every possible input (and some impossible ones, too), and you can see
how automated testing is really not optional, it is required.
Another side of automated testing concerns the fact that with a suite of automated tests, it is very easy to make sure that your software works at any level, simply by running the test suite at any time, and checking the results. In fact, this is a scenario which many software development companies use, running their automated software tests overnight, so that when the development teams arrive for work the next day, everyone can see what is broken and needs to be fixed.
A final point about automated tests is that when you have a complete and properly maintained set of tests for your software product, you will be much less afraid to make changes or improvements, since you can always run the tests to make sure nothing got broken by the changes.
With JavaScript, as with many other programming languages, there is a testing framework which facilitates this paradigm of automated testing, called QUnit. One of the best ways to learn to use this framework for testing your software is to download it and start using it. To do that, you set up a web page, include a couple of QUnit files and your JavaScript source code, and open the page in a browser. Pretty simple, eh? Here's an example:
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>QUnit basic example</title>
<link rel="stylesheet" href="qunit-1.12.0.css">
</head>
<body>
<div id="qunit"></div>
<div id="qunit-fixture"></div>
<script src="qunit-1.12.0.js"></script>
<script>
test( "a basic test example", function() {
var value = "hello";
equal( value, "hello", "We expect value to be hello" );
});
</script>
<script>
test( "another basic test example", function() {
var values = [1,2,3,4,5,6];
var roll = Math.floor(1 + (Math.random() * 6));
equal( roll, 1, "Rolled a 1, test succeeds" );
equal( roll, 2, "Rolled a 2, test succeeds" );
equal( roll, 3, "Rolled a 3, test succeeds" );
equal( roll, 4, "Rolled a 4, test succeeds" );
equal( roll, 5, "Rolled a 5, test succeeds" );
equal( roll, 6, "Rolled a 6, test succeeds" );
});
</script>
</body>
</html>
So, when you write a test, you simply embed it in your web page along with your script. Then when you just load up the web page the tests are run and the results are displayed. The same principle applies as what we are used to doing with JavaScript: make a change to the script, refresh the web page, and see the immediate result.
Here's another example which uses a separate script file for the JavaScript, so you can see how this would work:
<!doctype html>
<head>
<meta charset='utf-8'>
<script type='text/javascript' src='recursiveFibonacci.js'></script> <!-- Custom code for the menu bar -->
<script type='text/javascript' src='functionGroup.js'></script> <!-- Custom code for the menu bar -->
<!-- The next two lines are needed for the qunit tests -->
<link rel="stylesheet" href="qunit-1.12.0.css">
<script src="qunit-1.12.0.js"></script>
</head>
<body>
<h1>Sample HTML Test Page Using Q-Unit Test Framework</h1>
<p><a href=''>Here is a link</a> to the Q-Unit API documentation page that should be checked out. You can and should
visit this page, set up a web page, and test your code. I recommend you AT LEAST practice with this for your own
purposes of testing your functions as you write them. It is easy to maintain a simple test page like this one for
your own use, and simply copy and paste things for each successive project. Further, you should know the format by
heart for use on the final exam [subtle hint].
<div id="qunit"></div>
<div id="qunit-fixture"></div>
<script src="qunit-1.12.0.js"></script>
<script>
test( "a basic test example", function() {
var value = "hello";
equal( value, "hello", "We expect 'value' to be the string 'hello'" );
});
</script>
<script>
test( "tests the included function sayHi() from functionGroup.js", function() {
equal( sayHi("Bob"), "Hey there Bob, glad to see you today!", "We expect this to pass." );
});
// NOTE: you can have more than one test at a time inside the same script tag
test( "tests the included function getName() from functionGroup.js", function() {
equal( getName(), "Annie", "We expect a pass if the user enters the string 'Annie'." );
});
// NOTE: you can add extra comments inside the script tag as well...
test( "tests the included functions getName() and sayHi() BOTH at the same time", function() {
equal( sayHi( getName() ), "Hey there JimmyJoeDon, glad to see you today!",
"We expect this to pass if the user enters the string 'JimmyJoeDon'." );
});
// NOTE: you can have multiple equal or OK or any other kind of test in the same test block
test( "tests the included function containsA() from functionGroup.js", function() {
let z = prompt( "Enter a string to check for 'a': " );
let p = prompt( "Enter a string which is the same length\nas what you just entered: " );
let q = p.substring( 0, p.length - 1 );
equal( containsA( z ), true, "We expect this to pass if the string entered contains an 'a'." );
equal( areSameLength( z, z ), true, "We expect this to pass since the strings are the same. " );
equal( areSameLength( z, p ), true, "We expect this to pass ONLY if the strings are the same length." );
ok( areSameLength( z, z ), "We expect this to pass since the strings are the same and it's an 'OK' test." );
strictEqual( areSameLength( z, z ), true, "We expect this to pass since the strings are the same (strict)." );
strictEqual( areSameLength( z, p ), true, "We expect this to pass ONLY if the strings are the same length (strict)." );
notEqual( areSameLength( z, q ), true, "We expect this to pass ONLY if the strings are NOT the same length." );
});
// NOTE: you can do anything with the test inputs that is legal JavaScript
test( "tests the included function addEmUp() from functionGroup.js", function() {
equal( addEmUp( [1,2,3] ), 6, "We expect this to pass if the total is 6." );
let n = [1,2,3,4,5,6,7,8,9,10];
equal( addEmUp( n ), 55, "We expect this to pass if the total is 55." );
equal( addEmUp( n ) + 11, 66, "We expect this to pass if the total is 66." );
equal( Math.sqrt( addEmUp( n ) + 45 ), 10, "We expect this to pass if the returned value is 10." );
});
test( "we can also use the 'OK' test to simply put out a message.", function() {
ok( true, "The Q-Unit framework will also tell us about synatx errors! HANDY!!" );
});
</script>
<script>
// NOTE: we expect ALL of these to fail, except one, since the die only has one face value at a time
test( "another basic test example", function() {
let values = [1,2,3,4,5,6];
let roll = Math.floor(1 + (Math.random() * 6));
equal( roll, 1, "If a 1 is rolled, THIS test succeeds" );
equal( roll, 2, "If a 2 is rolled, THIS test succeeds" );
equal( roll, 3, "If a 3 is rolled, THIS test succeeds" );
equal( roll, 4, "If a 4 is rolled, THIS test succeeds" );
equal( roll, 5, "If a 5 is rolled, THIS test succeeds" );
equal( roll, 6, "If a 6 is rolled, THIS test succeeds" );
});
</script>
</body>
</html>
<
Here is the matching code from the javascript file:
/**
* A simple function that just says hello for the name passed in
*/
function sayHi( name ) {
return "Hey there " + name + ", glad to see you today!";
}
/**
* A simple function that just returns the name the user enters
*/
function getName() {
return prompt("What is your name?");
}
/**
* A simple function to see if the argument contains a lower case 'a'
*/
function containsA( input ) {
return (input.indexOf("a") !== -1);
}
/**
* A simple function to determine if two strings are equal in length
*/
function areSameLength( a, b ) {
console.log("Just a note to let us know the function was called..." );
return (a.length === b.length) ? true : false;
}
/**
* A slightly more complex function to add the values of an array
*/
function addEmUp( arrayToAdd ) {
let total = 0;
for( var i = 0; i < arrayToAdd.length; i++ ) {
total += arrayToAdd[i];
}
return total;
}
Your text book shows other code which includes ways to use both jQuery and Q-Unit directly from the Internet, which is a good way to do it so that you don't have to download anything. As long as the GitHub repository is online and working properly, everything will be hunky-dory. So, write Q-Unit tests for all your script functions and you will be far ahead on the bug-free working program curve!
Another way to test your code is to do what's called "instrumenting" the code. We've talked about
this concept earlier in the semester, but it's included here for completeness with testing. The idea is to
add in temporarily one or more alert() calls to output intermediate values so that you
can see if things are working as expected. This kind of thing can be very helpful when you are trying to see
if some computation is progressing normally, if a function is getting called, if a loop is being executed, or
which leg of an if statement is being taken. It's primitive, but just about every programmer
in the world today will (at least occasionally) use this method as a sanity check.
One last thing to re-visit before we close out is to take one more look at recursion. You remember that it is
the method in which a function calls itself. Here is an example for recursively finding the Nth
Fibonacci number, which is a pretty standard thing to show in Computer Science:
// Running the fibonacci sequence in recursive mode
// This version is based on the number sequence that defines the
// first fibonacci number as the first 1 in the sequence.
// In other words:
//
// 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55
// ^
// |
// first fibonacci number
//
// ...which means that the fifth one is 8
//
var fib = function( x ) {
if( x < 2 ) {
return 1;
} else {
return fib( x - 2 ) + fib( x - 1 );
}
}
Next week is the last week of class, so it will be dedicated to reviewing all the material we've covered this semester in preparation for your final exam. This should get you all well on the way on the path to full final exam success!!