Transcript .ppt

CS 5150
Software Engineering
Lecture 22
Reliability 3
Administrivia
•
Quiz 3 next Monday
•
•
•
•
CS 5150
Today is the deadline for rescheduling!
Design patterns
Security/privacy
Reliability
2
Final Milestone
•
Client satisfaction
•
•
CS 5150
In previous milestones my evaluation of
client satisfaction has been informal
This time I will ask them to send me an
email a few days after receiving the final
deliverable, with answers to a couple simple
questions about how well the project went
3
SE in the News
•
CS 5150
Petraeus and privacy
4
Testing Testing Testing
•
Testing comes in many flavors
•
•
•
•
•
•
•
•
•
CS 5150
User testing
Unit testing
Integration testing
Performance testing
Stress testing
Regression testing
Acceptance testing
Manual/automated testing
Close/open-box (black/white-box) testing
5
Fundamental Facts About Testing
•
•
Testing is indispensable for software
verification, but can never prove that non-trivial
software is defect-free
Each test proves that the system works in one
particular circumstance
•
•
CS 5150
Most non-trivial software has infinitely many
possible execution contexts (i.e., input,
environment, etc)
Every project should have a verification plan
that includes testing strategies
6
Unit Testing
•
•
•
•
•
CS 5150
Each unit test should verify that a single ‘unit’
of software works in isolation
Can be challenging to write unit tests for code
buried deep in a project
Overlap with type systems
Basic sanity checks
Should run automatically
•
e.g. overnight
7
Integration Testing
•
•
CS 5150
Motivation: components developed by different
people/teams; bugs can easily crop up at the
boundaries
Definition a little nebulous
•
More useful to explicitly do other kinds of
testing
8
Regression Testing
•
Every time you find a bug in your project,
make a test case that reproduces the bug
•
•
A good set of regression tests provide a
‘ratcheting’ mechanism
•
CS 5150
The test case should be committed to your
repository no later than the bug fix itself
Helps prevent the recurrence of bugs
9
Fuzz Testing
•
•
CS 5150
Run your program many times with a random
sequence of inputs
Black-box vs white-box
10
Manual Testing
•
Has a bad reputation in some computer
science circles, but is absolutely essential
•
•
CS 5150
Not all testing can be automated
Write test scripts
11
Dealing with Bugs
•
Any change in complex software has a nontrivial chance of introducing bugs
•
•
•
•
CS 5150
Major reason for managerial conservatism
Bug avoidance
Bug tolerance
Bug detection
12
Defensive Programming
•
Write code to check assumptions (numerical
value ranges, complex data structure
invariants)
•
•
•
•
CS 5150
Call it all over the place
JPL rules about assertion frequency
“Impossible” things can happen in libraries,
compilers and even hardware
Avoid risky programming techniques
•
Many coding standards ban ‘goto’ entirely
13
Fault Tolerance
•
•
•
•
CS 5150
Instead of bombing out when unexpected
conditions occur, find some way to keep going
Checkpoints
Logs
Fault tolerance mechanisms are extremely
difficult to test (so it’s best to keep them
simple)
14
Fixing Bugs
•
•
•
CS 5150
Reproducibility
Test case complexity
Workaround vs “proper” fix
15
Determinism => Reproducibility
•
•
It is very hard to fix bugs that cannot be
reproduced reliably
Do not introduce unnecessary nondeterminism into your programs
•
•
•
•
•
CS 5150
Random numbers
Address-based data structures
Multithreading
Time limits vs effort limits
Environment dependencies
16
Safety-Critical Systems
•
•
CS 5150
Much more emphasis on validation and
verification
Beyond the scope of 5150
17