|
|
xi | |
|
|
xiii | |
Foreword |
|
xix | |
Acknowledgements |
|
xxi | |
PART I: INTRODUCTION TO EXPERIMENTATION |
|
|
|
|
Pre-Scientific Status of Software Engineering |
|
|
3 | (3) |
|
Why Don't We Experiment In Se? |
|
|
6 | (4) |
|
Kinds of Empirical Studies |
|
|
10 | (2) |
|
Amplitude of Experimental Studies |
|
|
12 | (5) |
|
|
17 | (1) |
|
Who Does This Book Target? |
|
|
18 | (1) |
|
Objectives To Be Achieved By the Reader of This Book |
|
|
19 | (1) |
|
|
20 | (3) |
|
Why Experiment? The Role of Experimentation In Scientific and Technological Research |
|
|
|
|
23 | (1) |
|
Research and Experimentation |
|
|
23 | (3) |
|
The Social Aspect in Software Engineering |
|
|
26 | (1) |
|
The Experimentation/Learning Cycle |
|
|
27 | (6) |
|
|
33 | (2) |
|
Why Do Experiments Need To Be Replicated? |
|
|
35 | (5) |
|
Empirical Knowledge Versus Theoretical Knowledge |
|
|
40 | (5) |
|
|
|
|
45 | (1) |
|
Searching for Relationships Among Variables |
|
|
45 | (2) |
|
Strategy of Stepwise Refinement |
|
|
47 | (2) |
|
Phases of Experimentation |
|
|
49 | (2) |
|
Role of Statistics In Experimentation |
|
|
51 | (6) |
PART II: DESIGNING EXPERIMENTS |
|
|
Basic Notions of Experimental Design |
|
|
|
|
57 | (1) |
|
Experimental Design Terminology |
|
|
57 | (8) |
|
The Software Project As An Experiment |
|
|
65 | (5) |
|
Response Variables In Se Experimentation |
|
|
70 | (10) |
|
|
80 | (3) |
|
|
|
|
83 | (1) |
|
|
83 | (2) |
|
|
85 | (5) |
|
How To Avoid Variations of No Interest To The Experiment: Block Designs |
|
|
90 | (7) |
|
Experiments With Multiple Sources of Desired Variation: Factorial Designs |
|
|
97 | (5) |
|
What To Do When Factorial Alternatives Are Not Comparable: Nested Designs |
|
|
102 | (1) |
|
How To Reduce The Amount of Experiments: Fractional Designs |
|
|
103 | (1) |
|
Experiments With Several Desired and Undesired Variations: Factorial Block Designs |
|
|
104 | (9) |
|
Importance of Experimental Design and Steps |
|
|
113 | (3) |
|
Specific Considerations for Experimental Designs In Software Engineering |
|
|
116 | (3) |
|
|
119 | (6) |
PART III: ANALYSING THE EXPERIMENTAL DATA |
|
|
Basic Notions of Data Analysis |
|
|
|
|
125 | (1) |
|
Experimental Results As A Sample of A Population |
|
|
126 | (2) |
|
Statistical Hypotheses and Decision Making |
|
|
128 | (4) |
|
Data Analysis for Large Samples |
|
|
132 | (5) |
|
Data Analysis for Small Samples |
|
|
137 | (10) |
|
Readers' Guide To Part III |
|
|
147 | (4) |
|
|
151 | (2) |
|
Which Is The Better of Two Alternatives? Analysis of One-Factor Designs With Two Alternatives |
|
|
|
|
153 | (1) |
|
Statistical Significance of the Difference Between Two Alternatives Using Historical Data |
|
|
153 | (7) |
|
Significance of the Difference Between Two Alternatives When No Historical Data Are Available |
|
|
160 | (3) |
|
Analysis for Paired Comparison Designs |
|
|
163 | (2) |
|
One-Factor Analysis With Two Alternatives In Real Se Experiments |
|
|
165 | (8) |
|
|
173 | (2) |
|
Which of K Alternatives Is the Best? Analysis for One-Factor Designs and K Alternatives |
|
|
|
|
175 | (1) |
|
Identification of the Mathematical model |
|
|
176 | (3) |
|
Validation of the Basic Model That Relates the Experimental Variables |
|
|
179 | (7) |
|
Calculations the Factor- and Error-induced Variation in the Response Variable |
|
|
186 | (3) |
|
Calculating the Statistical Significance of the Factor-Induced Variation |
|
|
189 | (6) |
|
Recommendations Or Conclusions of the Analysis |
|
|
195 | (4) |
|
Analysis of One Factor With K Alternatives In Real Se Experiments |
|
|
199 | (2) |
|
|
201 | (2) |
|
Experiments With Undesired Variations: Analysis for Block Designs |
|
|
|
|
203 | (1) |
|
Analysis for Designs With A Single Blocking Variable |
|
|
203 | (13) |
|
Analysis for Designs With Two Blocking Variables |
|
|
216 | (3) |
|
Analysis for Two Blocking Variable Designs and Replication |
|
|
219 | (1) |
|
Analysis for Designs With More Than Two Blocking Variables |
|
|
220 | (7) |
|
Analysis When There Are Missing Data In Block Designs |
|
|
227 | (2) |
|
Analysis for Incomplete Block Designs |
|
|
229 | (3) |
|
|
232 | (3) |
|
Best Alternatives for More Than One Variable: Analysis for Factorial Designs |
|
|
|
|
235 | (1) |
|
Analysis of General Factorial Designs |
|
|
236 | (10) |
|
Analysis for Factorial Designs With Two Alternatives Per Factor |
|
|
246 | (23) |
|
Analysis for Factorial Designs Without Replication |
|
|
269 | (11) |
|
|
280 | (6) |
|
Analysis of Factorial Designs In Real Se Experiments |
|
|
286 | (3) |
|
|
289 | (4) |
|
Experiments With Incomparable Factor Alternatives: Analysis for Nested Designs |
|
|
|
|
293 | (1) |
|
Identification of the Mathematical Model |
|
|
294 | (1) |
|
|
294 | (1) |
|
Calculation of the Variation In the Response Variable Due To Factors and Error |
|
|
295 | (1) |
|
Statistical Significance of the Variation In The Response Variable |
|
|
296 | (1) |
|
|
297 | (2) |
|
Fewer Experiments: Analysis for Fractional Factorial Designs |
|
|
|
|
299 | (1) |
|
Choosing the Experiments In A 2k-p Fractional Factorial Design |
|
|
300 | (5) |
|
Analysis for 2k-p Designs |
|
|
305 | (5) |
|
|
310 | (3) |
|
Several Desired and Undesired Variations: Analysis for Factorial Block Designs |
|
|
|
|
313 | (1) |
|
Identification of the Mathematical Model |
|
|
314 | (2) |
|
Calculation of Response Variable Variability |
|
|
316 | (1) |
|
Statistical Significance of the Variation In the Response Variable |
|
|
317 | (3) |
|
Analysis of Factorial Block Designs In Real Se Experiments |
|
|
320 | (1) |
|
|
321 | (2) |
|
Non-Parametric Analysis Methods |
|
|
|
|
323 | (1) |
|
Non-Parametric Methods Applicable To Independent Samples |
|
|
324 | (4) |
|
Non-Parametric Methods Applicable To Related Samples |
|
|
328 | (2) |
|
Non-Parametric Analysis In Real Se Experiments |
|
|
330 | (4) |
|
|
334 | (3) |
|
How Many Times Should An Experiment Be Replicated? |
|
|
|
|
337 | (1) |
|
Importance of the Number of Replications In Experimentation |
|
|
338 | (1) |
|
The Value of the Means of the Alternatives To Be Used To Reject H0 Is Known |
|
|
338 | (3) |
|
The Value of the Difference Between Two Means of the Alternatives To Be Used To Reject H0 Is Known |
|
|
341 | (1) |
|
The Percentage Value To Be Exceeded By the Standard Deviation To Be Used To Reject H0 Is Known |
|
|
342 | (1) |
|
The Difference Between the Means of the Alternatives To Be Used To Reject H0 Is Known for More Than One Factor |
|
|
343 | (2) |
|
|
345 | (4) |
PART IV: CONCLUSIONS |
|
|
Some Recommendations On Experimenting |
|
|
|
|
349 | (1) |
|
Precautions To Be Taken Into Account In Se Experiments |
|
|
349 | (5) |
|
A Guide To Documenting Experimentation |
|
|
354 | (5) |
REFERENCES |
|
359 | |
ANNEXES |
|
|
Some Software Project Variables |
|
|
367 | |
|
Some Useful Latin Squares and How They Are Used To Build Greco-Latin and Hyper-Greco-Latin Squares |
|
|
379 | |
|
|
385 | |