Software engineering research and Computer Science research are currently in an immature state of understanding:
- there is not a first principle or universal theory that helps to decide beyond doubt what is right or wrong, what works and what not.
- still today many ideas from which originate most important advancements in Computer Science remain unproved conjectures as i.e. the Church–Turing conjecture, P ≠ NP problem, or the Brewer's conjecture
People, the high variability caused by the human factor, is heavily underestimated:
- The effectiveness of many software technologies depends heavily on the people who are using them [2].
- When researchers investigate programmer behavior, a messy area involving human factors, their methods must become ever more sophisticated. Above all, we must be aware of how much we can learn and where we must accept limits to our knowledge [2].
Substantial differences among projects limit the possibility to generalize the results of experiments
- Evidence has a context-specific component. On certain topics, there is little evidence that results from one project have been, or can be, generalized to others [2].
- Many experiments do not remotely resemble Professional Software Production as is in the IT Industry. Toy problems are used in artificial environments with inexperienced programmers, looks for example at the Basili and Selby study [2]. The exceptions exists and are rare [4][5].
In conclusion software engineering is
not mature enough for practice:
- David Budgen and Barbara Kitchenham ask: “Is evidence-based software engineering mature enough for practice and policy?” Their answer is “no, not yet”: the software engineering field needs to significantly restructure itself before we can show that results can be replicated in different projects [2][6].
- To date, we have not realized our dream of evidence that is elegant, statistically sound, and replicable [2].