Ph.D. Student SeminarPh.D. Status Report
November 19, 2003
Software Engineering group, IDI, NTNU
Ph.D. Status Report
Name Surname
Thomas Østerlie
Supervisors
Letizia Jaccheri, Eric Monteiro, Reidar Conradi
Year of Ph.D.
First year
Title of Ph.D.
‘The role of organizations in software process improvement’
Description of Ph.D. work
Motivation/background
A recent study on software process improvement (abbrev. SPI) concludes that “technical andorganizational problems must be jointly resolved toimprove the software process”(Dybå 2002).A survey of SPI initiativescalls for increased flexibility “in choosing andapplying SPI frameworks, learning from organizationalscientists”(Conradi and Fugetta 2003).There seems to be areciprocal relationship between organizational issuesand technology in the shaping of the softwaredevelopment process.However: what is the relationshipbetween organization and technology?
My preliminary research theme is the evolution of the software development process, with basis in a working hypothesis that the development process evolves as the product of an interplay between work practices, technology, and organizational issues.
With basis in previous work, I want to study the subject using the same approach as science studies have applied experimental science. This, however, is an untraditional approach to the young field of empirical software engineering. It is therefore, in an introductory stage, important to bridge the gap between science studies and empirical software engineering. While the differences are great, there are striking similarities between the object of study within the two traditions. Both are concerned with a practice, and both employ a micro level approach to the matter. Where empirical software engineering studies concrete techniques, like code reading (Latienberger et al. 1997) and pattern use (Vokac et al. 2003), science studies are interested in the interplay between experimental researchers and laboratory instruments (Latour 1987).
However, the biggest difference lies in the basic epistemology. This semester I have worked on trying to fit the epistemology of science studies with studies within software engineering, of which I try to present the preliminary results in this status report. What I attempt is to develop the theoretical basis for my future work, and argue for its relevance in doing studies within software engineering.
Research theme/focus
Hypotheses play an important role in empirical software engineering. The role of hypothesis is the topic of this part of the status report, as it plays a rather less prominent role in the way science studies approach their field studies. It is therefore important to explain the differences, and the implications of this on planning my data collection. This part of the status report is therefore organized as follows:
I will lay out some of the presupposition for experimental work, and the implications this has on the researcher's role in scientific work.This will be contrasted with science studies understanding of the experiment. Science studies play a dual role, as its ethnographic method also provides a possibility doing studies in software engineering. By giving science studies this dual role, the intention is to show why the ethnographic mode of enquiry relies on less on prior hypothesis than the experimental mode of enquiry.Instead, by showing that the experiment and the ethnography are diametrically different modes of enquiry and thereby giving the scientist radically different roles in the enquiry, the aim is to show the critical importance of prior hypothesis in experimental work and the critical importance of not having traditional hypothesis but broader research questions in doing ethnographic work.
1. The experiment
Empirical software engineering focuses predominantlyon experimentation. Empirical research methodologies as that proposed by Wohlin et al. (2000) and guidelines as those proposed by Kitchenham et al. (2000), though acknowledging other modes of empirical enquiry, primarily focuses on the experimental mode of enquiry.The field's vocabulary is tailored to planning, preparing, conducting and analysing experiments.In doing so, it enforces the presuppositions of the experimental method of enquiry.
The experiment is not simply about collecting facts, “but relevant facts” (Chalmers 1999, p.27).“When experimenting, subjects are assigned to different treatments” (Wohlin et al. 2000, p.9) measuring and recording the effect.Facts are produced inductively through analysing the recorded effects.The experiment can therefore be understood as a focused mode of enquiry, where facts are sought through controlling the experiment setting's variables.
The epistemology underpinning this approach is now presented, to argue the close relation between the experiment and the hypothesis.
1.1 The epistemology of experiments
Karl Popper, disenchanted with the idea of science being special because it is derived from facts, presented an epistemology based on falsification.Popper was especially disenchanted by the way certain sciences, like Marxism and Freudianism for instance, were able to explain any fact.It seemed to him that although formulated as powerful scientific theories confirmed by a whole range of scientific facts, they could in fact explain nothing because they could rule out nothing.A scientific theory can never be proven right, only proven wrong.There might always be undiscovered facts that can disprove a theory.That the facts have yet to be discovered, does not mean the theory is proven.A scientific theory in Popperian epistemology, is a natural law that can both explain and predict.
Popper scientific ideal was therefore of scientific theories that can only be verified through falsification.New theories must therefore be put through rigorous and ruthless testing.Theories that fail to stand up to the testing, must be discarded and replaced by improved theories.Through this kind of trial and error, science progresses towards a theory that stands up to the tests and is therefore deemed more likelyto be true because no theory can ever be truly verified.Also, it says nothing about how new theories are found.This is the scientist's role, to produce the relevant theories and questions in an attempt to falsify these theories.
Popper's view of science is tightly connected with the experimental scientific tradition.His epistemology is reflected in the way researchers conduct experiments.The experiment must be approached with a specific theory to be tested.The hypothesis is the focal point for the entire experiment, laying the foundations for the scientist's choice of variables, treatments and instruments.The theory is the hypothesis to be tested by the experiment.To falsify the theory, a null hypothesis is formulated.The null hypothesis assumes the diametrically opposite position of the hypothesis.The hypothesis is strengthened by falsifying the null hypothesis.The hypothesis can in no way said to be verified this way, but the more null hypothesis the scientist can falsify the more likely is the hypothesis.
2. Science studies and experiments
In “How experiments end”, Peter Gallison, himself an experimental physicist, focuses on the construction of experimental facts into theories.He represents an epistemology that is radically different from Popper's. Instead of understanding experimentation as a process of strengthening a theory through falsification, he understands scientific work as “how arguments emerge from the modern physical laboratory”(ibid, p. ix), about the construction of scientific facts.He researches how experiments end.The central question is: when do scientist consider a theory sufficiently proven, why, and how do they go about in proving it?
Gallison is a proponent of science studies' constructivist epistemology. Instead of viewing science as the exploration of the truth out there, the constructivists argue that scientific facts are constructed in the laboratory by scientists and their apparatus. Instead of trying to understand how the world out there can be best verified, science studies take the scientists through experiments and instruments constructs an understanding of the world out there. By shifting from the facts out there to the facts among us, science studies looks at how the process of running experiments and the instruments used actually shapes the facts that are produced. Instead of the scientists discovering facts, the scientists construct a fact together with their instruments.
Science studies are an effort to understand science through focusing on the scientific experiment, on the practice of doing scientific work. The object of study is not the facts themselves, but rather how the scientists make sense of the world, their “world building activities” (Latour 2000, p.20). Instead of trying to understand how the world out there can be best verified, science studies take the scientists through experiments and instruments constructs an understanding of the world out there. By shifting from the facts out there to the facts among us, science studies looks at how the process of running experiments and the instruments used actually shapes the facts that are produced. Instead of the scientists discovering facts, the scientists construct facts together with their instruments.
In science studies, the scientist's role is to understand how the experimental scientists go about constructing sufficient evidence to corroborate a hypothesis. The experimental scientists' world view is the object of study. In this way, the experimental scientists are actually those best to explain their world view (read: ontology).
The important move here is towards a study of the specific to say something about the general; the specific being the scientific practice of experimentation, and the general science at large. Science studies are a strictly empirical approach, recognizing that the only thing we as empirical scientists can say anything about is limited to our observations of things and people. To be able to study how people make sense of the world, the researcher must immerse himself in their ontology. This means that the researcher must think in the categories that make up the ontology, and thereby the world constructed by the scientists under study. The analyst becomes parts of what he analyses, and those being analysed become part of the analyser. This is not simply a rhetorical move, but rather a way of applying the constructivist epistemology onto the studier not simply the studied. If the experimental scientists are constructing facts about their object of study, the person studying the experimental scientists is in turn constructing facts about his object of study.
2.2 Implications for the role of hypothesis
That the scientists construct facts together with their instruments has repercussions for the way science studies perform their enquiries.
Where the hypothesis and the null hypothesis are the critical components in the experimental mode of enquiry, the scientists' role is to come up with hypothesis and to design an experiment to falsify this hypothesis. Science studies ethnography, on the other hand, is a mode of enquiry seeking a broad empirical foundation. Where the experiment is about not only collecting facts, ”but relevant facts” (Chalmers 1999, p.27), the ethnographic enquiry sorts out the relevance of facts after they have been gathered. Experimental science has its roots in the natural sciences, where the researcher's role is to pose the questions. Ethnography has its roots in the cultural sciences, where the researcher interacts with other people. His role is therefore different. Where the natural scientist has to interpret facts produced by nature, the ethnographer interprets facts produced by people together with people. Where the natural scientist must produce relevant questions himself, the ethnographer produces relevant questions in interfacing with his object of study.
In science studies, the scientist's role is to understand how the experimental scientists go about constructing sufficient evidence to corroborate a hypothesis. The experimental scientists' world view is the object of study. In this way, the experimentalscientists are actually those best to explain their world view (read:ontology). All the science study researcher can do is to besensitive to his subjects' descriptions. Entering the field with the intent to use his own classifications, will blind the ethnographer for the classifications of the ontology he enters. However, it has to be said, that this is not a naïve approach upholding the belief that ethnographers may approach their object of study as a tabula rasa. The first analysis of field data will be coloured by the ethnographer's own classifications (Neumann and Jørgensen 2003). These form the initial foundations upon which the enquiry must find place. By immersing in the ontology, the ethnographer will let go of some of his prior categories.
2.1 Relevance to a study in software engineering
What has this to do with software engineering?
With the constructivist approach a study within software engineering concerns itself with the world building activities of those developing software systems. Instead of studying how to identify system requirements, the constructivist approach could look at the micro-mechanics of how requirements are constructed by stakeholders. Instead of studying how to improve the software development process, the constructivist approach could look at the micro-mechanics of how development processes are constructed over time. The focus is all the time on the practice, on studying the interaction between people and things. Instead of asking how organizational factors influence the software development process, the question can be posed as: how has the concept of organizational factors influenced the construction of the software development process?
Open issues
Other issues that need to be addressed in bridging science studies and empirical software engineering are the role of theories, how to conceptualize practice and data analysis. A larger issue is the repercussions of entering the field with a priori categories that are applied to the field data (for more, refer to Foucault 1978). These issues will be the focus for the rest of this semester and well into next semester.
Research design
To be done.
Preliminary results
None
Preliminary conclusion
None
Status
In this section you record your current status and projected plan.
Papers matrix
Title of the paper / Publication / FutureOpen source development: Simply scratching an itch? / IRIS26 conference
Unveiling distributed development in open source projects: The practices of aligning, wedging, and / Open source workshop, at First International Conference on Communities and Technology / Will be submitted for Management Science, special issue on open source
There is no such thing as open source. Or: what do we mean by open source? / Will be submitted to IRIS27 conference
From the community of practice to practice of the community: On being specific on practice / Will be submitted to CSCW journal spring 2004
Credit plan
Course title / TermDIF8601: Object oriented systems / Spring 2003
Empirical software engineering / Spring 2003
IT3605: Avanserte emner i systemutvikling / Fall 2003
KULT8850: Forskning og samfunn / Fall 2003
TDT8081: Programvarearkitektur / Spring 2004
IT-emner / Spring 2004
Qualitative Methods in Information Systems Research / Fall 2004
Credit status
I have passed both DIF8601 and ‘Empirical Software Engineering’. I am doing ‘Forskning og samfunn’ this semester. Because of the new Ph.D. rules, all other courses except ‘IT-emner’ will be removed from my credit plan if my application is passed.
Activity last 6 months
After having had my exams in May, I have attended two conferences: IRIS26 and Communities and technologies 2003. I have presented papers at both conferences. In June I also attended the PhD Days by the Information Systems group at Institute of Informatics in Oslo.
The past months have been in preparation of my field work. Since my PhD work is an extension of what I did for my master thesis, I have had a simpler start as I know the object of study quite well. Choice of case has therefore be fairly simple. Focus has instead been to expand upon the theoretical basis for my master thesis. Much of this semester has therefore been devoted to charting the field of science studies. I have now come to the point where I need to a) put my preliminary theoretical foundations in writing, and b) seriously consider starting my field work.
Activity next 6 months
Next semester will be devoted to writing up the results of this semester’s reading. I will also start my field studies in the small. This can be considered more of an exploratory study, to chart the field so to say.
1