Nereu F. Kock Jr. , Robert J. McQueen, John L. Scott
How can action research be made more rigorous? We discuss in this paper action research, positivism and some major criticisms of action research by positivists. We then examine issues relating the conduct of IS research in organisations through multiple iterations in the action research cycle proposed by Susman and Evered. We argue that the progress through iterations allows the researcher to gradually broaden the research scope and in consequence add generality to the research findings. A brief illustrative case is provided with a study on groupware introduction in a large civil engineering company. In the light of this illustrative case we contend that effective application of the iterative approach to action research has the potential to bring research rigour up closer to standards acceptable by positivists and yet preserve the elements that characterise action research as such.
KEYWORDS: Brazil, Action Research, Information Systems Research, Positivism, Research Rigour
An important theme in information systems (IS) research has been the study of processes related to the development of IS applications and the effects that IS applications have on people, particularly in formal settings such as organisations. Reviews of IS research literature have shown that one particular research perspective has been dominant in the field, the positivist perspective (Orlikowski and Baroudi, 1991).
Ever since its emergence as a distinct line of research, notably after the end of the second world war, action research (AR) has been a controversial subject - it has been both evangelised by AR practitioners and strongly criticised by those who defend positivist research approaches. Practitioners have presented AR as an alternative to overcome the limitations posed by positivism, often giving the impression that AR and positivism are contradictory research movements (Reason, 1993; 1994). This view has led researchers into feeling compelled to take one side or another, as regards their main research orientation, and form research communities where either the word "AR" or "positivism" is seen as associated with "inappropriate" forms of inquiry (Heller, 1993). This conflict has been prejudicial to the AR practice in the IS field, which has been almost banned from top IS research journals. This has been particularly noticed with respect to leading American journals were AR has been reported as amounting to less than 1% of all published IS research (Orlikowski and Baroudi, 1991, p. 4).
We define AR, in the following section, by contrasting it with three other major IS research approaches. An analysis of AR and positivism then follows. In this analysis a brief historical review of these two research movements is provided, and common themes in the critique of AR by positivists are highlighted by focusing on the application of AR to IS research in organisations. A review of three main criticisms of AR by positivists follows this analysis. The review is conducted in the light of an illustrative AR project in a Brazilian organisation. We try to show that if AR is performed through iterations in the AR cycle, and that if those iterations search for an expansion in the scope of the research study, those main criticisms from positivists may no longer apply to AR. The main goal of this paper is to show that IS AR, if done in cycles and following some simple guidelines, has the potential to be conducted in ways that render it acceptable as a "rigorous" research approach from a positivist perspective while retaining the attributes that characterise it as AR.
The goal of our analysis implies a recognition that some criticisms of AR from positivists can in fact be used to improve AR's practice. However, in order to be selective in our analysis towards what seems to be more relevant from an AR perspective, we focus on those criticisms that have been recognised as valid in seminal papers about AR and its practice, and which have been acknowledged in the IS research literature as characterising the dichotomy between positivist research approaches and AR. That is, we are particularly concerned in this paper with criticisms from positivists that have been seen by AR practitioners themselves as warranting attention.
The growing importance of IS research in the 1980s and 1990s has led to a number of different research approaches and methods, usually adapted from other disciplines such as sociology, natural sciences, and business studies. As a result of that, a number of different IS research taxonomies have been proposed (Galliers, 1984; 1992; Cash and Lawrence, 1989). Most of these taxonomies, however, fail to present truly independent categories of research. For example, the Harvard colloquium on qualitative IS research methods (Cash and Lawrence, 1989) proposes a very broad category - qualitative research - as a distinctive research approach. Tesch (1990), however, argues that what is often referred to as qualitative research is primarily an approach to the collection and analysis of data that can be used in a number of research approaches, also typically seen as distinct, such as case research and AR. To avoid falling into the same trap, but conscious of the limitations that this decision poses, we will intentionally adopt a narrow scope by considering only four general research categories to provide a simple basis for understanding AR's place within the IS research field. A brief description of these categories, highlighting major contrasting characteristics, is provided next:
Experimental research. In experimental research the researcher has a strong control over the environment being observed. This research approach has its roots in the scientific practice of biologists and physicians, where variables are manipulated over time, associated numeric data is collected, and causal or correlational models are tested through statistical analysis. Typical instances are Chidambaram and Jones' (1993) study of the impact of communication medium and computer support on teams in dispersed and face-to-face meetings by comparing experiments where some groups used a group decision support system and others did not; and Gallupe et al.'s (1994) comparative study where production blocking was manipulated in three experiments so the performance of blocked and unblocked electronic brainstorming and verbal brainstorming groups could be assessed.
Survey research. This research approach has its roots in the work of economists and sociologists. In survey research the researcher typically has a considerable sample to be analysed, which suggests the use of questionnaires with questions that are easy to be answered and that permit quantitative evaluation "a posteriori". Survey research is typically applied to validate models or hypotheses. Typical instances are the survey involving 49 organisations in Southern California performed by Winter (1993), which shows that computers can act as symbols of status; and Brynjolfsson and Hitt's (1993) survey based on firm-level data from 1987 to 1991 about 380 large firms, which evaluates those firms' return on investment in IS.
Case research. This research approach has its root in business studies. Cases are analysed either to build up or validate models or theories, typically through collection of textual data through interviews. Typical instances are the on-site case research performed in a large corporation by Alavi (1993) where she assessed the utilisation of an electronic meeting system by staff; and the interview-based research performed by Trauth and O'Connor (1991) to analyse the influences and impacts of cultural, economic, and political factors on the establishment and evolution of information technology firms in the Republic of Ireland.
Action research. The origins of this research approach rest on socio-psychological studies of social and worklife issues. AR is often uniquely identified by its dual goal of both improving the organisation participating in the research project, usually referred to as client organisation, and at the same time generating knowledge. Although typically applying very little, if any, control on the environment being studied, the AR practitioner is expected to apply intervention on this environment. Typical instances are the study of asynchronous groupware support effects on process improvement groups conducted in a service organisation by Kock and McQueen (1995); and the research on the participatory development and introduction of an expert system in a welding plant conducted by Candlin and Wright (1991).
From the comparison of these four major research approaches one main characteristic, and strength, of AR becomes clear: it suggests intervention carried out in a way that may be beneficial to the organisation participating in the research study. AR recognises that even casual observation affects a system and therefore takes this effect inside its scope (Lincoln and Guba, 1985). The other three approaches are based only on detached observation and analysis, completely disregarding the possibility of positive intervention from the researcher. Previous work suggests that this distinctive characteristic of AR leads to the development of a stronger linkage between organisations and research centres and to organisational development and improvement (Ledford and Susan, 1993a; Sommer, 1987). Nevertheless, AR has been the target of severe criticism from positivists, who typically view experimental and survey research as the only "valid" modes of scientific inquiry.
Although there is some controversy about the origins of AR, it has been considered a distinctive form of research since the early 1940s. Kurt Lewin is generally regarded as one of its pioneers (Checkland, 1981; Argyris, Putnam and Smith, 1985) and the first person to use the term "action research" to refer to a specific research approach in which the researcher generates new social knowledge about a social system, while at the same time attempts to change it (Lewin, 1946; Peters and Robinson, 1984). A distinctive thrust of AR has also developed after World War II in Great Britain at the Tavistock Institute of Human Relations in London as a ground-breaking method to deal with sociological and psychological disorders arising from prison camps and war battlefields (Rapoport, 1970; Fox, 1990).
Early published material suggests that AR grew from a desire of researchers to deal with important social problems. However, shortly after its early development in the late 1940s, AR begun to be used in large scale to deal with intra-organisational and worklife problems. Most of the AR practice in the second half of the 20th century has continued and expanded this organisational and worklife focus, and one of the major topics of AR has been the issue of "job satisfaction" and its dependence upon several aspects of work situations (Gustavsen, 1993).
One of the reasons for the emergence of AR and its consequent use in the IS field is the recognition that a social system can be more deeply understood if the researcher is part of the socio-technical system being studied, which can be achieved through applying positive intervention on the system. This involvement is also believed to foster cooperation between researcher and those who are being studied, information exchange, and commitment towards both research quality and organisational development. This situation was illustrated by Eric Trist (Fox, 1990) in his account of one of the earliest studies with the characteristics of AR, conducted in Austria by Paul Lazarsfeld and Marie Jahoda, a project on long-term unemployment in a textile village outside Vienna whose workers have become unemployed overnight. The researchers had found that in order to get access to people and relevant research information they had to clearly show that they were doing something in the villagers interest. In doing so they eventually changed those people's own view about the very system that was believed to have caused them to be unemployed, improved their relationship with management, and eventually led workers and management through a process of cooperative solution of mutual problems.
An analysis of positivism as a distinct line of thought is important because, as mentioned before, AR is constantly referred to as opposed to positivist research methods (Kock and Corner, 1996). In fact, the two main positivist tenets - the belief that there are universal and permanent laws or principles that represent unidirectional causal relationships, and the belief that there is only one real "scientific" method to unveil those relationships (Walker, 1993; Guba and Lincoln, 1994) - are openly questioned by AR practitioners who, for example, do not accept the endorsement of "appropriate" research methods.
Descartes' treatise "Discourse on Method", published early in the 17th century, had a strong influence in shaping positivist thought. The Cartesian work supported methods using natural science and mathematics as much as possible to understand nature. In doing so, Descartes argued, human beings would be able to understand the world in an unbiased way, unaffected by the imperfections of their sensorial organs. This is also seen as likely to have allowed Descartes to carry on with his scientific inquiry without incurring the problems of his predecessors with the Church (Hirschheim, 1985).
Positivism developed with little opposition until the first half of the 19th century, with prominent contributions from Francis Bacon, Galileo, Newton, Hobbes and later from August Comte, who stated unequivocally that the study of human phenomena should reflect methods of physical science (Teichman and Evans, 1995). In the second half of the 19th century anti-positivists entered the research scene suggesting that individuals do not exist in isolation, and that they therefore need to be understood as part of the cultural and social environment from which they are part.
Following this vigorous rise of anti-positivism, positivism strengthened again in the 1920s with Bertrand Russell's logical positivism and later with the hypothetical-deductive and units of analysis methods. But this wave was again strongly criticised by several social scientists, giving way to the emergence of new knowledge building paradigms and epistemologies such as post-positivism, critical theory and constructivism. Among the shared assumptions of these emerging paradigms and epistemologies is the belief that there is a plurality of methods and that each method has its validity determined by the specific situation in which it is applied and by the type of knowledge sought (Vidich and Lyman, 1994, Harvey and Myers, 1995).
A number of AR practitioners have tried to bridge the gap between positivism and AR by describing "classical" AR as a form of field experiment which is by nature "low" in control, and citing the early work of AR practitioners such as Kurt Lewin to support their claims (Elden and Chisholm, 1993). However, this view has been questioned by others who state that the approach proposed by controlled experiments, even if this control is minimal, is often inappropriate as unilateral control of some variables by the researcher would prevent conclusions based on a "natural" process. The AR purpose, they maintain, is not experimenting but rather discovering correlational and/or causal couplings between variables in situations where learning and change flow naturally from research interventions.
Sample surveys and controlled experiments are often pointed by positivists as the preferred types of research and inferential statistics the method to discover causal laws. However, even though survey research and controlled experiments are seen as providing a rigorous basis for the statements that are made, AR practitioners point out that these methods cut the researcher off from the discovery of non-deterministic and reciprocal relations in social systems (Jonsonn, 1991).
It becomes clear from the discussion about the origins of both AR and positivism that while positivism is a distinct line of thought - an "epistemology" in the jargon of Philosophy - AR is no more than a research approach built on intervention coupled with careful study and analysis. So, why is AR so often seen as opposed to positivism? We argue that a main reason is because AR is one of the few, reasonably successful, research approaches with a relatively widespread adoption in some fields (Ledford and Susan, 1993) and a growing adoption in the IS field, in which the two main positivist tenets (discussed in the previous section) are vigorously questioned. Case research, although seen by some as resembling AR, has been widely used in ways that endorse the main tenets of positivism (Yin, 1994; Doolin, 1995).
Although being described as a successful research approach for more than 50 years (Elden and Chisholm, 1993), there is still plenty being said against AR by positivists. One of the main criticisms of AR by positivists is that in looking at AR strategies in a historical perspective, the development of sound research procedures, techniques and methodologies has not been the main goal. Rather, AR has been preoccupied with the action itself and its influence on the settings in which research is done (Gustavsen, 1993). This often leads IS researchers into seeing AR projects as merely the application of information technology to solve practical problems and at best generating normative approaches and methodologies, rather than valid research knowledge obtained in a rigorous way. Three main possible AR weaknesses emerge from the discussion by Orlikowski and Baroudi (1991) about the clash between positivist and non-positivist assumptions, and therefore seem to require particular attention in the development of methodological tools for improving AR rigour from a positivist perspective:
Contingency of the research findings. While important links between variables can be unveiled in AR that might not by the use of deterministic and targeted approaches (e.g. survey research), AR is often seen as inappropriate to produce models with high external validity, i.e. that are valid outside the context of the AR project (Cook and Campbell, 1976; Berkowitz and Donnerstein, 1982). This is because most AR projects involve a small number of client organisations in in-depth and often longitudinal studies (Galliers, 1992), and very seldom generalisability assessments across a number of organisations or industries.
Low control of the environment. This lack of control is one of the main reasons for AR being seen as inappropriate to test or produce strong theories, or build up research models based on solid evidence. The influence of a particular variable might take too long to be isolated in AR studies testing or refining a causal model where the extent to which a dependent variable is influenced by a set of independent variables needs to be carefully examined (Jonsonn, 1991).
Personal over-involvement. The usual personal over-involvement of researchers with client organisations in AR projects may hinder good research by introducing personal biases in the conclusions (Francis, 1991). This is particularly true in situations involving a conflict of interests. With respect to this Galliers (1992) points out that AR "... places a considerable responsibility on the researcher when objectives are at odds with other groupings." (p. 152).
Other alleged weaknesses have been discussed by Rapoport (1970). One of these is AR's typical unplanned and informal structure. The ad-hoc approach of AR, where most of the study is done in cycles with temporary reports, methodologies and frameworks, may be considered as lacking scientific discipline and consequently regarded of low academic interest . Another alleged weakness is AR's interference with the research environment that, while potentially beneficial to the client organisation, may bias research findings in ways that are difficult to be identified, and make them difficult to be replicated by other researchers in different settings. A third alleged weakness is the lengthy time required to conduct quality AR projects, which may not be acceptable by the research's sponsor or client. Two principles, usually followed in case research to avoid this are to perform a careful preliminary preparation of the research, and seek guidance from a structured methodology (Yin, 1989). These two principles, however, are still not commonly practised in AR.
AR, in contrast to positivist research approaches, tries to bridge the gap between scholars and practitioners. While typically scholars are preoccupied with philosophy and general theories, practitioners are more concerned with problem solving and bottom-line techniques. Sommer (1994) states that society is the victim of this dichotomy as research outcomes often end up forgotten on some dusty shelf without any practical application other than support for further theoretical research. This point is supported by Jonsonn (1991), who also maintains that the contribution to theory is not affected in AR because the study is done much more deeply as the researcher has an inner involvement with the environment. AR is seen as adding texture to theoretical notions and food for theoretical speculation, and a way of dealing with complexity in the presence of oversimplified and primitive theory. This seemingly romantic view of AR, however, apparently has done little to draw support from positivists, who often see AR benefits as short of balancing its excessively "soft" and imprecise nature.
In positivist research the outcome is confirmation or rejection of knowledge stated in the form of hypotheses or a model to be tested, while the main contribution sought by AR is to build up or enhance an existing model or theory by selective intervention. While sometimes referred to as "rigorous", the AR method is not perceived by positivists as "truly" objective, since it implies researcher-initiated modifications in a system that changes while being observed. Yet being a source of criticism from positivists who would rather observe the system from an "outsider" point of view, it does not mean that method is in AR a meaningless standard. By analogy Ledford and Susan (1993) point to the physics of subatomic particles, where the "subjects" (particles) react to attempts to measure them, the source of the famous Heisenberg principle. Yet, physicists continue to make progress in this field without abandoning former methodological standards.
One particular trait of AR that enjoys a particular prominence in the work by Susman and Evered (1978) is its cyclic nature; the notion that quality AR is conducted in cycles involving a succession of intervention and reflection stages. Although most AR projects either claim or imply that this trait is present, respective reports often hide the existence of cycles and the learning process that went on across cyclic iterations. For example, none of the AR projects described in the 1993 special issue on AR of the Human Relations journal (Brown, 1993; Engelstad and Gustavsen, 1993; Greenwood et al., 1993; Ledford and Mohrman, 1993; Levin, 1993) was described as a set of iterations in the AR cycle, even though all of them were described as sets of somewhat disconnected stages in which, with the exception of Ledford and Mohrman's study, there was no systematic collection of data to support study findings.
In contrast to the reports in the 1993 Human Relations special issue, we rely on the cyclic nature of AR to show that it can achieve the level of internal consistency and validity that positivists argue it cannot because of its (AR) own nature. In order to do so we briefly describe in the next section the process of building up model generality in AR. This is illustrated with a study of groupware support effects on a large civil engineering organisation. This illustration emphasises the learning process that goes on across successive iterations in the AR cycle. We then, later on in the paper, revisit the issue of "positivism versus AR" in the light of this illustrative example.
Susman and Evered (1978) view a general AR project as a cyclical process carried out through what these authors refer to as the AR cycle, comprising five stages: diagnosing, action planning, action taking, evaluating, and specifying learning. The diagnosing stage involves the identification and definition of an improvement opportunity or a general problem to be solved in the client organisation. The following stage, action planning, involves the consideration of alternative courses of action to attain the improvement or solve the problem identified. The action taking stage involves the selection and realisation of one of the courses of action considered in the previous stage. The evaluating stage involves the study of the outcomes of the selected course of action. Finally, the specifying learning stage involves the study of the outcomes of the evaluating stage and, based on this study, knowledge building in the form of a model describing the situation under study. Initially this model is expected to be only descriptive, rather than predictive, since the deep involvement of the researcher with the environment being studied leads, due to time constraints, to the study of a small number of instances of particular events. However, as the number of AR studies carried out on a similar topic grows, their resulting descriptive models can then be integrated into more general and predictive models, and eventually lead to "grand theories" (Strauss and Corbin, 1990).
The classical non-participatory approach to AR usually prescribes that all stages but one, the specifying learning stage, be carried out in cooperation with the client organisation (see Figure 1). More contemporary approaches to AR, such as participatory AR or PAR, strive for the full involvement of the client organisation in the specifying learning stage as well (Elden and Chisholm, 1993).
Figure 1: The AR Cycle
One of the reasons why AR is seen as preferably carried out in cycles is the opportunity that it allows for strengthening research findings by building on evidence gathered from previous iterations in the AR cycle. Ketchum and Trist (1992) see the frequency of the iterations in the AR cycle as likely to decrease as the match improves between the researcher's conception of the socio-technical system, expressed in the model comprising research findings, and that found as a result of the specifying learning stage in each cycle. This can be obtained by expanding the research scope, e.g. the areas of the client organisation involved in the research, and building up the generality of the results through the identification of invariable patterns.
This point is illustrated in Figure 2, which depicts the relationship between research scope and the generality of the model describing research findings. The rectangles in the cycles represent each of the AR cycle stages, where: "di" represents diagnosis, "ap" represents action planning, "at" represents action taking, "ev" represents evaluating, and "sl" represents specifying learning. The iterations are named cycle 1, cycle 2, ...to cycle "n", where "n" is the total number of iterations in the AR project.
Figure 2: Relationship between research scope and model generality
The point that the validity of AR findings resulting from multiple iterations in the AR cycle can benefit from an expansion of the research scope across iterations is also illustrated by the case described in the next section, which shows how successive iterations in the AR cycle were carried out in a state owned civil engineering company in Brazil.
In the early 1990s the Rio de Janeiro State Construction Company (EMOP) became responsible for the coordination of a project aimed at building 500 Integrated Public Education Centres (CIEPs). CIEPs provide cheap and locally accessible education for poor regions in the State of Rio de Janeiro. The responsibility for the CIEPs Project had to be carried out concurrently with EMOP's normal activities, aimed at supplying building maintenance services to most of the buildings belonging to the State of Rio de Janeiro, including public hospitals, primary and high schools, and historical buildings.
Most of EMOP's activities involved a heavy amount of coordination of third party activities. Every time a building maintenance or construction was required, four main tasks had to be successively carried out by EMOP: (1) Generate a time schedule for the project describing main activities and respective resources and deadlines; (2) Calculate a budget for the project; (3) Conduct a public bid with contractors and contract the best offer; and (4) Supervise the work of the contractor.
In order to carry out those four main activities a huge amount of data had to be processed against several computer databases on a daily basis. This was accomplished through several applications running on a central mainframe. The operation of these applications along with the control and service of requests from several other departments was fulfilled by EMOP's Central Processing Department (CPD). This concentration of activities created a bottleneck in the CPD. Several urgent requests used to be kept pending for several days, causing dissatisfaction from the primary users of the information.
To the problem of the centralisation of information in the CPD was added the cost of maintenance of the mainframe. As a strategy to force EMOP to upgrade to a new mainframe its vendor increased support and maintenance costs, and informed EMOP that the mainframe model in use was being taken out of its product line. The high cost of a new mainframe along with the need to decentralise the applications running on the old one led EMOP's management into considering downsizing all the applications to a recently installed local area network (LAN).
The researchers came to know EMOP by means of an international consulting company hired by EMOP's president to coordinate a major business process re-engineering effort in the organisation. The researchers were invited to join the consulting team as senior consultants. This effort aimed at streamlining EMOP's core processes so that it could cope with an increasing number of projects without having to expand its permanent staff basis.
The researchers were interested in studying the effect of groupware systems on project organisations, i.e. those whose main core processes involved the planning and coordination of projects. From this perspective EMOP was considered an appropriate site. The researchers decided to carry out the research study through several iterations in the AR cycle, concurrently with their consulting work.
A general model, stated as a set of hypotheses related to the general thesis that "groupware systems would positively affect productivity and quality of project-related activities", was developed as an initial research framework on which to build subsequent findings. This model was based on the practical experience of the researchers as well as on literature review.
Diagnosing. The consulting company, the researchers and EMOP's president formed an IS restructuring group (ISRG) which, after some meetings, identified two major problems with the current IS strategy of the company: (1) The centralisation of the applications in the CPD, because it was affecting productivity as well as being perceived as a strong source of dissatisfaction with the role played by information technology in the organisation; and (2) The concentration of core applications in the mainframe, because of high maintenance costs and the low quality of the support provided by the vendor.
Action planning. According to a major applications' downsizing plan, which would eventually free the organisation from depending on applications running on the mainframe, the ISRG agreed on that the downsizing process should start with some applications used by an area of the organisation called Prices Division. Approximately 20 employees would be affected by this pilot project. A group formed by the researchers, some of the consultants and some of the members of the CPD would be responsible for the project, which would comprise the installation at Prices Division of a number of LAN applications, such as a distributed database management system, a word processor, a spreadsheet and some groupware systems. The prospective groupware systems comprised an e-mail system with remote connection features and a distributed synchronous conferencing system. This plan was perceived by the researchers as an appropriate initial step towards refining their basic assumptions about the effects of groupware systems on project organisations.
Action taking. Some of the applications running on the mainframe and used by Prices Division were converted to the LAN using an automatic code converter. A distributed database management system, a word processor, and a spreadsheet were installed. An e-mail system and a synchronous conferencing system were also installed to provide a link between CPD and Prices Division.
Evaluating. Most of the new software systems installed presented problems, including the downsized applications. Participant observation and interviews showed that those failures reduced staff confidence and motivation towards the change process. Also, due to the lack of job-related training, it took some time for the users to learn how to use relatively complex new tools such as the database system and the spreadsheet. This was balanced by the success of the e-mail system, which in a short time have secured a critical mass of users for formal and informal communication. These applications also improved the communication and increased data exchange between the CPD and the Prices Division. The synchronous conferencing system, on the other hand, initially generated excitement but was after a while largely "ignored" by the users. One of the main reasons was that users typically saw unscheduled invitations to participate in synchronous computer-mediated discussions as "disruptive", perception that was either communicated to the callers explicitly or tacitly (e.g. by answering the phone but not computer calls).
Specifying learning. This first iteration in the AR cycle refined and, to a certain extent, reinforced some of the researcher's hypotheses, such as that: (1) The introduction of an e-mail system can facilitate the introduction of other software applications; (2) An e-mail system linking two different areas can foster the reduction of communication barriers; (3) E-mail systems can improve the productivity of activities that involve information exchange; and (4) Asynchronous groupware applications are likely to be perceived by users in general as less disruptive than synchronous groupware applications.
Diagnosing. The experience at Prices Division lead the president and the board of directors to decide installing workstations and the new software applications for their own personal use. There were seven directors altogether. This direct contact of the powerful board of directors with the technology and its effects was seen as useful to guarantee future support to the work of the consultants and researchers. However, the researchers were convinced that the use of technology should be directed to effectively solving problems faced by the directors rather than simply giving them access to general applications, since the success or failure of this endeavour would probably be associated with the effectiveness of the whole restructuring effort. The use of the groupware systems was seen primarily as a solution to some of the communication problems faced by the board of directors, due to the fact that they spent most of their working time outside the company's premises. This fact was seen by the directors as a cause for misunderstandings and decision delays. The groupware systems were seen as having the potential to allow them to communicate more with each other and speed up decisions. This new iteration in the AR cycle was consistent with the research goals and likely to provide insights on how a new group of users, the company's top management, would be affected by the use of groupware systems.
Action planning. It was decided that the first among the new software applications to be installed for the directors and the president would be the groupware systems, followed by the other applications. It was also decided that some of the downsized applications would initially be made available to the directors and president for read-only purposes.
Action taking. The workstations and software were installed. Each of the new users was trained individually by the researchers to allow for direct observation and unstructured interviews, as well as to use the short available time of the directors to learn how to use the new software applications as effectively as possible .
Evaluating. Among the new software applications installed the only regularly used by the president and board of directors was the e-mail system, for pre and post-meeting discussions. In unstructured interviews these users noted that the e-mail system made meetings much more efficient, and that errors and misunderstandings were decreased as a result of the availability of this new communication channel. Some of the directors asked close managers and assistants to operate downsized applications and provide them with information of higher level than that actually supplied by the applications. When interviewed, the directors and president declared that the data provided by some of the downsized applications was too "raw" and of little use to them without some refinement. They also declared that they did not use the synchronous conferencing system, being the main reason that they did not like to interrupt the work of their peer directors to talk electronically, nor be interrupted themselves during the few hours that they spent per day at EMOP's premises. They would rather send each other e-mail messages, which were perceived as less intrusive than an invitation to participate in a synchronous conference or a phone call.
Specifying learning. This second iteration in the AR cycle suggested some new hypotheses, such as that: (1) E-mail systems can improve the quality of senior management staff meetings by supporting pre and post-meeting discussion; and (2) E-mail systems can improve the efficiency of senior management staff communication. This iteration also provided support to the hypothesis that "asynchronous groupware systems are perceived by senior executives as less intrusive than synchronous groupware systems" and pointed to a possible generalisability of this hypothesis, based on the analysis of the previous iteration, for routine business discussions involving users in general. This iteration also reinforced a previous belief of the researchers that distributed database applications are generally of little value to high management staff, unless they incorporate user-friendly decision support features.
There were two more subsequent iterations which followed the same structure as the two previous iterations. The four iterations lasted approximately one year altogether. The third iteration involved the installation of a new software application, a distributed project scheduler customised to support group editing and storage of information about all projects coordinated by EMOP, initially made available only to EMOP's Planning Department. The fourth iteration in the AR cycle disseminated all the software applications, previously introduced only locally, throughout the organisation. These subsequent iterations generated new hypotheses, reinforced former ones, and also provided ground for refutation of some previous hypotheses. The new resulting set of hypotheses was seen as a tentative model of the effects of groupware on project companies. These hypotheses were later embodied as features in the design of an asynchronous groupware tool to support project companies (Kock, 1994).
How can the cyclical IS AR approach just illustrated help overcome the limitations often ascribed by positivists to AR as a valid and rigorous research approach? We now revisit three of what we view as among the main alleged AR weaknesses discussed in the section "Positivism versus AR", which also are opposed to the main characteristics of "appropriate" positivist IS research (Orlikowski and Baroudi, 1991, p. 10): contingency of the research findings, low control of the environment, personal over-involvement. This is done in the light of the illustrative example of AR project conducted at EMOP to provide a general answer for the question posed. Our goal is not to provide a justification for the use of AR to be tolerated by positivists, but rather to show that the cyclical approach to AR possesses implicit strengths that may render some of the criticism from positivists unfounded.
Research rigour is often seen as linked to the reliability of the instruments for data collection and analysis used in the research, and to the internal and external validity of the research findings. In addition, in most situations research rigour is ultimately aimed at increasing validity, particularly external validity, and research instrument reliability tests are often seen as just a means to increasing the validity of the final research findings (Cook and Campbell, 1976; Berkowitz and Donnerstein, 1982). To say that research findings are "highly contingent" is the same as saying that the findings have a low "external validity".
Internal validity is a measure of the internal consistency of the research findings and is not necessarily linked to external validity, which is a measure of the generality of the findings regarding situations other than the one studied. A high internal consistency of the findings is not always likely to increase their generalisability - e.g. although the research findings regarding IS effects on salespeople productivity in a car rental chain may be shown to be internally consistent, their replication in a bank chain or even in a different car rental chain may be uncertain. The threats to external validity in AR are often seen as caused by the focus of AR on in-depth study of a small number of socio-technical systems, e.g. one to three organisations.
Even though it involved only one organisation, our AR study at EMOP led to a high confidence about some particular research findings, and their likely replication in other organisations whether they are project organisations or not. One of these findings was that synchronous groupware systems are perceived as considerably more intrusive than asynchronous groupware systems, and that this characteristic would render them useless as a support tool for routine business meetings. While apparently trivial today, this research finding was counterintuitive when the research project was conducted (early 1990s), a time when software vendors and researchers widely advertised synchronous computer conferencing as likely to totally replace executive meetings in the years to come due to obvious gains in meeting productivity (Coleman, 1992).
The main reason for our confidence as to the high external validity of some of our research findings was the observation that they held across iterations. According to the cyclical approach to AR described, the successive move across iterations should strive to expand the research scope, which would lead to an increase in the generality of the findings. This effect is analogous to the one caused by choosing a wider sample population in statistical studies, although the "sample" is always likely to be considerably smaller in AR. The extra advantage of this approach to AR is that the experience acquired from previous iterations can help the researcher improve the effectiveness of the intervention in the client organisation in further iterations, and allow for the identification of not previously identified "relevant" units of analysis early on in the study - which is seldom possible in "one-shot" research projects.
Our experience indicates, however, that the key point to effectively use successive iterations in the AR cycle to improve research findings generality seems to be an appropriate design of the research intervention so successive iterations allow for the collection of data about the same units of analysis in relatively independent settings. In our study, for example, research scope was widened by bringing in new organisation departments into the research project. If we consider the organisational department as a unit of analysis of our research, we can say that the finding that "the introduction of an e-mail system can facilitate the introduction of other software applications" has held true for over a dozen instances of this unit of analysis even though only "one" organisation was studied. Since the "department" is one of the basic structural units of most organisations, we can even say that the internal consistency of our findings regarding this unit of analysis considerably increased the likelihood that these findings will hold true for other organisations as well.
It is undeniable that a low degree of control over variables of the socio-technical system being studied can hamper the test of causal links between these variables. Testing links between variables, however, requires both variables and links to be clearly stated before the research project starts. This is in turn likely to limit research findings by focusing the research on a limited set of variables and leaving out others that might be relevant for the understanding of the events under consideration. In some cases this may lead to thorough studies about issues that are irrelevant from an organisational perspective, while in others it may lead to the reporting of technology effects that do not hold in practical organisational situations because they are offset by other effects that were disregarded in the original study. This may be one of the reasons for the large number of contradictory effects reported in the empirical research literature on groupware systems, which has been based predominantly on experimental research studies (DeSanctis et al., 1993).
A high control over the environment is also likely to lead those involved to behave in an artificial way and thus irreversibly bias research results. This is one of the reasons why a number of methods commonly used in the natural sciences are rendered useless in social research and thus in IS research, which as well as management research is a form of social research. As Mintzberg (1979) and others pointed out, human behaviour is not as predictable as the behaviour of rats, and therefore many of the research findings based on statistical analysis of simple cause-and-effect relationships that naively disregarded this fact have been either misleading or irrelevant.
Given the problems above one can say that the low control over the environment being studied, characteristic of most AR projects, is more of an advantage than a disadvantage in the generation of relevant and valid knowledge. While comforting, this belief does not make any easier for AR practitioners the task of showing that research findings clearly follow from the evidence at hand. The low control of variables, which prevents manipulation to generate highly focused data, leads researchers into collecting a large amount of data about "everything" to avoid missing important evidence. The problem is that this data is often so sparse that nothing but anecdotal evidence can be provided as support for research findings. While this type of evidence has some value in itself, it is unacceptable as the "only" basis on which to frame a rigorous research argument.
We believe that the cyclical approach to AR goes some way towards meeting the demands for rigour that make AR appear to be a suspicious research approach when compared to controlled laboratory or field experiments. However, it is precisely the collection of data about the same unit of analysis across iterations, as briefly discussed in the previous section, that makes it possible. Not only does it allow for the collection of data from different sources about the same variables and events, a desirable form of triangulation in research data collection (Jick, 1979), the study of the same units of analysis across iterations also allows for comparison of IS effects in a longitudinal manner and thus the avoidance of cross-sectional biases (Galliers, 1992). A cross-sectional study is performed at a specific time, or within a very short time span, which is seen as one of the main limitations of this approach because common seasonal phenomena such as a high-season increase in departmental productivity, for example, can be seen as caused by disconnected variables that also vary seasonally, particular when the time span of seasonal variations is considerably longer than that of the whole research study. This can be avoided in cyclical AR projects because successive iterations allow not only for data collection at different times (e.g. different periods of the year), but also for long-term assessment of effects observed in early iterations.
While personal over-involvement from the part of the researcher is likely to bias research results, it is inherent in AR because it is impossible for a researcher to both be in a detached position and exert positive intervention on the socio-technical system being studied. This is particularly true when the number of situations experienced by the researcher is small and the intensity of this involvement is high. Research on human cognition has shown not only that human beings rely mostly on experiential learning for the acquisition of knowledge, but also that those experiences that are accompanied by intense emotional discharges (e.g. anger, fear) are remembered more vividly than those in which there is little emotion involved (Gioia and Sims, 1986). This phenomenon, which is one of the most general research findings in the field of human cognition, is likely to be a result of the evolutionary process we have gone through as a species, in which emotion is used to strengthen the memory connections to events related to survival.
Unfortunately, the phenomenon just described has its down side as it is also likely to distort the way in which people in general, and AR practitioners in particular, may perceive events and situations where there is a high degree of personal involvement, especially when these situations involve conflict, stress, or any events that may lead to an intense emotional response. Our involvement in the AR project carried out at EMOP indicates that emotion-ridden events are likely to be experienced in most AR projects, particularly because AR interventions foster change, and change is always met with resistance and apathy by some, and support and enthusiasm by others. The clash between those who believe that the status quo in the client organisation should be maintained and the change enthusiasts is likely to catch AR practitioners right in the middle.
The main benefit likely to accrue to AR practitioners as a result of successive iterations in the AR cycle is that disconfirmatory evidence in further iterations may help correct distortions in the findings of previous iterations caused by personal over-involvement. In our research study at EMOP for example we felt often inclined to favour certain explanations when they implicitly assigned responsibility for research intervention failures to staff in the organisation rather than members of the research team, also an instance of a common type of cognitive bias known as "self-serving attribution bias" (Sims and Lorenzi, 1992; Woofford, 1994). In a number of situations these explanations were modified based on observations in further iterations where we, often against our own will, were forced to recognise that the independent or intervening variable that caused the negative effect that led to the intervention failure was linked to our own behaviour or to software systems about which we had initially high personal expectations.
Our experience in the AR project at EMOP also suggests that, in order to escape the negative consequences of personal over-involvement, the researcher should avoid as much as possible personal identification with methodologies and software systems that are being introduced into the client organisation as part of the AR project. While several iterations in the AR cycle can help correct misconceptions, this is going to happen at a high price for the researcher if these misconceptions were caused by the overt adoption of a methodology or software system because they were presented by the researcher as the "most appropriate" for that specific situation. Although the client organisation often relies on the expert advice given by the researcher to make decisions regarding the changes to be implemented as part of an AR project, a number of those decisions may prove to be less than appropriate. Since the project is expected to be a also a "research" study, both client organisation and researcher are expected to learn from it, and this must be made clear to the client organisation from the outset of the AR project.
This paper puts forth the argument that the cyclical approach to AR as described by Susman and Evered (1978) has the potential to yield IS research that is closer to the standards set by positivists in their own terms than AR is normally given credit for. This is true particularly of three main positivist criticisms of AR: contingency of the research findings, low control of the environment, and personal over-involvement. The consideration of these three main criticisms is important in our assessment because they touch on what is fundamentally seen by positivists as "appropriate" IS research (Orlikowski and Baroudi, 1991, p. 10).
One might argue that the strengths and weaknesses of AR, and particularly its application to IS research, should be acknowledged in its own merits, and that therefore a discussion about AR strengths in the light of positivist beliefs is at best irrelevant. In fact, a number of AR schools, such as the PAR and the cooperative inquiry schools (Reason, 1988; McTaggart, 1991; Whyte et al., 1991), state quite clearly that their particular ways of doing AR constitute a desirable alternative to approaches used by positivists based on a distinct scientific philosophy. However these claims have often been seen with suspicion by positivists, and also misunderstood by those brought into AR by the promise that they could finally do "easy and non-statistical research" in the IS research field (Kock and Corner, 1996). We believe that AR practitioners should first understand the claims of positivists and their fundamental basis, and then whenever possible respond in neutral terms, as we try to do in this paper - i.e. avoiding terms associated with traditional discourse colonisation attempts from both sides.
Finally, we point out that a critical assessment of AR practices and methodologies may be hindered if those who defend AR as a valid and appropriate approach to conducting IS research turn their back on criticism coming from outside groups. AR practitioners, who are still a minority group in the IS research field, can feel compelled to unconditionally support everything that comes from their comrades, and assign "prejudice" and "background-biased beliefs" as the reasons for the criticism coming from outside their group, ignoring it without further consideration. However, it is our belief that we can learn more from critical comments than from encouragement, and thus that AR practitioners should try to thrive on external criticism by critically assessing whether that criticism is grounded or not. The grounded criticism by positivists, whether it is spelled out in a constructive manner or not, can certainly be used by AR practitioners to improve their practice.
We would like to thank the staff at EMOP who participated in the AR research project described in this paper, and the anonymous reviewers for their helpful comments. This study was partially funded by a research grant from the Ministry of Science and Technology of Brazil - CNPq.
Alavi, M. (1993), An Assessment of Electronic Meeting Systems in a Corporate Setting. Information & Management, 25(4), pp. 175-182.
Argyris, C., Putnam, R. and Smith, D.M. (1985), Action Science. San Francisco, CA: Jossey-Bass.
Berkowitz, L. and Donnerstein, E. (1982), External Validity is More Than Skin Deep: Some Answers to Criticisms of Laboratory Experiments. American Psychologist, 37(3), pp. 245-257.
Brown, L.D. (1993), Social Change Through Collective Reflection with Asian Nongovernmental Development Organizations. Human Relations, 46(2), pp. 249-273.
Brynjolfsson, E. and Hitt, L. (1993), Is Information Systems Spending Productive? New Evidence and New Results. In J.I. Degross, R.P. Bostrom and D. Robey (Eds), Proceedings of the 14th International Conference on Information Systems. New York, NY: The Association for Computing Machinery, pp. 47-64.
Candlin, D.B. and Wright, S. (1991), Managing the Introduction of Expert Systems. International Journal of Operations & Production Management, 12(1), pp. 46-59.
Cash Jr, J.I. and Lawrence, P.R. (Eds) (1989), The Information Systems Research Challenge: Qualitative Research Methods. Boston, MA: Harvard Business School.
Checkland, P. (1981), Systems Thinking, Systems Practice. New York, NY: John Wiley & Sons.
Chidambaram, L. and Jones, B. (1993), Impact of Communication Medium and Computer Support on Group Perceptions and Performance: A Comparison of Face-to-Face and Dispersed Meetings. MIS Quarterly, 17(4), pp. 465-491.
Coleman, D. (Ed) (1992), Proceedings of Groupware'92 Conference. San Mateo, CA: Morgan Kaufmann.
Cook, T.D. and Campbell, D.T. (1976), Four Kinds of Validity. In M.D. Dunnette (Ed), Handbook of Industrial and Organizational Psychology. Chicago, IL: Rand McNally, pp. 224-246.
DeSanctis, G., Poole, M.S., Dickson, G.W. and Jackson, B.M. (1993), Interpretive Analysis of Team Use of Group Technologies. Journal of Organizational Computing, 3(1), pp. 1-29.
Doolin, B. (1995), Alternative Views of Case Research in Information Systems. In G. Pervan and M. Newby (Eds), Proceedings of the 6th Australasian Conference on Information Systems. Perth, Australia: School of Information Systems, Curtin University, pp. 767-777.
Elden, M. and Chisholm, R.F. (1993), Emerging Varieties of Action Research. Human Relations, 46(2), pp. 121-141.
Engelstad, P.H. and Gustavsen, B. (1993), Swedish Network Development for Implementing National Work Reform Strategy. Human Relations, 46(2), pp. 219-248.
Fox, W.M. (1990), An Interview with Eric Trist, Father of the Sociotechnical Systems Approach. The Journal of Applied Behavioural Science, 26(2), pp. 259-279.
Francis, D. (1991), Moving from Non-Interventionist Research to Participatory Action. In C. Collins & P. Chippendale (Eds), Proceedings of The First World Congress on Action Research, V.2. Sunnybank Hills, Australia: Acorn, pp. 31-42.
Galliers, R.D. (1984), In Search of a Paradigm for Information Systems Research. In E. Mumford et al. (Eds), Research Methods in Information Systems. New York, NY: North-Holland, pp. 281-297.
Galliers, R.D. (1992), Choosing Information Systems Research Approaches. In R. Galliers (Ed), Information Systems Research. Boston., MA: Blackwell Scientific Publications, pp. 144-162.
Gallupe, R.B., Cooper, W.H., Grise, M. and Bastianutti, L.M. (1994), Blocking Electronic Brainstorms. Journal of Applied Psychology, 79(1), pp. 77-86.
Gioia, D.A. and Sims Jr., H.P. (1986), Social Cognition in Organizations. In H.P. Sims Jr. and D.A. Gioia (Eds), The Thinking Organization. San Francisco, CA: Jossey-Bass, pp. 1-48.
Greenwood, D.J., Whyte, W.F. and Harkavy, I. (1993), Participatory Action Research as a Process and as a Goal. Human Relations, 46(2), pp. 175-191.
Guba, E.G. and Lincoln, Y.S. (1994), Competing Paradigms in Qualitative Research. In N.K. Denzin and Y.S. Lincoln (Eds), Handbook of Qualitative Research. Newbury Park, CA: Sage, pp. 105-117.
Gustavsen, B. (1993), Action Research and the Generation of Knowledge. Human Relations, 46(11), pp. 1361-1365.
Harvey, L.J. and Myers, M.D. (1995), Scholarship and Practice: The Contribution of Ethnographic Research Methods to Bridging the Gap. Information Technology & People, 8(3), pp. 13-27.
Heller, F. (1993), Another Look at Action Research. Human Relations, 46(10), pp. 1235-1242.
Hirschheim, R.A. (1985), Information Systems Epistemology: An Historical Perspective. In E. Mumford et al (Eds), Research Methods in Information Systems. New York, NY: North-Holland, pp. 13-35.
Jick, T.D. (1979), Mixing Qualitative and Quantitative Methods: Triangulation in Action. Administrative Science Quarterly, 24, December, pp. 602-611.
Jonsonn, S. (1991), Action Research. In H. Nissen, H.K. Klein and R. Hirschheim (Eds), Information Systems Research: Contemporary Approaches and Emergent Traditions. New York, NY: North-Holland, pp.371-396.
Ketchum, L.D. and Trist, E. (1992), All Teams are not Created Equal. Newbury Park, CA: Sage.
Kock Jr., N.F. (1994), MultiMail: A Computer Tool to Support Asynchronous Group Interaction. In D. Wollman (Ed), Proceedings of the Mercosul Software Conference, New Technologies Track. Curitiba, Brazil: SUCESU-PR, 20 pp. (in Portuguese).
Kock Jr., N.F. and McQueen, R.J. (1995), Integrating Groupware Technology into a Business Process Improvement Framework. Information Technology & People, 8(4), pp. 19-34.
Kock Jr., N.F. and Corner, J.L. (1996), Myths in Organisational Action Research: Reflections on a Study of Computer-Supported Process Redesign Groups. Research Report No. 1996-4, Hamilton, New Zealand: Dept. of Management Systems, University of Waikato.
Ledford, G.E. and Mohrman, S.A. (1993), Self-Design for High Involvement: A Large-Scale Organizational Change. Human Relations, 46(2), pp. 143-173.
Ledford, G.E. and Susan, A.M. (1993), Looking Backward and Forward at Action Research: Reply Paper. Human Relations, 46(11), pp. 1349-1359.
Ledford, G.E. and Susan, A.M. (1993a), Looking Backward and Forward at Action Research. Human Relations, 46(1), pp. 143-173.
Levin, M. (1993), Creating Networks for Rural Economic Development in Norway. Human Relations, 46(2), pp. 193-218.
Lewin, K. (1946), Action Research and Minority Problems. In G.W. Lewin (Ed), Resolving Social Conflicts. New York, NY: Harper & Row, pp. 201-216.
Lincoln, Y.S. and Guba, E.G. (1985), Naturalistic Inquiry. Beverly Hills, CA: Sage.
McTaggart, R. (1991), Principles for Participatory Action Research. Adult Education Quarterly, 41(3), pp.168-187.
Mintzberg, H. (1979), An Emerging Strategy of "Direct" Research. Administrative Science Quarterly, 24(4), pp. 582-589.
Orlikowski, W.J. and Baroudi, J.J. (1991), Studying Information Technology in Organizations: Research Approaches and Assumptions. Information Systems Research, 2(1), pp. 1-28.
Peters, M. and Robinson, V. (1984), The Origins and Status of Action Research. The Journal of Applied Behavioral Science, 20(2), pp. 113-124.
Rapoport, R.N. (1970), Three Dilemmas in Action Research. Human Relations, 23(6), pp. 499-513.
Reason, P. (1988), The Co-operative Inquiry Group. In P. Reason (Ed), Human Inquiry in Action. Newbury Park, CA: Sage, pp. 18-39.
Reason, P. (1993), Sitting Between Appreciation and Disappointment: A Critique of the Special Edition of Human Relations on Action Research. Human Relations, 46(10), pp. 1253-1270.
Reason, P. (1994), Three Approaches to Participative Inquiry. In N.K. Denzin and Y.S. Lincoln (Eds), Handbook of Qualitative Research. Newbury Park, CA: Sage, pp. 324-329.
Sims Jr., H.P. and Lorenzi, P. (1992), The New Leadership Paradigm: Social Learning and Cognition in Organizations. Newbury Park, CA: Sage.
Sommer, R. (1987), An Experimental Investigation of the Action Research Approach. The Journal of Applied Behavioral Science, 23(2), pp.185-199.
Sommer, R. (1994), Serving Two Masters. The Journal of Consumer Affairs, 28(1), pp. 170-187.
Strauss, A. and Corbin, J. (1990), Basics of Qualitative Research: Grounded Theory Procedures and Techniques. Newbury Park, CA: Sage.
Susman G.I. and. Evered, R.D (1978), An Assessment of the Scientific Merits of Action Research. Administrative Science Quarterly, 23, December, pp. 582-603.
Teichman, J. and Evans, K.C. (1995), Philosophy: A Beginner's Guide. Oxford, UK: Blackwell.
Tesch, R. (1990), Qualitative Research: Analysis Types and Software Tools. New York, NY: The Falmer Press.
Trauth, E.M. and O'Connor, B. (1991), A Study of the Interaction Between Information Technology and Society: An Ilustration of Combined Qualitative Research Methods. In H. Nissen, H. K. Klein and R. Hirschheim (Eds), Information Systems Research: Contemporary Approaches and Emergent Traditions. New York, NY: North-Holland, pp. 131-143.
Vidich, A.J. and Lyman, S.M. (1994), Qualitative Methods: Their History in Sociology and Anthropology. In N.K. Denzin and Y.S. Lincoln (Eds), Handbook of Qualitative Research. Newbury Park, CA: Sage, pp. 23-59.
Walker, R. (1993), An Introduction to Applied Qualitative Research. In R. Walker (Ed), Applied Qualitative Research. Aldershot, UK: Gower, pp. 3-26.
Whyte, W.F., Greenwood, D.J. and Lazes, P. (1991), Participatory Action Research: Through Practice to Science in Social Research. In W.F. Whyte (Ed), Participatory Action Research. Newbury Park, CA: Sage, pp. 19-55.
Winter, S.J. (1993), The Symbolic Potential of Computer Technology: Diferences Among White-Collar Workers. In J.I. Degross, R.P. Bostrom and D. Robey (Eds), Proceedings of 14th International Conference on Information Systems, New York, NY: The Association for Computing Machinery, pp. 331-344.
Woofford, J.C. (1994), Getting Inside the Leader's Head: A Cognitive Processes Approach to Leadership. SAM Advanced Management Journal, 59(3), pp. 4-9.
Yin, R.K. (1989), Research Design Issues in Using the Case Study Method to Study Management Information Systems. In J.I. Cash and P.R. Lawrence (Eds), The Information Systems Research Challenge: Qualitative Research Methods. Boston, MA: Harvard Business School, pp. 1-6.
Yin, R. K. (1994), Case Study Research. Newbury Park, CA: Sage.