6th Global Conference on Business & EconomicsISBN : 0-9742114-6-X

The Evaluation of Information Systems Success:aNew Perspective

Hafid Agourram

King Fahd University of PetroleumMinerals, Saudi Arabia

Email:

(966) 3 860 2678

Amine Nehari Talet

King Fahd University of PetroleumMinerals,Saudi Arabia.

Email:

Mailing address:

PO. Box 805

King Fahd University of Petroleum & Minerals,

Dhahran 31261

Saudi Arabia.

Key words: Information System Success (IS), Alignment, Fit, National culture, Business

Strategy, Information System Strategy

Acknowledgement: We would like to thank King Fahd University of Petroleum and Minerals and specifically the College of Industrial Management for their generous supports.

Abstract

Information system success has been a challenging concept for researchersfor many years. Research on the definition and the evaluation ofinformation system (IS) success was growing yet confusing. The social nature of IS success concept is a major factor that may explain this confusion. From the global and multicultural perspectives, a recent study showed that the meaning of IS success is not homogenous across different national cultures.Organizations are more and more attracted by doing business in different cultures and become global. This attraction is also yielded by the successful technical capabilities of the new information technologies that support the globalization strategy. The existing models of IS success require a large amount of modifications and adaptationsbefore they can be applied in local and international organizations. This paper proposesa comprehensive framework that is easy to use, integrates existing knowledge on the alignment and IS success domains and can be applied to any local and national context. The power of the framework is in its practicality and flexibility.

Introduction

The use of information technology and information systems in today’s organization is growing in a phenomenon way. The advanced sophistication of the new information technologies combined with their yielding benefits are major factors that justify their massive use in almost every type of organization. Morton (1991) reported that the rate at which new technology is introduced is increasing 20 to 30 percent annually and the numbers are promising to grow in the future. However, IS investment decision makers are confused and frustrated because they are unable and find it very difficult to justify their IT and IS investments by using existing tools of IS performance and success.Researchers have developed many IS success models to help managers measure the performance of their IS (DeLone and McLean, 1992, 2002; Seddon, 1997; Robey and Boudreau, 200.). The major problems with these models are: First, they cannot be applied to all contexts and to all types of information systems. Second, these models show how to evaluate IS success independently from other organizational variables such as structure, strategy, and processes. Lastly, the values of these measurements may sometimes lead to confusion because they are not compared to a set of predefined expected values that define success to the organization. For example, it may be that an organization evaluated the success of its information system by using the impact of the system on the decision making process dimension and found that this impact is average. Is that a success or not? The answer is we don’t know because we luck a predefined level of the impact of the IS on its users to which we can compare the actual value. It has been largely documented that the organization and its information systems must fit each other; that is, one cannot be separated from the other. In this paper, we use this parallel philosophy between the organization and its IS to develop a very comprehensive process framework that uses existing knowledge on the fit theory. The framework can be used in any national and organizational culture and is very practical for managers who wish to evaluate IS success. The following sections will discuss existing IS success theory, the fit theory, our proposed framework, and thebenefits of the framework.

Information Systems success

Information system (IS) success and its determinants have long been considered critical to the field of information systems. Despite many attempts to model success (Delone and McLean, 1992, 2003), IS success definition and measure is still problematic for many factors. The first factor is the mixture of the technical and social aspects of an IS. IS success is then a perspective that emerges from the social and technical interplay within organizations (Kanellis, Lycett and Paul, 1998). Second, Alter (2000) argues that information technology and work practices are now so intertwined that it is difficult to identify their respective contribution to success. Other researchers link the difficulty of defining IS success to the methodological aspects involved in measuring IS success «Specifying a dependent variable is difficult because of the many theoretical and methodological issues involved in measuring IS success» (Garrity and Sanders; 1998, p. 14). Seddon, Staples, Patnayakuni, and Bowetell, (1999) argue that IS success is a fuzzy concept contingent upon different stakeholders and different types of IT. In the practice community, Markus and Tanis (2000) claimed that there is a fundamental gap in both practical and academic thinking about information systems lack of consensus and clarity concerning the meaning of success where information systems are concerned.

The problem of IS definition and measurement becomes more difficult and more complicated if we add the international dimension (Ishman,1998; Garrity and Sanders,1998, Pauleen et al., 2006). The international dimension includes cultural terms such as values and assumptions which may be at the heart of the differing perceptions and interpretations of IS success. Shing-Kao (1997) argues that «Research has shown that people notice, interpret and retain information based on their values, assumptions and expectations. Different assumptions and values lead to different ways of looking at the same thing» (p. 13).

This international differing perception of the meaning of a phenomenon is an important topic in International Management or Cross-cultural Management disciplines. Do theories and concepts born in a specific culture apply or have the same meaning in other cultures? Hofstede (1993), for example, after a large survey on work-related values in sixty countries, concluded that management theories and findings are not automatically transferable from one context to another. Shing-Kao (1997), Kedia and Bhagat (1988), and Robichaux and Cooper (1998) add that the majority of theories of management have a Western orientation.

Rosenzweig (1994) argues that a central concern in scientific research is external validity. That is, the extent to which a theorized or observed relationship among variables can be generalized to other settings. Rosenzweig (1994) claims that the main question should not be, «Are scientific management theories that interest us valid elsewhere? But how can we best understand management, as it exists around the world? » (p.37).

Researchers agreed that the measurement of IS success is not an easy task. The major problem with existing IS success model is that, first,they are very abstract and don’t rend service to managers who are always searching for more practical tools and techniques. Second, they are used independently from the organizational and national contexts. The organizational context has been largely documented by strategic alignment researchers such as Henderson, Venkatraman and Luftman and others who built a large body of knowledge that stressed the necessity to align and to fit IS and organizational strategies. On the other hand, the national context was always ignored by existing IS success models. Agourram (2004) conducted a cross cultural research that involves France, Germany, and Canada and found that IS success is perceived and defined differently in these cultures. Therefore, concludes the author, existing IS success models cannot be applied as they are in different national contexts. The following section discusses the alignment concept.

The alignment theory

Alignment of business and information strategies referred to the extent to which business strategies were enabled, supported, and stimulated by information strategies. Evidence for alignment was sought in the use of information or information technology, or both, which provided a comparative advantage to an organization over its competitors. The importance of aligning the objectives and strategies of an organization's information systems (IS) group with those of the broader organization has been recognized for some time (Lederer and Mendelow, 1986; Henderson, 1990; Chan and Huff, 1993; Henderson and Venkatraman, 1996; Luftman, 1996; Kearns and Lederer, 2000; Reich and Benbasat, 2000; Chan, 2002), and is usually referred to as strategic alignment. The early research into strategic alignment tended to be theoretical in nature (Henderson and Venkatraman, 1990) providing the platform on which later work was developed. Based on these early developments, a number of dimensions of strategic alignment have emerged. The social dimension of strategic alignment has been defined as "the state in which business and IT executives within an organizational unit understand and are committed to the business and IT mission, objectives, and plans" (Reich and Benbasat, 2000, p. 82). The intellectual dimension of strategic alignment is defined as "the state in which a high-quality set of interrelated IT and business plans exist" (Reich and Benbasat, 2000, p. 82). Put simply, research into the social dimension tends to "... focus on the people involved in the creation of alignment" whilst the intellectual dimension of alignment tends to "... concentrate on the content of plansand planning methodologies" (Reich and Benbasat, 1996). This concentration has naturally led to research at the CIO/CEO level, at which planning methodologies are chosen and strategies developed. Furthermore the little research that has been conducted on the social dimension of alignment has also been focused at this level (Reich and Benbasat, 1996, 2000; Chan, 2002). There is evidence to suggest that strategies developed at senior management level are often modified or even sabotaged at lower levels of the organization during implementation (Davies, 1993; Alien and Wilson, 2003; Nordstrom and Soderstrom, 2003). Chan and Huff (1992) have also made the observation that strategy is developed at executive level but essentially implemented at lower levels of an organization, with the consequence that greater attention needs to be paid to the dynamics of alignment at lower levels of the organization.

In studying strategic information systems (IS) alignment, it is important to examine the associated processes over time rather than viewing alignment as an isolated event. Organizations experience a series of interdependent changes in business strategy and Information System strategy which impose to increase the alignment between them. Alignment is a dynamic and emergent concept because of the ongoing change in business and information strategies. As business strategy changes, IS strategy must change in parallel. However, moving from one pattern of alignment between business and IS strategies turns out to be a rather difficult proposition. Consequently, organizations struggle to bring IS and business strategies into alignment and sometimes go through potentially problematic trajectories, where the business strategy changes but one (or more) of the IS strategy components fails to change appropriately.

The strategic use of information systems has been a fundamental concern for every organization. IS are used today to help organization implement and apply their strategies. They are used as major tools that help organizations gain competitive advantages over their competitors. Researchers such as Henderson and Venkatraman (1993) and Luftman(1993, 1998) have always stressed the fact that the realization of these competitive advantages is conditioned by the strategic use of IS. The alignment of organizational strategy, objectives and goals with those of IS, enhance the degree of the benefits these organizations can gain.Venkatraman (1993)developed the alignment model which is defined in terms of fourconstructs: business strategy, information technology strategy, organizational infrastructure and processes, and information technology infrastructure and processes. The model explains two fundamental characteristics of strategic management: strategic fit and function integration.

Greater alignment or "fit" between an organization's business strategy and IS strategy implies that the information systems are targeted on areas that are critical to successful business performance. "Alignment between business and IS strategies heightens managers' awareness and use of information systems, and it enables a firm to better use IS to help realize its goals and objectives or obtain a competitive advantage. Weak fit is problematic in two ways. First, it may create competing motivations for different people in different parts of the organization as with the tension between strategy and structure. Second, in the absence of a unifying theme or logic arising from tight fit, organizational players may be confused as to the appropriate decisions and behaviors in respect to achieving existing operational performance targets." (Rudy Hirschheim,Rajiv Sabherwal 2001).

. The following section discusses our framework that uses Venkatraman theory.

The new IS success framework

Figure 1 shows our proposed framework. The framework is a process and not a variantframework. The framework uses existing concepts from the alignmenttheory. The framework is a solution to many IS success evaluation problems. Its benefits willbe discussed in the last section.

1

OCTOBER15-17, 2006

GUTMAN CONFERENCE CENTER, USA

6th Global Conference on Business & EconomicsISBN : 0-9742114-6-X

The framework shows that ISsuccessor IS performance is a process which is made of the following phases:

  1. In phase 1, the strategic orientation of the organization is established. This document includes the directions and the plans as well as the goals of the organization. Parallel to the strategic orientation of the organization, IS people establish the strategic orientation of the information systems. These two processes are done at the same timeand the two must fit each other. (Jerry Luftman, 1993) hasextensively reported research work that discusses this first process.
  1. Phase 2 includes two steps that are also executed in parallel. In the first step and after having a clear idea on the direction of the organization, managers need to document theexpected outcomes of this direction; that is the expected performance of the organization. At the same time, IS professionals would document the expected performance of their information systems; that is IS success. IS expected performance must be aligned with the expected organizational performance. Moreover, the organizational expected performance determines the IS expected performance. If for example management consider that the business processes of the organization must be fluid and employ less people as one expected organizational performance element, this would push IS professionals to consider maximum automation of business processes as an element of the expected IS performance.
  1. In phase 3, business people implement organizational structures and procedures and design the new processes to help realize the expected organizational performance. At the same time, IS people implement the information systems that would help realize the expected IS performance. The implementation components and logic must fit the organizational infrastructures and procedures. In this step, and in case information systems are acquired and not developed in house, the acquired system built-in assumptions are taken into consideration in the IS implementation step.
  1. Phase 4 does not execute right away.It would take few years before we start actions in phase 4. In this phase, we compare the actual organizational performance with the expected organizational performance, and we compare the actual IS performance with the expected IS performance. Comparative research techniques can be used in this phase. In this phase, we don’t need to search for organizational and IS performance models and see it we can apply them to our organization. In this phase, the actual organizational performance is the DEGREE to which the actual organizational performance matches the expected organizational performance, and the actual IS performance is the DEGREE to which the actual IS performance matches the expected IS performance. If case the gaps between the expected and the actual performances is large, control is sent back to the first phase.

1

OCTOBER15-17, 2006

GUTMAN CONFERENCE CENTER, USA

6th Global Conference on Business & EconomicsISBN : 0-9742114-6-X

Conclusions

Our framework yields to the following benefits:

  • It pushes people to think about the outcomes and the results before the actual implementationof the systems. This is a major exercise that helps convince decision makers to invest in information systems,
  • The model requires a continuous fit and alignment between the organization and the information system which has been proved by research to be an excellent exercise, and
  • When it comes to evaluating the performance, we don’t need to borrow from very theoretical model and see if we can apply them to the actual context. The variety of information systems requires a dynamic IS evaluation system. Moreover, this phase which used to be the hardest and the most researchable becomes the easiest. We just compare what we have with what we expected. If the gap is large, actions can be easily implemented to reduce it and hence ameliorate the actual IS success.

References

  1. Alter, S. (2000). The Siamese Twin Problem: A Central Issue Ignored by Dimensions of Information Effectiveness. Communication of AIS. 2(20), 1-55.
  2. Agourram, H, :The Impact of National Culture on The Meaning of Information Systems Success, D.B.A Dissertation, University of Sherbrooke, May 2004.
  3. Alien, D. and Wilson, T., "Vertical trust/mistrust during information strategy formation", International Journal of Information Management, Vol. 23, pp. 223-37. 2003
  4. Chan, Y.E., "Why haven't we mastered alignment? The importance of the informal organization structure", MIS Quarterly Executive, Vol. 1 No. 2, pp. 97-112. 2002
  5. Chan, Y.E. and Huff, S.L., "Investigating information systems strategic alignment", Proceedings of the 14th International Conference on Information Systems, Orlando, Florida. 1993.
  6. Davies, R., "Making strategy happen: common patterns of strategic success", European Management Journal, Vol. 11 No. 2, pp. 201-13. 1993.
  7. DeLone, W.H and McLean, E.R. (1992). Information Systems Success: The Quest for The Dependent Variable. Information Systems Research, 3(1), 60-95.
  8. DeLone, W.H and McLean, E.R (2003). The Delone and McLean model of Information systems success: A ten-year update. Journal of Information Systems Management, 19(4), 9-26.
  9. Henderson, J.C and Venkatraman, N, Strategic alignment: Leveraging Information technology for transforming organizations, IBM Systems Journal, v32, no 1, 1993
  10. Garrity, E.J. and Sanders, G.L. (1998). Introductionto Information Systems Success Measurement. In E.J. Garrity and G.L, Sanders (dir), Information Systems Success Measurement (p. 1-12). Idea Group Publishing.
  11. Henderson, J.C. , :"Plugging into strategic partnerships: the critical IS connection", Skan Management Review, Vol. 3, pp. 7-18. 1990.
  12. Henderson, J. C., & Venkatraman, N. (1994). Strategic alignment: a model for organizational transformation via information technology. In T. J. Allen & M. S. Scott Morton (Eds.), Information technology and the corporation of the 1990s (pp. 202–220). New York: OxfordUniversity Press.
  13. Henderson, J.C. and Venkatraman, N.,: "Aligning business and IT strategies", in Luftman, J.F. (Ed.), Competing in the Information Age: Strategic Alignment in Practice, Oxford University Press, New York, NY, pp. 21-42. 1996.
  14. Hofstede, G. (1993). Cultures and organizations: software of the mind.London, McGraw-Hill.
  15. Ishman, M. (1998). Measuring Information Success at the individual level in cross-cultural Environments. In E.J. Garrity and G.L, Sanders (dir), Information Systems Success Measurement. Idea Group Publishing.
  16. Jerry N. Luftman, Paul R. Lewis, Scott H. Oldach: Transforming the Enterprise: The Alignment of Business and Information Technology Strategies. IBM Systems Journal 32(1): 198-221 (1993)
  17. Kanellis, P., Lycett, M. and Paul, R.J. (1998). An Interpretive Approach to the Measurement of Information System Success: From Concept to Practical Application In E.J. Garrity and G.L, Sanders (dir), Information Systems Success Measurement (p. 133-151). Idea Group Publishing.
  18. Kearns, G.S. and Lederer, A.L., :"The effect of strategic alignment on the use of IS-based resources for competitive advantage", Strategic Information Systems, Vol. 9, pp. 265-93. 2000
  19. Kedia, B.L. and Bhagat, R.S. (1988). Cultural Constraints on Transfer of Technology Across Nations: Implications for Research in International and Comparative Management. Academy of Management Review, 13(4), 559-571.
  20. Lederer, A.L. and Mendelow, A.L.: "Co-ordination of information systems plans with business plans", Journal of Management Information Systems, Vol. 6 No. 2, pp. 5-19. 1986
  21. Luftman, J, Lewis, P.R, Oldach, S.H, Transforming the enterprise: the alignment of business and information strategies. IBM Systems Journal, v32, n1, 1993
  22. Luftman, J., :"Applying the strategic alignment model", in Luftman, J. (Ed.), Competing in the Information Age: Strategic Alignment in Practice, Oxford University Press, New York, NY, pp. 43-69. 1996.
  23. Luftman, J. Assessing IT/business alignment, Information Systems in Management. v20, i4, 2003.
  24. Lyftman, J, Enablers and Inhibitors--- Despite Working on Different Levels, IT and Business Executives Must Communicate Better In Order For Companies to Experience the Benefits of Technology, Information week, Sept 14, n700, 1998.
  25. Markus, M.L and Tanis, C. (2000). The Enterprise System Experience – From Adoption to Success. In R. Zmud (dir.), Framing the Domain of IT Management (p. 173-207). Pinnaflex.
  26. Nordstrom, T. and Soderstrom, M., "Study of implementing an IT-impregnated corporate strategy", paper presented at the llth European Conference on Information Systems, Naples, 16-21 June. 2003
  27. Pauleen, D., Evaristo, R., Davison, R., Ang, S., Alanis, M., and Klein (2006), "Cultural Bias in Information Systems Research and Practice: Are You Coming from the same place I am?" Communications of the AIS, 17, 354-372.
  28. Reich, B.H. and Benbasat, I. , "Measuring the linkage between business and information technology objectives", MIS Quarterly, Vol. 20 No. 1, pp. 55-81. 1996
  29. Reich, B.H. and Benbasat, I., "Factors that influence the social dimension of alignment between business and information technology objectives", MIS Quarterly, Vol. 24 No. 1, pp. 81-113. 2000.
  30. Robichaux, B.P. and Cooper, R.B. (1998). GSS Participation: A Cultural Examination. Information and Management, 33, 287-300.
  31. Rosenzweig, P.M. (1994). When Can Management Science Research Be Generalized Internationally? Management Science, 40 (1), 28-39.
  32. Rudy Hirschheim,Rajiv Sabherwal :Detours in the path toward strategic information systems alignment,California Management Review.Berkeley: Vol.44,Iss.1;pg.87,23pgs Fall 2001.
  33. Seddon, P.B., Staples, S., Patnayakuni, R. and Bowetell, M. (1999). Dimensions of Information Success. Communication of the ACM. 2(20), 1-40.
  34. Shing-Kao, L. (1997). A study of National Culture versus Corporate Culture in International Management.Dissertation abstract international, NovaSouth-easternUniversity.
  35. Venkatraman, N., Henderson, J. C., & Oldach, S. : ‘Continuous Strategic Alignment: Exploiting Information Technology Capabilities for Competitive Success’, European Management Journal, vol. 11, no. 2, pp. 139 – 149. 1993

1