ECOMANAGEMENT N° SU 1104 (EN)
Evaluation of a telematic service : the EcoManagement project
- Training and counselling activities report -
WP n°8, Task n°8.2.8 (Deliverable n°8.13)
Jean-Jacques QUINTIN
Unité de Technologie de l'Education
Université de Mons-Hainaut
April 1998
Table of contents
General context of the evaluation of EcoManagement and specific objectives of the verification......
General context......
Specific objectives of the verification......
Stages of the evaluation of ecomanagement......
Methodology of the verification......
1.Conceptual framework......
2.Main questions of the evaluation of ecomanagement during the verification and the demonstration......
3.Organisation of the verification - sampling - population......
4.Organisation of the verification and protocols of evaluation......
5.Analysis of the results......
Intra-sites analysis......
FRANCE : Organisation 1......
Overall description of the organisation of the verification and context......
SME 1.1 : WÄRTSILA NSD-Diesel motors......
Teleassistant 1.1 : Philippe FESSARD (INFO PARTNERS)......
FRANCE : Organisation 2......
Overall description of the organisation of the verification and context......
SME 2.1 : RECY PLASTURGIE - User 2.1......
SME 2.2: GIP Ile de France - User 2.2......
SME 2.3: VALEO - User 2.3......
Teleassistant 2.1 : Dominique Barbe (C.N.E.D.)......
Germany : Organisation 3......
Overall description of the organisation of the verification and context......
SME 3.1 : Reichart Dienstleistungen und Elektronik-Recycling GmbH (RDE)......
Teleassistants......
Teleassistant 3.1: Susanne Sabrowsky of the TMS......
Teleassistant 3.2: Michaela Rosen of the TAR......
Teleassistant 3.3: R. Gralla of the TAR......
Teleassistant 3.4: Daniela Jürging of the TAR......
Teleassistant 3.5: Bauer from the TMS......
Teleassistant 3.6:Yvonne Stallknecht of the TÜV Management Systems......
Spain : Organisation 4...... 54
Overall description of the organisation of the verification and context...... 54
SME 4.1 : ICT...... 54
SME 4.2 : ICT...... 58
SME 4.3 : CETEMMSA...... 61
Teleassistant 4.1:Judith Sugrañes...... 65
Teleassistant 4.2:Victor Aroca...... 66
Greece : Organisation 5...... 68
Overall description of the organisation of the verification and context...... 68
SME 5.1 : TITAN S.A....... 69
Degree of satisfaction toward the support of an environmental management...... 71
SME 5.2 : Aluminium of Greece S.A....... 72
SME 5.3 : Chemical Industries of Northern Greece...... 75
Teleassistant 5.1 : Yanis Tsikinas...... 76
Teleassistant 5.2 : Kostas Symeonide...... 77
Inter-sites analysis : Degree of satisfaction toward the telematic aspect of EcoManagement...... 79
1.Parts of the system particularly appreciated or unappreciated....... 79
2.Satisfaction of the end-users toward the communication with the teleassistant...... 81
3.Satisfaction of the end-users toward the FAQ database...... 81
4.Satisfaction of the end-users toward the quality of the databases...... 83
5.Satisfaction of the end-users toward the quality of the training modules...... 83
6.Satisfaction of the end-users toward the quality of the counselling modules...... 85
7.Satisfaction of the end-users toward the method of environmental management...... 86
Conclusion and suggestion of adaptation...... 88
1.Sampling of users and SMEs...... 88
2.Organisation of the verification...... 88
3.Collection of information : the interview....... 89
4.The Ecomanagement Telematic System: functionalities...... 89
5.The Ecomanagement Telematic System: organisation of the service...... 89
6.The Ecomanagement Telematic System: Global concept and structure...... 89
7.The Ecomanagement Telematic System: FAQ database...... 90
8.The Telematic aspects of the verification: installation, setting up...... 90
9.The Telematic aspects of the verification: user attitudes...... 91
10.The user acceptance of the telecommunication....... 91
11.The Environmental management: methodology...... 91
12. Organisation of the demonstration phase....... 92
Annex I
Tests of connection and downloading speed...... 3
Organization of the verification phase...... 6
Guide for the verification phase...... 17
Test of theTelematic System...... 21
Validation intermédiaire du design pédagogique des modules de formation : L'eau, L'aire et Le sol...... 31
Annex II
Protocol of evaluation of EcoManagement during the verification...... 3
Protocol of interview: End users...... 10
Protocol of interview: Intermediate users...... 43
Protocol of interview: Head managers...... 63
General context of the evaluation of EcoManagement and specific objectives of the verification
General context
The main purpose of the work package 8 is to evaluate how the telematic service developed answers to the needs of the target population and how the project could improve the service so to better encounter those needs.
This service is mainly based on the use of a demonstrator (EcoManagement Telematic System) developed on the base of a precise description of what it should do. Those specifications were described in the deliverable n°4.5[1]. Those specifications were defined after the study of the needs of the target population. This analysis of the needs is synthesised in the deliverable n°3.4[2]. But ecomanagement is more than a telematic system, it also implies the intervention of different actors, teletrainers, telecounsellors, administrators, in such a way the end user will have a complete service in environment management. A short summary of this service is described in point 3.2 'Ecomanagement service: an organisation around the telematic system', p14.
The analysis of the verification phase has the purpose to bring information about the use of the demonstrator by a sampling of users from the target population, in such a way it will be possible to improve ecomanagement before the demonstration and the exploitation phases. Two main questions will guide the verification analysis:
-How the ecomanagement is functioning and what are the improvement necessary to improve the user acceptance ? This analysis will focus on the functioning of the ecomanagement telematic system.
-How the use of ecomanagement allows to improve the environmental management inside the SME and does it bring the impacts searched ? This analysis will focus on the impacts of the ecomanagement service.
Specific objectives of the verification
More precisely, the analysis will search information about the following points
From the end users:
1.Functioning of the ecomanagement telematic system
-Installation
-Functioning
-Usability
-User acceptance
-Physical performance (simulation)
2.Impacts of the use of ecomanagement
-Motivation to use the ecomanagement telematic system
-Motivation to implement the environment management
-Motivation to use telematic
From the intermediate users:
1.Functioning of the ecomanagement telematic system
-Installation
-Functioning
-Usability
-User acceptance
-Physical performance (simulation)
2.Impacts of the use of ecomanagement
-Change of attitude toward their trainer function
-Motivation for teleworking
-Motivation to use telematic
Stages of the evaluation of ecomanagement
The evaluation part of the ecomanagement project was considered as a permanently process allowing to constantly have information from the system into development in the purpose to improve it. The verification of the service in real conditions is therefore imbricated in other evaluation stages: 2 stages before (see below, stages 1 & 2) and is followed by the demonstration phase.
Stage n°1 :Local and intermediate validation of the ecomanagement system during its development.
The local verification focused on the design and the content of the modules (training and counselling). An 'intermediate' version of those different products allowed to analyse 'in local' the way user has to proceed to obtain an expected result and to improve the modules. The reader can find the result of the local and intermediate validation of the training modules in the annex[3] (French version).
Stage n°2 :Intermediate validation at distance of the Ecomanagement Telematic System.
This stage evaluated the aspects of the design intervening in the process of communication at distance while the next stage will analyse the way final users use the system at distance.
The appraisal group of this stage of verification was the partners of the project. The system was thus tested in internal simulating situations of telecounselling and teletraining.
The reader can find the result of this stage in the annex[4].
Stage n°3:Verification at distance and in real situation of the ecomanagement service.
This last stage of verification unfold in several countries with a sample of SMEs. The end of this stage is punctuated by this report, resuming the different recommendations of adaptation of the products.
Stage n°4:Demonstration
Methodology of the verification
1.Conceptual framework
The conceptual framework describes the main variables we will study in the verification and their assumed relationship. This framework is devised to be flexible and general enough to adapt itself during the collection of data.
To organise the informations and to analyse them, we built a framework close to those used to evaluate the implementation of innovation, to analyse the process of change in a structure and especially in the education field. Many studies report on the analysis of change in education (Huberman, 1973; Huberman, Miles, 1983; Tourneur, Beumier, Rochez-Nimal, 1986; Depover, Strebelle, Debrue, 1992). On the basis of those studies and on the characteristics of EcoManagement as a process of innovation, we built a theoretical model to organise the collection of information and the analysis of this process of change. As the figure below shows it, the model was devised around three axes, representing three groups of variables.
First axisContext & background variables
Second axisProcess variables. Use of the telematic System & Service)
Third axisProducts variables. Impacts of the process on the users and the SMEs
Unité de Technologie de l'Education de l'Université de Mons-Hainaut (UMH)D8.1330/04/98p.1(92)
Figure 1. Conceptual framework for the evaluation of the implementation of the ecomanagement service.
Unité de Technologie de l'Education de l'Université de Mons-Hainaut (UMH)D8.1330/04/98p.1(92)
1.1Context & background
Different studies (SARACHO, 1982; C.L.E.A., 1983; HOLMES, 1984) showed that contextual variables as motivation of the users, institutional structure, ... represent main factors to explain the way an innovation is processing. We selected the following variables for they seems to be interesting to explain some results regarding the impacts of ecomanagement upon the users.
-Telematic background of the users : end users and intermediate users (teleassistants)
-Telematic situation of the SME
-Environmental background of the users
-Environmental situation of the SMEs
-Support to the end users (environmental coordinator)
-Motivation of the users toward the innovation
1.2Process
Several authors (Zaltman & col, 1973; Yin & col, 1977; Berman & Mac Laughlin, 1978; Rosenblum & Louis, 1979; Fullan, 1982; Rogers 1983; Tourneur & col, Depover & col, 1992) considered a process of change as following several phases : pre-adoption (or 'adoption', 'initiation', 'mobilisation'), which mainly describes the innovation, implementation, which includes the first tries of the innovation and finally the routinisation or institutionalisation (or 'integration') which describes the process of change as integrated in the usual practices of an institution.
In the case of ecomanagement, the use of the Ecomanagement Telematic System and its service, is seen as a process of change in the SME and will follow the same phases as described above :
-Pre-adoption of the service: the service is described after a pro-active needs analysis and a definition of the specifications of the service and of the Ecomanagement Telematic System
-Implementation of the service: the service is implemented progressively (see ' Stages of the evaluation of ecomanagement' p.5) and finally tried for the first time in real conditions among a sampling of users. This is the object of the verification and this report will analyse the results from this specific phase.
-Routinisation of the service: if the previous stage succeeded to really implement the service in the SME, the routinisation has the opportunity to install the new service in the usual practice of the SME.
The implementation phase should also allow to have information to adapt the service to the target population. The ecomanagement phase of the project called 'demonstration' or 'experimentation' (see the Technical
annex of the ecomanagement project[5]) has the '... main objectives to test the organization of the service with a large enough number of SMEs ...'. The objectives of this project phase are '... to recruit a sufficiently large user group in each country, to study the impact on the family life and teleworking at home (for the telecounsellor and teletutors), to deliver the best schedule for teletutor and telecounsellor and to evaluate customer satisfaction and effectiveness of the service.'. So, the demonstration phase corresponds in fact to an new implementation phase for a large number of SMEs except for those which already participated to the verification. For those, it could be possible to arrive to a routinisation.
Figure Main Document Only. Relationship between phases of the innovation process and project phases (Work Packages).
1.3Products
The Deliverable D3.4 'Syntheses of needs analysis and contextual elements' and D4.5 'Specifications' constitute the products of the pre-adoption phase. The document D4.5 describes precisely what would be the innovation (the telematic system and the service).
Pre adoption
-Need analysis
-Specifications of the telematic system
-Specifications of the service
Implementation and routinisation
The process of innovation itself will be analysed through its unfoldment during the verification and demonstration phases (description of the service in each organisation, description of the system functionalities available and functioning) and through the impacts, progressively brought, upon the different appraisal groups : users (end users, teleassistants, administrators, ...), the SMEs and the environment. Indeed, the first effects of the implementation of ecomanagement will be sensitive on the end and intermediate users through their opinions about the system and service provided by ecomanagement. Then, we can expect some impacts on their way of handling environmental problems (knowledge, skills, ...) and finally on the environmental management of the SME itself having an impact on the rejections on the environment and on the waste recycling.
Here follows the different type of products by appraisal group:
Appraisal groups / ProductsSystem/Service / Description of the telematic system & the organisation of the service
Users / Analysis of the use of the system/serviceUser acceptance
Motivation
Attitude
SME / Implementation of an environmental management
Environment / Decrease of rejections, waste
Increase of recycled waste
2.Main questions of the evaluation of ecomanagement during the verification and the demonstration
2.1Main questions about the context of utilisation of EcoManagement and background
-What are the characteristics which have an influence toward the variables of impacts ?
-Which contextual characteristics influence the adoption of EcoManagement ?
2.2Main questions about the functioning of the demonstrator and the EcoManagement service
-Does the demonstrator do what it was planned it should (comparison to the specifications) ?
-Does the installation of the demonstrator pose any problem and if so, what is its impact and how to facilitate the installation ?
-Does the system functions properly, and if not, how to improve its functioning ?
-Does the use of the system pose any problem and if so, what is its impact and how to facilitate the use of the system ?
-Upon which services are the users satisfied and not satisfied ? In that case, how to improve those services ?
2.3Main questions about the impacts of the EcoManagement services
... on the users (end users and intermediate users)
Does the use of EcoManagement :
-increase the motivation to use it ?
-increase the capacity to use it ?
-increase the motivation to implement the environment management ?
-increase the motivation to use telematic tools ?
... on the intermediate users (only)
Does the use of EcoManagement :
-change the attitude toward their function ?
-increase the motivation for teleworking ?
... on the SMEs
Does the use of EcoManagement
-decrease the water consumption ?
-decrease the energy consumption ?
-decrease the waste consumption ?
-develop the an environmental management ?
-change the intention to respect the environment regulations ?
-change the intention to obtain an EcoLabel ?
3.Organisation of the verification - sampling - population
3.1The organisations and type of users
To participate to the verification phase, the partners from 4 different countries (France, Germany, Spain and Greece) organised themselves around 3 administrations. An administration is represented by the partner who implemented and managed a ecomanagement server during the verification.
France: Organisation 1 & 2
The partners from France (Cesi & FFP) organised the verification around the administration implemented in the Cesi. Three SMEs contacted by FFP and one SME by the Cesi participated to the verification. This represents what we called the 'Organisations'. As the figure below shows it, the end users from those SMEs could connect the ecomanagement server, contact the partner in case of problem. Beside this, contacts were planned with a teleassistant, essentially to test the videoconferencing and sharing system. The ecomanagement partners (Cesi and FFP) had an essential role. They were in charge to organise the verification and to collect the information. They helped the end user to install the software and hardware required to use ecomanagement, often a modem and a navigator as Netscape, they followed up the verification and they collected the informations at the end of it (interview).
Figure 2. Organisation of the verification in France.
Germany: Organisation 3
The administration system was implemented in the TÜV Akademy Rheinland (TAR), the ecomanagement partner who organised the verification in Germany. The verification in Germany unfold inside the partner company because at that time, the server had no 'firewall' yet and the partner decided not to allow the access to the server from outside the company. The user 3.1 came three times in the TAR to test the ecomanagement system and had a videoconferencing with a teleassistant from the TÜV. The other teleassistants tested the system from the TAR.
Figure 3. Organisation of the verification in Germany.
Spain and Greece: Organisation 4 & 5
The administration system was implemented in the ICT who organised the verification in Spain. The Greek users (end users and teleassistants) also had the possibility to access to the ICT server.
In Spain, only one user (SME 4.3) connected the server from his SME, outside ICT. The other used the ecomanagement system from ICT through a LAN. Two teleassistants tested the system and one of them had a videoconference with the user SME 4.3.
In Greece, all the users and teleassistants reached the ecomanagement system from the ICT server. We have no information about the way the Greek partner organised the verification neither the collection of information (interview).
Figure 4. Organisation of the verification in Spain and Greece.
3.2Ecomanagement service: an organisation around the telematic system.
The figure below, shows an illustration of what a 'theoretical' organisation could be to respect the specifications of the ecomanagement service (see the deliverable 'Specifications' D4.5).
A user (User x on the figure), the client, has the possibility to connect the ecomanagement system to access different type of information including a FAQ database and external database as the one tested during the verification: 'EAUDOC'. He also has the possibility to download a training or counselling modules and the method of environmental management. If he needs to contact a teleassistant (Teleassistant 1) and after looking for the most appropriate one through the consulting of the teleassistants database, he takes an appointment with him to have a videoconference meeting.