BEMA III

QUESTIONNAIRE

Note: text in italics indicates a new indicator; text in bold indicates a HMA strategic objective.

Include checkboxes for activities covered by 5.1 and 9.1, also detailed indication of agency size, functions and capacity

KPI / NAME OF INDICATOR / DRAFT PROPOSALS FOR AREAS TO BE COVERED BY INDICATOR
KPI 1 / Strategy and planning
Setting objectives and targets (1.1) / Long-term strategic planning, methodology used, inputs to strategy, staff involvement, public consultation, approval process, publication of strategic plan and annual work programmes, communication of plans to staff.
Planning for implementation of new legislation, new drug and device technologies.
Management decision processes (1.4) / Decision-making on large capital projects, implementation of legislation, recruitment.
Cascading to operational units and staff (1.1) / Links from strategic plan to annual work programme to department yearly plans and individual staff objectives.
Allocating resources (1.6) / Allocation of human, financial, and infrastructural resources to strategic and annual plans. Ability of the agency to manage its own resources. Dealing with government restrictions.
Monitoring performance and management actions (1.5) / Measurement of performance at different levels within the agency, against the plans. Managing projects: structures and systems in place. Types of performance indicators: output or outcomes? Methods for measurement, e.g. balanced scorecard. Systems for review and improvement. System for oversight of performance against plans.
Communicating results (1.1) / Communication of results at management level, to staff and to external stakeholders. Frequency of communication: monthly, quarterly, yearly.
KPI 2 / Leadership and culture
Leadership (1.2,1.8) / Leadership of top management, both formal and informal. Communication flows up and down the agency hierarchy.Staff satisfaction regarding infrastructure, working environment, management, colleagues (1.8, 5.5) Management response to staff concerns.
Culture / Top management’s creation of specific culture appropriate to regulatory bodies and national cultural norms. Means by which culture is created and maintained. Ability to change itwhen it is not suitable.
KPI 3 / Stakeholders
Identification (2.1) and communication (2.4) / Range of stakeholders identified by the agency. Methods for identifying their needs and expectations. Specific communication methods, regular and ad hoc, formal and informal, to different stakeholders. Methods by which stakeholders can give feedback to agency.
General communication from agency on its functions and role.
Specific communications to applicants and PV reporters explaining the agency’s requirements; assistance provided where needed for understanding, (including 9.2)
Policies on public consultations.
Extent of patient and public engagement.
Processes for ensuring effectiveness of interaction with stakeholders.
Transparency (2.3) / Policies and processes for ensuring access to information on applications, decisions, product information, agendas and minutes of meetings. Proactive as well as reactive, including processes for handling request under Freedom of Information act.
Contribution to the network (2.2) and to national health system. / Extent of contribution to EU network (medicines and devices), according to size, fair distribution of work, supporting small/new agencies, twinning, cross-agency training.
Policies and procedures for membership and contribution to EU working parties, feedback from meetings to agency staff.
Specific contribution to various formal and informal work-sharing initiatives; VHP for clinical trials; GMP inspections outside the EEA; extent of GCP inspections.
Contribution to national health system, extent of involvement and co-operation with national health agencies, role in public health initiatives.
Communication and education on safety and quality issues / Specific (non-urgent) quality and safety communications, to patients and HCPs (including vets and breeders), and feedback mechanisms (PV-RMP). (15.2)
ADR reporting systems (13.1)/quality defect reporting: different reporting methods available to stakeholders; extent of reporting by different stakeholder groups, including patients, vets and farmers; methods of promotion and education; means of ensuring quality of reports; feedback to reporters; assessment of value of consumer reportingand contribution to drug quality and safety.
KPI 4 / Quality management
Quality system (1.3) / Process-based and documented quality system. Extent of the system across the agency, and across core and support processes. Certification. Internal audits: extent of, CAPA system. Systems and structures for continuous improvements. View of QMS: just documents and audits (operational) or part of high-level management system?
KPI 5 / Risk management
Risk management system (3.1) / Formal, agency-wide risk management systems for managing the risk to the agency. Structures within which risks are managed. Management of risks in projects. Extent of use of risk management system and risk register. Assessment of its value in managing risks.
Conflicts of interest (3.2) / Internal staff and external experts. Policies for declarations both systematic and as potential conflicts arise. System for evaluation and assigning work. Record-keeping. Public transparency.
Protection of information (4.1) / Systems for protecting information: in databases, applications, files and working documents; in the agency and when files or documents are taken outside; identification of staff to outsiders, information sent to external experts. Maintaining confidentiality of information in archives or when stored electronically in data centres. Security of the systems used.
KPI 6 / Crisis management
Crisis management (involving agency infrastructure or staff) / Management of incidents and crises involving the agency’s infrastructure or staff. Business continuity arrangements; crisis management plans; disaster recovery plans. Ability of agency to manage crises, including those involving other MSs. System for review and ‘lessons learnt.’
Crisis management (involving risks to public health) (15.1 and 15.2) / Management of incidents and crises involving risks to public health. Crisis management plans specific to safety- or qualityrelated crises, identification of urgent safety or quality issues, process for taking a decision, process for taking regulatory action, systems and practices for involving multi-disciplinary staff across the agency and for obtaining external expert advice, communication to stakeholders including other MSs.
KPI 7 / Human resource management
Recruitment/provision of required expertise (5.3, 6.2, 13.9, 14.2) / System for recruitment, knowledge/understanding of current and future needs, extent of expertise available and needed. Resources needed for new drug and device technologies. Use of external experts, methods for contacting them.
Ability of the agency to recruit. Specific recruitment of assessors and PV assessors. Range of expertise available for assessment and PV assessment.
Learning and development (1.7, 6.1 and 6.3, 13.9) / Training system in place, use of competence frameworks (for competence or competencies?). Range of training provided (scientific, regulatory, soft skills, management, project management). Sources and types of training materials. Specific training and development of assessors and PV assessors.
Evaluation of the effectiveness of training and development.
KPI 8 / Operations management
Capacity management (5.2) / Methods of forecasting demand, using industry information, lifecycle forecasts. Fulfilment of legal authorisation obligations (and contribution to the network)
Measuring capacity, Estimating resource requirements. Strategies for dealing with uncertain and fluctuating demand.
Risk-based approaches (14.4) / Framework in place: structures, policies, approval systems for risk management tools, monitoring systems, improvements.
For clinical trials, assessment, pharmacovigilance, inspections and product testing.
Resource allocation (5.2) / Allocating assessors and PV assessors to specific work items, basis for allocation.
Timeline compliance (5.4) (6.4) (14.1) (16.4) (13.7) / Setting of KPIs, notifications of timelines, internally and externally, tracking systems, responsibilities, updating of information when dates change, communication to assessors and external experts, monitoring of compliance and corrective actions
KPI 9 / Information management
IT facilities / Systems that support the core work, workflow systems, e-submissions (12.1, 12.2), e-filing, databases including for ADRs (13.2, 13.3. 13.4). Appropriateness of technology to size of agency and workload. Planning for future IT needs.
IT information resources (5.5) / Resources available for assessors, inspectors and PV assessors: electronic submissions, review tools, intranet, web access to guidelines and scientific databases, access to journals, Sharing of information within departments and across different disciplines. Policies on information access across departments,
Data transfer to and from EU databases (12.3) (13.7) / Comprehensiveness and quality of data submitted to EU databases, responsibility for data input, agency policy and development of systems to ensure submission, accuracy and quality of data.
KPI 10 / Interfaces
Management of interfaces. / High-level policies for co-operation between different departments, high-level structures and systems for resolving issues.
Between quality assessors, GMP/GDP inspectors and OMCL (11.1 – 11,4)
(16.1 – 16.5) / Extent of interaction, responsibilities, communication, information flows/access, handling of critical findings., clarity of understanding
Between clinical assessors and GCP inspectors (and GMP inspectors)
(17.1 – 17.2) / Extent of interaction, responsibilities, communication, information flows/access, handling of critical findings., clarity of understanding
Between PV assessors, clinical assessors, PV inspectors (and GCP inspectors) (13.8) (18.1 - 18.3) / Extent of interaction, responsibilities, communication, information flows/access, handling of critical findings., clarity of understanding
KPI 11 / Scientific decision-making
Quality and consistency of assessment/inspection/PV assessment (routine and non-urgent)(7.2) (9.4) (14.1, 14.2)(includes sci adv) / Setting standards, agreeing approaches, communication between staff, record-keeping of assessment/inspection reports, guidance on preparing reports, peer review (extent/structure/basis for/feedback), maintaining regulatory and scientific memory, lessons learnt from previous activities, interdisciplinary working.
Additionally, for PV: real-time signal detection methods, evaluation and prioritisation system for detected signals, other sources used to validate the signal.
(Indicate where systems differ between assessment, inspection and PV assessment.)
Opinion-making by assessors, inspectors, PV assessors (7.1) (Includes sci adv) (9.3) (14.3) / Achieving integrated opinion, dealing with divergent views, incorporating advice from committees (advice not decisions), advice from experts,
System and procedures by which agreement is reached on action/no action in relation to safety signals. (13.5) (13.6)
Systems for reviewing the appropriateness of the assessors’/inspectors’ opinion, and corrective and improvement actions.
(Indicate where systems differ between assessment, inspection and PV assessment.)
Decision-making by the agency (3.3, 8.1) (includes sci adv) / Structures and levels of decision-making. Expertise used, including scientific committees, for decision-making. Information available at each level. Basis on which decisions are made.
(Indicate where systems differ between assessment, inspection and PV assessment.)
KPI 12 / Impact/effectiveness of scientific work / (Supporting SMEs, academia, orphan drugs, paediatrics, HCPs, enhancing product availability for patients)
Clinical trials and clinical investigations / Support for development of new medicines/combination therapies/advanced therapies, innovation, provision of direct support and advice to industry and academic sponsors, extent of interaction and co-operation with national and local ethics committees.
Scientific advice (9.5) / Impact of scientific advice on subsequent applications, review and improvement of system and processes for giving advice,
Medicines licensing / Impact of authorisation system on availability of medicines, MUMS. Extent of monitoring of medicines availability, interaction with government departments and other relevant national agencies to mitigate impact, review of options and improvement actions,
PV/safety decision (15.3) / Impact of decisions on use of products, tracking the effect of regulatory actions on prescribing, dispensing and use of products, reviewing impacts and making improvements to the systems for communicating decisions to relevant stakeholders.
KPI 13 / Pharmacovigilance
??

Not included

4 building security

13.2, 13.3, 13.4 13.7 (ADR databases and management in KPI 7)

19.2 (GLP, Is this in JAP?)

BEMA III Questionnaire – Version 0.2Page 1 of 3