Veterinary Medicines Regulatory Agency Self-Assessment Tool

Foreword

Many countries need a structured way to assess the effectiveness of the systems used to regulate veterinary medicines. Weaknesses in regulation can increase the risk of poor quality or falsified products, inappropriate availability and use of veterinary medicines and inadequate control of antimicrobial use.

The World Organisation for Animal Health (WOAH) supports countries in strengthening veterinary services and regulatory systems. The United Kingdom Veterinary Medicines Directorate developed the Veterinary Medicines Regulatory Agency Self-Assessment Tool (VMRA SAT), to help participating authorities examine their regulatory arrangements, identify strengths and gaps, and plan practical improvements.

VMRA SAT supports systematic reflection and national ownership. The quality of the result depends on objective input, appropriate evidence and constructive discussion among the organisations involved.

1. About VMRA SAT

1.1 Purpose

VMRA SAT provides a structured framework for an authority (or entities involved in the veterinary medicines regulatory system) to assess the maturity and implementation of its veterinary medicines regulatory system. It supports identification of strengths, gaps and priorities and provides a basis for an Institutional Development Plan.

The tool examines the legal framework, governance, organisational arrangements, regulatory processes, technical capacity, resources, quality systems and transparency associated with the regulation of veterinary medicinal products.

What VMRA SAT assesses

• VMRA SAT uses a clear assessment structure that moves from broad regulatory responsibilities to detailed requirements. It is organised into Functions, Indicators and Sub indicators, allowing a national regulatory authority to examine both its overall regulatory system and the individual arrangements that support it. This structure helps users identify strengths, recognise gaps and understand where improvement may be needed.
• Functions
• The eight Functions represent the main responsibilities involved in regulating veterinary medicines. Together, they provide a comprehensive view of the veterinary medicines regulatory system. The functions assessed are listed below. Please note that all these functions may be performed by a single institution or by multiple institutions. If a country is to conduct a robust self-assessment, all the institutions involved must take part in the self-assessment exercise.
• Veterinary medicines national regulatory system (VRS)
• Veterinary medicines Registration and marketing authorisation (VMA)
• Veterinary medicines Pharmacovigilance (VPV)
• Veterinary medicines Market surveillance and control (VMS)
• Veterinary medicines Licensing of establishments (VLE)
• Veterinary medicines Regulatory inspection (VRI)
• Veterinary medicines Laboratory testing (VLT)
• Veterinary medicines Batch release (VBR)

• Indicators
• Each Function is divided into Indicators. Indicators describe the main elements that should be in place for a regulatory responsibility to operate effectively. These may relate to areas such as legislation, governance, leadership, quality management, human resources, procedures, infrastructure, transparency and monitoring of performance. There are 13 indicators in total and not all are included in every Function. For example, in ‘01 VRS’, Indicator number 1 (VRS01) states: ‘Legal Provisions, regulations and guidelines required to define regulatory framework of veterinary national regulatory system (VRS).’

• Sub indicators
• Indicators are assessed through more detailed Sub indicators. Each of the 235 Sub indicators describe a specific requirement or practice that can be examined during the assessment. Users consider whether the requirement has been implemented and record the information and evidence supporting their judgement.
• Each Sub indicator is supported by a Fact Sheet containing information such as its purpose (Full fact sheets can be found here), requirements, suggested evidence, references, assessment criteria and relevant limitations or explanatory notes. This guidance helps users interpret the requirements consistently and base their assessment on verifiable information.
• Each sub-Indicator’s fact sheet has the following sections that contain information to assist the user populate the tool accurately.
• Scope – the product area(s) to which the sub indicator applies.
• Objective – what the sub-indicator is trying to achieve.
• Requirement - a concise requirement statement or requirement title that captures what must be in place to meet the sub indicator.
• References – lists useful reference material that may assist NRAs to complete its self-assessment.
• Framework - A field that is used to classify the type of element the sub-indicator relates to.
• Rating Scale – It sets out defined rating categories and their meanings.
• Limitations and Remarks – sets out any qualifying statements such as why a rating of ‘not applicable’ is not an option for a sub-indicator.
• Evidence to Review – lists what documents or other material should be reviewed and included as part of the assessment.
• Description – this field provides narrative explanatory guidance about what the sub-indicator covers and what the assessor should look for.
• NRA input - The NRA is expected to provide comprehensive information, explanations and clarifications in response to the sub-indicator's requirements. This should be based on the objective and description outlined for the sub-indicator. The NRA should deliver a full and clear description of how it fulfils the requirements. Where applicable, the NRA must also highlight any gaps, ambiguities or transitional arrangements that are currently in place. The information provided by the NRA is crucial in informing the appropriate status for the sub-indicator and, where applicable, the Institutional Development Plan (IDP).
• External Assessor input - Information, observations and judgements provided by the appointed external assessors during a benchmarking process, based on interviews, document review and verification activities. External assessor input is used to validate NRA input and to determine the final status of sub‑indicators.
• Reviewed documents - Documents and other forms of evidence that are provided to verify the implementation status of sub‑indicators. These are normally listed in the ‘Evidence to review’ section of the sub-indicator. (If it is a benchmarking exercise, these evidence should be made available to be reviewed by the external assessors).
• Status - The outcome of the assessment of the sub‑indicator, expressed through the rating scale categories, which reflect the level of implementation of the requirement. Status is determined based on the information in the NRA input and reviewed documents and following the guidance provided in the ‘Rating scale’. (In the case of benchmarking, assessor judgement will also be crucial).
• Institutional Development Plan (IDP) – this requires proactive identification of all gaps and areas for improvement highlighted through the sub-indicator assessment as identified in the NRA input and evidence review. For each gap, the necessary action should be clearly described, with a priority assigned, a realistic timeline established and required resources specified. Each IDP entry must be presented as a single, actionable item, with additional lines provided for further activities where appropriate. The format should enable straightforward tracking of progress and allow for updating of each action’s status as work advances. This approach ensures the IDP remains practical and focused, supporting the ongoing enhancement of regulatory capacity and facilitating effective monitoring of progress towards closing identified gaps.
• Strictly speaking the above is a recommendation or an action intended to close the gap identified. However, a when all the recommendations are compiled together, they form a structured plan that become the IDP which is developed following completion of the self-assessment. The IDP supports progressive improvement of regulatory capacity and maturity and provides a practical roadmap for national implementation.

1.3 What VMRA SAT is not

• It is not, by itself, a determination of legal compliance.
• It does not replace regulatory decisions, inspections, audits or specialist technical evaluations.
• A status selection without explanatory information and evidence does not constitute a robust assessment.

1.4 Assessment models

1.5 Confidentiality and ownership

Assessment information remains under the control of the participating authority. Files and reports should be shared only in accordance with arrangements agreed by that authority and any applicable national requirements. The assessment coordinator should agree access, storage, retention and publication arrangements before data collection begins.

2. Preparing for the assessment

2.1 Roles and responsibilities

2.2 Recommended preparation

1. Confirm the purpose, scope and assessment model.
2. Appoint the coordinator, function leads and NRA assessors.
3. Identify all organisations that carry out relevant regulatory functions.
4. Agree how decisions and disagreements will be resolved.
5. Allocate work at indicator level where separate files will later be merged.
6. Prepare an evidence register and secure access to documents.
7. Agree file naming, storage, access and backup arrangements.
8. Set review, moderation and approval points.
9. Confirm the software version, country data file and local IT requirements.

2.3 Readiness checklist

3. Installing and starting the software

3.1 Before installation

Use only the software package and country data file supplied or approved for the assessment. Confirm the applicable software version, supported operating system and any local IT security requirements. Do not use an executable received from an unverified source.

3.2 Windows desktop installation

1. Save the supplied compressed software package to an approved local folder.
2. Extract the complete package before attempting to run the application. Do not launch the application from inside the compressed folder.
3. Open the extracted folder and locate the VMRA SAT application file supplied for Windows.
4. Launch the application. If an organisational security warning appears, follow local IT procedures rather than bypassing the control.
5. Import the supplied country data file using the relevant import option.
6. Check the country name, software version and assessment state before entering information.
7. Close and reopen the application to confirm that the assessment can be accessed.

3.3 MacOS installation

Use the MacOS package and instructions supplied with the release. The application file and security steps differ from Windows. Please note that there are two versions for Mac, one for those with Intel processors and another one for those with Silicon chip processors. Follow organisational IT requirements and verify the country name and software version before entering information.

3.4 Importing the data file

Importing Data File

Click on the three horizontal bars

Click on Data source

Select the ‘Import external file’ and then click on the ‘Browse’ button.

Then Data source manager will open together with a question asking you if you’d like to save current file. Click ‘No’.

It will open your windows screen, go to the folder where you saved the data file, select the file, then click ‘Open’.

You should then see a populated front page with the country name, the dates etc.

If it is blank data file for a country, other than the front page the sub-indicators will be empty. However, if you opened a populated data file, then all the information previously saved should be there for each of the sub-indicators.

3.5 Starting or reopening an assessment

1. Open the VMRA SAT application.
2. Select or import the correct assessment file.
3. Confirm that the displayed country and assessment are correct.
4. Check the last saved or exported version against the file register.
5. Create a backup before making substantial changes to an existing assessment.

Critical check

4. Navigating VMRA SAT

The home screen presents the eight regulatory functions and gives access to assessment and administrative areas. The exact wording and position of controls must be checked against the released software.

Navigation principles

• Save information regularly and before moving between major sections. The software should save the information automatically but always save before moving on, just in case.
• Use the application navigation controls provided by VMRA SAT.
• Check that entered information remains visible after saving.
• Do not use Delete unless the effect is understood and a recoverable backup exists.
• If the screen differs materially from this manual, record the software version and consult the developer team.

5. Completing a sub indicator

5.1 Structure

VMRA SAT is structured into functions, indicators and sub indicators. Each sub indicator contains information that supports interpretation and assessment, including scope, objective, requirement, references, framework classification, rating scale, limitations and remarks, evidence to review, description, NRA input, external assessor input, reviewed documents, status and Institutional Development Plan fields.

5.2 Completion sequence

1. Read the scope to confirm the products, activities and institutions covered.
2. Read the objective, requirement and description together.
3. Review the rating scale and any limitations or remarks.
4. Identify the evidence needed to show both the existence and operation of the arrangement.
5. Enter a clear NRA narrative describing the current situation.
6. Reference the documents and records reviewed.
7. Identify gaps, ambiguities, exceptions and transitional arrangements.
8. Select the status supported by the narrative and evidence.
9. Add the evidence required by giving the unique file identifier, reference or URL and in the description, write specific description referring to the section, chapter or heading. Avoid making general description such as ‘legislation’.
10. Create one or more IDP actions for gaps requiring improvement.
11. Save the entry and perform a completeness check.

5.3 Writing NRA input

NRA input should explain how the requirement is met in practice. It should identify the responsible institution, applicable legal or procedural basis, operating process, implementation coverage, evidence reviewed, exceptions and known gaps. Avoid unsupported statements such as “compliant” or “in place”.

6. Determining status

The status records the level to which the requirement of a sub indicator is implemented. It is separate from the predetermined maturity level assigned to the sub indicator. The ‘Rating scale’ in the fact sheet provides additional information on the appropriate selection of the status.

6.1 Status decision questions

• Does the arrangement cover the full scope of the sub indicator?
• Is responsibility formally assigned?
• Is the necessary legal, policy or procedural basis in force?
• Is the arrangement operating in practice?
• Is implementation consistent and sustained?
• Is suitable evidence available?
• Are there exclusions, gaps or transitional arrangements?
• Does the rating scale specify additional conditions?

Where sufficient evidence is not available to demonstrate implementation, do not select Implemented. Select Not Applicable only when the sub indicator permits that status and the stated conditions are satisfied.

6.2 Maturity levels

Each sub indicator is assigned a predetermined maturity level reflecting the stage of regulatory system development at which the requirement would normally be expected. The user does not select the maturity level, and maturity level must not be confused with implementation status.

6.3 Maturity level calculation algorithms

VMRA SAT uses an algorithm to calculate the maturity level achieved for each regulatory function. The calculation uses the implementation status assigned to the sub indicators within that function.

For calculation purposes:

• Implemented has a score of 1
• Partially Implemented has a score of 0.5
• Not Implemented has a score of 0
• Not Applicable is excluded from the calculation where that status is permitted

The algorithm assesses the results against the requirements associated with the Pre Bronze, Bronze, Silver, Gold and Gold Plus maturity levels.

Cumulative calculation

Maturity is cumulative. A function cannot achieve a higher maturity level unless the applicable requirements from all preceding levels have also been met.

The calculated result is therefore the highest maturity level for which the relevant requirements and implementation thresholds have been satisfied. It is not a simple average of all sub indicator scores.

Maturity is calculated separately for each regulatory function.

Available algorithms

VMRA SAT provides three approaches to calculating maturity:

• Strict algorithm

Under the Strict algorithm, every applicable sub indicator assigned to the maturity level being assessed must be Implemented. All applicable sub indicators at the preceding maturity levels must also be Implemented.

A Partially Implemented or Not Implemented sub indicator prevents the function from achieving that maturity level. This approach identifies every remaining requirement that must be fully implemented before the function can progress.

• Flexible algorithm

The Flexible algorithm recognises that a regulatory function may demonstrate the characteristics of a maturity level while a limited proportion of requirements remains Partially Implemented or, at the higher maturity levels, Not Implemented.

The degree of flexibility varies by maturity level.

At Gold and Gold Plus, the permitted percentages are maximum values. The combination of Partially Implemented and Not Implemented sub indicators must remain within the applicable thresholds.

• Restricted Flexibility algorithm (also called Critical)

Restricted Flexibility (Critical) combines elements of the Strict and Flexible algorithms.

The applicable Flexible algorithm percentage thresholds are used but selected critical sub indicators must be Implemented. A function cannot achieve the relevant maturity level if an applicable critical sub indicator is Partially Implemented or Not Implemented, even where the overall percentage threshold would otherwise be met.

This approach permits limited flexibility while ensuring that requirements considered fundamental to regulatory capability have been fully implemented.

Pre Bronze requirements

Pre Bronze represents minimum operating capability. It requires the implementation of six foundational sub indicators covering:

• A legal foundation for veterinary medicines regulation
• An established source of funding
• Publicly available contact information for regulatory services
• An up-to-date list of veterinary medicines permitted on the national market

All applicable Pre Bronze requirements must be Implemented before the function can progress to Bronze.

Selecting an algorithm

The algorithm should be selected before interpreting or reporting the maturity results. The selected approach should reflect the purpose of the assessment, the authority’s regulatory development context and the level of assurance required.

The Strict algorithm demonstrates that every applicable requirement for the maturity level has been fully implemented. The Flexible algorithm recognises incremental progress where limited gaps remain. Restricted Flexibility provides an intermediate approach by allowing limited gaps while protecting critical requirements.

The selected algorithm affects the maturity designation and must therefore be stated whenever results are reported. For example:

Silver using the Restricted Flexibility algorithm

Results calculated using different algorithms should not be compared without clearly identifying the algorithm applied.

Interpreting the result

The calculated maturity level provides a summary of the implementation statuses recorded for a regulatory function. It does not replace review of:

• The underlying NRA input
• The evidence and reviewed documents
• The reasons for each status selection
• Identified implementation gaps
• Critical sub indicators that have not been implemented
• The associated Institutional Development Plan actions

A higher maturity designation indicates that more of the applicable requirements associated with that level and the preceding levels have been met. It does not independently verify the quality, consistency or effectiveness of regulatory performance.

7. Evidence and reviewed documents

7.1 Evidence principles

• Relevant: directly addresses the requirement.
• Authentic: can be traced to an authorised source.
• Current: reflects the arrangements assessed or clearly explains the period covered.
• Sufficient: demonstrates design and implementation, not merely intention.
• Representative: covers the relevant functions, products, locations or time periods.
• Traceable: recorded with enough information for another reviewer to locate it.

7.2 Types of evidence

7.3 Recording reviewed documents

For each item, record a concise title, reference or identifier, version, date, owner or issuing body, location and the part relevant to the sub indicator. If the information is published, provide the URL. Avoid attaching personal or operationally sensitive information unless necessary and authorised. In the description, provide specific information related to the sub-indicator and evidence requirement. Provide sections, chapters, headings so it is easy to identify.

If evidence is outdated, contradictory, incomplete or limited in scope, state this in the NRA input and reflect the limitation in the status and IDP.

For self-assessment exercise, no external body will be inspecting the contents of these documents but assessment coordinator or function lead within the organisation may wish to verify the content.

8. Institutional Development Plan

The Institutional Development Plan converts identified gaps into a prioritised and monitorable programme of improvement. Each entry should address one clear gap and specify a practical action.

8.1 Minimum content

• Gap or problem to be addressed
• Action and intended result
• Priority
• Responsible owner
• Required resources
• Target date or milestone
• Activity status

8.2 Writing an actionable entry

8.3 Prioritisation

Prioritisation should consider regulatory and public health risk, maturity level, legal obligations, dependencies, feasibility, available resources and the effect on other regulatory functions. The national authority should define and apply a consistent priority scale.

A recommendation describes what should improve. An IDP brings all agreed actions together into a structured national plan that can be assigned, monitored, updated and closed.

9. Collaborative assessments

9.1 Control model

Nominate national coordinator and maintain one controlled master assessment file. Contributors should work only in copies issued by the coordinator and should not exchange uncontrolled files directly.

9.2 Allocation and file naming

Where separate files will be merged, allocate complete indicators to individual contributors or teams. Avoid dividing the sub indicators of the same indicator between files. It will not be possible to merge files if they are done this way. Each indicator must be assigned to a single person regardless of how many sub-indicators it contains.

Suggested file name: Country_FunctionOrIndicator_Contributor_DDMMYY (e.g. UKVMD_VMA_010203_Ben_210425). In this example Ben covered VMA01, VMA02 and VMA03 indicators. Use an approved convention consistently and avoid overwriting the last accepted version.

9.3 Export workflow

1. The National coordinator creates and records the master file and a backup.
2. The National coordinator issues controlled working copies and records the allocated indicators.
3. Contributors complete only their allocated content and conduct a local check.
4. Contributors export or return the file using the release specific file naming method as described above.
5. The National coordinator checks the allocated indicators, status, evidence and IDP entries.
6. Overlapping or conflicting entries are referred to the relevant leads for resolution and an agreed version is saved as the final version.
7. Follow the steps below to export a file:

Exporting Data File

Click on the three horizontal bars

Click on Data source

Type in a new file name for the information to be exported Country_FunctionOrIndicator_Contributor_DDMMYY. Avoid having spaces between them, use underscore (_) or hyphen (-) instead. (E.g. UKVMD_VMA_010203_Ben_210425)

Then click on ‘Browse’ and it’ll ask you to select the folder you would like the file to be saved in.

Once the folder is selected, click on ‘Select folder’.

The export file is now saved in the folder selected.

Then attach this file to your function lead or national coordinator so all the files can be checked and then sent.

9.4 Quality checks

• All allocated indicators are present.
• No completed narrative has been left blank.
• Status selections have been assigned correctly.
• Reviewed document references under each sub indicator have been added.
• IDP actions are complete and not duplicated.
• The file naming convention has been followed.

10. Quality assurance, completion and reporting

10.1 Quality assurance review

10.2 Consensus and approval

Function leads should review proposed statuses and IDP actions with the relevant institutions. Material disagreements should be recorded and resolved through the agreed governance process. The final assessment and IDP should receive the level of national approval agreed at the planning stage.

10.3 Generating reports

1. Create a backup of the approved master assessment file.
2. Use the report function supplied in the released software.
3. Select the required Word or PowerPoint output, where available.
4. Confirm the country, assessment version, completion state and report scope.
5. Generate the report and save it using the agreed naming convention.
6. Check headings, charts, statuses, narrative, evidence references and IDP entries against the master file.
7. Correct source data in VMRA SAT rather than editing a report in a way that creates a conflicting version.
8. Record the approved report in the document register and apply the agreed sharing controls.

10.4 Interpreting outputs

Charts and maturity profiles should be interpreted alongside the underlying sub indicator evidence and narrative. A visual result is a summary of the assessment, not a substitute for reviewing the basis of each status. Avoid comparisons between authorities unless the assessment basis, scope, evidence standards and methodology are demonstrably comparable and the participating authorities have agreed the use.

10.5 Future reassessment

Retain the approved master file, final reports, evidence register, IDP and decision record in accordance with national retention requirements. At reassessment, use a controlled copy, verify the software and data versions, and record changes in evidence, implementation and IDP completion.

11. Troubleshooting and support

Support request information

• Country and assessment name
• Software version
• Operating system
• Manual version
• File type and version, without sending confidential content unless authorised
• Exact action attempted
• Exact error message
• Screenshot with sensitive information removed
• Whether the problem can be repeated
• Contact details and urgency

Insert the approved support email address, service hours and escalation arrangements before publication.

Appendix A. Quick start guide

1. Confirm scope, roles, governance and evidence arrangements.
2. Install the correct software release and import the correct country file.
3. Create the controlled master file and backup.
4. Allocate complete indicators to contributors.
5. Read each sub indicator in full before entering information.
6. Describe implementation, cite evidence, identify gaps and select status.
7. Create actionable IDP entries for gaps.
8. Merge controlled contributions into the master file and check each merge.
9. Complete cross function quality assurance and national approval.
10. Generate, verify, store and control reports and the final dataset.

Appendix B. Evidence register template

Appendix C. Status decision record

Appendix D. Glossary and abbreviations

Appendix E. Selected definitions

The following definitions are working definitions used for the purposes of VMRA SAT. Where applicable, definitions have been aligned with WOAH terminology. They should not all be interpreted as official WOAH definitions