EditverseCME needs assessments: structure, evidence and the independence standards
How to build a CME needs assessment from practice gap to learning objective, how to rank the evidence behind a gap, and what the ACCME Standards require.
Key points
- A needs assessment documents a professional practice gap, identifies the educational need behind it, and turns that need into learning objectives and an outcome measure.
- ACCME defines a professional practice gap as the difference between what a professional is doing or accomplishing and what is achievable on the basis of current professional knowledge. The educational need is classified as knowledge, competence or performance.
- Evidence for a gap is strongest when it measures actual practice in the learner group. Guideline text shows what should happen; it does not show that it is not happening.
- The ACCME Standards for Integrity and Independence in Accredited Continuing Education have five standards and took effect on 1 January 2022.
- An ineligible company, meaning one whose primary business is producing, marketing, selling, re-selling or distributing healthcare products used by or on patients, cannot influence the planning, faculty, delivery or evaluation of accredited education.
What the document is for
A continuing medical education needs assessment is the reasoned case for an educational activity. It is read by the accredited provider's planning committee, by reviewers of educational grant requests, and later by accreditation surveyors looking for evidence that the provider plans from the needs of its own learners. This article describes the structure for activities accredited within the ACCME system in the United States. Other accreditation systems use similar logic with their own terms.
The requirement comes from ACCME's core accreditation criteria. The provider incorporates into its activities the educational needs, in terms of knowledge, competence or performance, that underlie the professional practice gaps of its own learners. ACCME describes the process: the provider examines problems in practice or patient care and looks for knowledge, strategy, skill, performance or system deficits that could be contributing to them.8 Related criteria ask that activities be designed to change competence, performance or patient outcomes, that formats suit the setting and objectives, and that the provider analyses the changes achieved.9
The chain from gap to objective
Every needs assessment follows the same chain. Each link must be traceable to the one before it.
| # | Link | Question it answers | Test of a good entry |
|---|---|---|---|
| 1 | Best practice | What is achievable on current professional knowledge? | Cites a current guideline, trial or standard, with year |
| 2 | Current practice | What are these learners doing or achieving now? | Uses data from the target audience or a close proxy |
| 3 | Professional practice gap | What is the difference between 1 and 2? | States the gap as a measurable difference |
| 4 | Cause of the gap | Why does the gap exist? | Separates lack of knowledge, lack of skill or strategy, failure to apply, and system barriers |
| 5 | Educational need | Which cause can education address? | Classified as knowledge, competence or performance |
| 6 | Learning objective | What will learners be able to do afterwards? | Observable verb; matches the level of the need |
| 7 | Format | How will the education be delivered? | Fits the objective: case-based for competence, practice feedback for performance |
| 8 | Outcome measure | How will change be shown? | Measures the same level as the objective |
The most common fault is a break between links 4 and 5. If the cause of a gap is a formulary restriction or a staffing shortage, a lecture will not close it, and the assessment should say which part of the gap education can reach. The second most common fault is a mismatch between links 6 and 8: an objective written at the level of performance and an outcome measured with a knowledge quiz. Planning the assessment and the outcome evaluation together, as the published model for integrating planning and assessment in continuing education recommends, avoids this.10
Ranking the evidence for a gap
ACCME does not publish a hierarchy of evidence for practice gaps. The ranking below is a working method that follows from the definition: since a gap is a difference between observed and achievable practice, sources that observe practice in the learner group rank highest.
| Tier | Source | What it shows | Limits |
|---|---|---|---|
| 1 | Performance data from the learners' own setting | Audit, quality dashboard, registry or claims data for the target audience | May not be available outside health systems |
| 2 | Published practice data in a comparable population | National registry reports, quality programme results, peer-reviewed audits and surveys of practice | May be dated, or drawn from a different country or setting |
| 3 | Outcomes of previous activities | Pre-test results, case vignette performance and stated barriers from earlier education on the topic | Self-selected learners |
| 4 | Learner surveys and focus groups | Perceived needs and barriers | Perceived need is not measured practice |
| 5 | Expert opinion and planning committee judgement | Clinical context and plausibility | Weakest alone; useful for explaining causes |
| n/a | Guidelines, pivotal trials, new approvals | The best-practice side of the comparison | Not evidence of a gap by themselves |
A credible assessment combines at least one source from tiers 1 to 3 with tier 4 or 5 material that explains why the gap exists. Every source is cited to the same standard as a journal article, and evidence is described fairly, including data that suggest the gap is narrowing.
Worked example
The example is illustrative. The audit, survey and figures are invented to show how the chain is written; they are not real data.
| Link | Entry |
|---|---|
| Best practice | The current guideline recommendation on stroke risk assessment and oral anticoagulation in atrial fibrillation, quoted with the guideline's name, year and recommendation number. (In a real document this row carries the full citation.) |
| Current practice | Illustrative regional audit of 42 primary care practices: 61 percent of eligible patients had anticoagulation prescribed within 90 days of diagnosis. In 28 percent of records no stroke risk score was documented. |
| Gap | About four in ten eligible patients are not anticoagulated within 90 days, and risk is not documented in over a quarter. |
| Causes | Illustrative survey of 180 clinicians: 47 percent said they overweight bleeding risk in older patients; 33 percent were unsure how to dose in reduced renal function; 21 percent cited no recall system for follow-up. |
| Educational needs | Competence: weighing stroke against bleeding risk in older adults. Knowledge: dose selection by renal function. The recall system is a system barrier, noted but outside the scope of this activity. |
| Learning objectives | After the activity learners will be able to (1) calculate and document stroke and bleeding risk for a patient with newly diagnosed atrial fibrillation; (2) select an anticoagulant and dose according to renal function and interacting medicines; (3) explain the balance of benefit and risk to an older patient in plain terms. |
| Format | Case-based online module with four patient cases and decision points; optional practice audit template. |
| Outcome measures | Competence: case vignette decisions before and after. Performance, for practices using the audit template: proportion anticoagulated within 90 days at six months. |
The independence standards in one table
The ACCME Standards for Integrity and Independence in Accredited Continuing Education were released in December 2020 and went into effect on 1 January 2022, replacing the Standards for Commercial Support.1 They consist of a preamble, eligibility rules and five standards.
| Standard | Applies to | Core requirements |
|---|---|---|
| 1. Ensure content is valid3 | All accredited education | Patient care recommendations rest on current science, evidence and clinical reasoning and give a fair and balanced view of options. Research cited conforms to accepted standards of design, data collection, analysis and interpretation. New and evolving topics are identified as such. |
| 2. Prevent commercial bias and marketing4 | All accredited education | All decisions on planning, faculty selection, delivery and evaluation are made without influence or involvement from owners and employees of an ineligible company. Education is free of marketing or sales. Learner names and contact details are not shared with an ineligible company without the learner's explicit consent. |
| 3. Identify, mitigate and disclose relevant financial relationships5 | All accredited education, with listed exceptions | Collect all financial relationships with ineligible companies in the prior 24 months from everyone in control of content, with no minimum amount. Exclude owners and employees of ineligible companies, with three narrow exceptions. Identify which relationships are relevant, mitigate them before the person takes up the role, document the steps, and disclose to learners before the activity. |
| 4. Manage commercial support appropriately6 | Education receiving financial or in-kind support from ineligible companies | The provider makes all decisions on receipt and disbursement. A written agreement is executed before the education starts. The provider keeps an account of support and its use. Supporters are disclosed to learners by name, without logos, trade names or product messages. |
| 5. Manage ancillary activities7 | Where marketing or non-accredited education accompanies accredited education | Learners can easily tell accredited education from other activities. At live events, marketing and non-accredited sessions do not take place in the educational space within 30 minutes before or after an accredited activity. Educational materials carry no marketing. Ineligible companies may not provide access to, or distribute, accredited education to learners. |
What the standards mean for the needs assessment itself
ACCME's eligibility rules define an ineligible company as one whose primary business is producing, marketing, selling, re-selling or distributing healthcare products used by or on patients. The examples given include pharmaceutical companies, device manufacturers, diagnostic laboratories that sell proprietary products, and advertising, marketing or communication firms whose clients are ineligible companies. Owners and employees of ineligible companies are treated as having unresolvable financial relationships and are excluded from planning and faculty roles, apart from the limited circumstances in Standard 3.2
Four practical consequences follow for the document.
- The provider owns the assessment. The choice of gap, the interpretation of evidence and the objectives are decisions about planning, and under Standard 2 they are made without influence from an ineligible company.
- A supporter's interest area is not a gap. A grant request may be submitted to a company with products in the field, but the gap must stand on practice data and would read the same if no supporter existed.
- Balance starts here. Standard 1 requires a fair and balanced view of diagnostic and therapeutic options. An assessment that frames the gap as under-use of one product has already failed that test. Gaps are framed around patient outcomes and clinical decisions, and treatments are named by generic name and class.
- Authorship of the document is recorded. Anyone who drafts or reviews the assessment is in a position to shape content. Their financial relationships are collected and handled under Standard 3 like those of faculty.
Where writing support fits
The accredited provider directs the work and makes the planning decisions. Anyone engaged to search the evidence or draft the gap analysis works to the provider's instructions, and the provider applies ACCME's eligibility definitions and Standard 3 to that person or firm as it does to faculty.
Section template
- 1. Summary
- The gap, the audience and the proposed activity in one paragraph.
- 2. Target audience
- Professions, specialties and settings, and the estimated number of learners.
- 3. Best practice
- Current guideline and evidence statements with full references and years.
- 4. Current practice and gap
- Data by tier, source and date; the gap stated as a number where possible.
- 5. Causes and educational needs
- Each cause classified; needs labelled knowledge, competence or performance; system barriers noted.
- 6. Learning objectives
- Three to five, each mapped to a need.
- 7. Format and outcomes plan
- Why the format fits; what will be measured and when.
- 8. Independence record
- Who planned, who drafted, financial relationships collected, mitigation steps, and source of any commercial support.
- 9. References
- Numbered, verified against the source.
The same evidence discipline applies to non-accredited scientific documents. Editverse Medical prepares referenced letters, summaries and slide decks through its medical communications service; the corporate medical writing page describes how sources are checked.
What is the difference between a practice gap and an educational need?
The gap is the measured difference between current and achievable practice. The educational need is the reason for the gap that education can address, classified by ACCME as knowledge, competence or performance. One gap can have several needs, and some causes, such as system barriers, are not educational needs at all.
Can a pharmaceutical company write the needs assessment for an accredited activity?
No. Under ACCME Standard 2, decisions on planning, faculty, delivery and evaluation must be made without influence or involvement from the owners and employees of an ineligible company. A company may provide commercial support under a written agreement as set out in Standard 4, with the provider in control of all decisions.
How many references does a needs assessment need?
There is no ACCME number. What matters is that each claim about best practice and each claim about current practice has a source, and that at least one source measures practice in a population close to the target audience.
How far back must financial relationships be collected?
Standard 3 asks for all financial relationships with ineligible companies within the prior 24 months, with no minimum financial threshold, from planners, faculty and others in control of content.
References
- Accreditation Council for Continuing Medical Education. Standards for Integrity and Independence in Accredited Continuing Education. accme.org
- Accreditation Council for Continuing Medical Education. Eligibility. accme.org
- Accreditation Council for Continuing Medical Education. Standard 1: Ensure Content is Valid. accme.org
- Accreditation Council for Continuing Medical Education. Standard 2: Prevent Commercial Bias and Marketing in Accredited Continuing Education. accme.org
- Accreditation Council for Continuing Medical Education. Standard 3: Identify, Mitigate, and Disclose Relevant Financial Relationships. accme.org
- Accreditation Council for Continuing Medical Education. Standard 4: Manage Commercial Support Appropriately. accme.org
- Accreditation Council for Continuing Medical Education. Standard 5: Manage Ancillary Activities Offered in Conjunction with Accredited Continuing Education. accme.org
- Accreditation Council for Continuing Medical Education. Core Accreditation Criteria: Educational Needs. accme.org
- Accreditation Council for Continuing Medical Education. Accreditation Criteria. accme.org
- Moore DE Jr, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009;29(1):1-15. PMID 19288562 ยท DOI