Author: Sarah VanVickle Chaves, Director of IVD and Scientific Writing Services, Ph.D. Molecular and Cellular Biology
For medical device and in vitro diagnostic (IVD) manufacturers, identifying relevant literature is only the first step in a systematic literature review (SLR). Once publications have been found and screened, the evidence still needs to be extracted, critically appraised, weighted, and connected to the regulatory conclusions in the Clinical Evaluation Report (CER) or Performance Evaluation Report (PER).
This is where well-designed data extraction and appraisal tables become essential.
A Notified Body reviewer should be able to move from a statement in the CER or PER back through the evidence and understand: Which study supports the conclusion? What population and device were studied? What were the relevant outcomes? How reliable was the evidence? How applicable is it to the device under evaluation? And how much weight did the manufacturer give it?
Under the EU Medical Device Regulation (MDR), manufacturers must identify clinical evidence through a systematic scientific literature review, appraise all relevant clinical data for their suitability, and analyze those data to reach conclusions about safety, clinical performance, and clinical benefit.
For IVDs, the In Vitro Diagnostic Medical Devices Regulation (IVDR) similarly requires manufacturers to identify relevant data through systematic scientific literature review, appraise those data, and assess the totality of scientific validity, analytical performance, and clinical performance evidence.
Neither is a spreadsheet containing only author, year, and study title. A regulatory literature review needs to show what the evidence says, how it was evaluated, and why it deserves the weight assigned to it.
Why Data Extraction and Appraisal Matter Under the MDR
MDR Annex XIV establishes three distinct steps relevant to literature evidence:
- Identify available clinical data and clinical evidence gaps through systematic scientific literature review.
- Appraise all relevant clinical data by evaluating their suitability for establishing device safety and performance.
- Analyze all relevant clinical data to reach conclusions about safety, clinical performance, and clinical benefits.
Those are not interchangeable activities.
A publication can be relevant enough to pass literature screening but still provide weak evidence for a particular regulatory conclusion. For example, an article may discuss the appropriate device technology and indication but include only a small patient population, have limited follow-up, use an outdated version of the device, report outcomes that do not correspond to the manufacturer’s claims, provide insufficient information about adverse events, contain significant methodological limitations, or study a population that differs substantially from the intended population.
The study may still be relevant. But it should not necessarily receive the same evidentiary weight as a well-designed prospective investigation directly evaluating the subject device.
This distinction between relevance, methodological quality, applicability, and evidentiary weight is precisely what appraisal tables should make visible.
What Does the Notified Body Actually Assess?
Under MDR Annex VII, a Notified Body’s clinical evaluation assessment includes examination of the manufacturer’s appraisal and analysis of the available data and its relevance to demonstrating conformity with the applicable General Safety and Performance Requirements (GSPRs). The Notified Body assessment also covers literature-search methodology and documentation, equivalence, clinical investigations, post-market surveillance, PMCF, and the conclusions of the CER.
That means the reviewer is not simply checking whether literature exists. The reviewer needs to evaluate whether the manufacturer has used the literature appropriately.
TÜV SÜD similarly describes clinical evaluation as a systematic and planned process involving the continuous generation, collection, appraisal, analysis, and evaluation of clinical data. It emphasizes that the clinical evaluation should be thorough and objective and consider both favorable and unfavorable evidence.
BSI provides best-practice guidance addressing clinical evaluation submissions under MDR and emphasizes well-prepared clinical evaluation documentation for Notified Body review.
A practical goal of the extraction and appraisal tables, therefore, is to make the manufacturer’s reasoning auditable.
Data Extraction and Critical Appraisal Are Different
Data Extraction: What Did the Study Report?
Data extraction captures the relevant factual characteristics and results of the publication, such as study design, device studied, indication, patient population, sample size, comparator, follow-up, endpoints, effectiveness/performance results, adverse events, complications, and author conclusions.
Critical Appraisal: How Much Confidence Should We Place in It?
Appraisal evaluates the quality and usefulness of those data for the regulatory question. Relevant dimensions may include methodological quality, appropriateness of study design, risk of bias, adequacy of sample size, relevance of the population and device, adequacy of follow-up, clinical relevance of endpoints, statistical considerations, applicability to the intended purpose, relevance to safety/performance claims, and overall evidentiary weight.
What Should a Regulatory Data Extraction Table Include?
There is no single MDR-mandated spreadsheet template that every manufacturer must use. The fields should instead be driven by the device, intended purpose, clinical evaluation plan, regulatory claims, risks, and literature-review questions. Several categories are generally useful.
Publication Identification
- unique study/reference ID
- first author
- publication year
- complete citation
- DOI or PMID
- country or geographic region
- publication type
- literature search from which the publication originated
A unique identifier can follow the publication through search results, screening, extraction, appraisal, CER discussion, and the reference list, improving traceability.
Study Design
- randomized controlled trial
- prospective cohort
- retrospective cohort
- case-control study
- registry
- case series
- diagnostic accuracy study
- systematic review
- meta-analysis
- expert consensus or other design
Simply recording “clinical study” provides little information about evidentiary strength.
Device Identification
- device name
- manufacturer
- model/version, where reported
- device generation
- generic technology
- subject device, equivalent device, similar device, or other technology
- relevant technical characteristics
Device evidence is often version-dependent. For equivalent and similar devices, the relationship to the subject device should be explicit.
Population and Clinical Context
- indication
- disease severity
- inclusion and exclusion criteria
- age
- sex, where relevant
- relevant comorbidities
- anatomical characteristics
- baseline disease status
- previous treatment
- clinical setting
A study involving the correct device but the wrong population may have limited applicability to the CER.
Sample Size and Analysis Population
- total enrolled
- total treated/tested
- total analyzed
- losses to follow-up
- relevant subgroups
- reasons for major differences between enrollment and analysis populations
For IVDs, specimen counts, disease distribution, specimen type, and reference method may also be important.
Comparator or Reference Standard
- comparator device
- control group
- standard of care
- alternative intervention
- predicate technology
- reference method
- clinical reference standard
These data are especially important when establishing state-of-the-art benchmarks.
Follow-Up Duration
- planned follow-up
- actual follow-up
- mean/median follow-up
- follow-up range
- completeness of follow-up
Follow-up duration should be appropriate to the clinical risks and expected device lifetime.
Safety Outcomes
- adverse events
- serious adverse events
- device-related adverse events
- procedure-related events
- complications
- device deficiencies
- revisions
- explants
- reoperations
- mortality
- discontinuations
- newly identified or unexpected risks
A table that extracts performance while omitting adverse findings may create an incomplete evidence picture.
Performance and Clinical Benefit Outcomes
- technical success
- clinical success
- symptom improvement
- functional outcomes
- quality-of-life measures
- diagnostic accuracy
- sensitivity
- specificity
- survival
- freedom from reintervention
- recurrence
- healing
- hospitalization
The fields should align with the clinical evaluation plan, claims, and predefined safety and performance objectives.
What Changes for an IVDR PER?
For IVDs, the extraction framework should reflect the three pillars of performance evaluation under IVDR Article 56: scientific validity, analytical performance, and clinical performance. A single generic extraction table may not be optimal for all three.
Scientific Validity Evidence
For literature supporting scientific validity, consider extracting:
- analyte or marker
- associated clinical condition or physiological state
- biological/clinical rationale
- population
- study design
- strength of association
- relevant statistical results
- professional guideline or consensus support
- author conclusions
IVDR Annex XIII permits scientific validity to be supported through sources including peer-reviewed literature, information concerning devices measuring the same analyte or marker, professional consensus positions, proof-of-concept studies, and clinical performance studies.
Clinical Performance Evidence
- intended-use population
- specimen type
- comparator/reference method
- disease prevalence
- sensitivity
- specificity
- positive predictive value
- negative predictive value
- likelihood ratios
- ROC/AUC results
- confidence intervals
- cut-off values
- indeterminate results
- subgroup analyses
The amount and quality of evidence should support a qualified assessment of intended clinical benefit and safety. Quality and applicability—not merely quantity—are central to the evidence assessment.
Building the Appraisal Table
Once relevant data have been extracted, the next question is: How good is the evidence? A strong appraisal framework should evaluate several distinct dimensions rather than reducing every publication to an unexplained numerical score.
Methodological Quality
- appropriateness of study design
- prospective vs. retrospective design
- randomization
- blinding
- control of confounding
- selection bias
- completeness of follow-up
- outcome ascertainment
- appropriateness of statistical analysis
- reporting completeness
- potential conflicts of interest
The appropriate criteria should reflect the study design. A randomized trial should not necessarily be evaluated with exactly the same criteria as a retrospective registry or diagnostic accuracy study.
Scientific Validity
- Are endpoints clearly defined?
- Are statistical methods appropriate?
- Is sample size adequate?
- Are conclusions consistent with the results?
- Are important confounders addressed?
- Is follow-up sufficient for the outcome?
- Is there evidence of selective reporting?
Relevance to the Device Under Evaluation
Device relevance: Is the subject device, an equivalent device, a similar device, or another technology being studied?
Population relevance: Does the population correspond to the intended population?
Indication relevance: Is the clinical condition the same?
Endpoint relevance: Do the outcomes correspond to the manufacturer’s claims and clinical evaluation objectives?
Use relevance: Was the device used in a manner consistent with the intended purpose?
Contribution to Safety and Performance
A useful field is what the publication actually contributes to the clinical evaluation—for example, safety, clinical performance, clinical benefit, state of the art, adverse-event characterization, risk estimation, equivalence, benchmark development, or multiple categories.
What Should the Appraisal Output Look Like?
| Appraisal Dimension | Assessment |
| Study design | Prospective multicenter cohort |
| Methodological quality | High |
| Device relevance | Direct – subject device |
| Population relevance | High |
| Endpoint relevance | High |
| Follow-up adequacy | Moderate |
| Safety-data contribution | High |
| Performance-data contribution | High |
| Key limitations | 12-month follow-up; no randomized comparator |
| Overall evidentiary weight | High |
The exact terminology is less important than having a predefined and consistently applied methodology. A “high-quality” rating should mean the same thing from one publication to another.
Avoid the Black-Box Appraisal Score
A frequent weakness is assigning a publication a score such as “Appraisal score: 7/10” without explaining what 7 means. Questions immediately arise about which criteria generated the score, whether all criteria were weighted equally, what threshold is acceptable, and whether another reviewer could reproduce the result.
A numerical scoring system can be useful, but only when its logic is transparent. In many cases, a structured qualitative assessment of quality, relevance, and contribution communicates more than a single composite score.
Separate Quality From Relevance
This is one of the most important principles in regulatory appraisal. Consider two hypothetical studies:
Study A
- randomized controlled trial
- 500 patients
- excellent methodology
- evaluates a different device technology
Study B
- prospective single-arm study
- 120 patients
- moderate methodological quality
- evaluates the subject device for the exact intended indication
Study A may provide stronger evidence for the state of the art. Study B may provide more directly applicable evidence for subject-device safety and performance. If a single appraisal score combines quality and relevance, this distinction may disappear. A stronger framework evaluates them separately.
Do Not Hide Unfavorable Evidence
The MDR explicitly requires the clinical evaluation to be thorough and objective and to account for favorable and unfavorable data. The IVDR applies the same principle to performance evaluation.
Accordingly, extraction should capture unexpected adverse events, poor performance, device failures, negative studies, conflicting results, subgroup concerns, author-reported limitations, and conclusions that do not support the manufacturer’s desired narrative.
The purpose of appraisal is not to make unfavorable evidence disappear by assigning it a low score. Instead, the manufacturer should determine whether the evidence is methodologically credible, whether it applies to the device, and what it means for safety, performance, benefit-risk, risk management, labeling, PMCF/PMPF, or the clinical evidence strategy.
Link Appraisal to Evidence Weighting
Not every publication should influence the clinical evaluation equally. The appraisal methodology should therefore lead logically to evidence weighting.
Higher Weight
- strong methodology
- direct subject-device relevance
- intended-use population
- clinically relevant endpoints
- adequate sample size
- appropriate comparator
- sufficient follow-up
- complete reporting
Moderate Weight
- subject-device evidence with methodological limitations
- highly relevant similar technology
- smaller population
- useful safety information but limited performance data
- meaningful contribution to state-of-the-art characterization
Lower Weight
- substantial methodological limitations
- materially different device
- different indication
- minimal clinical detail
- primarily expert opinion
- limited applicability to the regulatory question
The weighting criteria should be established before the manufacturer knows whether a study’s results are favorable or unfavorable. Otherwise, evidence weighting can become an unintended mechanism for confirmation bias.
The Appraisal Table Should Connect to the CER or PER
A beautifully constructed spreadsheet that never connects to the clinical evaluation conclusions has limited regulatory value. The evidence pathway should remain traceable from literature search through screening, extraction, appraisal, weighting, synthesis, safety/performance conclusions, benefit-risk assessment, GSPR support, and PMCF/PMPF decisions.
A Practical Example
Consider a hypothetical implantable medical device. The manufacturer identifies 35 relevant publications. A strong extraction and appraisal process might classify them as follows:
| Evidence Group | Primary Contribution |
| Subject device – 8 publications | Direct safety and performance |
| Equivalent device – 5 publications | Supporting clinical evidence |
| Similar devices – 10 publications | SOTA, risks, benchmark outcomes |
| Alternative treatments – 7 publications | SOTA and benefit-risk context |
Guidelines/systematic reviews
– 5 publications
SOTA and clinical practice
Within the eight subject-device publications, the appraisal might determine that three provide high-weight evidence, three provide moderate-weight evidence, and two provide lower-weight evidence. The CER should not simply conclude that “eight publications support the device.” It should explain how the quality and applicability of those studies affect the totality of evidence.
Common Problems That Can Trigger Reviewer Questions
Problem 1: Extraction Table Contains Only Bibliographic Information
Why it matters: The reviewer cannot determine what evidence the publication contributes.
Better approach: Extract study design, population, device, endpoints, safety, performance, follow-up, and relevant results.
Problem 2: No Documented Appraisal Methodology
Why it matters: Evidence weighting can appear arbitrary.
Better approach: Define appraisal criteria before conducting the assessment and apply them consistently.
Problem 3: All Studies Receive the Same Weight
Why it matters: A retrospective case series and a well-designed prospective investigation do not necessarily provide equivalent evidence.
Better approach: Evaluate methodological quality, relevance, and contribution separately.
Problem 4: Device Relevance Is Unclear
Why it matters: The reviewer cannot distinguish subject-device evidence from equivalent, similar, or unrelated technology.
Better approach: Explicitly classify the device relationship for every publication.
Problem 5: Only Positive Results Are Extracted
Why it matters: This conflicts with the expectation for objective evaluation of favorable and unfavorable evidence.
Better approach: Extract relevant adverse, negative, and conflicting findings using the same methodology applied to favorable evidence.
Problem 6: Extraction and CER Numbers Do Not Match
Why it matters: Traceability is weakened when study-level data and the CER narrative differ without explanation.
Better approach: Implement quality control between source publications, extraction tables, evidence synthesis, and final regulatory documentation.
Problem 7: Tables Are So Large That They Become Unusable
Why it matters: More columns do not automatically mean better extraction.
Better approach: Extract what is necessary to answer the predefined clinical/performance questions and use separate modules where evidence types differ materially.
What Does a Notified Body Want to Be Able to Trace?
A useful way to design an extraction and appraisal framework is to imagine the reviewer selecting one major CER conclusion—for example, “The device demonstrates an acceptable rate of Procedure-Related
Complication X.” The reviewer should then be able to trace that statement backward through the publications, complication rates, definitions, follow-up periods, device relationships, methodological reliability, assigned weight, state-of-the-art benchmark, contradictory evidence, and the manufacturer’s final interpretation.
If the documentation can answer those questions without forcing the reviewer to reconstruct the entire analysis, the evidence package becomes substantially easier to assess.
Do Not Forget the Lifecycle
Extraction and appraisal tables should be designed for updating—not just initial submission. MDR clinical evaluation is a lifecycle process, and IVDR similarly requires clinical evidence and its assessment within the PER to be updated throughout the device lifecycle, including information generated through PMPF.
Useful lifecycle fields can include search/update cycle, date identified, previous/new publication, appraisal status, reviewer, QC reviewer, evidence category, and CER/PER version incorporated. This turns the extraction table into a maintainable evidence-management tool.
A Practical Checklist for Your Extraction and Appraisal Tables
| Question | What the Reviewer Should Be Able to See |
| Can every publication be uniquely identified? | Citation and unique reference ID |
| Is the device studied clear? | Device, model/version, manufacturer, relationship to subject device |
| Is the study population relevant? | Indication, demographics, disease characteristics |
| Is study design clear? | Design, sample size, comparator, follow-up |
| Are relevant outcomes extracted? | Safety, performance, clinical benefit |
| Are unfavorable findings captured? | Adverse events, failures, negative/contradictory results |
| Was methodological quality assessed? | Predefined quality criteria |
| Was regulatory relevance assessed? | Device, population, indication and endpoint relevance |
| Is evidence weighting transparent? | High/moderate/low or another predefined approach |
| Are limitations visible? | Study-specific limitations and applicability concerns |
| Can results be traced into the CER/PER? | Clear evidence IDs and synthesis references |
Can the table be updated? Versioning/search-cycle information
The Bottom Line
A Notified Body does not need an extraction table merely to prove that someone read the articles. The purpose is to demonstrate control over the evidence.
A defensible systematic literature review should make it possible to understand what evidence was identified, what each study reported, how reliable each source is, how relevant it is to the device and intended purpose, how much evidentiary weight it deserves, how favorable and unfavorable findings were handled, and how the totality of evidence supports the CER or PER conclusions.
MDR Annex XIV requires manufacturers to systematically identify, appraise, and analyze relevant clinical data. MDR Annex VII makes the appraisal and analysis of those data—and their relevance to conformity assessment—part of the Notified Body’s clinical evaluation assessment. For IVDs, IVDR Article 56 and Annex XIII similarly require a methodologically sound performance evaluation based on the assessment of relevant scientific validity, analytical performance, and clinical performance data.
How Criterion Edge Can Help
Systematic literature reviews for medical devices and IVDs require a combination of literature-search expertise, clinical interpretation, evidence appraisal, and regulatory writing.
Criterion Edge supports manufacturers with systematic literature review and clinical/performance evaluation activities, including evidence identification, screening, data extraction, critical appraisal, evidence weighting, state-of-the-art assessment, and integration of literature evidence into CER and PER documentation.
A structured extraction and appraisal methodology can make the resulting evidence package easier to review, easier to update, and easier to defend during conformity assessment.
References and Regulatory Resources
- Regulation (EU) 2017/745 (EU MDR) – Article 61 and Annex XIV, Clinical Evaluation and Post-Market Clinical Follow-up. Source
- Regulation (EU) 2017/745 (EU MDR) – Annex VII, requirements for Notified Body assessment of clinical evaluation. Source
- Regulation (EU) 2017/746 (EU IVDR) – Article 56, Performance Evaluation and Clinical Evidence. Source
- Regulation (EU) 2017/746 (EU IVDR) – Annex XIII, Performance Evaluation, Performance Studies and Post-Market Performance Follow-up. Source
- European Commission – MDCG guidance documents and other guidance. Source
- BSI – Best Practice Guidance for Clinical Evaluation Submission. Source
- BSI – Preparing a Clinical Evaluation Report. Source
- TÜV SÜD – Clinical Evaluation of Medical Devices. Source
- TÜV SÜD – Clinical Data for Medical Devices. Source
- TÜV SÜD – IVDR Annex XIII: Performance Evaluation, Performance Studies and PMPF. Source
This article is intended for general informational purposes and does not constitute regulatory or legal advice. Manufacturers should determine the evidence requirements and appraisal methodology appropriate to their specific device, intended purpose, risk classification, claims, conformity assessment route, and regulatory strategy.
