Majority of data informing NICE’s treatment appraisals of ‘poor quality’

Poor quality data has been used as the basis for two thirds of NICE’s appraisals for new treatments, according to a new study published in BMJ Open.

The study also found that there has been no improvement in the quality of data during the last 20 years, and it has remained ‘consistently poor’.

Study analysed data used for NICE’s appraisals over a 20-year period

NICE was set up in 1999 by the Government to decide which treatments are available on the NHS in England. Its appraisal committee advises the NHS on the clinical benefit and cost-effectiveness of both new and existing technologies by assessing clinical data.

The appraisal process involves advice from patient experts and clinical consultees, as well as reports from independent technology assessors. Changes have been made in recent years to also include survey data, national statistics and expert opinions.

While previous research has scrutinised NICE’s approval process, no previous studies have systematically analysed all NICE’s active technology appraisals since its inception more than 20 years ago.

Researchers from the London School of Economics and Political Science therefore set out to analyse all NICE’s active technology appraisals published between 2000 and 2019.

They scrutinised the clinical evidence submitted by the manufacturers and assessed its quality for decision making purposes.

The data was extracted from the independent assessment group and evidence review group reports and final appraisal determinations on the quality of submitted randomised controlled clinical trials and the overall quality of evidence submitted for decision-making.

For single technology appraisals (STAs) they also extracted data on quality of life evidence and comparative clinical evidence.

Quality of evidence found to be good in only 1% of cases

The evidence for 409 technology appraisals was analysed, including 104 multiple technology appraisals (MTA) and 305 single technology appraisals (STA).

The appraisals included 384 drugs, 25 non-pharmaceutical products, 14 medical devices, six other therapies, and five surgical procedures.

Each category was scored for quality: 2 for good; 1 for acceptable; 0 for poor; and −1 for unacceptable.

In two thirds (65%) of all appraisals, the overall quality of evidence was judged to be either poor (55%) or unacceptable (10%).

The quality of evidence was judged acceptable in a third (34%) of appraisals or good in only 1%.

Evidence quality has not improved over time

The researchers say the overall quality of evidence submitted to NICE has not changed over the last 20 years and has remained consistently poor.

While weak or insufficient evidence from poorly conducted clinical trials was often used because it was the only evidence available, the authors of the study say even when trials had been done well, the comparators were often ‘unsuitable for decision-making in the NHS context’.

For example, comparator data often did not reflect the UK population and routine treatment pathways. Indeed, indirect comparisons were used in 68% of STAs to establish the comparative clinical effectiveness of interventions.

They added that quality of life data was often of poor quality and clarity in reporting methodology and details by both manufacturers and assessment bodies varied significantly.

Drug manufacturers must put ‘more effort’ into generating high-quality evidence

While the researchers acknowledge various limitations to their study, including the subjective scoring system used for grading the quality of evidence, they highlight that the scores were peer reviewed by all the members of the research team to try and minimise bias.

They conclude: “We found that the primary components of clinical evidence … that influence patients and are crucial for NICE’s decision making framework are of poor quality.

“Since the evidence bar continues to be lowered, it is essential to have [health technology assessment] bodies and payers’ input to ensure that the generation of evidence submitted to NICE is strengthened.

“However, it is essential that stakeholders are aware of this and that organisations put more effort into generating high-quality evidence premarket and post-market entry.”

NICE questions why validated frameworks were not used to grade evidence

Dr Jacoline Bouvy, Programme Director, Medicines Evaluation at NICE, told PHT that while NICE ‘strongly agrees’ with the need for high quality evidence within health technology assessments, the committee does not agree that NICE’s evidence bar ‘continues to be lowered’.

She said: “NICE technology appraisals are founded on principles of independence, transparency and rigour that allow NICE to help practitioners and commissioners get the best care to patients fast while ensuring value for the taxpayer.

“The NICE methods manual is clear that despite calling for evidence of the highest possible standard, limitations in the evidence available to inform an evaluation are likely and sometimes inevitable. NICE committees still have to make recommendations in such cases, and can do so when limitations in the evidence are fully described, the risk of bias is characterised, and the resulting uncertainty in the evidence is fully considered.”

“NICE further highlights that Osipenko et al. have used a novel rating system for assessing evidence quality within NICE technology appraisal evaluations. Limited explanation is given around the development, testing or validation of their methodology or why widely used validated frameworks such as the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool or Cochrane Risk of Bias Tool (RoB 2) were not considered relevant or applicable.”