Real-World Evidence: A Reality Check

Real-world evidence (RWE) is a buzzword, and there’s nothing that American healthcare likes more than the zzzzz of a good buzzword.

But the buzz of RWE has some substance to it. RWE research is changing how medications and medical devices are approved and paid for. It’s nipping at the heels of the randomized clinical trial that remains the standard by which all medical research is judged. And RWE proponents are covetously eyeing the mountains of data that patient Fitbits, Apple watches and other devices could produce.

Last year Pfizer used RWE to win expanded approval for its breast cancer drug Ibrance (palbociclib), adding male breast cancer treatment as an indication. The FDA accepted the analysis of data from electronic health records (EHRs) and medical databases as proof of Ibrance’s efficacy. The growing acceptance of RWE is due, in part, to former FDA commissioner Scott Gottlieb, M.D., who welcomed using data from outside of clinical trials.

Over the past four years, Jeff Morgan, a managing director at Deloitte Consulting LLP, has seen a significant increase in RWE in life sciences. Morgan, who leads Deloitte’s RWE unit, called ConvergeHealth, sees it as part of a trend of people becoming more engaged in their healthcare and the data associated with it. “There [are] a lot more patient data available than ever [before].”

RWE may have some financial advantages for drug developers. Traditional research, anchored in clinical trials, is becoming increasingly expensive. A study published in the March 3, 2020, issue of JAMA investigated the costs of developing 63 of 355 new therapeutic drugs and biologic agents that the FDA approved between 2009 and 2018. The researchers calculated that the research cost per product was $985 million, a figure that includes expenditures on failed trials. Experts debate estimates such as those, and there are always questions about which costs should be included. But no one would question that drug development is an expensive proposition.

Proponents of RWE say it will yield information that is both more targeted and more thorough than the evidence that clinical trials can provide. But that doesn’t mean RWE research is easy to do — or inexpensive, notwithstanding the relative cost of the randomized trial. What’s more, EHRs and insurance claims may contain a wealth of information, but neither are designed for medical research. And if wearable devices were to live up to merely half of their hype, even more data would come pouring into the mix. Collecting, organizing and understanding data requires skill, time and money, and many current computer systems just aren’t up to dealing with that data firehose.

Critical to value-based pricing

Today RWE is predominantly used by the pharmaceutical industry to help demonstrate a product’s effectiveness to both prescribers and payers when it is used outside a controlled clinical trial. Patients who are enrolled in clinical trials of a drug may differ from those who are prescribed a medication after it has been on the market, notes Morgan. Those patients are often older than the study participants, and they tend to have more comorbidities.

RWE has been used for trial design, trial recruitment and marketing insights, says Dan Riskin, M.D., CEO of Verantos, an RWE technology company in Menlo Park, California. “None of these uses makes clinical assertions, saying one drug is better than another,” or that treatment with drug A will result in 20% better outcomes, he says. He considers the use of RWE to make clinical assertions as an “advanced use” case, when investigators pull data from multiple sources including EHR unstructured data, wearables, and the like. Advanced RWE might be used to expand indications and provide evidence-based results for use in value-based care, says Riskin, whose company provides these types of services.

A 2018 McKinsey & Co. report on RWE said that pharmaceutical companies have entered a second phase of RWE research that involves using it in a more integrated fashion. Companies have created centralized RWE teams that are often situated in their medical affairs departments. But McKinsey said there is still a problem in the industry of RWE expertise being scattered throughout the companies.

In addition to companies wanting other methods to prove safety and efficacy, payers are eyeing RWE because of drug costs. “The pressure to demonstrate the value of the product is of the highest importance,” says Morgan. “We will see a transformation in the reimbursement model from the traditional price-per-pill model. We’re seeing a lot of adoption and experimentation of value-based pricing.” And RWE research is critical to value-based pricing because it reveals the outcomes a drug produces once it is “in the wild” and on the market, in contrast to the tightly controlled environment of a clinical trial.Many value-based contracts hinge on whether there is a significant difference between the results a drug produced in clinical trials and those it produces in a particular payer’s population.

Armies of extractors

The amount of data that can be collected on patients is exploding. In some cases, the quality of the analysis is not keeping pace with the quantity of the data. It was simpler with “old school” RWE that relied on claims data, which Riskin says have their virtues. They’re readily available, and the databases include millions of people. But, he says, “[The data are] not very accurate. [They’re] not very rich.”

The FDA made great strides in collaborating with industry to arrive at an understanding of quality RWE. “If you look at their most recent framework in RWE, data validity, it’s trying to get at what is believable,” Riskin says. It’s a difficult task, he continues, one that requires “rich” data sources that include the narrative sections of doctor reports, representing what he says are 80% of the EHRs. According to Riskin, some large oncology companies are including that narrative data. “It’s rich information but it’s an expensive version of RWE.” The data must be moved and extracted, enriched and then tested for accuracy.

Although medical records data aggregators aren’t new, says Morgan, “we’re seeing a lot of experimentation with unstructured data and how you curate that.” Data curation includes machine learning and natural language processing to extract data from the written or typed narrative. Some organizations hire armies of human abstractors to manually abstract the data, he says. “There is no perfect solution.”

Better data access and formatting would be welcome, but each data set does what it’s supposed to do, says Riskin. “Claims data is perfectly situated for decisions on claims reimbursement. It’s not designed to help someone run a study,” he says. Doctor narratives are intended to be a quick reference to the patient’s important problems and what has been done to address them. A researcher would want to know more about the patient’s medical conditions, what was prescribed or tried before, and the doctor’s thought process for choosing one treatment over another.

Data are often messy, but those with expertise in knowing how to control for certain parameters or conditions can arrive at some accurate inferences, Morgan says. But there are plenty of pitfalls. Riskin points to standardized data fields, such as checked boxes, as an example. Consider when patients with cancer or heart failure are asked whether they smoke. If only 10% of smokers accurately mark the box for positive smoking status, is it fair to treat these boxes as valid data, Riskin asks. “I’ve seen entire models made on that one field where an engineer thought it was (accurate) because they checked the box.” But peer reviewed articles show that for conditions such as cancer and heart failure, the rate of filling out that box correctly is less than 50%.

Each type of medical information provides a piece of the puzzle. Morgan says there’s growing interest among RWE researchers in finding ways to link the data sources at the patient level by, for example, matching up claims data and health records that would weave together clinical and economic information. Those links can also be to national registries and patient-reported outcomes. “This is a situation where advanced RWE has a lot of room to grow,” says Riskin. With so much information available, it has to be done the right way, with a high-quality protocol, institutional review board approval when necessary, and compliance with patient privacy rules.

Considering quality

The FDA is still working on its guidance for RWE. A 2018 framework established some ways to evaluate RWE in support of new drug indications or post-approval study requirements, but the agency expects to issue a more complete guidance for RWE in the next year, says Riskin. The guidance will, no doubt, be influential in setting expectations for RWE. Still, researchers, drugmakers, clinicians and payers will probably be battling it out over RWE and the quality of the evidence it produces. Making those assessments may start from the perspective of evaluating studies of variable quality, and determining what is believable and what isn’t, according to Riskin. “(Verantos) put a stake in the ground to decide what we think is believable,” he says of his company. “We are very happy that regulators are now involved. They make it their business to determine what is believable.” For Verantos, a quality RWE study has a protocol measuring data accuracy and ensuring minimum requirements for that accurate measurement. With that, “you can create a high-quality study. You will have to do a lot to achieve that.”

RWE research may vary with the study’s objective. “The FDA’s term is ‘fit for purpose.’ It means that depending on the purpose, that determines what the validity of the data is to be,” says Riskin, noting that the data required for marketing insights are not the same as those required for regulatory purposes.

Cloud-based, rolling in

In the next few years, Morgan anticipates a massive shift in use. “There is great promise in these data, which can drive efficiencies in research and development. We can design trials that are more patient-centric, to decrease the burden on patients.”

In addition to EHRs, the data for RWE research will come from daily at-home measurements, such as heart and respiration rates. “It’s just going to help start answering questions that a few years ago you couldn’t answer,” Morgan says. Instead of an EHR providing specific patient information a dozen times a year, or less, at visits, the information could be streaming daily. That will strain current information systems; most are not yet set up to receive this data volume. Healthcare organizations will need to move from on-premise legacy systems to cloud systems.

As cloud-based platforms are continually adopted, the data will be analyzed by more than just data scientists, says Morgan. He explains that data analytics are being democratized by increasingly accessible technology, allowing more stakeholders to work with them.

The future involves providing therapies targeted for an individual’s situation. If a 45-year-old woman has hypertension and a history of diabetes, “I want to know what works well for other people like her,” Riskin says. That may be in the future, but that’s where this is heading, he says.