Cite this article

Cao Han, Yao Chen*, Yan Xiaoyan, Yu Yongpei, Shang Meixia. Exploration of Design Types and Statistical Analysis Methods of Licensed Medical Device Clinical Research Based on Boao Lecheng Real World Data [J]. China Food and Drug Administration. 2021.11(214): 6-13.

Exploration of Design Types and Statistical Analysis Methods for Clinical Research on Licensed Medical Devices Based on Boao Lecheng Real World Data

Study Design and Statistical Analysis for Clinical Research of Special Innovative Medical Devices Based on Real-World Data from Boao Lecheng

Cao Han

Beijing University First Hospital

CAO Han

Peking University First Hospital

Yao Chen*

Peking University First Hospital, Peking University Clinical Research Institute

Hainan Real World Data Research Institute

YAO Chen*

Peking University Clinical Research Institute, Peking

University First Hospital

Hainan Institute of Real World Data

Yan Xiaoyan

Peking University Clinical Research Institute

YAN Xiao-yan

Peking University Clinical Research Institute

Yu Yongpei

Peking University Clinical Research Institute

YU Yong-pei

Peking University Clinical Research Institute

Shang Meixia

Beijing University First Hospital

SHANG Mei-xia

Peking University First Hospital

Abstract

The demand for using real-world evidence from the Hainan Boao Lecheng International Medical Tourism Pilot Zone to support the domestic approval and listing of licensed medical devices is increasing, but the guiding principles for domestic and foreign real-world data to be used for medical device regulatory decision-making are very limited for Lecheng. How to carry out scientific and reasonable research design according to the characteristics of Lecheng and adopt corresponding rigorous statistical analysis has become an urgent problem to be solved in the real world data research of Lecheng. This article will analyze the key characteristics of Lecheng's real-world data research, and explore the types of research design and statistical analysis methods suitable for Lecheng based on these characteristics.

There has been increasing demand for using real-world data from Boao Lecheng to support approval of special innovative medical devices. However, existing guidance on using real-world evidence to support regulatory decision-making for medical devices has limited relevance to the situation in Lecheng . How to create scientific and reasonable study design and perform rigorous statistical analysis according to the characteristics of Lecheng has become an urgent problem. In the present study, we analyzed characteristics of real-world study in Lecheng, and explore the suitable study design and statistical analysis analysis methods.

Keywords Key words

Boao Lecheng; licensed medical devices; real-world data; type of study design; statistical analysis methods

Boao Lecheng; special innovative medical devices; real-world data; study design; statistical analysis

Real-world data (RWD) refers to various data related to patient health status and/or diagnosis and treatment and health care that are collected daily [1-3], and real-world research (real-world research) generated from it. -world study, RWS) and real-world evidence (RWE) value to the regulation of medical devices has also received extensive attention from researchers and regulators. From 2017 to 2018, the U.S. Food and Drug Administration (FDA) successively issued the "Use of Real-World Evidence to Support Regulatory Decision Making for Medical Devices" (Use of Real-World Evidence to Support Regulatory Decision Making for Medical Devices). 2] and "FDA's Real-World Evidence Program" (Framework for FDA's Real-World Evidence Program) [1], my country's National Medical Products Administration (NMPA) released in 2020 "Real-world data for medical treatment". Technical Guiding Principles for Device Clinical Evaluation (Trial) [3], which provide a basic framework for the exploration of using RWD/RWE to evaluate the effectiveness of medical devices.

Hainan Boao Lecheng International Medical Tourism Pilot Zone is one of the Hainan Free Trade Port strategies. In 2018, the State Council entrusted the Hainan Provincial Government with a unique policy on the licensed use of imported medical devices and drugs that are urgently needed for clinical use in Lecheng, making Lecheng the only area in my country that can use licensed medical devices that have been approved for marketing abroad and unregistered in China. At present, Lecheng has carried out real-world data research on two batches of medical device products, which has received extensive attention from medical enterprises and the society. Lecheng has unique policy support, and the guiding principles of relevant real-world data at home and abroad for drug and device regulatory decision-making are very limited for Lecheng. How to carry out scientific and reasonable research design according to the characteristics of Lecheng and adopt corresponding rigorous statistical analysis has become an urgent problem to be solved in the real world data research of Lecheng. key to conversion. This article will analyze the key characteristics of Lecheng's real-world data research, and explore the types of research design and statistical analysis methods suitable for Lecheng based on these characteristics.

1. The key features of Lecheng's real-world data research on licensed medical devices

Due to its unique policy support, as well as the characteristics of medical devices, patient sources, diagnosis and treatment locations, and data collection [4-5], Lecheng conducts real-world data research with the following three key characteristics: (1) Licensed medical devices have been approved abroad Listed on the market, there are foreign clinical data, and successful clinical application experience has been obtained, and there is a wealth of previous evidence; ② The real-world data research conducted by licensed medical devices in Lecheng is a supplementary material to support its domestic registration and listing. Domestically, the effectiveness and safety are comparable to those of foreign countries, and there is no need to repeat large-scale clinical research under the premise of ensuring the quality of research; ③ The use of licensed medical devices in Lecheng is for patients who urgently need such devices for diagnosis and treatment, and it is difficult to Randomization and establishment of concurrent internal controls usually only generate a single group of clinical indicator data for licensed medical devices. The above characteristics determine that the real-world data research carried out in Lecheng is different from the thinking framework of real-world data research under normal circumstances in other regions.

2. Design type suitable for real-world data research of Lecheng licensed medical devices

According to the three key characteristics of the real-world data research of Lecheng licensed medical devices, that is, it has overseas clinical research evidence, only domestic evidence is required as supplementary materials for registration, and only a single set of clinical index data for licensed medical devices can be generated in the real diagnosis and treatment environment of Lecheng. , single-group target value studies and single-group studies based on external controls are two suitable types of research designs.

2.1 Study on single group target value

A single-group target value study refers to a method to evaluate the effectiveness and safety of a product by examining whether the results of the main evaluation indicators of the product are within the pre-specified target value range through a single-group clinical study without concurrent control. A single set of objective performance criteria (OPC) refers to the standard that the product efficacy or safety indicators recognized in the professional field should meet [6], generally through clinical trial regulatory guidelines, industry standards or expert consensus and the history of similar products. The results of the study were determined [7]. In view of the characteristics that Lecheng licensed medical devices already have overseas clinical application experience, and it is difficult to randomize and set up internal controls over the same period, the evaluation indicators and (or) guidelines of overseas clinical studies are used as OPC effectiveness studies or observational studies, which can be used as Alternative strategies for pragmatic randomized controlled studies to provide key evidence for domestic registration of licensed medical devices. However, it should be noted that there is no parallel control in the single-group target value study, and confounding bias is difficult to control. In addition, when evaluating the correlation between adverse events and products, as well as the incidence of adverse events, there is a lack of reference and evidence from the control group [7]. Therefore, single-arm target value studies are more suitable for the evaluation of licensed medical devices with good safety profile and low incidence of adverse events. Although the single-group target value research design fits the characteristics of Lecheng real-world data research, due to the inherent limitations of this type of design, it is recommended that the sponsor of the corresponding licensed medical device clinical research should work with clinical medical experts and biostatisticians in the design stage of the research plan. Communicate and negotiate with regulatory authorities, and then carry out OPC-based effectiveness research or observational research after reaching a consensus to avoid product evaluation risks.

2.2 Single-group studies based on external controls

Since all the patients in Lecheng have urgent needs for licensed medical devices, it is not feasible to establish an internal control during the same period. The single-group target value study has certain limitations in its scope of application due to the lack of a control group to provide a reference. Therefore, it is considered that domestic medical institutions outside Lecheng will set up the population of routine diagnosis and treatment as an external control to evaluate the safety and effectiveness of licensed medical devices relative to routine diagnosis and treatment. Domestic disease registration data, natural population cohorts, and disease-specific cohorts have accumulated massive data [4], providing abundant resources for the establishment of external controls. In addition, foreign clinical research data of licensed medical devices should also be considered as an external control to conduct a non-inferiority test of the efficacy of the device in the domestic population of Lecheng. However, regardless of the source of the external control, it is not the same population as the Lecheng patients, and bias control is still a key issue affecting the results of the study. These factors affecting the results of the study cover a wide range, including demographic characteristics, diagnostic criteria, diagnostic techniques, disease stage or subtype, disease severity, concomitant medication, observation conditions, evaluation indicators and evaluation criteria. In addition, there may be many very important but unrecognized or unmeasured factors, etc. Therefore, when selecting an external control, a population with homogeneity of basic characteristics should be selected as much as possible. In the statistical analysis stage, appropriate statistical analysis methods should also be used to control bias. The following will specifically describe the commonly used statistics for bias control in real-world data research. learning method.

2.3 Types of extended study designs for single-group studies

The E5 guideline issued by the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH) in 1988 first proposed the concept of bridging study: a new drug has been approved in the original region In the future, if it is to be extended to a new region, the existing information of the clinical trials in the original region can be used to conduct additional small-scale experimental studies in the new region as needed. Through these experimental studies, it can be shown that the drug has the same safety and efficacy for the population in the new region. It can effectively and quickly extrapolate the drugs from the original region to the new region. The bridging idea has been widely used in the submission of international multi-regional clinical trials (MRCTs) to target regions for approval, that is, drug review agencies in the target regions require that in addition to the MRCT framework of appropriate samples, supplementary targets for the target region population This strategy has become the current trend of international research and development of new drugs [8]. The three-arm non-inferiority trial design provides positive inspiration for the above-mentioned bridging strategy in qualitative and quantitative evaluation of drug efficacy and safety in the target area: the trial design includes an experimental group, an active control group and a placebo group, which can both test the trial It can also test whether the experimental group is non-inferior to the positive control group, and at the same time, by proving the measurement sensitivity and stability of the positive control group, the effectiveness of the experimental drug can be further confirmed [9]. Some scholars[10] integrated the framework of MRCT+LCT into a new bridging strategy of three-arm non-inferiority trial design based on the three-arm non-inferiority trial design, that is, on the basis of MRCT drug effectiveness confirmation. In order to verify whether the drug is effective in LCT and non-inferior to the overall curative effect of MRCT, the existing MRCT efficacy information is used to carry out LCT for the target population.

All Lecheng licensed medical devices have been studied abroad to confirm their effectiveness and safety. It is necessary to verify whether they are effective in the Chinese population through the real-world data of Lecheng's small sample practice, and are not inferior to the devices in foreign countries. Effects in clinical studies. The bridging strategy of the quasi-three-arm non-inferiority trial design under the framework of MRCT+LCT provides a new idea for solving the above-mentioned problems in the evaluation of the effect of Lecheng licensed medical devices: combining the above mentioned problems in "2.2 Single-group study based on external control". Two types of externally controlled research designs, that is, a single-group study with domestic routine diagnosis and treatment as an external control and a single-group study with foreign clinical research data as an external control are integrated and transformed (Figure 1). The two conventional treatment groups at home and abroad are compared as the negative control arm in the three-arm non-inferiority trial design, and the qualitative judgment of whether the experimental device is effective is called the quasi-negative control arm; the Lecheng experimental device group is compared to the three-arm The experimental arm in the non-inferiority trial design; the experimental device group in the foreign clinical research is analogized as the positive control arm to quantitatively measure the effectiveness of the experimental device. Based on this framework, it is assumed that the efficacy of the experimental device in Lecheng is non-inferior to the results of foreign clinical studies, and the percentage of the efficacy difference between the experimental device and conventional treatment in the foreign clinical study is used as the non-inferiority threshold, which is confirmed by non-inferiority test. The efficacy of the test device in Lecheng: ① the effectiveness of the test device relative to conventional treatment; ② the percentage of the efficacy of the test device in foreign countries is retained; ③ non-inferior to the efficacy of the test device in foreign countries. This new research design framework still needs to verify its applicability with the actual case of Lecheng. In addition, both negative and positive controls in this study design are external controls, and bias is still inevitable. Therefore, in the real-world data research of Lecheng licensed medical devices, bias control is a key link that needs to be carefully considered for all research types.

3. Statistical analysis methods applicable to the real-world data research of Lecheng licensed medical devices

3.1 Statistical analysis methods commonly used in the study of single-group target values

The commonly used statistical analysis methods for evaluating the outcome indicators of single-group target value studies are mainly divided into two categories: hypothesis testing and confidence interval methods [7, 11-12]. The hypothesis test of a single-group target value study is a one-sided test. According to the purpose of the study, the outcome evaluation indicators can be divided into high-quality indicators (such as efficiency) and low-quality indicators (such as the incidence of adverse events). Let θ1 be the overall parameter of the outcome indicator, and θ0 be the OPC of the outcome indicator, the hypothesis test of the single-group target value study is: ① For high-quality indicators: H0: θ1 ≤ θ0, H1: θ1>θ0; ② For low-quality indicators : H0 : θ1 ≥ θ0, H1 : θ1 < θ0, the inspection level (α) is usually 0.025. When the one-sided test P ≤ α, H0 is rejected and H1 is accepted, and the test device is considered to meet the design requirements. When the outcome indicator is a qualitative indicator, a hypothesis test is performed on the rate: ① The target value π0 or the expected population parameter π1 is far from 0% or 100% (between 10% and 90%), and the sample size is large (at least 50 cases) , it is recommended to use the normal approximation method; ② When the target value π0 or the expected overall parameter π1 is close to 0% or 100% (<10% or >90%), the normal approximation method may increase the type I error, it is recommended to use a more Precise exact probability method.

The evaluation of outcome indicators in single-group target value studies can also be achieved by comparing the confidence intervals of the outcome indicators and the OPC. For high-quality indicators, when the lower limit of the two-sided confidence interval of the trial device's outcome indicator (1-2α)% is higher than the OPC, the trial device is considered to meet the design requirements; for low-quality indicators, when the trial device's outcome indicator (1-2α) )% when the upper limit of the two-sided confidence interval is lower than the OPC, the test device is considered to meet the design requirements. When the outcome indicator is a qualitative indicator, the confidence interval is estimated for the rate: ① the target value π0 or the expected overall parameter π1 is far from 0% or 100%, and the sample size is large, it is recommended to use the normal approximation method; ② when the target value π0 or It is expected that the overall parameter π1 is close to 0% or 100%, and it is recommended to use the Miettinen exact estimation method or the Wilson scoring interval method [13-14].

The Lecheng trial device research population may include multiple subgroups at the same time, such as different disease stages or subtypes and different disease severity. Different population composition may cause different target values. At this time, the weighted method can be used to calculate the target value of the composite endpoint of multiple subgroups, that is, according to the proportion of the sample size of each subgroup to the total sample size, the corresponding weight of the OPC of the corresponding subgroup is given, and then the weight of the overall study population is used. OPC was used as the standard for the evaluation of outcome measures for this trial device. For example, in a clinical study evaluating the efficacy of a certain iliac vein stent, according to the literature, the main evaluation index was the vascular patency rate after stent placement. The study population was divided into 2 subgroups: post-thrombotic (PT) and non-thrombotic (NT) subgroups, assuming that 75% of the subjects were PT and 25% The subjects of the study were the NT group. According to relevant literature estimates, the vascular patency rates in the PT group and NT group were 77.6% and 95.5%, respectively, and the weighted OPC was 82.1%. The comparison of outcome indicators and weighted OPC can still be achieved using the above-mentioned hypothesis testing and confidence interval methods.

3.2 Statistical analysis methods for single-group studies based on external controls and their extended study design types

The statistical analysis ideas based on the type of externally controlled study design can be summarized as non-inferiority test, that is, the efficacy of the Lecheng trial device is non-inferior to the efficacy of the trial device in foreign clinical studies. Therefore, the statistical method of non-inferiority test can be applied to the evaluation of the efficacy of Lecheng medical devices. The "Non-Inferiority Clinical Trials" (Non-Inferiority Clinical Trials) issued by the FDA proposes the use of the double confidence interval method for statistical inference of non-inferiority clinical trials [15]. xNT and xOT respectively represent the efficacy of the experimental device in Lecheng clinical research and foreign clinical research, and xNP and xOP respectively represent the efficacy of conventional therapy in domestic and foreign clinical research. Assuming that xNT, xOT, xNP, and xOP satisfy normal distributions N(μNT, σNT2), N(μOT, σOT2), N(μNP, ​​σNP2), and N(μOP, σOP2), respectively, the test device indicators are designated as high-quality indicators. Take the point estimate of the efficacy of the test device in foreign clinical research μOT - μOP

The lower limit of the confidence interval of 100(1-δ)% is taken as the effect of the test device applied abroad, which is recorded as M1, that is,

, which is a conservative estimate of the efficacy of the test device in foreign applications. For any 0 ≤ λ ≤ 1, when it is hoped that the efficacy of the experimental device in Lecheng can retain at least 100λ% of the efficacy of the overseas application, let M2=(1-λ)M1, and M2 is specified to be used in the non-inferiority test. The boundary value of , then the non-inferiority hypothesis test with double confidence interval method is:

H0 was rejected, and the efficacy of the experimental device in the Lecheng clinical study was considered to be non-inferior to the efficacy of its application in foreign countries.

In addition, according to the idea of ​​bridging research, the efficacy of the experimental device in Lecheng and the efficacy in foreign clinical research can also be determined. At present, the classical statistical methods for similarity testing mainly include three categories: two-stage design and group sequential design method, weighted z-test method, and reconstruction probability and generalization probability method [16-17]. The two-stage design and group sequential design method are clinical studies that take the bridging study as a whole and divide it into two parts: the original regional stage (foreign clinical research stage) and the new regional stage (Lecheng clinical research stage) [18-19 ]. When the clinical research phase of Lecheng is completed, statistical analysis is carried out on the accumulated data of foreign clinical research and clinical research of Lecheng. If the test statistic TN is greater than the predetermined threshold CN, it is considered that the foreign clinical research results of the test device can be extrapolated to China. The basic idea of ​​the weighted z-test method in bridging research is to add the approximate normality z-test statistic of the original area and the new area according to the pre-specified weight, and then perform hypothesis testing to obtain a new statistic [20]. zN and zO respectively represent the z-test statistic of the experimental equipment Lecheng clinical research and foreign clinical research, and w represents the weight, then the weighted statistic

. At this time, a hypothesis test is performed on zw, and if it is less than the predetermined test level α, the bridge is considered to be successful. Reconstruction probability refers to the probability of repeating the research results of the original region in the new region under the same conditions; generalization probability refers to the probability of obtaining curative effect in the new region under the condition that the existing information indicates that the curative effect of the new region and the original region may be different [ 21-23]. The essence of using the reconstruction probability and generalization probability method to test whether the bridge is successful is whether the new region can achieve the pre-specified repeatability probability. The above three types of classical statistical analysis methods are all frequency methods. Due to the need to rely on external information, it is easy to cause problems such as type I error expansion, which may cause these bridging strategies to fail the approval of the regulatory authorities [17].

The idea of ​​bridging research is to support the rapid listing of medicinal devices in new regions through the information on the effectiveness and safety of medicinal devices already mastered in the original region, which is very consistent with the Bayes statistical concept [17]. Lecheng experimental medical devices usually have foreign clinical research results, so compared with the classical frequency method, the Bayes method may be more useful in the real-world data research of Lecheng licensed medical devices. The basic idea applied by empirical Bayes in bridging research is to use the posterior probability of population parameters to establish similarity judgment criteria [24]. The specific method is to take the efficacy difference between the experimental device group and the conventional treatment group in the original region as the prior of the new region, use the efficacy difference between the experimental device group and the conventional treatment group in the new region as the existing data, and use the empirical Bayes method to calculate the new region. The posterior probability of the difference between the experimental device group and the conventional treatment group. μNT and μP represent the overall mean of the new regional experimental device group and the conventional treatment group, respectively, and the posterior probability of the new regional efficacy must satisfy: PSI=Pr { μNT-μP>0|bridging data and prior}>1-α, where PSI can be regarded as a criterion of similarity. If the idea of ​​non-inferiority test is further introduced, that is, calculating the probability that the difference between the efficacy of the new region and the original region is greater than a fixed non-inferiority threshold, the quantitative measurement of the effectiveness of the experimental device in the new region can be achieved. At this time, the posterior probability of the efficacy of the new region must satisfy: PSI=Pr{μNT-μOT>-δ|bridging data and prior}>1-α, where μOT is the overall mean of the experimental device group in the original region, and δ is the pre-specified value The non-inferiority margin of . However, the above similarity judgment method largely relies on the sample information of the original region, and the data sample size of the original region is often much larger than that of the new region, making it dominant in the calculation of the posterior distribution, and the results of the new region are often It is impossible to "reverse" the original regional results, and it is easy to draw similar conclusions between regions. The mixed prior Bayes method uses two distribution functions that are added according to a certain weight to form the prior distribution of the overall parameters, so as to weaken the "dominant" influence caused by the excessively large sample content in the original area [25]. One of the distribution functions is determined from the data known in the original region, while the other distribution function uses an uninformative prior distribution. However, since the value of the weight needs to be specified manually, it often lacks practical significance, so it is difficult to choose its reasonable value in practical application.

It should be noted that the above statistical analysis methods are all qualitative judgments and non-inferiority verifications of the efficacy of the experimental device in the new region and the original region, and the interpretation of the sensitivity and stability of the experimental device in the new region relative to conventional therapy is very limited. For the situation of Lecheng, we can refer to the non-inferiority study design of quasi-three-arm bridging (Figure 1), but the relevant hypothesis testing and applicable statistical analysis methods still need to be further derived and evaluated.

3.3 Common statistical analysis methods for confounding bias control

Since most real-world data studies do not have a randomization process (with the exception of pragmatic clinical trials), confounding bias is a problem common to nearly all types of real-world study designs. The China Real World Data and Research Alliance (ChinaREAL) published the "Technical Specifications for Statistical Analysis of Evaluation of Treatment Outcomes Based on Real World Data" in 2019 [26], and the "Real World Data for Medical Device Clinical Research" published by NMPA in 2020. The Guidelines for Evaluation Techniques (Trial) [3] both provide detailed descriptions of confounding bias control strategies and commonly used statistical analysis methods in real-world data research. Effective statistical description and selection of appropriate statistical models are the key aspects of confounding bias control. Commonly used multivariate analysis methods include multiple linear regression models, logistic regression models, cox proportional hazards regression models, and corresponding multilevel models. In addition to traditional multivariate analysis methods, models based on causal inference are also widely used in statistical analysis of efficacy evaluation, such as propensity scores, instrumental variables, marginal structural models and structural equation models [26]. For the real-world data research of Lecheng licensed medical devices, since most of the research designs are single-group studies, the above statistical analysis methods to control confounding may be limited in the scope of application, and may be applied in the research design of establishing external controls. For most single-group studies in Lecheng, stratified analysis according to possible confounders and comprehensive sensitivity analysis may be more feasible strategies for identifying and controlling confounding bias.

4. Conclusion

The real-world data research of licensed medical devices in the Lecheng International Medical Tourism Pilot Zone in Boao, Hainan is currently in its infancy, and it is necessary to construct as soon as possible a statistical analysis method applicable to the design, evaluation and derivation of real-world data in Lecheng, and effectively combine the existing clinical research abroad. Data to achieve rapid approval and listing of licensed medical devices in China. By analyzing the key characteristics of the real-world data research of Lecheng licensed medical devices, this paper proposes a relatively reasonable and feasible research design and corresponding statistical analysis ideas, but it still needs to be compared and evaluated through the application of Lecheng's real cases.

About the first author

Cao Han, postdoctoral fellow, Department of Medical Statistics, Peking University First Hospital. Specialties: Statistical Design and Analysis of Clinical Research

About the corresponding author

Yao Chen, Master of Health Statistics, Professor, Doctoral Supervisor of Clinical Research Methodology, Director of Medical Statistics Office of Peking University First Hospital, Deputy Director of Peking University Clinical Research Institute and Deputy Director of Hainan Real World Data Research Institute. Specialties: Statistical Design and Analysis of Clinical Research