NHQDR Methodology Report
Methods Used on the Website of the National Healthcare Quality and Disparities Reports
- What’s New
- Organization of the NHQDR Measures Within the Integrated NHQDR Website
- Available Comparisons
- National View
- State View
- Data Query
- Internet Citation
The integrated National Healthcare Quality and Disparities Reports (NHQDR, previously NHQR/NHDR) website provides comprehensive information about health care developments and overviews for policymakers, legislators, and reporters. It also contains detailed data tables for researchers. Comparative information, such as trends over time and current distance from achievable benchmarks, is easily accessible. The user can also "drill down" to more detailed information to demonstrate connections between the broader and more elemental levels.
The design of this website was based on recommendations from the National Academy of Medicine (NAM) (formerly Institute of Medicine) on how to improve upon the NHQDR reports and related products. The aim was to facilitate greater impact regarding quality improvement and disparities elimination. A few key goals served as the foundation of the integrated NHQDR website design:
- Bring together the NHQDR and related products in a way that produces a cohesive story about healthcare quality
- Present information in forward-looking and action-oriented formats
- Highlight equity as an integral part of overall quality, rather than as a separate story line
- Present data as a solution to health care disparities, with prospects for evidence-based practices
- Improve overall navigation and usability of the site for various audiences
The NHQDR website integrates information that was previously accessible from three different websites:
The following additional features were added to address the NAM recommendations:
- Benchmarks based on performance of the top 10 percent of States, to encourage achievable goals
- Displays that provide consistent definitions and comparisons across national and State levels
- Displays of racial and ethnic comparisons, as a part of the larger quality evaluation
- Provision of subject areas and topics that simplify access to the data and information
This Methods Report includes the following sections:
- An overview of the organization and presentation of materials
- An introduction to each of the five sections of the integrated NHQDR website—Reports, National View, State View, Data Query, and Resources
- A description of the methods behind the various graphics and data presentations within the five sections
This section covers major changes on the website, data, and analysis methods, with the newest changes first.
Data Limitations and Changes: Before 2018, the NHQDR had about 50 core measures using Healthcare Cost and Utilization Project (HCUP) data. However, the HCUP State data and trend data were not available for the 2019 report due to the change from ICD-91 diagnosis codes to ICD-10 codes and changes to the AHRQ Quality Indicators (same as the 2018 report). Different from the 2018 report, HCUP risk-adjusted benchmarks were used for some of the Quality Indicators for subgroup comparisons under the National View panel. National data and State data are available for the two HCUP opioid measures.
Behavioral Risk Factor Surveillance System data became available for the 2019 report and are included in all analyses.
Medical Expenditure Panel Survey activity limitation data were not available for the 2019 report or for the website.
The "Opioids" panel was new for the 2018 report. This panel provides data, analytic results, charts, and State quartile maps for measures related to opioid use or abuse. Users can select a measure on the first page and look for more details by selecting demographic areas. The 2018 report has seven opioid measures; two are core measures and five are supplemental measures. The supplemental measures are included in the website analysis, not in the report.
Data Limitations and Changes: Before 2018, the NHQDR had about 50 core measures using Healthcare Cost and Utilization Project (HCUP) data. However, the HCUP State data and trend data were not available for the 2018 report due to the change from ICD-9 diagnosis codes to ICD-10 codes and Quality Indicator changes. Only 2016 national data were included on the website.
Data for about 20 nursing home care measures and 30 home health care measures were not available for the 2016 and 2017 reports. For the 2018 report, 2013-2016 data were available and were included in the comparisons as well as trend analysis.
Data availability may affect the analysis results for various reasons, including differences in properties of measures and source data and in population subgroups in the data tables.
Consistency Between the Report and the Website: Since 2016, we have been trying to make the analysis methods and results consistent between the website and the report. However, inconsistencies still exist. For 2018, nursing home data became available after the report was prepared and we decided to include them in the website data analysis. We also decided to include all data for the opioid supplemental measures in the website data analysis because opioid-related issues are a Department of Health and Human Services priority.
Benchmark Year: The benchmark year for the 2018 report did not move forward by 1 year. In previous years, the benchmark year usually moved forward by a year. The benchmark year was 2014 for the 2016 report, and 2015 for the 2017 report. For 2018, the benchmark year was still 2015. Because of this change, more measures, States, and subgroups reached the benchmark in 2018 compared with previous years.
From the 2017 report, data for about 200 supplemental measures are included on the Data Query page. These measures are grouped in two additional subject areas and a number of topics:
- Other Primary Measures
- Composite Measures
- Other Measures
- Supplemental Measures
- Access to Care
- Person-Centered Care
- Care Coordination
- Effective Treatment
- Healthy Living
- Care Affordability
1 ICD-9 is the International Classification of Diseases, Ninth Revision. ICD-10 is the 10th revision.
return to the top of this section
Organization of the NHDQR Measures Within the Integrated NHQDR Website
The NHQDR reports include approximately 300 different measures that are collected from more than three dozen organizations, including the Agency for Healthcare Research and Quality (AHRQ), the Centers for Disease Control and Prevention (CDC), the National Center for Health Statistics (NCHS), and the Centers for Medicare & Medicaid Services (CMS). The term measure is used to define a specific metric, such as “Adults who received a blood cholesterol measurement in the last 5 years” or “Deaths per 1,000 adult hospital admissions with congestive heart failure.”
To facilitate the evaluation of comprehensive overviews and trends, the 300 individual measures are categorized into seven areas:
- Priority areas
- Diseases and conditions
- Priority Populations
- Health Insurance
- Access to Care
- Type of Care
- Setting of Care
Each of the seven subject areas is further divided into three or more topics, which are listed below.
NHQDR Subject Areas and Topics
Diseases and Conditions
Chronic Kidney Disease
HIV and AIDS
Mental Health and Substance Abuse
Residents of Rural Areas
Adults With Basic Activity Limitations
Adults With Complex Activity Limitations
NHPI (previously NHOPI)
Access to Care
Types of Care
Setting of Care
Home Health and Hospice
A measure is often included in more than one section. For example, the measure “Adults who received a blood cholesterol measurement in the last 5 years” is included under the Cardiovascular Disease within Diseases and Conditions, under Prevention within Type of Care, and under Ambulatory within Setting of Care.
For all of the subject areas except Priority Populations and Health Insurance, the collection of measures is based on topic areas in the NHQDR. The measures that are reported under Priority Populations and Health Insurance are limited to those that have sufficient data to include them as a reporting category. For example, for “Adults who received a blood cholesterol measurement in the last 5 years,” an estimate for females is included under the Priority Population of Women; however, this measure is not included under the Priority Population of NHPI (Native Hawaiian/Pacific Islander), because the data source does not provide an estimate for this racial group. In other words, only a subset of NHQDR measures has sufficient data to support reporting under Priority Populations and Health Insurance.
return to the top of this section
Within the different subject areas and topics, the integrated NHQDR website presents three types of comparative information-
- National and State-level comparisons with achievable benchmarks
- National trends over time
- State-level comparisons with the national overall estimates (i.e., State Snapshots)
National trends are always highlighted in the NHQDR reports. State-level comparisons with all-State averages had been highlighted in the State Snapshot. The addition of the comparison with achievable benchmarks was based on the IOM recommendations to promote best-in-class achievement at both the national and State levels.
return to the top of this section
National and State-Level Comparisons With Achievable Benchmarks
The incorporation of comparisons with achievable benchmarks sets standards based on the performance of the top 10 percent of States. These standards are considered achievable because they have already been attained by the best performing States.
State-level benchmarks are only determined for NHQDR measures that have State-level data for at least 30 States. The benchmark for a measure is calculated as an average of the top 10 percent of reporting States. For example, if data from 50 States are available for a measure, then the benchmark is the average of the measure estimate across the best five States. States without 2013 or later data were excluded from the benchmark calculation for the 2016 report. For the 2017 and 2018 report, 2015 data were selected first if available; otherwise, 2016 or 2014 data were selected.
Measure estimates for the Nation, topics, State overall, and topics within each State are then compared with the State-level benchmark for that measure. Based on this comparison, national and State-level measure estimates are assigned to one of three categories: achieved the benchmark or better, close to the benchmark, or far away from the benchmark. These categories are defined below.
- Achieved the benchmark or better. The value for a measure is at least 90 percent of the benchmark value. This includes the case in which the measure's value is equal to or better than the benchmark.
- Close to the benchmark. The value for a measure is between 50 percent and 90 percent of the benchmark (i.e., the value for a measure is at least 50% of the benchmark value but has not reached the 90% criterion for having achieved the benchmark).
- Far away from the benchmark. The value for a measure has not achieved 50 percent of the benchmark.
When comparing estimates with the benchmarks, one critical consideration is the direction of the outcome. Specifically, a low value is the desired outcome for some measures, such as mortality, unmet needs, and communication problems. In contrast, a high value is the desired outcome for other measures, such as receiving recommended tests or reporting good communication. Consider a measure in which a low number is desired, such as “Adults who needed to see a specialist in the last 12 months who found it difficult to see a specialist.” If the benchmark is 18.1 and the national estimate is 14.6, then the measure achieved the benchmark—its value is 19 percent lower than the benchmark (the preferred direction). If the national estimate is 22.0, then it would be considered close to the benchmark—its value is 20 percent higher than the benchmark. If the national estimate is greater than 27.2, the measure would be considered far away from the benchmark—its value is more than 50 percent larger.
return to the top of this section
National Trends Over Time
National trends are determined for NHQDR measures with at least 4 years of data. The time span can vary across measures. Beginning with the 2016 report, the average annual percent change (AAPC) has been estimated using unweighted log-linear regression.
Model: ln(M) = β0 + β1Y,
where ln(M) is the natural logarithm of the aligned rate, β0 is the intercept or constant, and β1 is the coefficient corresponding to year Y
- Improving = Average annual percentage change >1% per year in a favorable direction and p <0.10.2.
- Worsening = Average annual percentage change >1% per year in an unfavorable direction and p <0.10.
- No Change = Average annual percentage change ≤1% per year or p ≥0.10.
State Snapshot Comparisons
Under State Snapshot, each State’s overall performance is compared with the national overall estimate. State performance by topic is also compared with the national overall estimate not the national estimate for a particular topic. This approach is in contrast to the comparison with an achievable benchmark in which the comparison is with the average of the top 10 percent of States.
The national overall estimate is the estimate from micro data. If it is not available, the all-State average is calculated as the national overall estimate.
For each measure, State performance is categorized as better-than-average, average, or worse-than-average relative to the all-State average.
These comparison categories are defined based on a statistical test for differences:
- Better-than-average. The State rate on an NHQDR measure is better than the all-State average, and that difference is statistically significant.
- Average. The State rate on an NHQDR measure is not statistically different from the all-State average.
- Worse-than-average. The State rate on an NHQDR measure is worse than the all-State average, and that difference is statistically significant.
Beginning with the 2016 report, two criteria have been used to define the difference between the State rate and the national average:
- The absolute difference must be statistically significant with p <0.05 on a two-tailed test.
- The relative difference must be at least 10% when framed negatively.
Across a group of measures within a subject area and topic, each State receives a performance meter score. First, points are assigned to each measure as follows:
- 1 point for each State-level measure that was better than the all-State average
- 0.5 points for each State-level measure that was average, relative to the all-State average
- 0 points for each State-level measure that was worse than the all-State average
Next, the points are combined into a meter score:
((A*1) + (B*0.5) + (C*0)) * 100 A + B + C
2. A probability of 0.10 was selected as the significance level because the magnitude of the standard errors varied considerably by type of data.
A = number of better-than-average NHQDR measures in the summary
B = number of average NHQDR measures in the summary
C = number of worse-than-average NHQDR measures in the summary
Example: North Dakota has 37 better-than-average measures, 61 average measures, and 11 worse-than-average measures. Thus, A = 37, B = 61, and C = 11.
(37 * 1) + (61 * 0.5) + (11 * 0)) * 100/(37 + 61 + 11)) = (37 + 30.5 + 0) * 100/109 = 67.5 * 100/109 = 6,750/109 = 61.93
Meter scores range from 0 (all measures are worse than average) to 100 (all measures are better than average). Scores between 0 and 100 represent the mix of measures that are worse than average, average, and better than average. Higher scores represent better performance, because the score increases with the number of measures that are average and increases more rapidly with the number of measures that are better than average.
A 180-degree colored semicircle divided into five categories is used for visual presentation of the data. The five categories are:
- Very Weak: 0 ≤ score < 20
- Weak: 20 ≤ score < 40
- Average: 40 ≤ score < 60
- Strong: 60 ≤ score < 80
- Very Strong: 80 ≤ score ≤ 100
The meter score for a given measure is depicted on the semicircle as a performance meter arrow as shown in Figure 1. A solid arrow is used for the most recent year of available data, if at least five measures are available. A dashed arrow is used to show performance for the baseline year, when the baseline has more than two-thirds of the measures available in the most recent year. This criterion is applied to ensure similar comparisons between the baseline and the most recent year. For the 2016 report, measures with the latest year before 2013 were excluded from the most recent year’s comparison.
The State overall meter scores are more reliable and stable than the State topic scores mainly because the number of measures with valid data for each topic within a State is usually much smaller than that for the State overall. The number of data years and measure types with valid data may also affect the meter scores. Please keep these factors in mind when interpreting the meter scores and check the underlying data for details.
Figure 1: Performance Meter
The National View section of the NHQDR integrated website provides an overview of performance across NHQDR measures. This perspective shows areas of strengths and weaknesses at a glance. There are two types of comparative information provided in this section—
This section provides two types of comparative information:
- Comparison with “achievable benchmarks” for current data
- Trends over time
This information is available across all NHQDR measures and by subject area and topic.
Comparisons With Achievable Benchmarks
On the National View page, there are comparisons with achievable benchmarks: across all measures, which are organized in several categories; and by measure areas and topics. The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons With Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement (Figure 2). If a measure does not have an available benchmark (i.e., no State data from which to calculate a benchmark), it is not represented in the summary graphic. In the example in Figure 2, benchmarks are available for 129 measures: 23 measures that are far away from the benchmark, 49 measures that are close to the benchmark, and 57 measures that have achieved or performed better than the benchmark.
Figure 2: National View Benchmark Summary, All Measures
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 3). For measures that report data, the comparison to benchmarks is provided for individuals who are Hispanic and non-Hispanic White, Black, and Asian and Pacific Islander.
Figure 3: National View Benchmark Summary, by Race and Ethnicity
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 4). For measures that report data, the comparison with benchmarks is provided for low-income and high-income communities.
Figure 4: National View Benchmark Summary, by Community Income
The user can obtain information about (1) which measures are included in the different benchmark achievement categories and (2) the value of the achievable benchmarks, by using the Benchmark Achievement filter and reviewing data under “View Underlying Data” An example is provided in Figure 5.
Figure 5: Data Table Underlying the Achievable Benchmark Bar Charts
Trends Over Time
Within the National View, the user may also view summary results of trending overtime overall measures and by measure areas and topics. The methods for the trend comparison are detailed under the section of this report titled National. Since the 2016 report, the trending method is the same as for the report.
The trends summary graphic displays the total frequencies of each category of achievement. Measures that do not have at least 4 years of data to document the trend are shown in the “Data Not Available” column. Select a topic to get the trend summary graphic for each topic. In Figure 6, there are 101 measures showing improvement over time, 81 measures with no change, and 17 measures worsening over time. There is no trending information for 51 measures, which represent 20 percent of all of the measures in this figure— 51/(51 + 101+81+17).
Figure 6: National View Summary of Trends in Measures Over Time
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the annual average percent change, by using the Trend filter or viewing the “View Underlying Data” table. An example is provided in Figure 7.
Figure 7: Data Table Underlying the Trend Bar Charts
The State View section of the NHQDR integrated website is similar to the National View in that it provides an overview of performance across NHQDR measures and by measure areas and topics but, in addition, provides an overview of a State’s overall performance and a State’s performance for each topic. This approach allows the user to see each State’s areas of strengths and weaknesses at a glance. Two types of comparative information are provided in this section—
These Dashboards provide two types of comparative information:
- Comparison with “achievable benchmarks” for current data
- State Snapshot comparisons with all-State averages
This information is available across all State-level measures and by subject area and topic.
Users begin by selecting a State from the State filter drop-down box (Figure 8).
Figure 8: State View State Selection
State Comparisons With Achievable Benchmarks
Similar to the National View page, the first comparative graphic under State View is a summary of quality measures compared with achievable benchmarks. Three summaries are available: across all State-specific measures, by race and ethnicity, and by community income. Summary by race/ethnicity is also available by default. Select a measure area, then a topic for summary results for a particular topic.
The methods for the benchmark comparison are detailed under the section of this report titled National and State-Level Comparisons With Achievable Benchmarks.
The first summary graphic displays the total frequencies of each category of achievement, where State-specific data and benchmarks are available (Figure 9). If a measure does not have an available benchmark, it is not represented in the summary graphic. In the example in Figure 8, benchmarks are available for 131 measures reported for the State: 30 measures that are far away from the benchmark, 35 measures that are close to the benchmark, and 66 measures that have achieved or performed better than the benchmark.
Figure 9: State View State Summary Graphic Compared With Benchmarks
The second summary graphic displays the total frequencies of each category of achievement by race and ethnicity (Figure 10). For measures that report data for the selected State, the comparison with benchmarks is provided for individuals who are White, Black, Hispanic, and Asian and Pacific Islander.
Figure 10: State View Benchmark Summary, by Race and Ethnicity
The third (final) summary graphic displays the total frequencies of each category of achievement by community income (Figure 11). For measures that report data for the selected State, the comparison with benchmarks is provided for low-income and high-income communities.
Figure 11: State View Benchmark Summary, by Community Income
The user can obtain information about (1) which measures are included in the different achievement categories and (2) the value of the achievable benchmarks, by selecting the Population Group, and Achievement Benchmark viewing the “View Underlying Data.” table. An example is provided in Figure 12.
Figure 12: Data Table Underlying the Achievable Benchmark Bar Charts
The State Snapshot Dashboard provides insight into a selected State’s performance overall or by topic within a State by comparing it with the national average for the available measures. The methods for the State Snapshot are detailed under the section of this report titled State Snapshot Comparisons.
Figure 13 shows the State Snapshot performance meter for a sample State. The user can see that the State performed better in the most recent data year than it had performed in the past (baseline year) for the "Adult hemodialysis patients who use arteriovenous fistulas as a primary mode of vascular access" measure. The most recent score was 69.4, which is better than its baseline score of 65.7. In addition, the State’s performance is in the “average” range relative to the comparisons with the all-State averages. Beneath the performance meters, there is a table that lists the meter score based on the filter selections. The data table can be filtered by State. Select a measure area and a topic under “Subject Area” and "Topic" for summary results for a particular topic.
Figure 13: State Snapshot Performance Meter
The user can obtain information about (1) which measures are included in the performance meter score and (2) the average annual percent change, by sorting the Recent Performance column in the “View Underlying Data” table. An example is provided in Figure 14.
Figure 14: Data Table Underlying the State Snapshot Performance Meter
The State Snapshot Dashboard provides “snapshot” information across all subject areas and topics on one web page. The first graphic is the State Snapshot performance meter across all available State-level measures. Next, rectangular versions of the performance meter present information on each subject area and topic. The methods for the State Snapshot meters are detailed under the section above called State Snapshot Comparisons.
Figure 15 shows a portion of a sample State Snapshot Dashboard. The inverted triangles (black) indicate performance in the most recent data year; the regular triangles (translucent) indicate performance in the baseline year. The State’s performance improved in the areas of cancer and chronic kidney disease. Performance declined for mental health and substance use disorders.
For cancer, the State’s performance improved from the weak range to the average range from the baseline to the most recent year. Quality performance in the areas of respiratory diseases has not changed from the baseline year to the most recent year, as indicated by two triangles in the same location. The meters for HIV/AIDS and for diabetes do not have a baseline year for comparisons. If a topic is not displayed this is because the State has no data on these measures.
Figure 15: Example of a State’s Dashboard Meters
The data query section of the NHQDR integrated website provides detailed analytic capabilities that allow the user to view individual measures rather than summaries of performance. Figure 16 contains a view of the "Data Query Tool" filter selections available. The user has options to view measures available for National Trends, National Categories, National Disparities, State Trends, and State Categories.
- Geographic area - National or a State is noted on the buttons for selection, including National Trends, National Categories, National Disparities, State Trends, State Categories.
- Subject Area - filter by diseases and conditions, priority populations, health insurance, access to care, types of care, or settings of care
- Topic - filter within a Subject Area.
- Measure - filter by specific measures available. Type keyword text within the box directly to locate measures with specific text.
- Category - Refine table and graph results using Categories available per measure for National Categories, National Disparities, and State Categories.
These selections define the list of available measures. Only one measure can be selected at a time. The user can choose to examine the measure over time (Trending) or by Categories as well as a Dispartities focus. The available categories and filters depend on the available data reported to the NHQDR from the original data source.
Figure 16: Data Query Filters
Based on the user’s selections, the data query dashboard will return an underlying data table of the information. The information is a subset of the rows in the full NHQDR data table for the chosen measure. The results of a query on the National measure “Adult current smokers with a doctor's office or clinic visit in the last 12 months who received advice to quit smoking” are shown in Figure 17.
Links for the measure source and specification are available by selecting the “Download Data” button under the data table.
Figure 17: Data Query Table Results
Users can view a graphical depiction of the selected data query above the data table. When the query is specific to trends over time, a line graph is displayed (Figure 18). When one or two categories are selected, bar graphs are used to depict the data.
Figure 18: Data Query Graphic for Results by Category
Methods Used on the Website of the National Healthcare Quality and Disparities Reports. December 2021. Rockville, MD: Agency for Healthcare Research and Quality.