Evaluation of Potential ACO Performance Bias from Retrospective & Prospective Assignment

INTRODUCTION
STUDY DATA & METHODS
RESULTS
POLICY MAKING IMPLICATIONS

Executive Summary

Current claims-based attribution methodologies require ACOs to make certain tradeoffs. Prospective attribution has the advantage of timeliness while retrospective attribution has the advantage of precise overlap between assigned beneficiaries and beneficiaries receiving primary care services from the ACO during the performance year. The NAACOS Cost and Quality Data Warehouse, containing the comprehensive Medicare fee-for-service (FFS) claims dataset from 2013 through 2015, was used to quantitatively investigate differences in key metrics that impact ACO performance for 322 Medicare Shared Savings Program (MSSP) ACOs. This is the first study to simulate the prospectively and retrospectively assigned beneficiary populations for actual ACOs. Key metrics were evaluated for each of the 322 ACOs during the 2015 performance year. Despite significant differences between prospective and retrospective assignment, neither methodology shows performance bias.

It was discovered that 88 percent of ACOs would have experienced lower assignment under prospective assignment. Prospective assigned beneficiary count is 10 percent lower than retrospective assignment across all studied ACOs. This poses a concern for smaller ACOs that participate in prospectively assigned tracks. On average, 72 percent of ACOs would have experienced higher risk-adjusted per beneficiary per year (PBPY) costs when transitioning to prospective assignment. Risk-adjusted PBPY costs are 2 percent higher than retrospective risk-adjusted PBPY costs overall. The higher PBPY starting point may benefit the ACO if it can identify opportunities for optimizing patient care and appropriately capturing risk adjustment. An 0.8 percentage point higher assigned death rate from 3.5 percent to 4.3 percent was observed under prospective assignment placing higher emphasis on end-of-life expense management for ACOs that select prospective assignment.

Of those studied, the average ACO would have experienced only 71 percent overlap between the prospectively assigned and retrospectively assigned populations. Both the prospective and retrospective assignment methodologies demonstrate significant mismatch between intervenable and attributed populations. This suggests that more focus should be placed on non-claims-based methods of attribution, such as voluntary alignment, to achieve sustained success under the MSSP. Table 1 below shows decision factors to weigh to help ACOs choose assignment methodology.

Table 1: High-level Factors to Consider Between Prospective and Retrospective Assignment

Decision Factor Description
ACO Size If the ACO has close to 5,000 assigned beneficiaries, then the ACO may favor retrospective assignment because beneficiary count is lower under prospective assignment versus retrospective assignment.
ACO Market Competitiveness If competing ACOs in the same local market are prospectively assigned, then the ACO may favor prospective assignment because prospective assignment has precedence over retrospective assignment.
Local Medicare Enrollment Growth If there is a high rate of new Medicare enrollment in the local market or “age-ins,” the ACO may favor retrospective assignment to the new Medicare beneficiary market share.
Duration of Medical Management Programs If the ACO’s medical management programs require a long duration to produce a return on investment, then the ACO may favor prospective assignment because it allows early identification of assigned beneficiaries for risk identification and stratification to medical management programs.
Member Churn Rate If there is high member churn to other ACOs, then the ACO may favor retrospective assignment because the ACO will not be financially responsible for beneficiaries who change primary care physicians, move during the performance year or don’t have an office visit.
Beneficiary Engagement If there is low assigned beneficiary engagement, then the ACO may favor retrospective assignment because the ACO will not be financially responsible for beneficiaries who don’t have an office visit throughout the performance year.
End-of-life Management Program If the ACO has a robust end-of-life management or outpatient advance directives program implemented, the ACO may favor prospective assignment because the ACO is responsible for a higher assigned rate of deaths having end-of-life expenses.

Introduction and Background

The choice between prospective and retrospective attribution is a critical consideration during ACO track selection. NAACOS provides a comprehensive resource containing definitions and qualitative comparisons between prospective and retrospective attribution for the MSSP and Next Generation Model. Figure 1 below from the NAACOS resource shows the difference between the attribution timeframes for the ACO models for a 2018 performance year.

Figure 1: ACO Attribution Timeframes

ACO Attribution Timeframes

ACOs historically have had little quantitative information to support the decision between prospective and retrospective assignment. Lewis et al. provides quantitative differences under the Pioneer ACO Program context using the 100 percent Medicare FFS claims data from 2008 and 2009 to simulate ACO networks. The differences in attribution rates and Primary Care Provider (PCP) visits of attributed beneficiaries were studied.

The purpose of this NAACOS study and resource is to further quantify the aforementioned qualitative differences between prospective and retrospective assignment ACOs to better inform ACO track selection. This is the first study to recalculate attribution under prospective and retrospective assignment over the same time period for actual ACOs. The quantitative metrics concerned include assigned beneficiary count, assigned beneficiary overlap, risk score, PBPY medical expenditures, risk-adjusted PBPY medical expenditures, and annual death rate.

Study Data & Methods

The study used the NAACOS Cost and Quality Data Warehouse which contains 100 percent of ACO assigned beneficiaries in addition to 80 percent of those eligible for assignment, but not assigned from 2011 to 2015. Data elements include all institutional and provider bills for those included in the sample. The assigned beneficiary count, assigned beneficiary overlap, risk score and PBPY medical expenditures were studied for the 2015 performance year, during which 392 ACOs participated in the program. Four ACOs were eliminated from this study because they did not have complete data in the NAACOS Data Warehouse.

Furthermore, validation checks were used to compare actual retrospective assigned beneficiary counts from the MSSP 2015 ACO public use file (PUF) with calculated retrospective assigned beneficiary counts and PBPY medical expenditures. This resulted in the exclusion of an additional 66 ACOs that exceeded a 20 percent validation threshold. Variances between the NACOS Data Warehouse and PUF could have been caused by substance abuse claims contained within the PUF and not the NACOS Data Warehouse, large claims included in the NAACOS Data Warehouse but not the PUF, or provider participant list changes from 2013 through 2015. The final sample includes 322 ACOs.

Claims-based prospectively and retrospectively assigned populations were calculated using CMS published rules for the 322 ACOs in 2015. Since the NAACOS Data Warehouse is based on assignable beneficiaries, 2013 and 2014 claims data were available to calculate the October 1, 2013 through September 30, 2014 prospective assignment window for the 2015 performance year as if the retrospective assignment ACO were instead based on prospective assignment.

ACOs frequently change provider participant composition between performance years. Changes in provider participants from the study period October 1, 2013 through December 31, 2015 were handled by only considering the provider tax identification numbers (TINs) that were continuous participants during the entire study period. This methodology allows individual providers to change participation within the study period and is consistent with CMS policy. This caused discrepancies between the PUF and the NAACOS Data Warehouse because the PUF includes all participant TINs as they are added or removed throughout the 2013, 2014 and 2015 performance years.

Six metrics of interest were calculated under prospective and retrospective populations for each ACO to quantify the 2015 impact of prospective versus retrospective assignment. These metrics include total assigned beneficiary count, assigned beneficiary overlap or match percentage, risk score, PBPY medical expenditures, risk-adjusted PBPY medical expenditures, and assigned beneficiary death rate. The assigned beneficiary match percentage is defined as the percentage of prospectively assigned beneficiaries who were also retrospectively assigned as a proportion of the total number of retrospectively assigned beneficiaries.

Results

Quantitative results for beneficiary assignment counts and overlap appear in Table 2 below. The overall difference in assigned beneficiary count is 5,538,098 under prospective assignment and 7,003,882 under retrospective assignment (79 percent). Despite prospective assignment having precedence over retrospective assignment, 88 percent of ACOs would have experienced lower assignment under prospective assignment. Prospective assigned beneficiary count is 10 percent lower than retrospective assignment across the studied ACOs. Reasons for this include the static nature of prospective assignment, loss of opportunity to capture increasing enrollment in Medicare, and inability to add beneficiaries throughout the performance year.

The average ACO would have experienced only 71 percent of overlap between the prospectively assigned and retrospectively assigned populations. Reasons for beneficiaries being prospectively assigned but not retrospectively assigned include change in provider and recovery from an acute episode without seeking care in the following year. Lewis et al. looked at a similar measure of overlap and found that 31 percent of patients did not receive primary care from ACO participants during the performance year let alone become assigned to the ACO. This is at least partially offset by the phenomenon that retrospective assignment ACOs may invest in managing the care of the patient throughout the performance year only to ultimately lose their attribution to a competing ACO.

Table 2: Assigned Beneficiary Count Differences and Degree of Overlap Between Prospectively and Retrospectively Assigned Populations

Metric
(N=322 ACOs)
Prospective Retrospective Difference
Assigned Beneficiary Count
Total Assigned Beneficiary 5,538,098 7,003,882 79%
ACO Average Assigned Beneficiaries 15,799 19,663 90%
Number of ACOs with Lower Beneficiary Count 283 39 88%
Assigned Beneficiary Match %
Total Match Percent 4,608,256 7,003,882 66%
ACO Average Match Percent 71% 100% N/A

Table 3 below shows that 64 percent of ACOs have higher PBPY medical expense under prospective assignment. Prospectively assigned ACOs have 3 percent higher PBPY medical expense overall. The exact cause for this is unknown, but it is hypothesized that a slightly older population and higher assigned death rate may be contributing factors.

There was not a consistent pattern between the average HCC risk score of the prospectively and retrospectively assigned populations. Nearly half of the time the risk score would be lower, and the other half it would be higher under prospective assignment consistent with random churn of assigned beneficiary population. Note that the raw risk score was used for this study since the renormalization factor cancels for the same ACO.

Seventy-two percent of ACOs studied would have experienced higher risk-adjusted PBPY costs when transitioning to prospective assignment. Risk-adjusted PBPY costs are 2 percent higher than retrospective risk-adjusted PBPY cost overall. Depending on the ACO’s ability to identify the source of the higher cost, the higher risk-adjusted PBPY cost may present an opportunity for ACOs to improve the efficiency of care for prospectively assigned beneficiaries. Since benchmarking and performance year evaluation methodology is symmetric, it is expected that both the benchmark and performance year risk-adjusted PBPY costs would be higher under prospective assignment.

Table 3: PBPY and Risk Score Differences Between Prospectively and Retrospectively Assigned Populations

Metric
(N=322 ACOs)
Prospective Retrospective Difference
PBPY ($)
Average PBPY $9,149 $9,149 103%
Number with Higher Prospective PBPY 213 119 64%
ACO Average Risk Score
Average Risk Score 1.23 1.24 .99
Number with Higher Prospective Risk Score 163 159 51%
Risk-Adjusted PBPY ($)
Average Risk-Adjusted PBPY $8,171 Column 3 Value 8 102%
Number with Higher Prospective Risk-Adjusted PBPY 233 89 72%

The MSSP ACO program excludes beneficiaries who die prior to the performance year from final assignment. Beneficiaries who die during the performance year count for partial credit in the shared savings formula. Table 4 below shows the difference in assigned death rates between prospectively and retrospectively assigned populations. It reveals that there is an approximate 0.8 percent higher assigned death rate under prospective assignment, which helps to explain the higher PBPY medical expenses. The main effect is due to an overlap in the assignment and performance window featured in retrospective assignment. That is, some beneficiaries who would have been prospectively assigned may have died before incurring the particular primary care services that would have caused them to be retrospectively assigned, effectively lowering the observed death rate of retrospectively assigned beneficiaries. The overall Medicare FFS average death rate is closer to the prospectively assigned rate further supporting that the retrospective death rate is artificially lowered by the retrospective beneficiary sampling methodology. Additionally, there is a minor demographic effect. Medicare beneficiaries who newly enroll in Medicare throughout the performance year are unable to be attributed leading to a higher average age under prospective assignment. Further research is needed to fully theoretically explain these empirical results.

Table 4: Difference in Death Rates Between Prospective and Retrospective Assignment ACOs

2015 Deaths 2015 Total Assigned 2015 aAssigned Death Rate
2015 Prospective 235,751 5,538,098 4.3%
2015 Retrospective 245,294 7,003,882 3.5%

Policy Making Implications

There is no one-size-fits-all claims-based attribution approach that is optimal for every ACO. This suggests that ACOs should have the flexibility to choose the attribution approach that fits their local market dynamics, available resources and patient population independent of which ACO model or track is selected. NAACOS has repeatedly advocated for the choice of assignment methodology, regardless of the specific model or track in which an ACO participates. CMS should also exclude beneficiaries who move from the ACO’s extended service area from assignment, a policy which applies to the Next Generation Model and should be added to MSSP and future Medicare ACO models.

Additionally, the quantitative evidence detailed in this study supports that neither claims-based attribution approach is ideal for all ACOs. However, neither attribution methodology puts an ACO at a general disadvantage from being able to achieve performance goals such as earning shared savings. As such, more emphasis should be placed on exploring improvements to the attribution methodology. For example, a successful implementation of a voluntary alignment process, has the potential to improve assignment as well as emphasize an ACO’s relationship with its beneficiaries and further engage them in their care and involvement in the ACO.

References:

Valerie A. Lewis, Asha Belle McClurg, Jeremy Smith, Elliott S. Fisher, Julie P.W. Bynum. Attributing Patients To Accountable Care Organizations: Performance Year Approach Aligns Stakeholders‘ Interests, Health Aff (Millwood). 2013 Mar; 32(3): 587–595. doi: 10.1377/hlthaff.2012.0489

National Association of ACOs (NAACOS) member resource. The Impact of Retrospective Versus Prospective Attribution on Your ACO, https://naacos.memberclicks.net/the-impact-of-retrospective-versus-prospective-attribution-on-your-aco

Technical Appendix:

(p4) Total Assigned Beneficiary Difference
Total Assigned Beneficiary Difference
(p4) ACO Average Assigned Beneficiary Difference =

ACO Average Assigned Beneficiary Difference

(p4) Average ACO Match Percent =
Total Assigned Beneficiary Difference
(p5) Average PBPY Difference =

 Average PBPY Difference

(p5) Average Risk Score Difference =

 Average PBPY Difference

(p5) Average Risk-Adjusted PBPY Difference

 Average PBPY Difference