The assessment of U.S. 50-top hospitals, annually ranked by U.S. News & World Report, are based on subjective interpretations, rather than quality. And these subjective reputations have very little association to objective quality measures among the top 50 hospitals, according to a study published April 20 in the Annals of Internal Medicine.

“Hospital leaders generally believe that the U.S. News & World Report rankings are accurate and use them as marketing tools to attract patients,” the authors wrote.

Currently, the U.S. News & World Report rankings measure equal parts of a facility’s structure including process and outcomes. For the most part these ratings are determined by analyzing objective hospital measures such as staff ratios, volume, adjusted mortality and adverse event rates.

However, process rankings are based on the subjective views of 250-board certified physicians who are selected at random to assess five hospitals in their specialty that “provided the best care to patients.”

To determine the accuracy of the annual rankings, particularly the subjective aspects, Ashwini Sehgam, MD, of Case Western Reserve University and MetroHealth Medial Center in Cleveland, and colleagues quantified the role reputation plays when identifying the U.S. News & World Report's top 50 hospitals.

The study uses the pre-set rankings to depict degrees of variation within survey results. “Understanding the role of reputation may help users interpret the rankings more knowledgeably and may inform efforts to improve the assessment of hospital quality,” the authors wrote.

During the study, the researchers used descriptive statistics to analyze the reputation scores for 12 specialties: cancer; diabetes; endocrine disorders; digestive disorders; ear, nose and throat; geriatric care; gynecology; heart surgery; kidney disorders; neurology and neurosurgery orthopedics; respiratory disorders; and urology.

Additionally, the researchers compared reputation scores from 100 randomly selected unranked hospitals and ranked each specialty based on reputation scores alone.

The researchers found that of the rankings in which the researchers based reputation scores alone—100 percent concurred with U.S. News & World Report rankings for the top hospital for each of the 12 specialties.

In addition, the results showed that the researchers' results concurred with U.S. News & World Report rankings for the top five hospitals, top 10 hospitals and top 20 hospitals, 97, 91, and 89 percent of the time.

Other results showed that only 64 percent of the reputation scores of the top 50 hospitals were below 5.0. Additionally, 31 percent of reputation scores were 5 – 30 and 4 percent were greater than 30.

These reputation scores were based off descriptive statistics (mean, SD, and percentage), and calculated by a coefficient of variation.

Researchers found a large variation across the top ranked hospitals in each specialty. For highly ranked cancer facilities the degree of variation was 177 percent and reputation scores ranged from 6 percent to 63 percent for other components, according to the author.

“By combining several subjective and objective measures, U.S. News & World Report’s rankings appear to be a rigorous, complex, and multidimensional index of hospital quality.” However the authors noted that “the relative standings of the top 50 hospitals largely reflect the subjective reputation of those hospitals.”

“This subjective reputation does not already capture objective measures of hospital quality; in fact, little relationship exists between reputation and objective measures in the top 50 hospitals,” the authors noted.

For 100 randomly selected unranked hospitals, researchers found that these hospitals all had reputation scores equal to zero. “These data suggest that hospitals lacking national recognition are unlikely to be highly ranked by U.S. News & World Report,” the authors wrote.

“High rankings also may enhance reputation, which in turn sustains or enhances rankings in subsequent years. Users should understand that the relative standings of U.S. News & World Report’s top 50 hospitals largely indicate national reputation, not objective measures of hospital quality,” said the authors.

The authors warned that highly rated hospitals should be weary of rankings based solely on subjective reputation and said that “reputation-based rankings may act as barriers to improving hospital quality, because even highly successful quality improvement programs are unlikely to enhance a hospital’s rankings.”

While U.S. News & World Report claims that its rankings are based “largely on hard data,” the authors offered that to make this claim, the basis of reputation-based rankings should be eliminated.

"The current rankings fall short of being an evidence-based system that data-conscious consumers, value-based purchasers and reform-minded policymakers can rely on for healthcare decision,” the authors concluded.