The acid test for private equity has always been whether it could continue to generate the premium returns that PE firms promised their limited partners (LPs). The industry generally made good on that pledge, usually outperforming all other asset classes, on average, across the ups and downs of the business cycles. However, as we point out in Bain’s Global Private Equity Report 2012, returns since the 2008 market meltdown have gotten little support from the tepid economic recovery or volatile debt and equity markets.

The dynamics influencing private equity returns suggest that the industry has become trapped by its own success. An ever-increasing number of PE firms able to raise deep reserves of dry powder compete fiercely for deals, keeping asset prices high and leaving little or no room for return multiples to expand at a time of sluggish GDP growth, higher-cost debt and lower levels of leverage. Those dynamics do not appear likely to change anytime soon.

The challenging overall outlook for returns means that, more than ever before, LPs need to be able to identify and work with top-quartile general partners (GPs) that are not dependent on market beta to boost their results. The private equity performance gap between the best and the rest is huge (see figure below). Indeed, the performance leaders generate a disproportionately large share of the value-creating alpha that accounts for PE’s superior returns. A recent analysis by Oliver Gottschalg, a professor at the École des Hautes Études Commerciales in Paris, found that fund managers in the top half of the rankings generated 18% of alpha, and an elite group of performance superstars—GPs whose returns rank in the top 5% relative to the peer group—generated alpha worth an impressive 40 percentage points in returns.

How can LPs separate the few true winners from the pack of followers? A successful track record is one important guide—but only a partial one. GPs that have managed a top-quartile fund have a better than six-in-10 probability that their successor fund will also be an above average performer. But the persistence of a record of past success may be weakening over time. A recent Bain analysis, which compared the performance of pairs of successive buyout funds from vintages of 2000 and earlier with fund pairings from vintages of 2001 and later, found that 37% of successor funds from the pre-2000 pairings were able to repeat their predecessor’s top-quartile performance. Only about 27% of the successor funds of the post-2001 pairings did as well.

It is too early to determine whether the drop-off in top-quartile consistency is a one-time blip or a longer term trend. But LPs need to be open to the possibility that the rules of the game have changed and subject even the best-performing GPs to a deeper level of scrutiny each time they come back to the market.

What specific traits should LPs look for? First, they want to ensure that GPs can build durable partnerships with strong portfolio-company management teams that possess distinctive operational expertise and have incentives in place that motivate them to succeed. Second, they need to weigh the internal organizational dynamics of the PE firm as a whole. Finally, they need to look beyond the superficial results of past deals and rigorously analyze how the GP brought its capabilities to bear to generate alpha.

In the challenging three years since the private equity recovery has tried to assert itself, both GPs and LPs have come to recognize that they can no longer count on market beta to be a wind at their backs. For GPs that aspire to be performance leaders, it is time to roll up their sleeves and demonstrate that they can build great companies.

This post was written by Graham Elton, Bill Halloran, Hugh MacArthur and Suvir Varma, leaders of Bain & Company’s Private Equity Group.