Several years of subpar results from the private-equity industry's biggest players are testing the patience of some of their most-loyal investors: the public-employee pension funds that have come to count on these buyout firms for healthy returns.

A new report by a consultant to the California State Teachers' Retirement System, or Calstrs, shows that returns from large US buyout funds are lagging behind many of the pension's internal benchmarks.