Royal Bank of Canada is facing a conundrum about one of its fastest-growing businesses: when to apply the brakes.

The bank's capital-markets business, after fighting for years to become a global player, threatens to become a victim of its own success. Earnings from capital markets are bumping up against a self-imposed limit that is key to the risk control on which Canada's banks built their recent reputation.