In the big business of advising on mergers and acquisitions, smaller is increasingly better.

A group of small M&A-advisory shops, known as boutiques, have muscled in on the signature Wall Street offering and now routinely take lucrative deal assignments from the big banks that have for years dominated the business.