It is the age-old art of sealing a deal: Making your product seem more desirable than it may actually be.

But on Wall Street, where traders for decades have tried to cajole clients into buying or selling, the question of how far they can go has become pivotal.