SYDNEY—Australia’s securities regulator has launched an inquiry into share-market operator ASX after what it called repeated and serious failures that raise doubts over its ability to maintain critical trading infrastructure.

The Australian Securities and Investments Commission on Monday said it had decided on the inquiry due to concerns shared by the Reserve Bank of Australia.