The challenge to Hewlett Packard Enterprise’s acquisition of Juniper Networks came as many in corporate America had expected a lighter touch under a new administration.

The agency plans to argue that the company illegally penalizes customers that try to use rival payment processors.

Wall Street is abuzz about a potential deal, but the large price tag for the app is one of many factors that could limit the...