Labor groups said working conditions had not changed enough to warrant the removal. The company is partly owned by donors to President Trump.