Jagmeet Singh ‘not at all’ worried whether losing a quarter of incumbents reflects leadership