'Our federal caucus is in a precarious position,