Well for one, most of them aren't suited for that level of Team Work. There's only so much Nagato to go around, and the weaker people are susceptible to moves like Shinra Tensei and Chibaku Tensei, on top of being turned into chakra batteries for Nagato or controlled/subdued through Rinnegan Genjutsu/intimidation.Even excluding the current top tiers, who all don’t really have a reason to fight him funnily enough, assuming all the other characters ganged up on him, Nagato would likely still be getting clapped around. If anything, there’d have to be a reason why they don’t do that.
Essentially too many people become a liability, and people who aren't at least somewhat adept at Taijutsu or non-chakra-based attacks are basically just feeding Nagato chakra.