It's not as simple as you'd like to think. For one, the castle ends up incredibly vulnerable when all the heavy hitters, metas and magic users leave. People who can't fight or defend themselves can't be left unguarded all the time. The First Gens have stormed the castle in the past and we were lucky that none of us were around at the time.

We also have to travel on foot for the most part. It takes days to wander around some areas. Exploration of the land can take years for all we know. With people coming and going all the time, the groups need those capable of keeping up physically and even emotionally for whatever else is out there.