I think it's more if you leave the castle and someone you care about shows up here, then you're going to miss out on them if you're living far away. Though I guess since the journals still work it might be possible to know when they show up and go back to take them away from the castle.

Also the fact that powers don't really work away from the castle, and it kind of sucks.

What kind of game though?