The Magic Circles are the Secret to the Longevity of the Gorosei
Through unknown means, Saturn was able to appear on Egghead by emerging from a “Magic Circle” on the ground as if through a portal; said circle included a pentagram in its middle with five number 5s in-between its jags.
If we go back to the end of Chapter 233 (Gorosei’s introduction) we see some more magical circles lining the walls.
So those magic circles may be slowing the aging process of the Gorosei. That would explain why they always stay in Pangaea Castle. But when they leave that room they start to age again. Their appearance didn’t seem to age after the 2 year time skip.
Saturn even states that it’s been a long time since he’s been to the surface. So it’s very possible that it’s been hundreds of years since they have left Pangaea Castle. With these magic circles it could very make it where the Gorosei are untouched by time but not have eternal youth or immortality.
*Theory by dlajoie1_