In chapter 920 Kinemon revealed that Momonosuke, Kanjuro, Raizo, Kiku, and himself have time traveled 20 years from the past.
So, this mysterious guy (here highlighted in green) was with them right before the time travelling but after arriving 20 years later, he wasn’t around.
In chapter 939 we discover that the mysterious guy is one of Oden’s Nine Red Scabbards, Kawamatsu. Hiyori tells Zoro how she escaped from Oden Castle’s destruction 20 years ago with the help of Kawamatsu.
Hiyori remembers the death of her parents and disappearance of her brother 20 years ago, where she did not believe Momonosuke’s promise to return in 20 years and lost her will to live. Zoro asks why she did not go with him, and Hiyori reveals that it was to ensure the Kozuki Family’s bloodline would survive if the plan did not go right.
She was taken care of by the retainer Kawamatsu, who managed to secretly take her out of the burning castle by digging down into the water channel and swimming through it. They were separated when she was 13.
Finally, I want you all to remember the mysterious character in Chapter 631 cover page. This cover page shows Crocus and Laboon for the first time after the timeskip. Crocus is drinking with an unknown person wearing a Samurai or Japanese clothes which it seems related to Wano Kingdom.
I highly suspect that this is the same guy we are talking about. Look at that straw hat! Look at that design! It’s Kawamatsu!!!!
*Theory by JomaBusa
