I get why the Gorosei didn’t target Luffy earlier. That has to do with 3 things: not having information, their objectives and lack of opportunity.
Let’s do a recap.
1- When Shanks stole the Gomu Gomu no Mi, they probably knew the isles that Shanks visited but by no means they had enough personnel to send to all those locations. Goa was probably the hardest one to spy because of Dragon and Garp.
2- So they never really knew that Luffy ate the Gomu Gomu no Mi. They probably only discovered that after Alabasta.
3- Now they could have hunt Luffy since Alabasta, or put a tremendous bounty on him. But Luffy was a rookie, giving a rookie a high bounty would probably raise a lot of questions that should not be made. Also even Luffy’s bounty post Alabasta maybe had some interference because those bounty posters took a while to reach Jaya for example.
4- They could have sent assassins, but that would probably create problems with Dragon and Garp. Also they could only assassinato Luffy on land, because finding someone on the sea is very hard. After Alabasta Luffy got to Skypiea (disappeared), Water 7, disappeared again in Thriller Bark, briefly apperared in Sabaody and disappeared yet again thanks to Kuma. Nobody expected Luffy to drop from sky into Marineford. The only window of opportunity for the Gorosei was Water 7, but then they opted to get Robin.
5- The Gorosei objectives is mainly preventing knowledge about the Ancient Kingdom and prevent Imu being toppled. They can achieve this by killing Robin. That would be the optimal course of action: killing Robin and maintaining the secret about Gomu Gomu no Mi. That’s why they kidnapped Robin and not Luffy on Water 7.
6- Finally, post time skip everything happened very fast until Luffy got to Wano. And then they had the very first opportunity because as they even said, no one would suspect anything if Luffy died in a battle with Big Mom and Kaido. Anything could have happened.
I don’t think it is strange that they took so long considering that most of time they didn’t know where Luffy was and when they knew there were other courses of action that were more optimal (killing Robin) or killing Luffy would raise too many questions from top players (Dragon, Shanks) and even from some Marines (Garp).
*by Tolkius