The spell ‘resurrection’ can bring someone back from the dead using only a small portion of their body. Now I ask- If someone is resurrected more often than once, does it matter which body is used?

Scenario: Bob the fighter is slain in combat. The enemy rips off his head and keeps it as a trophy. The rest of the body is later used by the party to resurrect him. Now, the party prepares for revenge. They strike at the Ogre in question, and defeat him. As a fitting reward, and to remember this battle, Bob takes ‘his’ skull from the Ogre and keeps it with him. Years later, Bob dies again. His body is not accessable to the rest of the party. All they have is the skull of his first incarnation. Could this skull be used to resurrect him despite it not being part of his body last time he was alive?