That's up to you and how much patience you've got. I don't bother personally, and have never yet completed all four sets of collectibles. The best I ever managed was three, plus ten beetles. After completing all the tasks I generally start a new game in a different slot, and pop back every now and then to refuel the fire, start villagers in new trades, bang the gong, etc. While I'm there I'll keep looking out for missing collectibles, but won't obsess about it. Some people have posted here that they've been looking for a particular collectible for months, but I just don't have the patience or that.
_________________________
To err is human; to arr is pirate.