Only one villager needs to pick up the gong piece, and it doesn't need to be a builder once you've opened the crate.

Maybe putting more than one villager on the piece is causing them to drop it?