I yell at my villagers every few minutes that no matter how many times they try they can't push the boulder. I pick them up and throw them into the ocean after they go back a second time.