Ooooooh! You're mean! Poor little villagers aren't allowed to relax in your game?

Seriously, the villagers will spend some of their time doing other things once their urgent needs are met. You can micro-manage them (as I still sometimes do), or you can just relax and let them do what they want to do.