Here is list of actions that villager will do when they are not working:
Dancing
Rain Dancing
Relaxing
Unsure of what to do
Bored
Telling a story
Telling a joke
Chatting
Worrying about foods
Worrying about housing
Teaching
Exercising
Laundry
Going swimming
Going to Villager Center
Drinking well water
Chasing the Butterflies
Watering crops (hmm does that count as working)