Two things:

1. The fact that when your kids have moved out, you're just waiting for the parents to die. It's not so bad in the beginning, as you can still buy upgrades for the house, repair floors, walls etc, but once you have everything, it's just tedious.

2. The fact that some collectibles only appear when it's raining, and often not even then. I like the collectible aspect, but not this part. I'm on Generation 12 of my current family and I have two bugs. Beyond frustrating.