Do you have some families where it's daytime for them when it's daytime for you and some families where it's daytime for them when it's nighttime for you? Like, out of 5 slots, 3 are day when it's day for you and 2 are night when it's night for you? Because I extensively tried this and it won't work. Even if you don't bother the day ones when you're playing the night ones, the day peeps suffer.

What I ended up doing was installing the game on a second user account on my computer. So, during the day I have all day families on my "regular" account that I've always had. Then, when I want to sit down and play in the evening, I leave that game closed and instead I go into my second user account and play those families. I haven't had anymore problems with tiredness or exhaustion since I started doing that.