Are you changing the game speed right after you switch players? If you are, that's the reason you're seeing time pass. As I mentioned before, if you go to the menu and change players, then change the game speed before you click Play, the game will ignore that you had it paused all that time and update all of your fishies at the new speed setting when you click Play. You must click Play immediately after switching players if you want to preserve your game speed setting. Once you've clicked Play, you can go back to the menu and change the game speed.