It does seem to happen when you open the game after it's been closed for a while -- the game goes back and calculates all the food that would have been eaten while the game was closed, and then removes it. If you've been gone a while, or if you have other events pop up when you restart, the food calculation might take a minute or two.

Try using a slower game speed when the game is closed (or you can't watch it) -- and try micromanaging a bit. Keep everyone at the research table (no building, no farming, no breeding) until the crops come in, then let one or two villagers harvest.

Good luck!
-- jen