Most mobile devices don't get restarted often, nor are they cleared of apps being in the "background" (it's more involved than that, but this term will work). After time, having so many apps open will "bog down" or slow down the device, or even make apps crash. In most cases, it's an issue that can cause performance problems.

However, in your case, it looks like it's helpful. If you'd like to see your villagers move slower, don't power off/start your device, and over time they'll get slow. If you see the game crash though, you might have to close a few apps.

(For everyone else, clearing out your apps can help your device's performance - but don't use task killers! Those can often delete your save game data. Instead, close apps according to your device's manual.)