That has happened to me when I was "keeping" villagers on task at the research table, by continuously dropping those who would walk away back to the table.
A couple of times doing this, when the table/lab was really full, I couldn't tell that a few of them
got the wrong idea, and when they were walking off to the Lovely Hut I thought they were just going off task. When I picked the ladies up, POOF there was the kid!
