I know what you mean, i accedently drag the wrong villager(Male) into another and now i have 5 women off their jobs caring for babies... Kinda funny though