Yes, that's how you know you've reached the maximum number of villagers. If you train those worried villagers to do something else (like healing or farming), they'll usually stop worrying.