That's right. The trees never run out of fruit. However, not many food points are gained, espcially as it takes the farmers so long to shake down the fruit, cook it, then take it to the food bin. With a moderately-sized village you'd need several farmers to keep the village properly supplied relying solely on the yellow fruit. That of course might slow down the village's development elsewhere as they could be better deployed as builders or researchers.