No, I've generally got by with just the one fruit tree - plus of course, a lot of mushrooms! Once, with a lot of luck in getting duplicate rare collectibles, I got fishing before the (first) tree had even matured! It is important not to expand the tribe beyoyond the original eight villagers, although I did manage with "newcomers" from events a couple of times.