The fertilizer should be used on fully mature flowers. It won't work on immature flowers, or the vegetables. Certain flowers are more fragile than others, and the different fertilizers help keep them stable, but you should experiment to see which one works best on which flower (no sense in using a Tier 4 fertilizer on a Tier 2 flower!)

I should mention that all flowers will be vibrant while in inventory, but when they are planted, they might be brown, if they need water or fertilizer (if necessary).