How your soil was built? I do believe that adding organic matter as soon as possible is the best option, and for sure later next to middle flower adding more is a great options. It does take a long time to degrade and absorb by plant, so no worry by adding earlier. Also care with microorganism so they can decompose that matter and make them available.