Feel free to remove any and all shade leaves at any point in time. I remove any leaves that cover potential branch and bud nodes throughout the entirety of my grow. I also remove the top to leaves of every branch, including the main stalk, when the plant is as little as a couple weeks old. It makes the plant react as if it were topped, but without ever topping it. This means you aren't removing any vertical growth or potential branches. The plant also recovers much faster than if it were topped, so i think it's a win-win.
Once the plant is flipped into flower, i always do a heavy defoliation during week three to help with airflow and light. You don't want any leaves covering your nuggies. Some people even remove practically every leaf on their plant during this defoliation and end up with fat buds! So, the people that say you need all the leaves your plant produces, are just wrong. In my personal opinion, that is.