Plants do get damaged and die off it's normal, but since you have had the aphid issue this is more than likely the result of that, and yes you should remove damaged or dying leaves not only because they are a magnet for pest but it also makes the plant focus on recovering the leaves instead of actually growing. Hopefully, I answered your question and remember Happy Growing and Free your mind one puff at a time.