I think somethign is wrong... if your soil was a pH of 4, your plant wouldn't look healthy at all..
callibrate your device, or find somethign with better resolution to test... some .1 resolution strips work well enough, for example.
don't stop adjusting pH .. in soil you want 6.5-6.8. keep doing that with anything you add or if water-only.
With that said, if it is how you describe, it's a soil issue, not what you are putting in now (although what you put in could have caused it... if that makes sense - and not due to pH but due to chemicals invovled -- "organic" is still a chemical that adheres to basic laws of chemistry, thermodynamics, physics etc, just in case you are an "organic" person)
amending with some garden lime can combat a consistently acidifying soil... you have to do this when you mix the soil though.. top-dressing may help this early, but only watering for a very long time (not all at once, obviously). There are other things you can use to prevent this, if it is common.
the other way is to get different ferts or use them differently... or buy different soil / mix it up differently. it shouldn't be common, but can happen to anyone if rare.
.