I use a decent skimmer; I use GAC in a fluidized reactor; I don't feed alot
It's this. You are removing what little particle food (skimmer) and dissolved food (GAC) you do have, that is needed for growth. Since you have no nutrient/algae problems, you can be pumping in much more food per day, which will benefit more that just the sps. Also, by feeding more, your scrubber will grow darker (less "efficient" filtering), which will allow the DOC (and bacteria) to increase further. Lastly, increased feeding/DOC/N/P will allow softies/lps to grow, and maybe even nps if you have enough particles and flow. Try that in zeo/pellet tanks.
Skimmer/GAC is never a good combination with scrubbers, when it comes to stony growth. It defeats the very purpose of the algae, nutrition-wise. Zeo/pellet systems, when they are not bleaching corals, are providing lots of particles (bacteria). With scrubber-only systems, you can't bleach the corals (due to self-limiting algal growth), but you can remove too much of the DOC/particles supplied by the algae.