Google 'Spongodes montipora', the polyps ARE brown. LOL. I used to think like you are thinking many years ago, then I learned better and stopped listening to the 'experts' and started listening to my tank.
Bottom line, corals adapt, but corals do NOT like big changes in anything. If I drop my phosphates from .5 down to .03, the corals will bleach out (due to not enough food), if I raise my phosphates from .03 to .5, my corals will brown out (due to too much food, making the zooxanthallae populations explode, which is what gives corals the brown color), if I let corals adapt over time to .5 phosphates, they color up and look fantastic. Zooxanthallae is algae, and algae eats N/P, right? It all comes down to finding a good balance in the tank, whatever methods a person decides to use. There is no right or wrong way to run a reef tank as long as whatever method you choose works.
This is what my 75G used to look like, running a skimmer, reactor, and caulerpa for filtration, before I got into ATS's.