I am not sure which limit you reached, however there is indeed some hard limits in TFormula and TTreeFormula that you can exceed (nor customize short of recompile ROOT).
You may want to consider switch to using a draw script (see the TTree::Draw reference guide documentation) or the result of MakeProxy or the result of MakeSelector.