If you compile your code you use the standard C++ way of getting a nan, e.g. for a double variable
const double NAN = std::numeric_limits<double>::quiet_NaN();
This unfortunately doesn’t seem to work in CINT (neither does a possible way in C via nan() from math.h).
If you really need to have this work in interpreted code (i.e. not compiled), you can as a temporary workaround use results of undefined operations. Since with any of these your code becomes undefined as far as ROOT and C++ are concerned you should at least use a variable/define, so you can migrate any resulting breakage more easily. Using compiled code saves you from all this since in C++ there are ways to access a possible NaN value.
const double NAN = sqrt(-1);
If you can find something on that in the ROOT documentation, then go ahead and use it. Otherwise I would expect this to break anytime things in that part of the code get shuffled around.