ROOT JSON and NaN/Inf

ROOT
1

Hi all,

I tried converting a large number of TH1 derivatives into JSON using ROOT.TBufferJSON.ConvertToJSON(x) (from Python) and found that the resulting JSON is mal-formed.

This is due to NaN and Infinitiy values in the TH1s, especially +/-Inf is common in TProfile axis bounds. TBufferJSON represents them as “nan”/“inf”/"-inf" (without the quotes)[0], which it seems even JSROOT is unable to understand [1].

The Python JSON module expresses these as “NaN”/“Infinity”/"-Infinity", which is also not valid JSON [2] (the standard does simply not allow such values), but a syntax allowed in JavaScript and apparently accepted by a number of JSON parsers as an extension. So this would probably be a better choice.

Though some parsers are picky and will not allow any such values (looking esp. at databases, like the SQLite JSON extension [3] or Postgres JSONB [4]), so providing a “pure” output mode that removes Infs and NaNs (at the risk of losing information) might be worth considering as well.

What is your opinion on these things? Can we get this fixed on ROOT’s side?

Thanks,

Marcel

[0] Sample:

...
0, 0, 0, 0, 0],
  "fErrorMode" : 0,
  "fZmin" : -inf,
  "fZmax" : inf,
  "fTsumwz" : 12337.6925694346,
  "fTsumwz2" : 13893.578925121,
  "fBinSumw2" : []
}

[1] https://goo.gl/6LRsTU (CMS DQM GUI, needs certificate, might not work for non-CMS members)
[2] http://json.org/ , <link to rfc7159.txt>
[3] <sqlite docs>
[4] <postgres docs>
(sorry, not allowed to put links)

ROOT Version: 6.12/07 (provided by CMSSW)
Platform: slc6_amd64_gcc700
Compiler: GCC

2

Hi Marcel,

Why you want to save NaN or Infinity in JSON?
To my knowledge, JSON standard does not support them.

Are there real use-cases - like meaningful state of any ROOT object?

Regards,
Sergey

3

Hi Marcel,

I tried to check ROOT sources - there are no places where Infinity values assigned to persistent data members.

Probably, you could try to avoid such values in your code - at least for case of objects which should be delivered to JSROOT. While there is no equivalent of “Infinity” and “NaN” in JSON, will be very “expensive” to provide ROOT-special solution.

Regards,
Sergey

4

Hi Sergey,

as you might remember from the ROOT workshop [1], we are sitting on a pile of about 44e9 ROOT histograms from the last ~5 years, and I can’t really do much about what’s inside there.

For the inf’s, it is quite obvious where they come from: when you have a TProfile, and don’t want to lose the entries outside the Y (or Z) range, easiest is to make the range inifinite by setting +inf and -inf for the limits – and that is what many people have done over the years. (There might be other ways to do it, but now it is too late, the objects are there and work fine in classic ROOT).

For the NaNs, I am not sure where they come from, but I know for sure that every batch of CMS DQM output contains a few histograms with NaN bin values. These might be used to make bins white (for unused) on our red/green (for error/ok) color scale used in summary plots.

I could live with these values not being handled correctly (e.g. replace them with 1e38, -1e38, 0), but I really think that JSROOT should be able to read whatever TBufferJSON produces.

Thanks,

Marcel

[1] https://indico.cern.ch/event/697389/contributions/3108820/attachments/1714101/2764583/cms-dqm-rootworkshop.pdf

5

Yes, this can be used as workaround for infinite values.
With NaNs there is no good solution - one cannot use null or 0 at this place.

6

Yes, replacing NaN with anything will screw up the semantics. Using a JavaScript null or empty object might be the most “correct” thing to do, but will be a pain to work with…

Still, I think we need a way to handle this, maybe optionally.

7

Hi Marcel,

I submit PR to solve problem with infinite values:

I uses value 2e308 instead. It is ok for JSON, but when parsed by JavaScript, it replaced with Infinite values again. Means transport from C++ to JavaScript ROOT will work now.

Same patch will be applied for 6.16 branch.

Regards,
Sergey

8

Thanks Sergey!

I worked around by a bit of regex-replacing, which makes things work for now but will fail in certain cases. Good to know this can go away at some point.

Cheers,

Marcel

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.