TClass names for non-C++ types

hi there,

I am trying to provide full inter-operability with ROOT, from Go.
I can read and write ROOT objects all right (mostly.)

and now, I am in the middle of providing write ability to Go-based types.
most of this is working:

  • generating a TStreamerInfo
  • generating all the needed TStreamerElement
  • saving these in the ROOT file TStreamerInfos

there is currently one snag:

  • how to make the name of the Go type fully inter-operable with what TClass and friends expect?

A Go type is fully qualified like that: <package-name>.<type-name>.
where package-name can be, e.g. go-hep.org/x/hep/hbook and type-name: H1D.
so, for example, the fully qualified name of that type would be: go-hep.org/x/hep/hbook.H1D in a Go program.

Obviously, the /, . and - characters confuse TClass and friends.
e.g:

>>> import ROOT
>>> f = ROOT.TFile.Open("out.root")
TClass::Init:0: RuntimeWarning: no dictionary for class go-hep.org is available
>>> o=f.Get("my-evt")
<ROOT.TClass object ("go-hep.org") at 0x55d97d4aa150>
>>> f.ls("*")
TFile**		out.root	out.root
 TFile*		out.root	out.root
  KEY: go-hep.org/x/hep/groot/redm.Event	my-evt;1	
>>> ROOT.TClass.GetClass("go-hep.org")
<ROOT.TClass object ("go-hep.org") at 0x55729e691c00>
>>> cl.Print()
OBJ: TClass	go-hep.org

where out.root contains this:

$> root-ls -sinfos ./out.root
[...]
 StreamerInfo for "go-hep.org/x/hep/groot/redm.Event" version=1 title="go-hep.org/x/hep/groot/redm.Event"
  Name  TString offset=  0 type= 65 size= 24  
  run   long    offset=  0 type=  4 size=  8  
  evt   long    offset=  0 type=  4 size=  8  
[...]

would anyone have any guidance on how to handle this?

should I try to put all the Go types in a “special” ::golang namespace and translate all characters that C++ can’t handle to something it can?
say:

Go:  go-hep.org/x/hep/groot/redm.Event
C++: ::golang::go_hep_org::x::hep::groot::redm::Event

?
(could I use “equivalent” UTF-8 characters for - and . ?)

cheers,
-s

PS: I would expect that people playing with Python (via uproot), Rust or Julia will encounter this type of issue at some point too…


ROOT Version: 6.14/04
Platform: Linux
Compiler: Go-1.11.x


Hi Sebastien,

should I try to put all the Go types in a “special” ::golang namespace and translate all characters that C++ can’t handle to something it can?

I agree that your proposal goes in the right direction. One thing I would not do is prefix with ::golang which in first approximation an implementation detail. Using:

Go:  go-hep.org/x/hep/groot/redm.Event
C++: go_hep_org::x::hep::groot::redm::Event

would allow to read the data in the C++ class:

namespace go_hep_org {
namespace x {
namespace hep {
namespace groot { 
namespace redm {
class Event { .... };
}
}
}
}
}

Cheers,
Philippe.

hi Philippe,

yes, at first approximation this is indeed an implementation detail.

but not knowing a given TStreamerInfo actually comes from a Go-generated type, we loose information and end up with some hysteresis effect:

  • how could groot know that go_hep_org::x::hep::groot::redm::Event is actually a Go type and e.g. edm::reco::Particle is a C++ type?

I could also imagine that one would like to perform some special operation for a streamer that comes from Rust, Julia or Go (e.g. to register it with some Rust,… or Go type registry or some garbage collector hook, etc…)

is there a place somewhere in TStreamerInfo where one could stash this kind of metadata? or a way to encode it somewhere that would be backward and forward compatible? or some syntax to use?

PS: I was suggesting using ::golang because it reminded me of the issue we had with string and std::string, where one reflection type system would use the former while the other would use the latter, leading to some inefficiencies. I fear one could end up in such a situation.

In my opinion the file should not know where the objects comes from.

I could also imagine that one would like to perform some special operation for a streamer that comes from Rust, Julia or Go

It shouldn’t because the data ought to be the same no matter what wrote it.

( e.g. to register it with some Rust,… or Go type registry or some garbage collector hook, etc…)

This should be the reverse. I.e: here is a StreamerInfo with a given name, does the current process know about it natively, if not then emulate it. [In the C++ library, the algo is something like, do we have a TClass for it, if not do we know how to autoload a library for it, if not does the interpreter knows about it, if not then we emulate that class]

is there a place somewhere in TStreamerInfo where one could stash this kind of metadata? or a way to encode it somewhere that would be backward and forward compatible? or some syntax to use?

I think only the StreamerInfo’s title fit that bill.

Cheers,
Philippe.

ok.

I’ll play a bit with the TStreamerInfo's title to see what gives.

alternatively, I guess I could also save a “special” TStreamerInfo where all the needed Go specific stuff could be encoded.

e.g. to disambiguate between:

  • a_b_org::edm::Foo: a C++ class named Foo in the namespace a_b_org::edm; and
  • a_b_org::edm::Foo: a Go type named Foo from the package a-b.org/edm

e.g. to disambiguate between:

One question is whether you ever need to. I.e. do you envision a case where both would live in the same process and be semantically independent.

Cheers,
Philippe.

For example: from C++ you would have no choice but to read a a_b_org::edm::Foo into the C++ version and from Go you read it into the go-version, which if both exist would (better be) memory layout identical (or at least trivially translatable by the Go-to-C++ interfacing layer.

well, I don’t know, yet.
in a non-distributed application (well, every application is time-distributed…) it may not be the case, but in a distributed, polyglot, application such as AliceO2, where a process in an analysis pipeline may receive data from any number of processes upstream – with processes that can be written in any number of languages – this might be a more common situation.

yes, they better should, of course :slight_smile:
there’s also the issue that, when given a_b_c_org::edm::Foo, it may very well refer to either:

  • a-b.c.org/edm.Foo
  • a.b-c.org/edm.Foo

in the POOL old days, we had a set of special top-level data items (##POOL_Links, and other ##POOL_xyz, IIRC) that would encode similar metadata.
I think I’ll go this way (after a good night sleep, I realized I couldn’t encode all of what I wanted in a “polymorph” TStreamerInfo, without confusing ROOT b/c of an ever changing Checksum…)

This brings me to another question: how are the TStreamerInfo checksums computed ?
I vaguely recall there was some (unwanted?) change in how they were created at some point.
is there some specs or algorithm that describes what goes in? (and how?)

cheers,
-s

You are right and you ought to encode ‘.’ and ‘-’ differently.

well, I don’t know, yet.

I still think that you should not need it. Currently the I/O is producing a ‘platform independent binary form’ of the objects and thus you do not want to be encoding things differently depending on which language is writing it. [And even a newer ‘zero-copy’ mode would ‘just’ have to have the 2 sides to agree that they are they have the same data layout no matter which language they use]

but in a distributed, polyglot, application such as AliceO2 … this might be a more common situation.

Humm … I am not sure what you mean … Are you envisioning that the various element would reuse the same names for 2 semantically different types? [Since the I/O layer does the translation between platform independent binary form and in-memory layout, it does not matter if the 2 process have different memory layout [unless they want to use the yet-to-be-implemented-zero-copy-format]]

This brings me to another question: how are the TStreamerInfo checksums computed ?

I don’t think we got around to describe using english sentences. The authoritative answer is at: UInt_t TClass::GetCheckSum(ECheckSum code, Bool_t &isvalid) const

Cheers,
Philippe.

mission accomplished :slight_smile:

given the following Go type definition:

package redm

//go:generate root-gen-streamers -p go-hep.org/x/hep/groot/redm -t Event,HLV -o pkg_gen.go

// Event is a simple type to exercize streamers generation.
type Event struct {
	name string `groot:"Name"`
	run  int64
	evt  int64

	u8  uint8
	u16 uint16
	u32 uint32
	u64 uint64

	i8  int8
	i16 int16
	i32 int32
	i64 int64

	f32 float32
	f64 float64

	b bool

	HLV HLV

	ArrU8  [10]uint8
	ArrU16 [10]uint16
	ArrU32 [10]uint32
	ArrU64 [10]uint64

	ArrI8  [10]int8
	ArrI16 [10]int16
	ArrI32 [10]int32
	ArrI64 [10]int64

	ArrF32 [10]float32
	ArrF64 [10]float64

	ArrBs [4]bool
}

func (*Event) RVersion() int16 { return 1 }
func (*Event) Class() string   { return "go-hep.org/x/hep/groot/redm.Event" }

// HLV is a simple type to exercize streamers generation.
type HLV struct {
	px, py, pz, e float64
}

func (*HLV) RVersion() int16 { return 1 }
func (*HLV) Class() string   { return "go-hep.org/x/hep/groot/redm.HLV" }

and the following equivalent C++ definition event.h:

#include "TString.h"

namespace go_hep_org { namespace x { 
namespace hep { namespace groot { namespace redm {

	class HLV {
	public:
		double px,py,pz,e;
	};

	class Event {
	public:
		TString                              Name;
		long                                 run;
		long                                 evt;
		unsigned char                        u8;
		unsigned short                       u16;
		unsigned int                         u32;
		unsigned long                        u64;
		char                                 i8;
		short                                i16;
		int                                  i32;
		long                                 i64;
		float                                f32;
		double                               f64;
		bool                                 b;
		go_hep_org::x::hep::groot::redm::HLV HLV;
		unsigned char                        ArrU8[10];
		unsigned short                       ArrU16[10];
		unsigned int                         ArrU32[10];
		unsigned long                        ArrU64[10];
		char                                 ArrI8[10];
		short                                ArrI16[10];
		int                                  ArrI32[10];
		long                                 ArrI64 [10];
		float                                ArrF32[10];
		double                               ArrF64[10];
		bool                                 ArrBs[4];
	};
}}}}}

streamers are automatically generated, either at runtime or at buildtime, like so (for the latter):

$> go generate ./redm
$> go get ./redm

then, running the following Go program:

package main

import (
	"log"

	"go-hep.org/x/hep/groot"
	"go-hep.org/x/hep/groot/redm"
)

func main() {
	f, err := groot.Create("out.root")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	evt := redm.New("hello", 42, 666)
	err = f.Put("my-evt", evt)
	if err != nil {
		log.Fatal(err)
	}

	err = f.Close()
	if err != nil {
		log.Fatal(err)
	}
}

produces the following:

$> go run ./test-streamers.go
$> root-ls -sinfos ./out.root
=== [./out.root] ===
version: 61404
streamer-infos:
 StreamerInfo for "go_hep_org::x::hep::groot::redm::Event" version=1 title="Go;go-hep.org/x/hep/groot/redm.Event"
  TString                              Name    offset=  0 type= 65 size= 24  
  long                                 run     offset=  0 type=  4 size=  8  
  long                                 evt     offset=  0 type=  4 size=  8  
  unsigned char                        u8      offset=  0 type= 11 size=  1  
  unsigned short                       u16     offset=  0 type= 12 size=  2  
  unsigned int                         u32     offset=  0 type= 13 size=  4  
  unsigned long                        u64     offset=  0 type= 14 size=  8  
  char                                 i8      offset=  0 type=  1 size=  1  
  short                                i16     offset=  0 type=  2 size=  2  
  int                                  i32     offset=  0 type=  3 size=  4  
  long                                 i64     offset=  0 type=  4 size=  8  
  float                                f32     offset=  0 type=  5 size=  4  
  double                               f64     offset=  0 type=  8 size=  8  
  bool                                 b       offset=  0 type= 18 size=  1  
  go_hep_org::x::hep::groot::redm::HLV HLV     offset=  0 type= 62 size= 32  
  unsigned char                        ArrU8   offset=  0 type= 31 size= 10  
  unsigned short                       ArrU16  offset=  0 type= 32 size= 20  
  unsigned int                         ArrU32  offset=  0 type= 33 size= 40  
  unsigned long                        ArrU64  offset=  0 type= 34 size= 80  
  char                                 ArrI8   offset=  0 type= 21 size= 10  
  short                                ArrI16  offset=  0 type= 22 size= 20  
  int                                  ArrI32  offset=  0 type= 23 size= 40  
  long                                 ArrI64  offset=  0 type= 24 size= 80  
  float                                ArrF32  offset=  0 type= 25 size= 40  
  double                               ArrF64  offset=  0 type= 28 size= 80  
  bool                                 ArrBs   offset=  0 type= 38 size=  4  
 StreamerInfo for "TString" version=2 title=""
 StreamerInfo for "go_hep_org::x::hep::groot::redm::HLV" version=1 title="Go;go-hep.org/x/hep/groot/redm.HLV"
  double px      offset=  0 type=  8 size=  8  
  double py      offset=  0 type=  8 size=  8  
  double pz      offset=  0 type=  8 size=  8  
  double e       offset=  0 type=  8 size=  8  
---
go_hep_org::x::hep::groot::redm::Event my-evt          (cycle=1)

with the following content:

$> root-dump ./out.root
>>> file[./out.root]
key[000]: my-evt;1 "" (go_hep_org::x::hep::groot::redm::Event) => &{hello 42 666 8 16 32 64 -8 -16 -32 -64 32.32 64.64 true {1 2 3 4} [1 2 3 4 5 6 7 8 9 0] [1 2 3 4 5 6 7 8 9 0] [1 2 3 4 5 6 7 8 9 0] [1 2 3 4 5 6 7 8 9 0] [-1 -2 -3 -4 -5 -6 -7 -8 -9 0] [-1 -2 -3 -4 -5 -6 -7 -8 -9 0] [-1 -2 -3 -4 -5 -6 -7 -8 -9 0] [-1 -2 -3 -4 -5 -6 -7 -8 -9 0] [-1.1 -2.2 -3.3 -4.4 -5.5 -6.6 -7.7 -8.8 -9.9 0] [-1.1 -2.2 -3.3 -4.4 -5.5 -6.6 -7.7 -8.8 -9.9 0] [true false false true]}

which can be read from ROOT/C++ like so:

$> root
root [0] .L ./testdata/event.h++
Info in <TUnixSystem::ACLiC>: creating shared library /home/binet/work/gonum/src/go-hep.org/x/hep/groot/././testdata/event_h.so
Warning in cling::IncrementalParser::CheckABICompatibility():
  Possible C++ standard library mismatch, compiled with __GLIBCXX__ '20180831'
  Extraction of runtime standard library version was: '20181127'
root [1] auto f = TFile::Open("./out.root");
root [2] auto k = f->GetKey("my-evt");
root [3] auto ee = k->ReadObject<go_hep_org::x::hep::groot::redm::Event>()
(go_hep_org::x::hep::groot::redm::Event *) @0x7ffd104d0e10
root [4] ee->Name
(TString &) "hello"[5]
root [5] ee->evt
(long) 666
root [6] ee->run
(long) 42
root [7] ee->ArrF64
(double [10]) { -1.1000000, -2.2000000, -3.3000000, -4.4000000, -5.5000000, -6.6000000, -7.7000000, -8.8000000, -9.9000000, 0.0000000 }
root [8] ee->HLV.px
(double) 1.0000000
root [9] ee->HLV.py
(double) 2.0000000
root [10] ee->HLV.pz
(double) 3.0000000
root [11] ee->HLV.e
(double) 4.0000000

how cool is that?!

there are still a few interesting things to handle:

  • Go slices
  • Go types that “inherit” from TObjects
  • STL containers
  • polymorphic containers

but hey… I can basically write almost any kind of Go values in a ROOT file (at the top level.)

Congratulation!.

Cheers,
Philippe.
PS. In go-hep.org/x/hep/groot/redm.Event what is the semantic of the /x/ part?

eXtra code.

it’s also a convenient way to separate end-points in the go-hep.org server:

  • everything under /x/ is Go code
  • everything else is doc, blogs, etc…

I piggy backed on golang.org/x/sync (and others) where /x/ contains extra code, not part of the Go standard library but still interesting and well maintained.

Thank you. Philippe.

no, thank you!
I really appreciate the time you took to help me figure this out.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.