Segfault when writing STL map to TBufferFile

mjkramer · August 5, 2014, 4:44am

Hello,

I’m getting a segfault when I try to serialize an STL map (with std::string keys and values) into a TBufferFile in order to send it over a Unix socket. The following code reproduces the crash:

#include <map>
#include <string>
#include <TBufferFile.h>
#include <TApplication.h>
#include <TROOT.h>

using namespace std;

int main(int argc, char** argv)
{
  TApplication app("test", &argc, argv);

  map<string, string> m;
  m["foo"] = "bar";

  TClass *c = gROOT->FindSTLClass("std::map<std::string,std::string>", true);
  TBufferFile b(TBuffer::kWrite);
  b.WriteObjectAny(&m, c);

  return 0;
}

Is there anything obviously wrong with my approach? It actually seems to work fine inside CINT/Cling, although I suspect this might be related to some sneaky behind-the-scenes conversions between std::string and TString. I am having no problem serializing/transmitting/reconstructing objects that derive from TObject. It’s just this STL map that’s giving me trouble. FindSTLClass does in fact return a valid TClass*, and GetName() returns “map<string,string”>.

I suppose that, in the worst case, I can switch to a TMap, but I’d really like to avoid that. If anyone can help me, I’d greatly appreciate it. For what it’s worth, below is a stack trace. The crash occurs in both ROOT 5.34 and 6.00.

#0  strlen () at ../sysdeps/x86_64/strlen.S:106
#1  0x00007ffff75cec75 in TString::TString(char const*) () from /home/mkramer/physics/root/lib/libCore.so
#2  0x00007ffff6fc859c in TEmulatedMapProxy::WriteMap(unsigned int, TBuffer&) ()
   from /home/mkramer/physics/root/lib/libRIO.so
#3  0x00007ffff6fc87b0 in TEmulatedMapProxy::Streamer(TBuffer&) () from /home/mkramer/physics/root/lib/libRIO.so
#4  0x00007ffff6fbd308 in TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) ()
   from /home/mkramer/physics/root/lib/libRIO.so
#5  0x00007ffff6fba691 in TBufferFile::WriteObjectClass(void const*, TClass const*) ()
   from /home/mkramer/physics/root/lib/libRIO.so
#6  0x00007ffff6fb74ed in TBufferFile::WriteObjectAny(void const*, TClass const*) ()
   from /home/mkramer/physics/root/lib/libRIO.so
#7  0x00000000004058b9 in main (argc=1, argv=0x7fffffffd768) at stlbufcrash.cc:18

mjkramer · August 5, 2014, 7:28am

Okay, so I’ve experimented a bit with whatever possibilities I could think of. Still, nothing works.

I tried generating a dictionary with a LinkDef.h file:

#include <string>
#include <pair>
#include <map>

#ifdef CINT

#pragma link C++ class std::string;
#pragma link C++ class std::pair<std:;string, std::string>;
#pragma link C++ class std::map<std::string, std::string>;

#endif

I then did

and compiled mapdict.cxx into the program. Now TBufferFile::WriteObjectAny doesn’t crash, hooray! However, the resulting buffer’s size (as reported by TBufferFile::Length) is always the same (31), no matter how much data the std::map contains. Furthermore, if I try to reconstruct the map from the buffer’s contents:

TBufferFile br(TBuffer::kRead, b.Length(), b.Buffer());
map<string,string>* data = (map<string,string>*) br.ReadObjectAny(c);

…I get a crash at the call to ReadObjectAny, which isn’t surprising, since the constant length of the buffer is already a clear sign that something’s going wrong.

Alright, let’s try using AutoDict instead. I opened up ROOT, created a map<string,string>, and quit. I then had the following files (plus .d and .so versions):

AutoDict_allocator_pair_const_string_string___.cxx
AutoDict_map_string_string_less_string__allocator_pair_const_string_string_____.cxx
AutoDict_pair_const_string_string_.cxx

I tried compiling these into the program, instead of my mapdict.cxx, but now I’m once again crashing on the call to WriteObjectAny.

It’s gotta be possible to serialize STL containers. I’ve seen it done before. I’d really love to know what I’m doing wrong here.

Danilo · August 5, 2014, 12:20pm

Hi,

you don’t need to geenrate the dictionary: root provides it natively for map<string,string>.
So, copy pasting the lines in root6, at the prompt:

   -----------------------------------------------------------------------
  | Welcome to ROOT 6.01/02                           http://root.cern.ch |
  |                                          (c) 1995-2014, The ROOT Team |
  | Built for linuxx8664gcc                                               |
  | From heads/ProtoEnumDani@v6-00-01-552-g1f8776a, Aug 05 2014, 13:56:03 |
  | Try '.help', '.demo', '.license', '.credits', '.quit'/'.q'            |
   -----------------------------------------------------------------------

root [0]   map<string, string> m;
root [1]   m["foo"] = "bar";
root [2] 
root [2]   TClass *c = gROOT->FindSTLClass("std::map<std::string,std::string>", true);
root [3]   TBufferFile b(TBuffer::kWrite);
root [4]   b.WriteObjectAny(&m, c);
root [5] 
root [5] 
root [5]   TBufferFile br(TBuffer::kRead, b.Length(), b.Buffer(), false);
root [6]   map<string,string>* data = (map<string,string>*) br.ReadObjectAny(c);
root [7]   std::cout << (*data)["foo"] << "\n";
bar
root [8]

and compiling this with root5

#include <map>
#include <string>
#include <iostream>
#include <TBufferFile.h>
#include <TApplication.h>
#include <TROOT.h>

using namespace std;
int main(int argc, char** argv)
{
  map<string, string> m;
  m["foo"] = "bar";

  TClass *c = gROOT->FindSTLClass("std::map<std::string,std::string>", true);
  TBufferFile b(TBuffer::kWrite);
  b.WriteObjectAny(&m, c);


  TBufferFile br(TBuffer::kRead, b.Length(), b.Buffer(), false);
  map<string,string>* data = (map<string,string>*) br.ReadObjectAny(c);
  std::cout << (*data)["foo"] << "\n";

  return 0;
}

works.
Now, as you can see the adopt flag of the br TBuffer was set to false. This was done to avoid a double delete. In presence of sockets this setting might be different.

Cheers,
Danilo

mjkramer · August 5, 2014, 9:42pm

Hi Danilo,

Thank you for your help. However, when I tried your second example (which I copied and pasted without any changes), it still segfaulted (specifically, in the call to WriteObjectAny). I did not attempt to compile in any extra dictionaries; I just did g++ test.cc -o test root-config --libs --cflags. I tried it on three different machines running Ubuntu and Scientific Linux, with ROOT versions 5.28, 5.30, and 5.34. The result was the same in every case.

Please let me know if there’s anything else I should try, or if I should just go ahead and file a bug report. Thanks!

Edit: Also, I tried instantiating a TApplication at the beginning of main(), but it made no difference.

mjkramer · August 5, 2014, 10:35pm

Well, in the 20 minutes since my last post, I’ve made a bit of progress. The following code does work:

#include <map>
#include <string>
#include <iostream>

#include <TBufferFile.h>
#include <TApplication.h>
#include <TROOT.h>

using namespace std;

int main(int argc, char** argv)
{
  TApplication app("test", &argc, argv);

  gROOT->ProcessLine("#include <map>");
  gROOT->ProcessLine("std::map<std::string, std::string> asdf");

  TClass* c = gROOT->FindSTLClass("std::map<std::string, std::string>", true);

  map<string, string> m;
  m["foo"] = "bar";
  m["baz"] = "quux";

  TBufferFile b(TBuffer::kWrite);
  b.WriteObjectAny(&m, c);

  TBufferFile br(TBuffer::kRead, b.Length(), b.Buffer(), false);
  map<string, string>& m2 = *(map<string,string>*) br.ReadObjectAny(c);

  cout << m2["foo"] << " " << m2["baz"] << endl;
  return 0;
}

If I remove either of the two ProcessLine calls, it crashes. After running the program, the directory is polluted with the same AutoDict files I mentioned in an earlier post:

AutoDict_allocator_pair_const_string_string___.cxx,.d, .so
AutoDict_map_string_string_less_string__allocator_pair_const_string_string_____.cxx,.d,.so
AutoDict_pair_const_string_string_.cxx,.d,.so

I’ve tried recompiling the program and linking the AutoDict files directly into it, but it still generates them when run inside an empty directory. Hmm…

Anyway, it seems apparent that ROOT does not already contain the full set of dictionaries needed to handle a std::map<std::string, std::string>. My question now is, how can I link these dictionaries into the program and avoid generating all of these AutoDict files at runtime? If I comment out the two ProcessLine calls, and compile the AutoDict_.cxx into the program, it crashes. If I leave the two calls, and compile in AutoDict_.cxx, the program works, but it still generates new AutoDict files.

One thing that does work is to include the ProcessLine calls, but first call gSystem->Load() on the AutoDict .so files at runtime. In this case, the AutoDict files are not regenerated. This isn’t too bad of a solution. However, I’d still prefer to statically link these dictionaries into the binary. I thought that compiling the Autodict*.cxx files into the executable would work, but I guess I still need to call something in order to load those dictionaries from within the program’s address space.

mjkramer · August 5, 2014, 11:19pm

Sorry for posting three times in a row, but I’ve mostly solved the problem now! After looking at the AutoDict*.cxx files that ROOT generated when I told CINT to create a map<string,string>, I put together the following LinkDef.h files:

#include <map>

#ifdef __CINT__

#pragma link C++ nestedclasses;
#pragma link C++ nestedtypedefs;

#pragma link C++ class pair<string, string>+;
#pragma link C++ class pair<string, string>::*+;

#pragma link C++ class pair<const string,string>+;
#pragma link C++ class pair<const string,string>::*+;

#pragma link C++ class allocator<pair<const string,string> >+;
#pragma link C++ class allocator<pair<const string,string> >::*+;

#pragma link C++ class map<string,string,less<string>,allocator<pair<const string,string> > >+;
#pragma link C++ class map<string,string,less<string>,allocator<pair<const string,string> > >::*;
#pragma link C++ operators map<string,string,less<string>,allocator<pair<const string,string> > >::iterator;
#pragma link C++ operators map<string,string,less<string>,allocator<pair<const string,string> > >::const_iterator;
#pragma link C++ operators map<string,string,less<string>,allocator<pair<const string,string> > >::reverse_iterator;

#endif

Then I generated a dictionary with rootcint -f dict.cxx -c LinkDef.h. Unfortunately, the dictionary wouldn’t compile because CINT wasn’t smart enough to include the map header in dict.h. So I added #include to the very top of dict.h (right above the DO NOT EDIT warning ). Then I compiled dict.cxx into my program and it worked! The code of the program is

#include <string>
#include <iostream>

#include <TBufferFile.h>
#include <TApplication.h>
#include <TROOT.h>
#include <TSystem.h>

using namespace std;

int main(int argc, char** argv)
{
  TApplication app("test", &argc, argv);

  TClass* c = gROOT->FindSTLClass("std::map<std::string, std::string>", true);

  map<string, string> m;
  m["foo"] = "bar";
  m["baz"] = "quux";

  TBufferFile b(TBuffer::kWrite);
  b.WriteObjectAny(&m, c);

  TBufferFile br(TBuffer::kRead, b.Length(), b.Buffer(), false);
  map<string, string>& m2 = *(map<string,string>*) br.ReadObjectAny(c);

  for (map<string, string>::iterator it = m2.begin();
       it != m2.end(); ++it)
    cout << it->first << " = " << it->second << endl;

  return 0;
}

No segfaults, no AutoDict pollution, hooray! This is pretty much the “ideal” solution I was looking for, with the dictionaries statically compiled into my program. The only slight downside is the need to manually add #include to the dictionary. I will file a CINT bug report when I get a chance, unless someone points out that I’m doing something wrong.

Anyway, this thread can be marked as solved, although I hope that the ROOT developers will think about including the aforementioned dictionaries in ROOT by default, so that others don’t have to go through the frustration I’ve experienced.

Danilo · August 6, 2014, 6:31am

Hi,

for the sake of clarity and for the record: ROOT 5.34.19 (probably even previous versions) do natively provide dictionaries for map<string,string>. It is therefore not necessary to create any new dictionary.
The program

#include <map>
#include <string>
#include <iostream>
#include <TBufferFile.h>
#include <TApplication.h>
#include <TROOT.h>

using namespace std;
int main(int argc, char** argv)
{
  map<string, string> m;
  m["foo"] = "bar";

  TClass *c = gROOT->FindSTLClass("std::map<std::string,std::string>", true);
  TBufferFile b(TBuffer::kWrite);
  b.WriteObjectAny(&m, c);


  TBufferFile br(TBuffer::kRead, b.Length(), b.Buffer(), false);
  map<string,string>* data = (map<string,string>*) br.ReadObjectAny(c);
  std::cout << (*data)["foo"] << "\n";

  return 0;
}

compiles successfully with

clang++ -o example example.cpp `root-config --cflags --libs`

on any recent linux/OSx platform and runs successfully serialising and de-serialising the map.

Wile_E_Coyote · August 6, 2014, 11:01am

This is Ubuntu 14.04.1 LTS / x86_64 with gcc 4.8.2 and ROOT v5-34-00-patches as of 2014.08.04 (a “debug” built) here … [code][…]$ g++ -o example example.cpp root-config --cflags --libs
[…]$ ./example

*** Break *** segmentation violation

===========================================================
There was a crash (kSigSegmentationViolation).
This is the entire stack trace of all threads:

#0 0x00007fcb0b58799c in __libc_waitpid (pid=4785, stat_loc=stat_loc
entry=0x7fffda5629f0, options=options
entry=0) at …/sysdeps/unix/sysv/linux/waitpid.c:31
#1 0x00007fcb0b50c5a2 in do_system (line=) at …/sysdeps/posix/system.c:148
#2 0x00007fcb0c74e680 in TUnixSystem::Exec (this=0xd80ec0, shellcmd=0x14abfe0 “/…/etc/gdb-backtrace.sh 4782 1>&2”) at /…/core/unix/src/TUnixSystem.cxx:2171
#3 0x00007fcb0c74ef58 in TUnixSystem::StackTrace (this=0xd80ec0) at /…/core/unix/src/TUnixSystem.cxx:2418
#4 0x00007fcb0c74c845 in TUnixSystem::DispatchSignals (this=0xd80ec0, sig=kSigSegmentationViolation) at /…/core/unix/src/TUnixSystem.cxx:1293
#5 0x00007fcb0c74a71b in SigHandler (sig=kSigSegmentationViolation) at /…/core/unix/src/TUnixSystem.cxx:439
#6 0x00007fcb0c7528e8 in sighandler (sig=11) at /…/core/unix/src/TUnixSystem.cxx:3737
#7
#8 strlen () at …/sysdeps/x86_64/strlen.S:137
#9 0x00007fcb0c69c855 in TString::TString (this=0x7fffda565100, cs=0x10000ffff <error: Cannot access memory at address 0x10000ffff>) at /…/core/base/src/TString.cxx:115
#10 0x00007fcb0bee18ce in TEmulatedMapProxy::WriteMap (this=0x14a76c0, nElements=2992282291, b=…) at /…/io/io/src/TEmulatedMapProxy.cxx:192
#11 0x00007fcb0bee1be0 in TEmulatedMapProxy::Streamer (this=0x14a76c0, b=…) at /…/io/io/src/TEmulatedMapProxy.cxx:248
#12 0x00007fcb0bed1114 in TCollectionStreamer::Streamer (this=0x14a7610, buff=…, pObj=0x7fffda5653c0, onFileClass=0x0) at /…/io/io/src/TCollectionProxyFactory.cxx:183
#13 0x00007fcb0bed18e1 in TCollectionClassStreamer::Stream (this=0x14a75f0, b=…, obj=0x7fffda5653c0, onfileClass=0x0) at include/TCollectionProxyFactory.h:201
#14 0x00007fcb0c721ffe in TClass::StreamerExternal (this=0xf20690, object=0x7fffda5653c0, b=…, onfile_class=0x0) at /…/core/meta/src/TClass.cxx:5391
#15 0x00007fcb0c722300 in TClass::StreamerDefault (this=0xf20690, object=0x7fffda5653c0, b=…, onfile_class=0x0) at /…/core/meta/src/TClass.cxx:5468
#16 0x00007fcb0cb2c559 in TClass::Streamer (this=0xf20690, obj=0x7fffda5653c0, b=…, onfile_class=0x0) at include/TClass.h:442
#17 0x00007fcb0bec9df2 in TBufferFile::WriteObjectClass (this=0x7fffda5653f0, actualObjectStart=0x7fffda5653c0, actualClass=0xf20690) at /…/io/io/src/TBufferFile.cxx:2528
#18 0x00007fcb0bec9fe5 in TBufferFile::WriteObjectAny (this=0x7fffda5653f0, obj=0x7fffda5653c0, ptrClass=0xf20690) at /…/io/io/src/TBufferFile.cxx:2585
#19 0x000000000040413c in main ()

The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.

#8 strlen () at …/sysdeps/x86_64/strlen.S:137
#9 0x00007fcb0c69c855 in TString::TString (this=0x7fffda565100, cs=0x10000ffff <error: Cannot access memory at address 0x10000ffff>) at /…/core/base/src/TString.cxx:115
#10 0x00007fcb0bee18ce in TEmulatedMapProxy::WriteMap (this=0x14a76c0, nElements=2992282291, b=…) at /…/io/io/src/TEmulatedMapProxy.cxx:192
#11 0x00007fcb0bee1be0 in TEmulatedMapProxy::Streamer (this=0x14a76c0, b=…) at /…/io/io/src/TEmulatedMapProxy.cxx:248
#12 0x00007fcb0bed1114 in TCollectionStreamer::Streamer (this=0x14a7610, buff=…, pObj=0x7fffda5653c0, onFileClass=0x0) at /…/io/io/src/TCollectionProxyFactory.cxx:183
#13 0x00007fcb0bed18e1 in TCollectionClassStreamer::Stream (this=0x14a75f0, b=…, obj=0x7fffda5653c0, onfileClass=0x0) at include/TCollectionProxyFactory.h:201
#14 0x00007fcb0c721ffe in TClass::StreamerExternal (this=0xf20690, object=0x7fffda5653c0, b=…, onfile_class=0x0) at /…/core/meta/src/TClass.cxx:5391
#15 0x00007fcb0c722300 in TClass::StreamerDefault (this=0xf20690, object=0x7fffda5653c0, b=…, onfile_class=0x0) at /…/core/meta/src/TClass.cxx:5468
#16 0x00007fcb0cb2c559 in TClass::Streamer (this=0xf20690, obj=0x7fffda5653c0, b=…, onfile_class=0x0) at include/TClass.h:442
#17 0x00007fcb0bec9df2 in TBufferFile::WriteObjectClass (this=0x7fffda5653f0, actualObjectStart=0x7fffda5653c0, actualClass=0xf20690) at /…/io/io/src/TBufferFile.cxx:2528
#18 0x00007fcb0bec9fe5 in TBufferFile::WriteObjectAny (this=0x7fffda5653f0, obj=0x7fffda5653c0, ptrClass=0xf20690) at /…/io/io/src/TBufferFile.cxx:2585
#19 0x000000000040413c in main ()
===========================================================[/code]

Segfault when writing STL map to TBufferFile

=========================================================== There was a crash (kSigSegmentationViolation). This is the entire stack trace of all threads:

The lines below might hint at the cause of the crash. If they do not help you then please submit a bug report at http://root.cern.ch/bugs. Please post the ENTIRE stack trace from above as an attachment in addition to anything else that might help us fixing this issue.

===========================================================
There was a crash (kSigSegmentationViolation).
This is the entire stack trace of all threads:

The lines below might hint at the cause of the crash.
If they do not help you then please submit a bug report at
http://root.cern.ch/bugs. Please post the ENTIRE stack trace
from above as an attachment in addition to anything else
that might help us fixing this issue.