How to debug a C++ notebook crash?

ROOT
1

Hello,

I’m trying out the ROOT C++ notebook with a third party library, and in some situations, I get the following message:

The kernel appears to have died. It will restart automatically.

There is no further output or log that appears on the notebook, nor the terminal in which I launched the notebook. In the terminal, the only thing that appears is:

WARNING:root:kernel 30dd17e3-9198-4ce2-bef8-a984f165624d restarted

Is there a way to get more debug information on why the kernel died?

It seems that one of the calls into the third-party library function is causing the kernel death. However, this is not consistent, because sometimes I call into the same function that does not result in the kernel dying (using different library-specific parameters). I always load the third-party library by the way, and it seems to be always found.

Cheers,
Ahmad

_ROOT Version: v6.18.00
_Platform: Ubuntu 16.04
Compiler: Not Provided

2

Hi,
If you do not see any information in the notebook nor in the terminal, you can try to run the same code from the ROOT prompt (just type root from your terminal) and check if you see anything else there.

3

Hi,

Didn’t think about that, thanks. But ‘unfortunately’ in the interpreter it seems to run just fine…

Any other tips?

4

That is weird, could you double check that the ROOT C++ notebook and your terminal run with the exact same environment?

You started the Jupyter server with root --notebook?

5

I can confirm that the notebook and the interpreter are running within the same environment (checked by running sytem("printenv") in both and comparing the output).

You started the Jupyter server with root --notebook ?

Yep.

6

Ok, could you isolate the statement that triggers the segfault? You said it happens when you call into this other library?

7

The statement is already isolated, but it’s a high-level API call, so I can’t tell which function is causing the kernel to die further down the call tree. Also, I am not sure if it’s a segfault at this point, since there is no output. If it is a segfault, then I’d expect a stack trace to appear in the notebook terminal.

Is there a way to pass a flag to JupyROOT to enable a verbose output or a debug flag? I remember that Jupyter has a --debug flag for this type of situations.

8

As an update, I found [1] that you can also launch JupyROOT with the jupyter command, and then are able to pass the --debug flag. However, this does not result in any useful output to debug the crashing kernel issue.

Any other ideas?

[1] https://github.com/root-project/root/tree/master/bindings/pyroot/JupyROOT

9

Another idea would be to run the notebook with nbconvert (instead of interactively), and check if that spits out any output.

https://nbconvert.readthedocs.io/en/latest/execute_api.html

10

Thanks again for your reply. Unfortunately, I didn’t get any wiser from the output. I get:

$ jupyter nbconvert --ExecutePreprocessor.timeout=600 --to notebook --execute diffusion.h.ipynb 
[NbConvertApp] Converting notebook diffusion.h.ipynb to notebook
[NbConvertApp] Executing notebook with kernel: root
[NbConvertApp] ERROR | Timeout waiting for execute reply (600s).
Traceback (most recent call last):
  File "/usr/local/bin/jupyter-nbconvert", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/nbconvertapp.py", line 325, in start
    self.convert_notebooks()
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/nbconvertapp.py", line 493, in convert_notebooks
    self.convert_single_notebook(notebook_filename)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/nbconvertapp.py", line 464, in convert_single_notebook
    output, resources = self.export_single_notebook(notebook_filename, resources, input_buffer=input_buffer)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/nbconvertapp.py", line 393, in export_single_notebook
    output, resources = self.exporter.from_filename(notebook_filename, resources=resources)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/exporters/exporter.py", line 174, in from_filename
    return self.from_file(f, resources=resources, **kw)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/exporters/exporter.py", line 192, in from_file
    return self.from_notebook_node(nbformat.read(file_stream, as_version=4), resources=resources, **kw)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/exporters/notebook.py", line 31, in from_notebook_node
    nb_copy, resources = super(NotebookExporter, self).from_notebook_node(nb, resources, **kw)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/exporters/exporter.py", line 134, in from_notebook_node
    nb_copy, resources = self._preprocess(nb_copy, resources)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/exporters/exporter.py", line 311, in _preprocess
    nbc, resc = preprocessor(nbc, resc)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/base.py", line 47, in __call__
    return self.preprocess(nb, resources)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/execute.py", line 262, in preprocess
    nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/base.py", line 69, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/execute.py", line 280, in preprocess_cell
    reply, outputs = self.run_cell(cell, cell_index)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/execute.py", line 348, in run_cell
    exec_reply = self._wait_for_reply(msg_id, cell)
  File "/usr/local/lib/python2.7/dist-packages/nbconvert/preprocessors/execute.py", line 337, in _wait_for_reply
    raise exception("Cell execution timed out")
RuntimeError: Cell execution timed out

I’m pretty sure the cell that it is trying to execute is the one that causes the crash in the first play

Btw, the compiled binary of the same code I’m trying to run in the notebook finishes within 1 second…

Ahmad

11

I really ran out of ideas… Another cause could be that the failing cell is producing an output that is too big (too long text, too big image) but I guess that is not the case?

In the end, what the ROOT C++ kernel is doing is, from Python, running your code with ROOT.gInterpreter.ProcessLine("string with the code of your cell").

12

I ended up debugging the library by commenting out parts that I expected to be causing the crash. It turned out that in the end the problem was that the third-party library was performing a system call that would lead to a GUI to pop up (required for visualization purposes). Since we don’t need that visualization within the notebook, I turned that off, and now the notebooks run just fine.

Thanks again for your help :wink:

13

This one was tricky :slight_smile: glad to hear you found the solution!

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.