The number of dense units for classification

TMVA
1

Hi,

I have a simple question about classification. For a binary classification, the number of units of the last dense layer should be 1 or 2? In tutorial “TMVA_RNN_Classification.C”, the last layer is a dense layer with 2 units (see code below). Why it is not 1? Because you want to have a single probability for both signal and background? I have tried to use 1 instead, the program fails to run. Can someone please guide me about this issue? Thank you in advance.

import keras
from keras.models import Sequential
from keras.optimizers import adam_v2
from keras.layers import Input, Dense, Dropout, Flatten, SimpleRNN, GRU, LSTM, Reshape, BatchNormalization

model = keras.models.Sequential()
model.add(Reshape((10, 30), input_shape = (10*30, )))
model.add(LSTM(units=10, return_sequences=True) )
model.add(Flatten())
model.add(Dense(64, activation = ‘tanh’))
model.add(Dense(2, activation = ‘sigmoid’))
model.compile(loss = ‘binary_crossentropy’, optimizer = adam_v2.Adam(lr = 0.001), metrics = [‘accuracy’])
modelName = ‘model_LSTM.h5’
model.save(modelName)
model.summary()

2

I think @moneta can help.

3

Hi,

You can have 1 unit at the end which will output the probability of the event to be of type signal or 2 units, where the first will give the signal probability and the second one the background probability (= 1 - signal_prob).
The model should work also with one. Which error are you having ?

Lorenzo

4

Hi moneta,

I simply change the number of units from 2 to 1. The modified code is:
“model.add(Dense(1, activation = ‘sigmoid’))”

The error messages are listed below:
: Failed to run python code: history = model.fit(trainX, trainY, sample_weight=trainWeights, batch_size=batchSize, epochs=numEpochs, verbose=verbose, validation_data=(valX, valY, valWeights), callbacks=callbacks)
: Python error message:
Traceback (most recent call last):
File “”, line 1, in
File “/home/guang/.local/lib/python3.6/site-packages/keras/engine/training.py”, line 1184, in fit
tmp_logs = self.train_function(iterator)
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py”, line 885, in call
result = self._call(*args, **kwds)
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py”, line 933, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py”, line 760, in _initialize
*args, **kwds))
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py”, line 3066, in _get_concrete_function_internal_garbage_collected
graph_function, _ = self._maybe_define_function(args, kwargs)
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py”, line 3463, in _maybe_define_function
graph_function = self._create_graph_function(args, kwargs)
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/eager/function.py”, line 3308, in _create_graph_function
capture_by_value=self._capture_by_value),
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py”, line 1007, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py”, line 668, in wrapped_fn
out = weak_wrapped_fn().wrapped(*args, **kwds)
File “/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py”, line 994, in wrapper
raise e.ag_error_metadata.to_exception(e)
ValueError: in user code:

/home/guang/.local/lib/python3.6/site-packages/keras/engine/training.py:853 train_function  *
    return step_function(self, iterator)
/home/guang/.local/lib/python3.6/site-packages/keras/engine/training.py:842 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:1286 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2849 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:3632 _call_for_each_replica
    return fn(*args, **kwargs)
/home/guang/.local/lib/python3.6/site-packages/keras/engine/training.py:835 run_step  **
    outputs = model.train_step(data)
/home/guang/.local/lib/python3.6/site-packages/keras/engine/training.py:789 train_step
    y, y_pred, sample_weight, regularization_losses=self.losses)
/home/guang/.local/lib/python3.6/site-packages/keras/engine/compile_utils.py:201 __call__
    loss_value = loss_obj(y_t, y_p, sample_weight=sw)
/home/guang/.local/lib/python3.6/site-packages/keras/losses.py:141 __call__
    losses = call_fn(y_true, y_pred)
/home/guang/.local/lib/python3.6/site-packages/keras/losses.py:245 call  **
    return ag_fn(y_true, y_pred, **self._fn_kwargs)
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:206 wrapper
    return target(*args, **kwargs)
/home/guang/.local/lib/python3.6/site-packages/keras/losses.py:1809 binary_crossentropy
    backend.binary_crossentropy(y_true, y_pred, from_logits=from_logits),
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:206 wrapper
    return target(*args, **kwargs)
/home/guang/.local/lib/python3.6/site-packages/keras/backend.py:5000 binary_crossentropy
    return tf.nn.sigmoid_cross_entropy_with_logits(labels=target, logits=output)
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:206 wrapper
    return target(*args, **kwargs)
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:246 sigmoid_cross_entropy_with_logits_v2
    logits=logits, labels=labels, name=name)
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/util/dispatch.py:206 wrapper
    return target(*args, **kwargs)
/home/guang/.local/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:133 sigmoid_cross_entropy_with_logits
    (logits.get_shape(), labels.get_shape()))

ValueError: logits and labels must have the same shape ((100, 1) vs (100, 2))

: Failed to train model
***> abort program execution
terminate called after throwing an instance of ‘std::runtime_error’
what(): FATAL error

5

Hello,
Thank you for reporting the error.
It is correct, the TMVA Keras creates internally an array of labels that is used for training and it has the shape (number of events, number of classes), therefore you would need to have two output units for binary classification. It can work with one output unit, but you would need to have the corrrect label array.
So it will work if you use directly Keras, but not in the TMVA interface.
If this for you it is important, to have only one output unit in binary classification, I could try to fix this error.

Best regards

Lorenzo