I believe this has to do with the two coordinate systems. One is the normalised device coordinates (NDC) see here, the other one is the coordinate system spanned by the histogram.
If you draw a line “just like this”, it’s drawn in the histogram coordinate system, but if there’s no histogram, that might fail. If you go for NDCs, the line will be in the same place on the canvas independent of the underlying pad.