How can I run multiple PyROOT (Python) scripts using this script?

Newbie
1

Hi. I am using Python to try to run multiple Python scripts that are in different directories. It is important for me that I run these scripts from these respective directories (don’t want to run them from the same directory). Each script produces some plots which are then moved to another directory called plots. Essentially, I give a list of directory paths for the scripts and a list of the scripts I want to run. I then import and run the main function of each script (i.e. I run the scripts). Finally, when the script is finished, the corresponding produced plots are moved to the plots directory.

My problem is that the scripts do not always run. Sometimes I get "ImportError: No module named bla_bla" even though it runs properly if I only run that script without any other scripts. Also, some of my scripts do similar functions (I fit specific histograms using a fitter). If I do a fitting with one script, when I try to do the fit again in another script, I get an error "Error in <TFractionFitter::CheckConsistency>: Empty data histogram", which would suggest some memory leaking, maybe? Is there a way to “reset” everything before running the new script so I would avoid such an error? I suspect the error could be stemming from the lambda functions.

I have included my code here. Any suggestions for improvement are very welcome.

#!/usr/bin/env python
#============================= Libraries=============================
import sys 
import os
import shutil
import importlib
import pandas as pd

#=============================list of paths
#WARNING: VARIABLE NAME MUST END WITH 'path'
main_directory = '/home/epp/phrqtm/PhD/gitlab/'
#1
plot_isolation_path = main_directory+'PlotIsolation'
#2
z_tree_path = main_directory+'Ztree' 
#3
pull_plot_path = main_directory+'Pullplot' 
#4
bkg_cuts_path = main_directory+'BkgCuts'

paths_array =  map(lambda x : x[1], filter(lambda x : x[0].endswith('path'), globals().items()))

#=============================list of scripts
#WARNING: VARIABLE NAME MUST END WITH 'script'

#1
plot_isolation_script = 'plot_isolation' #produces 2D histogram of plot isolation vs 1/pT (mu)
#2
z_tree_script='signal_yields' #produces signal yields as a function of pt and eta cuts 
#3
pull_plot_script = 'pull_plot' #produces pull plot diagrams (chi) for the fits of the different histograms created using the [pT,eta] bins
#4
bkg_cuts_script = 'bkg_cuts'

scripts_array =  map(lambda x : x[1], filter(lambda x : x[0].endswith('script'), globals().items()))

#=============================dataframe
data = {'path':paths_array, 
        'script':scripts_array} 

df = pd.DataFrame(data)
print(df)
#================================================================================    
#======================================LOOP======================================
#================================================================================
target_dir_array_latex_format=[]
for i in range (len(df)):
    path=df.loc[i,'path']
    script=df.loc[i,'script']

#=============================Import scripts
    import_array = {script} #need an array for the import module bit to work
    sys.path.append(os.path.abspath(path)) #access the script path
    imported_script = {element: importlib.import_module(element) for element in import_array}
    globals().update(imported_script) #import script

#=============================Run Scripts
    print('**********************')
    print('**********************')
    print(script + " script has started from path: " + path)

    eval(script+'.main()') #run the main function in the script
     

#=============================Move plots
    source_dir = './' #current directory
    target_dir = '../plots/'+script+'/'
    target_dir_array_latex_format.append(' {../'+target_dir+'} ')

    if not os.path.exists(target_dir):
        os.makedirs(target_dir)

    file_names = os.listdir(source_dir)   
    for file_name in file_names:
        if '.png' in file_name or '.pdf' in file_name: 
            shutil.move(os.path.join(source_dir, file_name), os.path.join(target_dir, file_name))
    print('**********************')
    print(script + " script has finished and plots are moved.")
    print('**********************')
    print('**********************')
2

Hello,

If you want to start from scratch every time you run a different Python script, one option is to turn that code into a shell script and create a new Python process every time. Alternatively, you can spawn subprocesses from the Python script above:

https://docs.python.org/3/library/subprocess.html

1 Like
3

Thank you! That worked.