Multiligand file, MPI, and OSError: [Errno 24] Too many open files

Computer-Aided Drug Design at the Durrant Lab › Forums › Gypsum-DL › Multiligand file, MPI, and OSError: [Errno 24] Too many open files

Viewing 4 reply threads
  • Author
    Posts
    • July 13, 2021 at 2:15 pm #38722
      Filip
      Guest

      Hi, first of all, thanks for this software!

      I’m using gypsum_dl to process file with 23k SMILES, with MPI on cluster with 28 cpus assigned. After some time the program crashes with the error below.

      For the test input file with several ligands it works fine.
      best,
      Filip


      Ionizing all molecules...
      Traceback (most recent call last):
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/runpy.py", line 197, in _run_module_as_main
      return _run_code(code, main_globals, None,
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/site-packages/mpi4py/main.py", line 7, in <module>
      main()
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/site-packages/mpi4py/run.py", line 196, in main
      run_command_line(args)
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/site-packages/mpi4py/run.py", line 47, in run_command_line
      run_path(sys.argv[0], run_name='main')
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/runpy.py", line 268, in run_path
      return _run_module_code(code, init_globals, run_name,
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/runpy.py", line 97, in _run_module_code
      _run_code(code, mod_globals, init_globals,
      File "/cluster/miniconda3/envs/gypsum/lib/python3.9/runpy.py", line 87, in _run_code
      exec(code, run_globals)
      File "/cluster/projects/gypsum_dl/run_gypsum_dl.py", line 312, in <module>
      prepare_molecules(INPUTS)
      File "/cluster/projects/gypsum_dl/gypsum_dl/Start.py", line 286, in prepare_molecules
      params["Parallelizer"].run(job_input, execute_gypsum_dl)
      File "/cluster/projects/gypsum_dl/gypsum_dl/Parallelizer.py", line 327, in run
      return self.parallel_obj.run(func, args)
      File "/cluster/projects/gypsum_dl/gypsum_dl/Parallelizer.py", line 593, in run
      result_chunk = [func(*arg) for arg in args_chunk]
      File "/cluster/projects/gypsum_dl/gypsum_dl/Parallelizer.py", line 593, in <listcomp>
      result_chunk = [func(*arg) for arg in args_chunk]
      File "/cluster/projects/gypsum_dl/gypsum_dl/Start.py", line 315, in execute_gypsum_dl
      prepare_smiles(contnrs, params)
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/PrepareSmiles.py", line 76, in prepare_smiles
      add_hydrogens(
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/AddHydrogens.py", line 98, in add_hydrogens
      results.append(parallel_add_H(i[0], i[1]))
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/AddHydrogens.py", line 167, in parallel_add_H
      rdkit_mols = [Chem.MolFromSmiles(smi.strip()) for smi in smis]
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/AddHydrogens.py", line 167, in <listcomp>
      rdkit_mols = [Chem.MolFromSmiles(smi.strip()) for smi in smis]
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/dimorphite_dl/dimorphite_dl.py", line 570, in next
      return self.next()
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/dimorphite_dl/dimorphite_dl.py", line 592, in next
      smile_and_datum = self.args["smiles_and_data"].next()
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/dimorphite_dl/dimorphite_dl.py", line 481, in next
      mol = UtilFuncs.convert_smiles_str_to_mol(smiles_str)
      File "/cluster/projects/gypsum_dl/gypsum_dl/Steps/SMILES/dimorphite_dl/dimorphite_dl.py", line 387, in convert_smiles_str_to_mol
      stderr_pipe = os.pipe()

      OSError: [Errno 24] Too many open files

      MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
      with errorcode 1.

      NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
      You may or may not see output from other processes, depending on

      exactly when Open MPI kills them.

    • July 13, 2021 at 2:17 pm #38752
      ikk6
      Keymaster

      Hello. Any chance you could share the input file? Or provide us with the sample one that fails, if your data is sensitive.

    • July 14, 2021 at 1:26 pm #38840
      Filip
      Guest

      Thanks for your prompt reply. Unfortunately I can’t, but I will try to reproduce the problem with a similar in size subset from ZINC.

    • July 28, 2021 at 5:07 pm #39980
      Filip
      Guest

      Hi,
      I performed various tests (with ZINC library and re-testing with the original library), and everything works fine. I guess the observed error was connected with the filesystem itself or with quota on the HPC.

      best,
      Filip

    • July 28, 2021 at 5:08 pm #40156
      ikk6
      Keymaster

      Hello, Filip.
      Thank you for the update and good luck with your project!
      Yuri

  • Author
    Posts
Viewing 4 reply threads
  • The topic ‘Multiligand file, MPI, and OSError: [Errno 24] Too many open files’ is closed to new replies.