Adding new workers and resource management to new dataflow engine written in Python: Pydra
Contributor: Aditya Agarwal
This project aims to extend Pydra's worker classes to handle a wider range of systems by utilizing PSI/J, which offers a unified API for various schedulers. The objective of this project is to create a Pydra worker class capable of submitting and monitor jobs on different HPC systems, such as Theta, using PSI/J. Furthermore, the contributor intends to use Bokeh to construct an interactive web-based dashboard that will monitor the jobs being submitted and executed by the Pydra worker. The expected outcome of this project is a Pydra worker that can use PSI/J to submit and track jobs on various HPC systems, an updated Dask worker, as well as a jobs monitoring dashboard. The proposed methodology involves designing and implementing the Pydra worker class, evaluating its functionality and performance on various systems, comparing it to existing Pydra workers, and constructing a Pydra dashboard using Bokeh to monitor the jobs. Finally, the contributor will document the usage and installation of the Pydra worker with PSI/J, as well as the Pydra dashboard, and provide users with examples and tutorials.
- Interactive web-based dashboard for Pydra workers
- Pydra integration with PSI/J for submission and tracking of jobs
- Bokeh implementation in Pydra dashboard
- Examples and tutorials for users