# taskqueue [![](https://www.r-pkg.org/badges/version/taskqueue?color=green)](https://cran.r-project.org/package=taskqueue) [![R-CMD-check.yaml](https://github.com/byzheng/taskqueue/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/byzheng/taskqueue/actions/workflows/R-CMD-check.yaml) [![](http://cranlogs.r-pkg.org/badges/grand-total/taskqueue?color=green)](https://cran.r-project.org/package=taskqueue) [![](http://cranlogs.r-pkg.org/badges/last-month/taskqueue?color=green)](https://cran.r-project.org/package=taskqueue) [![](http://cranlogs.r-pkg.org/badges/last-week/taskqueue?color=green)](https://cran.r-project.org/package=taskqueue) Task Queue is implemented in R for asynchronous tasks based on [PostgreSQL](https://www.postgresql.org/) database. This package is only suitable for parallel computing without any communication among parallel tasks (i.e. [Embarrassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel)). ## Challenge of parallel computing in R Several R packages have been using for parallel computing (e.g. [High-Performance and Parallel Computing with R](https://cran.r-project.org/web/views/HighPerformanceComputing.html) which are not suitable for asynchronous tasks. - Uneven load among cores/workers. Some workers run faster than others and then wait for others to stop. - Cannot utilise the new available workers after a parallel task is started. `taskqueue` is designed to utilise all available computing resources until all tasks are finished through dynamic allocating tasks to workers. ## Installation Install the developing version from [Github](https://github.com/byzheng/taskqueue). ``` r remotes::install_github('byzheng/taskqueue') ``` ## Getting Started Please refer to - [Getting Started](https://taskqueue.bangyou.me/vignettes/getting-started.html) vignette for more details. - [Simple workflow example](https://taskqueue.bangyou.me/articles/simple-workflow-example.html) article for a complete simplified workflow. # Package index ## All functions - [`db_clean()`](https://taskqueue.bangyou.me/reference/db_clean.md) : Clean All Tables and Definitions from Database - [`db_connect()`](https://taskqueue.bangyou.me/reference/db_connect.md) : Connect to PostgreSQL Database - [`db_disconnect()`](https://taskqueue.bangyou.me/reference/db_disconnect.md) : Disconnect from PostgreSQL Database - [`db_init()`](https://taskqueue.bangyou.me/reference/db_init.md) : Initialize PostgreSQL Database for taskqueue - [`db_sql()`](https://taskqueue.bangyou.me/reference/db_sql.md) : A Wrapper function for DBI interface - [`.check_absolute_path()`](https://taskqueue.bangyou.me/reference/dot-check_absolute_path.md) : Check absolute path for system - [`.check_linux_absolute_path()`](https://taskqueue.bangyou.me/reference/dot-check_linux_absolute_path.md) : Check absolute path for Linux - [`.check_windows_absolute_path()`](https://taskqueue.bangyou.me/reference/dot-check_windows_absolute_path.md) : Check absolute path for Windows - [`is_db_connect()`](https://taskqueue.bangyou.me/reference/is_db_connect.md) : Test Database Connection - [`project_add()`](https://taskqueue.bangyou.me/reference/project_add.md) : Create a New Project - [`project_delete()`](https://taskqueue.bangyou.me/reference/project_delete.md) : Delete a Project - [`project_get()`](https://taskqueue.bangyou.me/reference/project_get.md) : Get Project Information - [`project_list()`](https://taskqueue.bangyou.me/reference/project_list.md) : List All Projects - [`project_reset()`](https://taskqueue.bangyou.me/reference/project_reset.md) : Reset a Project - [`project_resource_add()`](https://taskqueue.bangyou.me/reference/project_resource_add.md) : Assign a Resource to a Project - [`project_resource_add_jobs()`](https://taskqueue.bangyou.me/reference/project_resource_add_jobs.md) : Manage SLURM Job List for Project Resource - [`project_resource_get()`](https://taskqueue.bangyou.me/reference/project_resource_get.md) : Get resources of a project - [`project_resource_log_delete()`](https://taskqueue.bangyou.me/reference/project_resource_log_delete.md) : Delete Log Files for a Project Resource - [`project_start()`](https://taskqueue.bangyou.me/reference/project_start.md) : Start a Project - [`project_status()`](https://taskqueue.bangyou.me/reference/project_status.md) : Display Project Status - [`project_stop()`](https://taskqueue.bangyou.me/reference/project_stop.md) : Stop a Project - [`resource_add()`](https://taskqueue.bangyou.me/reference/resource_add.md) : Add a New Computing Resource - [`resource_get()`](https://taskqueue.bangyou.me/reference/resource_get.md) : Get Information for a Specific Resource - [`resource_list()`](https://taskqueue.bangyou.me/reference/resource_list.md) : List All Computing Resources - [`shiny_app()`](https://taskqueue.bangyou.me/reference/shiny_app.md) : Launch Shiny App for Monitoring Projects - [`table_exist()`](https://taskqueue.bangyou.me/reference/table_exist.md) : Check whether a table is existed - [`task_add()`](https://taskqueue.bangyou.me/reference/task_add.md) : Add Tasks to a Project - [`task_clean()`](https://taskqueue.bangyou.me/reference/task_clean.md) : Remove All Tasks from a Project - [`task_get()`](https://taskqueue.bangyou.me/reference/task_get.md) : Get Detailed Task Information - [`task_reset()`](https://taskqueue.bangyou.me/reference/task_reset.md) : Reset Task Status to Idle - [`task_status()`](https://taskqueue.bangyou.me/reference/task_status.md) : Get Task Status Summary - [`taskqueue_options()`](https://taskqueue.bangyou.me/reference/taskqueue_options.md) : Set or Get taskqueue Options - [`taskqueue_reset()`](https://taskqueue.bangyou.me/reference/taskqueue_reset.md) : Reset taskqueue Options to Defaults - [`tq_apply()`](https://taskqueue.bangyou.me/reference/tq_apply.md) : Apply a Function with Task Queue (Simplified Workflow) - [`worker()`](https://taskqueue.bangyou.me/reference/worker.md) : Execute Tasks as a Worker - [`worker_slurm()`](https://taskqueue.bangyou.me/reference/worker_slurm.md) : Create a worker on slurm cluster # Articles ### All vignettes - [Getting Started with taskqueue](https://taskqueue.bangyou.me/articles/getting-started.md): - [PostgreSQL Setup for taskqueue](https://taskqueue.bangyou.me/articles/postgresql-setup.md): - [Simple Workflow with tq_apply](https://taskqueue.bangyou.me/articles/simple-workflow.md): - [SSH Setup for taskqueue](https://taskqueue.bangyou.me/articles/ssh-setup.md):