Skip to contents

Getting Started with taskqueue

Source: vignettes/getting-started.Rmd
getting-started.Rmd

Introduction

taskqueue is an R package for asynchronous parallel computing based on PostgreSQL database. It is designed to dynamically allocate tasks to workers, efficiently utilizing all available computing resources until all tasks are completed.

This package is suitable for Embarrassingly parallel problems - parallel computing without any communication among parallel tasks.

Prerequisites

Before using taskqueue, ensure you have:

  1. PostgreSQL installed and configured (see PostgreSQL Setup vignette)
  2. SSH access configured for remote resources (see SSH Setup vignette)

Installation

Install the development version from GitHub:

devtools::install_github('byzheng/taskqueue')

Basic Workflow

1. Initialize Database 

library(taskqueue)

# Initialize the database structure
db_init()

2. Define a Resource

A computing resource is a facility/computer that can run multiple jobs/workers.

resource_add(
  name = "hpc", 
  type = "slurm", 
  host = "hpc.example.com", 
  nodename = "hpc",
  workers = 500,
  log_folder = "/home/user/log_folder/"
)

Parameters:

  • name: Resource name
  • type: Resource type (currently only slurm is supported)
  • host: Network name to access the resource
  • nodename: Obtained by Sys.info() on the resource
  • workers: Maximum number of available cores
  • log_folder: Folder to store log files (important for troubleshooting)

Note: log_folder is split by project. It’s better to use a high-speed hard drive due to frequent I/O operations.

3. Create a Project

taskqueue manages tasks by project. Each project has its own resources, working directory, runtime requirements, and configurations.

# Create a project with common requirements
project_add("test_project", memory = 20)

4. Assign Resource to Project 

project_resource_add(
  project = "test_project", 
  resource = "hpc"
)

5. Add Tasks 

# Add 100 tasks to the project
task_add("test_project", num = 100, clean = TRUE)

6. Develop Your Worker Function

Create a function that:

  • Takes task id as the first argument
  • Expects no return values
  • Saves final output to the file system
  • Checks whether the task is already finished
library(taskqueue)

fun_test <- function(i) {
  # Check output file
  out_file <- sprintf("%s.Rds", i)
  if (file.exists(out_file)) {
    return()
  }
  
  # Perform the actual computing
  # ... your computation here ...
  
  # Save final output
  saveRDS(i, out_file)
}

# Test locally
worker("test_project", fun_test)

7. Deploy to HPC

After developing and testing your function, save it to a file (e.g., rcode.R) and deploy to your HPC resource:

# Reset task status (if needed)
project_reset("test_project")

# Start the project
project_start("test_project")

# Schedule tasks on slurm resource
worker_slurm("test_project", "hpc", "rcode.R", modules = "sqlite/3.43.1")

# Check task status
task_status("test_project")

# Stop the project when done
project_stop("test_project")

Task Status

Each task has one of four statuses:

  • idle: Task is not running
  • working: Task is currently running on a worker
  • failed: Task failed for some reason (check the log folder for troubleshooting)
  • finished: Task completed without errors

Common Operations

Check Task Status 

task_status("test_project")

Reset Tasks

Reset all tasks in a project:

project_reset("test_project")

Reset only failed or working tasks:

project_reset("test_project", status = "failed")
project_reset("test_project", status = "working")

Manage Projects 

# List all projects
project_list()

# Get project details
project_get("test_project")

# Delete a project
project_delete("test_project")

Example: Complete Workflow

Here’s a complete example of using taskqueue:

library(taskqueue)

# 1. Initialize database (first time only)
db_init()

# 2. Add resource
resource_add(
  name = "my_hpc", 
  type = "slurm",
  host = "hpc.university.edu",
  nodename = "hpc",
  workers = 200,
  log_folder = "/home/user/taskqueue_logs/"
)

# 3. Create project
project_add("simulation_study", memory = 16)

# 4. Assign resource to project
project_resource_add(project = "simulation_study", resource = "my_hpc")

# 5. Add tasks
task_add("simulation_study", num = 1000, clean = TRUE)

# 6. Create worker function (save as worker_script.R)
library(taskqueue)

run_simulation <- function(task_id) {
    out_file <- sprintf("results/sim_%04d.Rds", task_id)
    if (file.exists(out_file)) return()
    
    # Run your simulation
    result <- your_simulation_function(task_id)
    
    # Save results
    saveRDS(result, out_file)
}

worker("simulation_study", run_simulation)

# 7. Deploy to HPC
project_start("simulation_study")
worker_slurm("simulation_study", "my_hpc", "worker_script.R")

# 8. Monitor progress
task_status("simulation_study")

# 9. Stop when complete
project_stop("simulation_study")

Tips

  1. Test Locally First: Always test your worker function locally with worker() before deploying to HPC
  2. Check Logs: If tasks fail, check the log folder for error messages
  3. Idempotent Functions: Make your worker functions idempotent (can be run multiple times safely) by checking if output already exists
  4. Resource Monitoring: Use task_status() regularly to monitor progress
  5. Clean Start: Use project_reset() to restart failed tasks