Interactive GPU Jobs
Introduction
This guide shows you how to run interactive GPU jobs on CHTC's High Throughput Computing (HTC) system.
CHTC's HTC system is primarily meant for submitting batches of large workloads. However, a few resources are set aside for testing and troubleshooting GPU jobs interactively.
Availablility
Shared GPUs
We have dedicated slots for interactive jobs on CHTC's shared GPU machines:
| Number of slots | GPU model | Number of GPUs per slot | GPU Memory |
|---|---|---|---|
| 4 | NVIDIA GeForce RTX 2080 Ti | 1 | 10.6 GB |
| 1 | NVIDIA A100 | 1 | 39.5 GB |
The slots in the table above only run interactive GPU jobs. Interactive GPU jobs can still run on other slots on shared GPU machines (including backfill slots), depending on availability upon submission.
All interactive GPU jobs have a maximum runtime of 4 hours.
You will not be able to run condor_ssh_to_job on noninteractive GPU jobs.
Researcher-owned GPUs
If you have access to researcher-owned GPU machines, you may submit interactive jobs within those machines' resource limits. condor_ssh_to_job is permitted on these machines.
Submit an interactive GPU job
-
Ensure your resource requirements allow you to match to an interactive GPU slot.
-
Submit your job with the
-icommand.condor_submit -i <submit_file> -
When your job starts, you will see a welcome message and your prompt will change, for example:
Waiting for job to start... Welcome to slot1_1@gpulab2001.chtc.wisc.edu! == NOTICE: THIS NODE IS ON PUPPET ENVIRONMENT "puppet8" == [netid@gpulab2001 scratch]$ -
When you are done with your interactive session, type
exitto return back to the Access Point.