Basic Scripting and Job Submission with Arguments

Code

Our executable written in shell, echo-next.sh:

#!/bin/bash
echo $1

We can use it on the command line after changing the file permissions to executable:

[user@login]$ chmod +x echo-next.sh
[user@login]$ ./echo-next.sh data.csv
data.csv

In bash, $1 references the first argument to a script. If you have more arguments, $2 will refer to the second, $3 to the third, etc. There are also ways to refer to the whole list of arguments if needed.

Wrapper script

We can now write a wrapper script! This script checks that one argument is passed, and also allows room for pre- and post-processing in this simple workflow.

Our wrapper script written in bash, wrapper.sh:

#!/bin/bash

# Check if an argument is provided
if [ $# -ne 1 ]; then
    echo 'Please use one argument. Usage: wrapper.sh [arg]'
    exit 1
fi

# Set filename variable to the first argument to the bash script
filename=$1

echo 'Pre-processing could go here.'

# Run code
./echo-next.sh ${filename}

echo 'Post-processing could go here.'

Ensure wrapper.sh is executable with chmod +x wrapper.sh before running it on the command line:

[user@login]$ chmod +x wrapper.sh
[user@login]$ ./wrapper.sh
Please use one argument. Usage: wrapper.sh [arg]
[user@login]$ ./wrapper.sh data.csv
Pre-processing could go here
data.csv
Post-processing could go here

Passing arguments with the HTCondor submit file

Now that we've understood how arguments work in shell script and how to write simple wrapper scripts that pass those arguments, we can pass arguments in HTCondor's submit file by specifying the arguments attribute, as shown in this excerpt:

# Create custom variable called "data"
data = data.csv

# Specify the executable & arguments
executable = wrapper.sh
arguments = $(data)

Code

Our executable written in Julia, echo-next.jl:

filename = ARGS[1]
println(filename)

The ARGS constant is an array -- if you have multiple arguments, you can reference them using Julia's usual notation (ARGS[1], ARGS[2], etc.) See Julia documentation on arguments for details.

We can use the script on the command line, assuming Julia is installed and on the PATH:

[user@login]$ julia echo-next.jl data.csv
data.csv

Wrapper script

We can now write a wrapper script! This script checks that one argument is passed, and also allows room for pre- and post-processing in this simple workflow.

Our wrapper script written in bash, wrapper.sh:

#!/bin/bash

# Check if an argument is provided
if [ $# -ne 1 ]; then
    echo 'Please use one argument. Usage: wrapper.sh [arg]'
    exit 1
fi

# Set filename variable to the first argument to the bash script
filename=$1

echo 'Pre-processing could go here.'

# Run code
julia echo-next.jl ${filename}

echo 'Post-processing could go here.'

Ensure wrapper.sh is executable with chmod +x wrapper.sh before running it on the command line:

[user@login]$ chmod +x wrapper.sh
[user@login]$ ./wrapper.sh
Please use one argument. Usage: wrapper.sh [arg]
[user@login]$ ./wrapper.sh data.csv
Pre-processing could go here.
data.csv
Post-processing could go here.

Passing arguments with the HTCondor submit file

Now that we've understood how arguments work in Julia and how to write simple wrapper scripts that pass those arguments, we can pass arguments in HTCondor's submit file by specifying the arguments attribute, as shown in this excerpt:

# Create custom variable called "data"
data = data.csv

# Specify the executable & arguments
executable = wrapper.sh
arguments = $(data)

Code

Our executable written in Python, echo-next.py:

import sys
filename = sys.argv[1]
print(filename)

sys.argv is a list -- if you have multiple arguments, you can reference each item in the list using Python's usual notation (sys.argv[1], sys.argv[2], etc.). See Python documentation on sys.argv for details.

We can use the script on the command line, assuming Python is installed and on the PATH:

[user@login]$ python3 echo-next.py data.csv
data.csv

Wrapper script

We can now write a wrapper script! This script checks that one argument is passed, and also allows room for pre- and post-processing in this simple workflow.

Our wrapper script written in bash, wrapper.sh:

#!/bin/bash

# Check if an argument is provided
if [ $# -ne 1 ]; then
    echo 'Please use one argument. Usage: wrapper.sh [arg]'
    exit 1
fi

# Set filename variable to the first argument to the bash script
filename=$1

echo 'Pre-processing could go here.'

# Run code
python3 echo-next.py ${filename}

echo 'Post-processing could go here.'

Ensure wrapper.sh is executable with chmod +x wrapper.sh before running it on the command line:

[user@login]$ chmod +x wrapper.sh
[user@login]$ ./wrapper.sh
Please use one argument. Usage: wrapper.sh [arg]
[user@login]$ ./wrapper.sh data.csv
Pre-processing could go here.
data.csv
Post-processing could go here.

Passing arguments with the HTCondor submit file

Now that we've understood how arguments work in Python and how to write simple wrapper scripts that pass those arguments, we can pass arguments in HTCondor's submit file by specifying the arguments attribute, as shown in this excerpt:

# Create custom variable called "data"
data = data.csv

# Specify the executable & arguments
executable = wrapper.sh
arguments = $(data)

Code

Our executable written in R, echo-next.R:

args <- commandArgs(trailingOnly = TRUE)
filename <- args[1]
cat(filename,"\n")

The output of commandArgs is a vector -- if you have multiple arguments, you can reference each item using R's usual notation (args[1], args[2], etc.). See R documentation on commandArgs for details.

We can use the script on the command line, assuming R is installed and on the PATH:

[user@login]$ Rscript echo-next.R data.csv
data.csv

Wrapper script

We can now write a wrapper script! This script checks that one argument is passed, and also allows room for pre- and post-processing in this simple workflow.

Our wrapper script written in bash, wrapper.sh:

#!/bin/bash

# Check if an argument is provided
if [ $# -ne 1 ]; then
    echo 'Please use one argument. Usage: wrapper.sh [arg]'
    exit 1
fi

# Set filename variable to the first argument to the bash script
filename=$1

echo 'Pre-processing could go here.'

# Run code
Rscript echo-next.R ${filename}

echo 'Post-processing could go here.'

Ensure wrapper.sh is executable with chmod +x wrapper.sh before running it on the command line:

[user@login]$ chmod +x wrapper.sh
[user@login]$ ./wrapper.sh
Please use one argument. Usage: wrapper.sh [arg]
[user@login]$ ./wrapper.sh data.csv
Pre-processing could go here.
data.csv
Post-processing could go here.

Passing arguments with the HTCondor submit file

Now that we've understood how arguments work in R and how to write simple wrapper scripts that pass those arguments, we can pass arguments in HTCondor's submit file by specifying the arguments attribute, as shown in this excerpt:

# Create custom variable called "data"
data = data.csv

# Specify the executable & arguments
executable = wrapper.sh
arguments = $(data)