# Cloud Computing for HPC Users

## HPC vs the DNAnexus Platform

| Component                  | HPC                                                  | DNAnexus Platform                                                         |
| -------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------- |
| Driver/ Requestor          | Head Node of Cluster                                 | API Server                                                                |
| Submission Script Language | Portable Bash System (PBS) or SLURM                  | dx-toolkit                                                                |
| Worker                     | Requested from pool of machines in private cluster   | requested from pool of machines in AWS/ Azure                             |
| Shared Storage             | Shared file system for all nodes (Lustre, GPFS, etc) | Project storage (Amazon S3/ Azure storage)                                |
| Worker File I/O            | Handled by Shared file system                        | needs to be transferred to and from project storage my commands on worker |

## Key Players with an HPC

* With an HPC, there is a collection of specialized hardware, including mainframe computers, as well as a distributed processing software framework so that the incredibly large computer system can handle massive amounts of data and processing at high speeds.
* The goal of an HPC is to have the files on the hardware and to also do the analysis on it. In this way, it is similar to a local computer, but with more specialty hardware and software to have more data and processing power.

![](https://3504452945-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fp8rQlIsJAoT4gb1fg2yr%2Fuploads%2Fgit-blob-322b14feecd098ebc3e6edb4c627aa08ad9266fd%2Fcloud8-01.png?alt=media)

* Your computer: this communicates with the HPC cluster for resources
* HPC Cluster
  * Shared Storage: common area for where files are stored. You may have directories branching out by users or in another format
  * Head Node: manages the workers and the shared storage
  * HPC Worker: is where we do our computation and is part of the HPC cluster.
* These work together to increase processing power and to have jobs and queues so that when the amount of workers that are needed are available, the jobs can run.

## Key Players in Cloud Computing

* In comparison, cloud computing adds layers into analysis to increase computational power and storage.
* This relationship and the layers involved are in the figure below:

  ![](https://3504452945-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fp8rQlIsJAoT4gb1fg2yr%2Fuploads%2Fgit-blob-7214f430e14df324f41d4750b479fa9bf5e9c948%2Fcloud4.png?alt=media)
* Let's contrast this with processing a file on the DNAnexus platform.
  * We'll start with our computer, the DNAnexus platform, and a file from project storage.
  * We first use the dx run command, requesting to run an app on a file in project storage. This request is then sent to the platform, and an appropriate worker from the pool of workers is made available.
  * When the worker is available, we can transfer a file from the project to the worker.
  * The platform handles installing the app and its software environment to the worker as well.
  * Once our app is ready and our file is set, we can run the computation on the worker.
  * Any files that we generate must be transferred back into project storage.

## Key Differences

* HPC jobs are limited by how many workers are physically present on the HPC.
* Traditionally, cloud computing has better architecture than an HPC, so the jobs are more efficient.

## Transferring Files

* One common barrier is getting our files onto the worker from project storage, and then doing computations with them on the worker. The last barrier we'll review is getting the file outputs we've generated from the worker back into the project storage.
* Cloud computing has a nestedness to it and transferring files can make learning it difficult.
* A mental model of how cloud computing works can help us overcome these barriers.

### Resolution:

* Cloud computing is indirect, and you need to think 2 steps ahead.
* Here is the visual for thinking about the steps for file management:

  ![](https://3504452945-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fp8rQlIsJAoT4gb1fg2yr%2Fuploads%2Fgit-blob-dd13d997bace125fac6b734e01f918d80cd31c4e%2Fcloud5.png?alt=media)

## Running apps

Creating apps and running them is covered later in the documentation.

Apps serve to (at minimum):

1. Request an EC2/Azure **worker**
2. Configure the **worker's environment**
3. Establish **data transfer**

### Why do this with DNAnexus?

* Highly secure platform with built-in compliance infrastructure
* Fully configurable platform
  * User can run **single scripts** to **fully-automated, production-level workflows**
* Data transfer designed to be fast and efficient
  * Read and analyze massive files directly using dxfuse
* Instances are configured for you via **apps**
  * Variety of ways to configure your own environments
* Access to the wealth of [AWS/Azure resources](https://documentation.dnanexus.com/developer/api/running-analyses/instance-types)
  * Largest Azure instances: **\~4Tb RAM**
  * Largest AWS instances: **\~2Tb RAM**

### Equivalent Commands

| Task        | dx-toolkit                 | PBS            | SLURM            |
| ----------- | -------------------------- | -------------- | ---------------- |
| Run Job     | dx run \<app-id> \<script> | qsub \<script> | sbatch \<script> |
| Monitor Job | dx find jobs               | qstat          | squeue           |
| Kill Job    | dx terminate \<jobid>      | qdel \<jobid>  | scancel \<jobid> |

### Practical Approaches

* Single Job
  * Use \`**dx run**\` on the CLI directly
  * Use \`**dx run**\` in a shell script
* [Batch Processing](https://documentation.dnanexus.com/user/running-apps-and-workflows/running-batch-jobs)
  * Use a shell script to use \`**dx run**\` on multiple files
  * Use dxFUSE to directly access files (read only)
  * [**dx generate-batch-inputs**](https://documentation.dnanexus.com/user/running-apps-and-workflows/running-batch-jobs#batching-multiple-inputs)/ **dx run --batch-tsv**

## Batch Processing Comparisons

| Component | HPC Recipe                             | Cloud Recipe                                                                 |
| --------- | -------------------------------------- | ---------------------------------------------------------------------------- |
| 1         | List Files                             | List Files                                                                   |
| 2         | Request 1 worker/ file                 | Use loop for each file: 1) use dx run, 2) transfer file, and 3) run commands |
| 3         | use array ids to process 1 file/worker |                                                                              |
| 4         | submit job to head node                |                                                                              |

## Resources

[Full Documentation](https://documentation.dnanexus.com/)

To create a support ticket if there are technical issues:

1. Go to the Help header (same section where Projects and Tools are) inside the platform
2. Select "Contact Support"
3. Fill in the Subject and Message to submit a support ticket.