Bring WDL apps to the CGC

📘

To get access to this feature, please contact our Support Team.

Overview

Workflow Description Language (WDL) is an open, standardized, human readable and writable language for expressing tasks and workflows. WDL is designed to be a general-purpose workflow language, but it is most widely used in the field of bioinformatics. If you already have tools or execution pipelines that are described in WDL, this page will provide details on how you can push them to the CGC and run your analysis at scale, using the full potential and benefits of the Seven Bridges execution environment.

Prerequisites

• An account on the CGC.
• Installed sbpack. This is a command-line tool you'll use to adapt and push WDL apps to the CGC. For more details on what sbpack can do, how to install it and its main use cases, see About sbpack below.
• (Optional) Downloaded WOMtool. The WOMtool jar executable is used to generate the JSON input file that specifies all of the WDL app's inputs.

About sbpack

The primary use of sbpack is to provide an easy way to upload (sbpack) and download (sbpull) apps to/from any Seven Bridges powered platform. Since it is a command-line tool, it can be particularly useful as a part of continuous development and integration pipelines for bioinformatics apps, as it allows seamless automated deployment of new app versions to the CGC. It works with apps described using the following workflow description standards:

• Common Workflow Language (CWL). Apart from enabling the standard app pull and push flows, also provides advanced functionalities such as resolution of linked processes, schemadefs and $includes and $imports.
• Nextflow. Adapts, prepares and pushes Nextflow apps for execution in Seven Bridges environments using a special sbpack_nf command.
• Workflow Description Language (WDL). Uses the sbpack_wdl command to convert and push WDL apps for execution in Seven Bridges environments.

To install sbpack, use the standard install method through pip:

pip install sbpack

Procedure

The procedure of publishing WDL apps for use on the CGC is a process that consists of the following two stages:

• Initial app conversion. In this step, your WDL app will be converted to a format that is executable on the CGC. However, to optimize your app for execution on the CGC and make the most out of the CGC's execution environment it is strongly recommended to go through the next step.
• Optimizing the converted app for execution in Seven Bridges environments. The app that has been initially converted now contains an additional configuration file that you will use to define CGC-specific options and fully optimize it for use in the Seven Bridges execution environment. Once the optimized configuration is prepared, the app configuration is pushed to the CGC again.

Initial app conversion

This step adapts the WDL app for execution on the CGC. It is performed by executing the sbpack_wdl command in the following format:

sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --womtool-input WOMTOOL_INPUT

In the command above, replace the placeholders as follows:

• PROFILE_NAME is the CGC profile containing the CGC API endpoint and authentication token, as set in the Seven Bridges credentials file.
• APPID specifies the identifier of the app on the CGC, in the {user}/{project}/{app_name} format. The {user} part is your CGC username. The {project} part is the project to which you want to push the app and {app_name} is the name you want to assign to the app. For example the full app ID can be rfranklin/my-new-project/my-wdl-app. If the specified app ID does not exist, it will be created. If it exists, a new revision (version) of the app will be created.
• WORKFLOW_PATH needs to be replaced with the path where the WDL app files are located on your local machine. The WORKFLOW_PATH folder should include all the dependencies that the workflow needs in order to run.
• ENTRYPOINT should be replaced with the path to the actual .wdl file, relative to the root of WORKFLOW_PATH.
• WOMTOOL_INPUT should be replaced with the path to the JSON inputs file containing your app's inputs schema generated using WOMtool. If you don't have the inputs file, replace --womtool-input with the --womtool-path argument and provide the path to the WOMtool jar executable on your local machine, which will generate the inputs and pass them to sbpack_wdl automatically.

Here is a sample of the command:

sbpack_wdl --profile cgc --appid rfranklin/wdl-project/test-app --workflow-path /Users/rfranklin/apps/wdl/demo --entrypoint cellranger_workflow.wdl --womtool-input /Users/rfranklin/apps/wdl/demo/inputs.json

Once executed successfully, this command will convert the WDL app for use on the CGC and push it to the CGC project specified as the value of the --appid argument. The local directory specified as the value of --workflow-path will now contain an additional sb_wdl_schema.yaml file. The file contains configuration parameters that can be adjusted and optimized for execution on the CGC.

Optionally, to avoid pushing the app to the CGC at this stage and perform optimizations for the Seven Bridges execution environment beforehand, use the --dump-sb-app flag at the end of the command. For a full list of available arguments to the sbpack_wdl command, see the sbpack_wdl command reference.

Optimizing the converted app for execution in Seven Bridges environments

When you have performed the initial conversion step, the generated sb_wdl_schema.yaml file is important as it contains confguration parameters that will help you optimize the app for execution on the CGC. The file consists of the following major sections:

• The initial section that includes general app information and the documentation content describing the app (if any):
  • app_content: Contains details about app's package and WDL file:  
  • code_package: CGC ID of the file that contains the WDL code. This is replaced by the git_pull key if the --no-package option was used to set a git repository as the source of the app's code. See the sbpack_wdl command reference for details.
  • entrypoint: Relative path to the file containing the WDL code, relative to the root directory of the ZIP file defined in code_package.
  • class: Defines the type of workflow description language used for the app. The value will always be wdl for WDL apps.
  • cwlVersion: Defines the version of CWL used to describe the app. The value will always be None for WDL apps.
  • doc (Optional): The Markdown-formatted text describing the app. 
  • The inputs section that defines details of the app inputs.
  • The outputs section that defines details of app outputs.
  • The requirements section that defines app execution requirements such as initial working directory.

Configuring inputs

Each of the app inputs that is present in the inputs section contains the following basic details:

• id: Unique identifier of the input.
• inputBinding: Defines the mapping of the input to the command line of the app that is being executed. If inputBinding is omitted, input is made available in the execution environment, but is not passed on to the WDL executor.
  • prefix: The ID of the WDL workflow input, in the form that would be added to the WDL inputs JSON file. For example, if you provide a value for an input that is labelled as workflow.input_file in the WDL inputs JSON file, the prefix would be defined as workflow.input_file.
• default: The default value for the input. If the input value is not set on task execution, this default value is taken and passed on to the executor as defined with inputBinding.prefix.
• label: Text description of the input. 
• sbg:toolDefaultValue: Default value of the input in the WDL workflow. Value provided here is not used in execution and is descriptive (for information purpose) only.
• sbg:fileTypes: Comma separated (with spaces) value of file extensions that are used in the file picker when setting up tasks. For example: sbg:fileTypes: “FASTQ, FASTQ.GZ”.
• type: The type of value expected on the input. If there is a question mark at the end, for example int?, the input is optional. Otherwise, the input is mandatory. Learn more about available types of inputs.

To accommodate for the transition between WDL and the Seven Bridges execution environment, the sb_wdl_schema.yaml file will always contain an additional input whose ID is auxiliary_files, which contains the list of files not added as explicit workflow inputs but are required for workflow execution. To enable proper execution on the CGC, please do not remove this input from sb_wdl_schema.yaml.

Note: File and Directory inputs

In some WDL versions, Directory inputs are marked as File. The type of such inputs will consequently be File on the CGC. If the input should actually be a directory, simply change its type property from File to Directory in sb_wdl_schema.yaml.

📘

There may be cases where input types are not properly set by sbpack_wdl during initial conversion, especially for complex input types. These cases may need to be handled manually based on each use case.

Configuring outputs

In addition to executing WDL apps on the CGC, you also need to optimize app outputs to produce and save only files that match the defined criteria, To achieve this and be able to further configure your app outputs, see the details about configuration parameters contained in the outputs section of the sb_wdl_schema.yaml file:

• id: Unique identifier of the output. You can change this value to provide a more adequate and descriptive one if necessary.
• outputBinding: Defines the glob expression or pattern that will be used to select the output file.
  • glob: The glob expression that defines the items to keep as outputs on the output port.
• type: The type of output value.

Example: Configuring an output

The sb_wdl_schema.yaml file always contains one automatically generated app output:

outputs:
  - id: wdl_workdir
    outputBinding:
        glob: '*'
    type: File[]

To configure your outputs, create an entry for each of the output ports you want to have, following the pattern below:

outputs:
  - id: csv_file
    outputBinding:
        glob: '*.csv'
    type: File

In the example above, replace csv_file with an ID that describes your output and replace *.csv with the glob that matches the type and naming of the data you want to produce on the output. Learn more about available types of outputs.

Example: Configuring a dynamic output directory name

You can also use the sb_wdl_schema.yaml file to set the name of the output directory by defining it in an app input, provided that the tool itself supports the option of defining the output directory name using the corresponding input argument and its value. The first step is to define the input that takes the output directory name (in the sb_wdl_schema.yaml inputs section):

inputs:
  - id: my_workflow_outdir_name
    inputBinding:
        prefix: my_workflow.outdir_name
    type: string?

Once you have defined the input, define an output, where the output directory glob will be a variable that gets the value defined in the input above.

outputs:
  - id: output_directory
    outputBinding:
        glob: '$(inputs.outdir_name)'
    type: Directory

The $(inputs.outdir) value is a variable that will be replaced with the actual value entered in the outdir input when the app is executed.

Configuring requirements

The requirements sections is primarily used for the following execution-related parameters:

• Setting input staging (making input files available in the app's working directory).
• Setting computation instances that are used for app executions on the CGC.

Setting input staging

Files that are named as inputs to a tool are not, by default, in the tool's working directory. In most apps this access is sufficient, since most tools only need to read their input files, process the data contained in them, and write new output files on the basis of this data to their working directory. However, in some cases an app might require input files to be placed directly in its working directory. If this is the case with your app, modify the requirements section in the sb_wdl_schema.yaml file as follows:

requirements:
  - class: InitialWorkDirRequirement
    listing:
      - '$(inputs.tsv_files)'

Entries under listing define files and directories that will be made available in the app’s working directory before the command is executed. The files and directories are usually defined as variables named after their respective input IDs.

Another useful option is creation of a file directly in the working directory. This is done by defining entryname and entry keys in the InitialWorkDirRequirement class, as follows:

requirements:
- class: InitialWorkDirRequirement
  listing:
  - entryname: input_nanoseq.csv   
    entry: |
    ${
    if (inputs.auxiliary_files && !inputs.in_csv_file){
        var content = 'group,barcode';
        for (var i = 0; i  < inputs.auxiliary_files.length; i++){
            if (inputs.auxiliary_files[i].metadata['barcode']){
                var barcode = inputs.auxiliary_files[i].metadata['barcode'];
            }
            else {
                var barcode = '';
            }
            if (inputs.auxiliary_files[i].metadata['group']){
                var group = inputs.auxiliary_files[i].metadata['group'];
            }
            else {
                var group = '';
            }
            content = content.concat(group,',',barcode,'\\n');
        }
        return content
    }
    else {
        return ''
    }

In the example code above, entryname defines the name of the file generated in the working directory, which is input_nanoseq.csv, while entry contains a Javascript expression that populates the generated file by getting barcode and group metadata values from input files and concatenating them in a single CSV file. As we're using a Javascript expression, it is recommended to use the default option of generating a YAML schema when using the sbpack_wdl command, as YAML is more convenient and reduces the possibility of making errors compared to using Javascript expressions in a JSON file. The expression can be defined to match your needs and intended use. Read more about dynamic expressions in tool descriptions or see some of the most common expression examples in our Javascript Cookbook.

Setting execution instances

Another useful option that is available for configuration in the hints section is the definition of the computation instance used for app execution on the CGC. This is also done by defining key-value pairs as follows:

hints:
  - class: sbg:AWSInstanceType
    value: c4.8xlarge;ebs-gp2;2000

In this case, the workflow uses a c4.8xlarge instance with 2000 GB of attached EBS storage. The value consists of the following three parts (separated by ;):

• Instance type, e.g. c4.8xlarge.
• Attached disk type: always ebs-gp2 for all instances with EBS storage.
• Disk size in GB.

See the list of AWS instances that are available for task execution on the CGC. 

Pushing the optimized app configuration to the CGC

When you are done with changes to the sb_wdl_schema.yaml file, push the optimized app configuration to the CGC. As we just making configuration changes to an app that has already been pushed to the CGC, this can be done using the regular sbpack command in the following format:

sbpack PROFILE_NAME APPID CONFIG_FILE

In the command above, PROFILE_NAME refers to the CGC profile containing the CGC API endpoint and authentication token, as set in the Seven Bridges credentials file. The  APPID parameter specifies the identifier of the app on the CGC. Use the same  APPID you used in the initial conversion step. Finally, CONFIG_FILE is the sb_wdl_schema.yaml in which you made app execution optimizations. The final command should be, for example:

sbpack cgc-profile rfranklin/wdl-project/test-app sb_wdl_schema.yaml

This pushes the modified app configuration to the CGC and creates a new revision (version) of the app. Once this is done, you are ready to run a task using the app.

Updating already converted and optimized apps

If you have already converted your app, made optimizations in the sb_wdl_schema.yaml file, and pushed the app to the CGC, all subsequent updates to the app's WDL code and the process of propagating the update to the CGC are quite straightforward. If the updates you made do not require changes to manually configured parameters in the sb_wdl_schema.yaml file (such as inputs, outputs, requirements, etc.), create a new code package by running a command in the following format:

sbpack_wdl --profile PROFILE_NAME --appid APPID --workflow-path WORKFLOW_PATH --entrypoint ENTRYPOINT --sb-schema SB_SCHEMA

This command is almost the same as the initial app conversion step, but differs in the additional --sb-schema argument. This argument allows you to provide and reuse an existing sb_wdl_schema.yaml configuration file where you have already made optimizations (configuration of inputs, outputs, requirements, etc.) for the execution of your app on the CGC. The command will generate a new code package based on your updated WDL code provided through --workflow-path and --entrypoint and the YAML or JSON configuration file provided through --sb-schema, and push the updated app to the CGC creating a new revision (version).

Copying WDL apps between projects on the CGC

When an app is on the CGC, you can copy it and use it on other CGC projects. To copy WDL apps between projects, use the sbcopy command that is a part of the sbpack utility:

sbcopy [--profile PROFILE] --appid APPID --projectid PROJECTID

The command takes the following arguments:

• PROFILE: refers to the CGC profile containing the CGC API endpoint and authentication token, as set in the Seven Bridges credentials file.
• APPID: specifies the identifier of the app on the CGC. Takes the form {user}/{project}/{app_id}. The {user} part is your CGC username. The {project} part is the source project where the app is located and {app_id} is the ID of the app you want to copy; for example rfranklin/my-new-project/my-wdl-app.
• PROJECTID: is the identifier of the destination project where the app will be copied. Takes the form {user}/{project}.

The final command should be, for example:

sbcopy cgc-profile rfranklin/wdl-project/test-app jsmith/my-wdl-project

📘

Note that WDL app copies made through standard visual interface or API methods instead of using sbcopy will still point to the originally pushed code package and the original project where it is located. This might cause failures due to lack of permissions, if users who need to run the copied instances of the app aren't added to the project where the original code package is located. To avoid this, please use sbcopy to copy WDL apps between projects on the CGC, as described above.

sbpack_wdl command reference

Here is a list describing all available arguments od the sbpack_wdl command that is used to convert and push WDL apps for execution on the CGC.

Important notes for executing WDL apps on the CGC

• Workflows are executed in Local mode. Make sure your workflow can run in Local mode before porting it to the CGC.
• Use of Docker is required. See how to create and upload a Docker image containing your app and make sure to edit the WDL code to use the newly created image. If the Docker image is not specified for a process, a default alpine image will be used.
• Execution is done in a dedicated working directory and all the WDL work is done in the wdl_workdir directory inside the working directory. Avoid using and relying on hard-coded paths in workflows.

Differences between running CWL and WDL apps on the CGC

Executions on the CGC normally result in separate jobs (steps) being created for each tool in the workflow. When a WDL pipeline is executed on the CGC, a single execution job will be created, regardless of the number of tools within the pipeline. This "one app, one job" approach results in the following differences compared to CWL app executions:

• Memoization is not available for WDL apps. As memoization relies on using previous job outputs to skip identical jobs in new executions, it is not useful in situations when there is only one job in an app execution.
• Handling of spot instance interruption can't be used with WDL apps. If a spot instance gets terminated when a WDL app is running, the app would have to be rerun on an on-demand instance from the beginning, which would result in increased costs instead of savings. Therefore, it is recommended to disable spot instances for WDL app executions.
• Task stats and logs are organized differently. As task statistics and logs are usually shown and organized per job, WDL apps will have cumulative stats for the single job in the execution, while logs will be saved in a single folder.