Import from a DRS server

OVERVIEW

Cancer Genomics Cloud Powered by Seven Bridges (CGC) allows you to import DRS files from either open external sources or the following compatible Seven Bridges environments:

• CAVATICA Powered by Seven Bridges
• BioData Catalyst Powered by Seven Bridges

After importing the files, you can then use them as any other file on the CGC. Please note that the actual files physically remain hosted on the source platform and are only linked to CGC.

Learn more about the procedure for generating a DRS manifest.

PREREQUISITES

In order to be able to import a DRS file, your account on the CGC has to be connected to the DRS server of the environment you are importing data from.

Furthermore, please make sure multi-factor authentication is enabled on both platforms you are connecting.

Access the import feature

1. Open the project to which you're importing files.
2. Select the Files tab and then click + Add Files.
3. Click Add Files > GA4GH Data Repository Service (DRS), as shown below.

The page for importing files is displayed. The following two options are available:

• Paste DRS URIs - use this option to paste the list of DRS URIs of the files you want to import (see below).
• From a manifest file - use this option to import files by using a manifest (see below).

Import files by pasting DRS URIs

1. Enter DRS URIs into the text area (see Form a DRS URI below).

📘

Important notes

The maximum number of links you can put inside the text area is 500. To import again, you will have to wait for 5 minutes.

In addition, please bear in mind that all other API calls you make within this time frame will also count toward this limit. URLs should be entered one per row.

2. Optional: Enter tags in the Tag files box (see below). 
3. Resolve naming conflicts-  choose the method for resolving a naming conflict; read more below.

📘

Allowed characters in file names are all alphanumerics and special characters except slash (/). File names can’t be longer than 255 characters.Allowed characters in folder names are a-z, A-Z, 0-9, and special characters (_), (-), and (.). Folder names can’t be longer than 255 characters.

4. Give your consent regarding importing sensitive data by checking the related option.
5. Click Import to start the upload.

Import from a manifest file

To import files from a manifest file:

1. Access your project.
2. Click the Files tab.
3. Click Add files and choose option "GA4GH Data Repository Service (DRS)".

Learn more:

• Procedure for generating a DRS manifest.
• Important notes regarding the manifest file format.

4. Click Browse manifest and choose the manifest file from your computer.

5, (Optional) Tag your files (see below).

6. Choose the method for resolving naming conflicts (see below).
7. Give your consent regarding importing sensitive data by checking the related option.
8. Click Import.

The files are imported to your project.

Manifest file format

The following rules apply for creating the manifest file.

ADD TAGS TO THE FILES

To make it easier to visually identify and organize files you can tag them before the import. Separate tags with commas or press enter.

All tags which are already in the selected project are autosuggested in the drop-down menu. Learn more about tagging your files. 

RESOLVE NAMING CONFLICTS

Naming conflicts happen when you try to import an item (file or folder) from a DRS server to a project's root folder or subfolder, where an item with the same name already exists. The list below illustrates the possible cases of naming conflicts and the way they are resolved depending on which option is selected in the Resolve naming conflicts dropdown:

1. Importing a file that has the same name as an existing file at the destination path:
   • Skip (default): The file won't be imported. This is the default behavior.
   • Auto-rename: The imported file will be automatically renamed by adding a prefix (underscore and serial number).
   • Overwrite: The existing file will be overwritten by the file imported from the server.
2. Importing a file that has the same name as an existing folder at the destination path:
   • Skip (default): The file won't be imported.
   • Auto-rename: The imported file will be automatically renamed by adding a prefix (underscore and serial number).
   • Overwrite: The file won't be imported.

The following table illustrates the entire naming conflict resolution mechanism:

Check import status

To check the progress of your import, click the notification bell in the upper right corner. The "Activity" area is displayed.

Click Open activity center to see more detailed information about each of the imports.

Click on an import in the left pane, and the right pane will show detailed information about that import including:

• DRS URI - the URI the file was imported from
• Status - the status of the import for each of the files; in case of a failed import, the following information will also be available:
  • Error Code - click to see detailed information about the error code
  • Error Message - the description of the error

Form a DRS URI on a compatible Seven Bridges environment

This step takes place on CAVATICA Powered by Seven Bridges or BioData Catalyst Powered by Seven Bridges.

A DRS URI is formed by appending the ID of the file you want to import, to the DRS endpoint of the environment you are importing the files from.

To form a DRS URI for importing a file:

1. Use the DRS endpoint for the environment you are importing the file from:
   CAVATICA Powered by Seven Bridges: drs://cavatica-ga4gh-api.sbgenomics.com
   BioData Catalyst Powered by Seven Bridges: drs://ga4gh-api.sb.biodatacatalyst.nhlbi.nih.gov
   You can also find the endpoint when logged in to the chosen environment, in the Account Settings page under Dataset Access.
2. Get the ID of the file from the URL, once you have clicked on a file to see its details.

3. Form the DRS URI by following the <drs-endpoint>/<file-id> pattern. For example, a full DRS URI for a file you want to import from BioData Catalyst Powered by Seven Bridges would be: drs://ga4gh-api.sb.biodatacatalyst.nhlbi.nih.gov/5cd573bce4b0359df2c239ae.
4. (Optional) In the Add tags field add the keywords (tags) that describe the imported items.
5. Resolve naming conflicts - Select the action to be taken if a naming conflict occurs. Available actions are Skip (default option), Auto-rename and Overwrite. Read more about naming conflicts resolution.
6. Click the checkbox to confirm that you understand the terms of use.
7. Click Import. The file import process starts.