{"_id":"57852b1ba795200e00f14939","version":{"_id":"55faf11ba62ba1170021a9aa","project":"55faf11ba62ba1170021a9a7","__v":45,"createdAt":"2015-09-17T16:58:03.490Z","releaseDate":"2015-09-17T16:58:03.490Z","categories":["55faf11ca62ba1170021a9ab","55faf8f4d0e22017005b8272","55faf91aa62ba1170021a9b5","55faf929a8a7770d00c2c0bd","55faf932a8a7770d00c2c0bf","55faf94b17b9d00d00969f47","55faf958d0e22017005b8274","55faf95fa8a7770d00c2c0c0","55faf96917b9d00d00969f48","55faf970a8a7770d00c2c0c1","55faf98c825d5f19001fa3a6","55faf99aa62ba1170021a9b8","55faf99fa62ba1170021a9b9","55faf9aa17b9d00d00969f49","55faf9b6a8a7770d00c2c0c3","55faf9bda62ba1170021a9ba","5604570090ee490d00440551","5637e8b2fbe1c50d008cb078","5649bb624fa1460d00780add","5671974d1b6b730d008b4823","5671979d60c8e70d006c9760","568e8eef70ca1f0d0035808e","56d0a2081ecc471500f1795e","56d4a0adde40c70b00823ea3","56d96b03dd90610b00270849","56fbb83d8f21c817002af880","573c811bee2b3b2200422be1","576bc92afb62dd20001cda85","5771811e27a5c20e00030dcd","5785191af3a10c0e009b75b0","57bdf84d5d48411900cd8dc0","57ff5c5dc135231700aed806","5804caf792398f0f00e77521","58458b4fba4f1c0f009692bb","586d3c287c6b5b2300c05055","58ef66d88646742f009a0216","58f5d52d7891630f00fe4e77","59a555bccdbd85001bfb1442","5a2a81f688574d001e9934f5","5b080c8d7833b20003ddbb6f","5c222bed4bc358002f21459a","5c22412594a2a5005cc9e919","5c41ae1c33592700190a291e","5c8a525e2ba7b2003f9b153c","5cbf14d58c79c700ef2b502e"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"category":{"_id":"5785191af3a10c0e009b75b0","version":"55faf11ba62ba1170021a9aa","__v":0,"project":"55faf11ba62ba1170021a9a7","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-07-12T16:21:46.337Z","from_sync":false,"order":28,"slug":"connect-cloud-storage","title":"CONNECT CLOUD STORAGE"},"parentDoc":null,"project":"55faf11ba62ba1170021a9a7","__v":1,"githubsync":"","user":"5613e4f8fdd08f2b00437620","updates":[],"next":{"pages":[],"description":""},"createdAt":"2016-07-12T17:38:35.002Z","link_external":false,"link_url":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":7,"body":"<a name=\"overview\"></a>\n##Overview\n\nThe Volumes API contains two types of calls: one to connect and manage cloud storage, and the other to **import** data from a connected cloud account.\n \nBefore you can start working with your cloud storage via the CGC, you need to authorize the CGC to access and query objects on that cloud storage on your behalf. This is done by creating a \"volume\". A volume enables you to treat the cloud repository associated with it as external storage for the CGC. You can import files from the volume to the CGC to use them as inputs for computation.\n\n<a name=\"procedure\"></a>\n##Procedure\n\nThis short tutorial will guide you through setting up a volume for a GCS bucket.  You'll register your GCS bucket as a volume and make an object from the GCS bucket available on the CGC.\n\nOnce a volume has been created, you can issue import operations to make data appear on the CGC.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"In this tutorial we assume you want to connect to an Google Cloud Storage bucket. The procedure will be slightly different for other cloud storage providers, such as an [Amazon S3 bucket](doc:aws-cloud-storage-tutorial). For more information, please refer to our list of [supported cloud storage providers](doc:supported-cloud-storage-providers).\"\n}\n[/block]\n<a name=\"prerequisites\"></a>\n##Prerequisites\n\nTo complete this tutorial, you will need:\n1. A [Google (GCP)](https://cloud.google.com/) account.\n2. One or more buckets on this GCP account via [Google Cloud Storage (GCS)](https://cloud.google.com/storage/).\n3. One or more objects (files) in your target bucket.\n4. An authentication token for the CGC. Learn more about [getting your authentication token](get-your-authentication-token).\n\n<a name=\"register\"></a>\n##Step 1: Register a GCS bucket as a volume\n\nTo set up a volume, you have to first register a GCS bucket as a volume. Volumes mediate access between the CGC and your buckets, which are local units of storage in GCS.\n\nYou can register a GCS bucket as a volume through the following steps below.\n\n<a name=\"create\"></a>\n###1a: Create an IAM (Identity and Access Management) user\n\n1. Log into the [Google Cloud Platform console](https://console.cloud.google.com).\n2. From the menu on the left select **IAM & Admin** > **Service accounts**.\n3. Click **+ Create service account** below the search bar.\n4. Fill in account details:\n    * **Service account name** - Descriptive name to label the account.\n    * **Service account ID** - Generated automatically based on the entered service account name. Can be modified if necessary.\n    * **Service account description** - More elaborate description of the account’s purpose.\n5. Click **Create**. The **Service account permissions** screen opens.\n6. In the **Select a role** dropdown, select **Storage** > **Storage Object Viewer**.\n7. Click **Continue**. The final screen of the wizard opens.\n8. In the **Create key** section, click **+ Create key**. Key options are displayed on the right.\n9. In the **Key type** list select **JSON**.\n10. Click **Create**. Your browser will download a JSON file containing the credentials for this user. Keep this file safe.\n\n<a name=\"authorize\"></a>\n###1b: Authorize this IAM user to access your bucket\n\n1. On the [Google Cloud Platform](https://console.cloud.google.com/) console, click <img src=\"https://files.readme.io/32ef300-gcp-hamburger.png\"\nheight=\"17px\" width=\"auto\" align=\"inline\" style=\"margin:1px\"/> in the top-left corner and navigate to the **Storage** section\n2. Select **Storage** > **Browser**.\n3. Locate your bucket and click the three vertical dots to the far right of your bucket's name.\n4. Click **Edit bucket permissions**.\n5. Click **Add members**.\n6. In the **New members** field enter the service account client's email. This email is located in the JSON file downloaded in the previous section.\n7. From the **Select a role** drop-down menu, select **Storage Legacy** > **Storage Legacy Bucket Reader**.\n8. Click **Save**. You have now authorized the newly-created IAM user to access the storage bucket.\n\n<a name=\"register-bucket\"></a>\n###1c: Register a bucket\n\nAt this point, you can associate the bucket with your CGC account by registering it as a volume.\n\nTo register your bucket as a volume, make the API request to [Create a volume](doc:create-a-volume-v2), as shown in the HTTP request below. Be sure to paste in your authentication token for the X-SBG-Auth-Token key.\n\nThis request also requires a request body. Provide a `name` for your new volume, an optional description, and an object (`service`) containing the information in the table below. Specify the `access_mode` as `RO` for read-only permissions. Be sure to supply a `bucket` name and substitute in your own `credentials`.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Key\",\n    \"h-1\": \"Description of value\",\n    \"0-0\": \"`type`\\n_required_\",\n    \"0-1\": \"This must be set to `gcs`.\",\n    \"1-0\": \"`bucket`\\n_required_\",\n    \"1-1\": \"The name of your GCS bucket.\",\n    \"2-0\": \"`prefix`\\n_default: empty string_\",\n    \"3-0\": \"`client_email`\",\n    \"4-0\": \"`private_key`\",\n    \"2-1\": \"If provided, the value of this parameter will be used to modify any object key before an operation is performed on the bucket.\\n\\nEven though Google Cloud Platform is not truly a folder-based store and allows for almost arbitrarily named keys, the prefix is treated as a folder name. This means that after applying the prefix to the name of the object the resulting key will be normalized to conform to the standard path-based naming schema for files.\\n\\nFor example, if you set the `prefix` for a volume to `a10`, and import a file with `location` set to `test.fastq` from the volume to the CGC, then the object that will be referred to by the newly-created alias will be `a10/test.fast`.\",\n    \"3-1\": \"The client email address for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials.\",\n    \"4-1\": \"The private key for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials.\"\n  },\n  \"cols\": 2,\n  \"rows\": 5\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"POST /v2/storage/volumes HTTP/1.1\\nHost: cgc-api.sbgenomics.com\\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\\ncontent-type: application/json\",\n      \"language\": \"http\",\n      \"name\": \"Create a volume\"\n    }\n  ]\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"name\\\": \\\"tutorial_volume\\\",\\n  \\\"description\\\": \\\"New GCS volume\\\",\\n  \\\"service\\\": {\\n    \\\"type\\\": \\\"gcs\\\",\\n    \\\"bucket\\\": \\\"test-bucket\\\",\\n    \\\"prefix\\\": \\\"\\\",\\n    \\\"credentials\\\": {\\n      \\\"client_email\\\": \\\"user:::at:::service.iam.gserviceaccount.com\\\",\\n      \\\"private_key\\\": \\\"-----BEGIN PRIVATE KEY-----\\\\nMIIEpQIBAAKCAQEAsXj4E7swo97szcOrAcraSbsGnNuTU1b/4llyspDa0lltZIKL\\\\nfl5s3QoqbjUWqAZXkJexei85g49ULD8BGKH2r4EF+XyKcpoon4uIFcbmYcmsUXM\\\\nJ3ujgyL5DbWnQZ6GrqgFNRFVVz/PuvTZOd6KFCrjbbtCxfKoXQrmCwFC/4NlFR3v\\\\n1kavU81w201Mied3e+pxjfiQKAJOoy5I7kfuH20xfzHXWR2YHdQGbzOUZyPgmzZ6\\\\nH6Ry39b7bgLVbyk3++e13KrsTEf58rRzUHLzlcUDcGyf8iTO2vA2qzcbrbovwqJr\\\\n7H4ZfFllDMYQ/ISj4cmi+sz/hR43LUK86emrXwIDAQABAoIBADBr2fvAMbINsZm+\\\\njjTh/ObrAWXgvvSZIx3F2/Z+cUW9Ioyu1ZJ3/uncMTF6iKD1ggSwbqVQIq7zKaWP\\\\ndGNZ4sk62PEQSx8924iiNsGaIqyj5FmvuoD3SeiorR0hd+3+a67RpwIQpaE1ht7y\\\\nmSYh4riX7w9sbU6G44rnQ1azVG1UHvk5ieOD4OPvJopuc6D6ow1oJOnHE0k8v3HY\\\\n1FpLdWCL6nSERqXOI5w+tllG4NMUmTZ2jhaBSEM4PIJVO+24TM3XFCcvhZ7ipPMF\\\\nP5B8hV4hDA4Av1Ei7iuRZlJsH4sRrtHJE3/FZLgqHRRvt/7w4c1xnwirNghtTNMb\\\\nXVoaS/ECgYEA15vL3l22mIoePlcCxIgVCAxhKm6TVQZsAE2EaeVsJKDl0AgCtn/1\\\\nThMIPPGkO8jmjqHGgA+FhjoUQuCCdIuON00mUpmUxZlwI5+uknuK597/zAjd6W8s\\\\n7p9apvBUDfod0hwF9Jfw+aUtZm6EAUNR1Odbb+bpXp1luwfcesHe4QcCgYEA0rg8\\\\nZBBwh2DetU6wWh2JIejBH5SfRUqtEwo5WiEZhrEQLazcpX4w5uvESnT+xd7qx3yC\\\\n/vyzqmy+YwP92Ql0vZApdQoyKGHVntY/o3HYxZD3x+7BKThUs747WjdSo8SwBkSr\\\\nxEzLBgTqqcho6UXvYTTEAg11F5yNYzbvVf4vROkCgYEAh6XtTamIB9Bd1rrHcv5q\\\\nvPWM7DVFXGj96fLbLAS7VRAlhgyEKG2417YBqNYejb6Hz5TYXhll2F0SAkFd0hU7\\\\nFG/lfHJDt04hz0fXfTFc4yTZqnSpqQPZMQfw8LajK2gA+v/Gf2xYn7fcKGW/h0vj\\\\nYB9u16hfirdcGZ+Ih3MR1mECgYEAnq1b1KJIirlYm8FYrVOGe4FxRF2/ngdA05Ck\\\\nZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALJlQZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+CxZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MjZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALSi0sVSXpA=\\\\n-----END PRIVATE KEY-----\\\"\\n    }\\n  },\\n  \\\"access_mode\\\": \\\"RO\\\"\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Create a volume request body\"\n    }\n  ]\n}\n[/block]\nYou'll see a response providing the details for your newly created volume, as shown below.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"href\\\": \\\"https://cgc-api.sbgenomics.com/v2/storage/volumes/rfranklin/tutorial_volume\\\",\\n  \\\"id\\\": \\\"rfranklin/tutorial_volume\\\",\\n  \\\"name\\\": \\\"tutorial_volume\\\",\\n  \\\"description\\\": \\\"New GCS volume\\\",\\n  \\\"access_mode\\\": \\\"RO\\\",\\n  \\\"service\\\": {\\n    \\\"type\\\": \\\"gcs\\\",\\n    \\\"bucket\\\": \\\"test-bucket\\\",\\n    \\\"prefix\\\": \\\"\\\",\\n    \\\"credentials\\\": {\\n      \\\"client_email\\\": \\\"user@service.iam.gserviceaccount.com\\\"\\n    }\\n  },\\n  \\\"created_on\\\": \\\"2019-05-26T16:44:20Z\\\",\\n  \\\"modified_on\\\": \\\"2019-05-26T16:44:20Z\\\",\\n  \\\"active\\\": true\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Response body\"\n    }\n  ]\n}\n[/block]\n<a name=\"import\"></a>\n##Step 2: Make an object from the bucket available on the CGC\n\nNow that we have a volume, we can make data objects from the bucket associated with the volume available as \"aliases\" on the CGC. Aliases point to files stored on your cloud storage bucket and can be copied, executed, and organized like normal files on the CGC. We call this operation \"importing\". Learn more about working with [aliases](doc:aliases).\n\nTo import a data object from your volume as an alias on the CGC, follow the steps below.\n\n<a name=\"launch-import\"></a>\n###2a: Launch an import job\n\nTo import a file, make the API request to [start an import job](start-an-import-job-v2) as shown below. In the body of the request, include the key-value pairs in the table below.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Key\",\n    \"h-1\": \"Description of value\",\n    \"0-0\": \"`volume_id`\\n_required_\",\n    \"0-1\": \"Volume ID from which to import the file. This consists of your username followed by the volume's name, such as `rfranklin/tutorial_volume`.\",\n    \"1-0\": \"`location`\\n_required_\",\n    \"2-0\": \"`destination`\\n_required_\",\n    \"3-0\": \"`project`\\n_required_\",\n    \"4-0\": \"`name`\",\n    \"5-0\": \"`overwrite`\",\n    \"5-1\": \"Specify as `true` to overwrite the file if the file with the same name already exists in the destination.\",\n    \"4-1\": \"The name of the alias to create. This name should be unique to the project. If the name is already in use in the project, you should use the overwrite query parameter in this call to force any file with that name to be deleted before the alias is created.\\n\\nIf name is omitted, the alias name will default to the last segment of the complete location (including the `prefix`) on the volume. Segments are considered to be separated with forward slashes ('/').\",\n    \"3-1\": \"The project in which to create the alias. This consists of your username followed by your project's short name, such as `rfranklin/my-project`.\",\n    \"2-1\": \"This object should describe the destination for the imported file on the CGC.\",\n    \"1-1\": \"Volume-specific location pointing to the file to import. This location should be recognizable to the underlying cloud service as a valid key or path to the file.\\n\\nPlease note that if this volume was configured with a `prefix` parameter when it was created, the `prefix` will be prepended to location before attempting to locate the file on the volume.\"\n  },\n  \"cols\": 2,\n  \"rows\": 6\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"POST /v2/storage/imports HTTP/1.1\\nHost: cgc-api.sbgenomics.com\\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\\ncontent-type: application/json\",\n      \"language\": \"http\",\n      \"name\": \"Start an import job\"\n    }\n  ]\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{ \\n   \\\"source\\\":{ \\n      \\\"volume\\\":\\\"rfranklin/tutorial_volume\\\",\\n      \\\"location\\\":\\\"example_human_Illumina.pe_1.fastq\\\"\\n   },\\n   \\\"destination\\\":{ \\n      \\\"project\\\":\\\"rfranklin/my-project\\\",\\n      \\\"name\\\":\\\"my_imported_example_human_Illumina.pe_1.fastq\\\"\\n   },\\n   \\\"overwrite\\\": true\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Start an import job request body\"\n    }\n  ]\n}\n[/block]\nThe returned response details the status of your import, as shown below.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"href\\\": \\\"https://cgc-api.sbgenomics.com/v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\\\",\\n  \\\"id\\\": \\\"SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\\\",\\n  \\\"state\\\": \\\"PENDING\\\",\\n  \\\"overwrite\\\": true,\\n  \\\"source\\\": {\\n    \\\"volume\\\": \\\"rfranklin/tutorial_volume\\\",\\n    \\\"location\\\": \\\"example_human_Illumina.pe_1.fastq\\\"\\n  },\\n  \\\"destination\\\": {\\n    \\\"project\\\": \\\"rfranklin/my-project\\\",\\n    \\\"name\\\": \\\"my_uploaded_example_human_Illumina.pe_1.fastq\\\"\\n  }\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Response body\"\n    }\n  ]\n}\n[/block]\nLocate the `id` property in the response and copy this value to your clipboard. This `id` is an identifier for the import job, and we will need it in the following step.\n\n<a name=\"check-import\"></a>\n###2b: Check if the import job has completed\n\nTo check if the import job has completed, make the API request to [get details of an import job](doc:get-details-of-an-import-job-v2), as shown below. Simply append the import job `id` obtained in the step above to the path.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"GET /v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN HTTP/1.1\\nHost: cgc-api.sbgenomics.com\\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\\ncontent-type: application/json\",\n      \"language\": \"http\",\n      \"name\": \"Get details of an import job\"\n    }\n  ]\n}\n[/block]\nThe returned response details the `state` of your import. If the state is `COMPLETED`, your import has successfully finished. If the `state` is `PENDING`, wait a few seconds and repeat this step.\n\nYou should now have a freshly-created alias in your project. To verify that a file has been imported, visit this project in your browser and look for a file with the same name as the key of the object in your bucket.","excerpt":"","slug":"google-cloud-storage-tutorial","type":"basic","title":"Google Cloud Storage tutorial"}

Google Cloud Storage tutorial


<a name="overview"></a> ##Overview The Volumes API contains two types of calls: one to connect and manage cloud storage, and the other to **import** data from a connected cloud account. Before you can start working with your cloud storage via the CGC, you need to authorize the CGC to access and query objects on that cloud storage on your behalf. This is done by creating a "volume". A volume enables you to treat the cloud repository associated with it as external storage for the CGC. You can import files from the volume to the CGC to use them as inputs for computation. <a name="procedure"></a> ##Procedure This short tutorial will guide you through setting up a volume for a GCS bucket. You'll register your GCS bucket as a volume and make an object from the GCS bucket available on the CGC. Once a volume has been created, you can issue import operations to make data appear on the CGC. [block:callout] { "type": "info", "body": "In this tutorial we assume you want to connect to an Google Cloud Storage bucket. The procedure will be slightly different for other cloud storage providers, such as an [Amazon S3 bucket](doc:aws-cloud-storage-tutorial). For more information, please refer to our list of [supported cloud storage providers](doc:supported-cloud-storage-providers)." } [/block] <a name="prerequisites"></a> ##Prerequisites To complete this tutorial, you will need: 1. A [Google (GCP)](https://cloud.google.com/) account. 2. One or more buckets on this GCP account via [Google Cloud Storage (GCS)](https://cloud.google.com/storage/). 3. One or more objects (files) in your target bucket. 4. An authentication token for the CGC. Learn more about [getting your authentication token](get-your-authentication-token). <a name="register"></a> ##Step 1: Register a GCS bucket as a volume To set up a volume, you have to first register a GCS bucket as a volume. Volumes mediate access between the CGC and your buckets, which are local units of storage in GCS. You can register a GCS bucket as a volume through the following steps below. <a name="create"></a> ###1a: Create an IAM (Identity and Access Management) user 1. Log into the [Google Cloud Platform console](https://console.cloud.google.com). 2. From the menu on the left select **IAM & Admin** > **Service accounts**. 3. Click **+ Create service account** below the search bar. 4. Fill in account details: * **Service account name** - Descriptive name to label the account. * **Service account ID** - Generated automatically based on the entered service account name. Can be modified if necessary. * **Service account description** - More elaborate description of the account’s purpose. 5. Click **Create**. The **Service account permissions** screen opens. 6. In the **Select a role** dropdown, select **Storage** > **Storage Object Viewer**. 7. Click **Continue**. The final screen of the wizard opens. 8. In the **Create key** section, click **+ Create key**. Key options are displayed on the right. 9. In the **Key type** list select **JSON**. 10. Click **Create**. Your browser will download a JSON file containing the credentials for this user. Keep this file safe. <a name="authorize"></a> ###1b: Authorize this IAM user to access your bucket 1. On the [Google Cloud Platform](https://console.cloud.google.com/) console, click <img src="https://files.readme.io/32ef300-gcp-hamburger.png" height="17px" width="auto" align="inline" style="margin:1px"/> in the top-left corner and navigate to the **Storage** section 2. Select **Storage** > **Browser**. 3. Locate your bucket and click the three vertical dots to the far right of your bucket's name. 4. Click **Edit bucket permissions**. 5. Click **Add members**. 6. In the **New members** field enter the service account client's email. This email is located in the JSON file downloaded in the previous section. 7. From the **Select a role** drop-down menu, select **Storage Legacy** > **Storage Legacy Bucket Reader**. 8. Click **Save**. You have now authorized the newly-created IAM user to access the storage bucket. <a name="register-bucket"></a> ###1c: Register a bucket At this point, you can associate the bucket with your CGC account by registering it as a volume. To register your bucket as a volume, make the API request to [Create a volume](doc:create-a-volume-v2), as shown in the HTTP request below. Be sure to paste in your authentication token for the X-SBG-Auth-Token key. This request also requires a request body. Provide a `name` for your new volume, an optional description, and an object (`service`) containing the information in the table below. Specify the `access_mode` as `RO` for read-only permissions. Be sure to supply a `bucket` name and substitute in your own `credentials`. [block:parameters] { "data": { "h-0": "Key", "h-1": "Description of value", "0-0": "`type`\n_required_", "0-1": "This must be set to `gcs`.", "1-0": "`bucket`\n_required_", "1-1": "The name of your GCS bucket.", "2-0": "`prefix`\n_default: empty string_", "3-0": "`client_email`", "4-0": "`private_key`", "2-1": "If provided, the value of this parameter will be used to modify any object key before an operation is performed on the bucket.\n\nEven though Google Cloud Platform is not truly a folder-based store and allows for almost arbitrarily named keys, the prefix is treated as a folder name. This means that after applying the prefix to the name of the object the resulting key will be normalized to conform to the standard path-based naming schema for files.\n\nFor example, if you set the `prefix` for a volume to `a10`, and import a file with `location` set to `test.fastq` from the volume to the CGC, then the object that will be referred to by the newly-created alias will be `a10/test.fast`.", "3-1": "The client email address for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials.", "4-1": "The private key for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials." }, "cols": 2, "rows": 5 } [/block] [block:code] { "codes": [ { "code": "POST /v2/storage/volumes HTTP/1.1\nHost: cgc-api.sbgenomics.com\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\ncontent-type: application/json", "language": "http", "name": "Create a volume" } ] } [/block] [block:code] { "codes": [ { "code": "{\n \"name\": \"tutorial_volume\",\n \"description\": \"New GCS volume\",\n \"service\": {\n \"type\": \"gcs\",\n \"bucket\": \"test-bucket\",\n \"prefix\": \"\",\n \"credentials\": {\n \"client_email\": \"user@service.iam.gserviceaccount.com\",\n \"private_key\": \"-----BEGIN PRIVATE KEY-----\\nMIIEpQIBAAKCAQEAsXj4E7swo97szcOrAcraSbsGnNuTU1b/4llyspDa0lltZIKL\\nfl5s3QoqbjUWqAZXkJexei85g49ULD8BGKH2r4EF+XyKcpoon4uIFcbmYcmsUXM\\nJ3ujgyL5DbWnQZ6GrqgFNRFVVz/PuvTZOd6KFCrjbbtCxfKoXQrmCwFC/4NlFR3v\\n1kavU81w201Mied3e+pxjfiQKAJOoy5I7kfuH20xfzHXWR2YHdQGbzOUZyPgmzZ6\\nH6Ry39b7bgLVbyk3++e13KrsTEf58rRzUHLzlcUDcGyf8iTO2vA2qzcbrbovwqJr\\n7H4ZfFllDMYQ/ISj4cmi+sz/hR43LUK86emrXwIDAQABAoIBADBr2fvAMbINsZm+\\njjTh/ObrAWXgvvSZIx3F2/Z+cUW9Ioyu1ZJ3/uncMTF6iKD1ggSwbqVQIq7zKaWP\\ndGNZ4sk62PEQSx8924iiNsGaIqyj5FmvuoD3SeiorR0hd+3+a67RpwIQpaE1ht7y\\nmSYh4riX7w9sbU6G44rnQ1azVG1UHvk5ieOD4OPvJopuc6D6ow1oJOnHE0k8v3HY\\n1FpLdWCL6nSERqXOI5w+tllG4NMUmTZ2jhaBSEM4PIJVO+24TM3XFCcvhZ7ipPMF\\nP5B8hV4hDA4Av1Ei7iuRZlJsH4sRrtHJE3/FZLgqHRRvt/7w4c1xnwirNghtTNMb\\nXVoaS/ECgYEA15vL3l22mIoePlcCxIgVCAxhKm6TVQZsAE2EaeVsJKDl0AgCtn/1\\nThMIPPGkO8jmjqHGgA+FhjoUQuCCdIuON00mUpmUxZlwI5+uknuK597/zAjd6W8s\\n7p9apvBUDfod0hwF9Jfw+aUtZm6EAUNR1Odbb+bpXp1luwfcesHe4QcCgYEA0rg8\\nZBBwh2DetU6wWh2JIejBH5SfRUqtEwo5WiEZhrEQLazcpX4w5uvESnT+xd7qx3yC\\n/vyzqmy+YwP92Ql0vZApdQoyKGHVntY/o3HYxZD3x+7BKThUs747WjdSo8SwBkSr\\nxEzLBgTqqcho6UXvYTTEAg11F5yNYzbvVf4vROkCgYEAh6XtTamIB9Bd1rrHcv5q\\nvPWM7DVFXGj96fLbLAS7VRAlhgyEKG2417YBqNYejb6Hz5TYXhll2F0SAkFd0hU7\\nFG/lfHJDt04hz0fXfTFc4yTZqnSpqQPZMQfw8LajK2gA+v/Gf2xYn7fcKGW/h0vj\\nYB9u16hfirdcGZ+Ih3MR1mECgYEAnq1b1KJIirlYm8FYrVOGe4FxRF2/ngdA05Ck\\nZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALJlQZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+CxZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MjZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALSi0sVSXpA=\\n-----END PRIVATE KEY-----\"\n }\n },\n \"access_mode\": \"RO\"\n}", "language": "json", "name": "Create a volume request body" } ] } [/block] You'll see a response providing the details for your newly created volume, as shown below. [block:code] { "codes": [ { "code": "{\n \"href\": \"https://cgc-api.sbgenomics.com/v2/storage/volumes/rfranklin/tutorial_volume\",\n \"id\": \"rfranklin/tutorial_volume\",\n \"name\": \"tutorial_volume\",\n \"description\": \"New GCS volume\",\n \"access_mode\": \"RO\",\n \"service\": {\n \"type\": \"gcs\",\n \"bucket\": \"test-bucket\",\n \"prefix\": \"\",\n \"credentials\": {\n \"client_email\": \"user@service.iam.gserviceaccount.com\"\n }\n },\n \"created_on\": \"2019-05-26T16:44:20Z\",\n \"modified_on\": \"2019-05-26T16:44:20Z\",\n \"active\": true\n}", "language": "json", "name": "Response body" } ] } [/block] <a name="import"></a> ##Step 2: Make an object from the bucket available on the CGC Now that we have a volume, we can make data objects from the bucket associated with the volume available as "aliases" on the CGC. Aliases point to files stored on your cloud storage bucket and can be copied, executed, and organized like normal files on the CGC. We call this operation "importing". Learn more about working with [aliases](doc:aliases). To import a data object from your volume as an alias on the CGC, follow the steps below. <a name="launch-import"></a> ###2a: Launch an import job To import a file, make the API request to [start an import job](start-an-import-job-v2) as shown below. In the body of the request, include the key-value pairs in the table below. [block:parameters] { "data": { "h-0": "Key", "h-1": "Description of value", "0-0": "`volume_id`\n_required_", "0-1": "Volume ID from which to import the file. This consists of your username followed by the volume's name, such as `rfranklin/tutorial_volume`.", "1-0": "`location`\n_required_", "2-0": "`destination`\n_required_", "3-0": "`project`\n_required_", "4-0": "`name`", "5-0": "`overwrite`", "5-1": "Specify as `true` to overwrite the file if the file with the same name already exists in the destination.", "4-1": "The name of the alias to create. This name should be unique to the project. If the name is already in use in the project, you should use the overwrite query parameter in this call to force any file with that name to be deleted before the alias is created.\n\nIf name is omitted, the alias name will default to the last segment of the complete location (including the `prefix`) on the volume. Segments are considered to be separated with forward slashes ('/').", "3-1": "The project in which to create the alias. This consists of your username followed by your project's short name, such as `rfranklin/my-project`.", "2-1": "This object should describe the destination for the imported file on the CGC.", "1-1": "Volume-specific location pointing to the file to import. This location should be recognizable to the underlying cloud service as a valid key or path to the file.\n\nPlease note that if this volume was configured with a `prefix` parameter when it was created, the `prefix` will be prepended to location before attempting to locate the file on the volume." }, "cols": 2, "rows": 6 } [/block] [block:code] { "codes": [ { "code": "POST /v2/storage/imports HTTP/1.1\nHost: cgc-api.sbgenomics.com\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\ncontent-type: application/json", "language": "http", "name": "Start an import job" } ] } [/block] [block:code] { "codes": [ { "code": "{ \n \"source\":{ \n \"volume\":\"rfranklin/tutorial_volume\",\n \"location\":\"example_human_Illumina.pe_1.fastq\"\n },\n \"destination\":{ \n \"project\":\"rfranklin/my-project\",\n \"name\":\"my_imported_example_human_Illumina.pe_1.fastq\"\n },\n \"overwrite\": true\n}", "language": "json", "name": "Start an import job request body" } ] } [/block] The returned response details the status of your import, as shown below. [block:code] { "codes": [ { "code": "{\n \"href\": \"https://cgc-api.sbgenomics.com/v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\",\n \"id\": \"SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\",\n \"state\": \"PENDING\",\n \"overwrite\": true,\n \"source\": {\n \"volume\": \"rfranklin/tutorial_volume\",\n \"location\": \"example_human_Illumina.pe_1.fastq\"\n },\n \"destination\": {\n \"project\": \"rfranklin/my-project\",\n \"name\": \"my_uploaded_example_human_Illumina.pe_1.fastq\"\n }\n}", "language": "json", "name": "Response body" } ] } [/block] Locate the `id` property in the response and copy this value to your clipboard. This `id` is an identifier for the import job, and we will need it in the following step. <a name="check-import"></a> ###2b: Check if the import job has completed To check if the import job has completed, make the API request to [get details of an import job](doc:get-details-of-an-import-job-v2), as shown below. Simply append the import job `id` obtained in the step above to the path. [block:code] { "codes": [ { "code": "GET /v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN HTTP/1.1\nHost: cgc-api.sbgenomics.com\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\ncontent-type: application/json", "language": "http", "name": "Get details of an import job" } ] } [/block] The returned response details the `state` of your import. If the state is `COMPLETED`, your import has successfully finished. If the `state` is `PENDING`, wait a few seconds and repeat this step. You should now have a freshly-created alias in your project. To verify that a file has been imported, visit this project in your browser and look for a file with the same name as the key of the object in your bucket.