{"__v":0,"_id":"57852b1ba795200e00f14939","category":{"project":"55faf11ba62ba1170021a9a7","version":"55faf11ba62ba1170021a9aa","_id":"5785191af3a10c0e009b75b0","__v":0,"sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-07-12T16:21:46.337Z","from_sync":false,"order":24,"slug":"connect-cloud-storage","title":"CONNECT CLOUD STORAGE"},"parentDoc":null,"project":"55faf11ba62ba1170021a9a7","user":"5613e4f8fdd08f2b00437620","version":{"__v":37,"_id":"55faf11ba62ba1170021a9aa","project":"55faf11ba62ba1170021a9a7","createdAt":"2015-09-17T16:58:03.490Z","releaseDate":"2015-09-17T16:58:03.490Z","categories":["55faf11ca62ba1170021a9ab","55faf8f4d0e22017005b8272","55faf91aa62ba1170021a9b5","55faf929a8a7770d00c2c0bd","55faf932a8a7770d00c2c0bf","55faf94b17b9d00d00969f47","55faf958d0e22017005b8274","55faf95fa8a7770d00c2c0c0","55faf96917b9d00d00969f48","55faf970a8a7770d00c2c0c1","55faf98c825d5f19001fa3a6","55faf99aa62ba1170021a9b8","55faf99fa62ba1170021a9b9","55faf9aa17b9d00d00969f49","55faf9b6a8a7770d00c2c0c3","55faf9bda62ba1170021a9ba","5604570090ee490d00440551","5637e8b2fbe1c50d008cb078","5649bb624fa1460d00780add","5671974d1b6b730d008b4823","5671979d60c8e70d006c9760","568e8eef70ca1f0d0035808e","56d0a2081ecc471500f1795e","56d4a0adde40c70b00823ea3","56d96b03dd90610b00270849","56fbb83d8f21c817002af880","573c811bee2b3b2200422be1","576bc92afb62dd20001cda85","5771811e27a5c20e00030dcd","5785191af3a10c0e009b75b0","57bdf84d5d48411900cd8dc0","57ff5c5dc135231700aed806","5804caf792398f0f00e77521","58458b4fba4f1c0f009692bb","586d3c287c6b5b2300c05055","58ef66d88646742f009a0216","58f5d52d7891630f00fe4e77"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"updates":[],"next":{"pages":[],"description":""},"createdAt":"2016-07-12T17:38:35.002Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":7,"body":"[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"On this page:\",\n  \"body\": \"* [Overview](#overview)\\n* [Procedure](#procedure)\\n* [Prerequisites](#prerequisites)\\n* [Step 1: Register an S3 bucket as a volume](#register)\\n * [1a: Create an IAM (Identity and Access Management) user](#create)\\n * [1b: Authorize this IAM user to access your bucket](#authorize)\\n * [1c: Register a bucket](#register-bucket)\\n* [Step 2: Make an object from the bucket available on the CGC](#import)\\n * [2a: Launch an import job](#launch-import)\\n * [2b: Check if the import job has completed](#check-import)\"\n}\n[/block]\n<a name=\"overview\"></a>\n##Overview\n\nThe Volumes API contains two types of calls: one to connect and manage cloud storage, and the other to import and export data to and from a connected cloud account.\n\nBefore you can start working with your cloud storage via the CGC you need to authorize the CGC to access and query objects on that cloud storage on your behalf. This is done by creating a \"volume\". A volume enables you to treat the cloud repository associated with it as external storage for the CGC. You can 'import' files from the volume to the CGC to use them as inputs for computation. Similarly, you can write files from the CGC to your cloud storage by 'exporting' them to your volume. Learn more about [working with volumes](volumes).\n\nThe CGC uses Amazon Web Services as a cloud infrastructure provider. This affects the cloud storage you can access and associate with your CGC account. For instance, you have full read-write access to your data stored in Amazon Web Services' S3 and read-only access to data stored in Google Cloud Storage.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"procedure\"></a>\n##Procedure\n\nThis short tutorial will guide you through setting up a volume for a GCS bucket. You'll register your GCS bucket as a volume and make an object from the GCS bucket available on the CGC.\n\nNote that as the CGC runs on AWS, you cannot export a file from the CGC to a GCS bucket, as this involves writing to that GCS bucket.\n\nOnce a volume is created, you can issue import and export operations to make data appear on the CGC or to move your CGC files to the underlying cloud storage provider.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"In this tutorial we assume you want to connect to an Google Cloud Storage bucket. The procedure will be slightly different for other cloud storage providers, such as a [Amazon S3 bucket](aws-cloud-storage-tutorial). For more information, please refer to our list of [supported cloud storage providers](supported-cloud-storage-providers).\"\n}\n[/block]\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"prerequisites\"></a>\n##Prerequisites\n\nTo complete this tutorial, you will need:\n1.An [Google (GCP)](https://cloud.google.com/) account.\n2. One or more buckets on this GCP account via [Google Cloud Storage (GCS)](https://cloud.google.com/storage/).\n3. One or more objects (files) in your target bucket.\n4. An authentication token for the CGC. Learn more about [getting your authentication token](get-your-authentication-token). Note that you cannot write to a GCS bucket from the CGC.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"register\"></a>\n##Step 1: Register an GCS bucket as a volume\n\nTo set up a volume, you have to first register a GCS bucket as a volume. Volumes mediate access between the CGC and your buckets, which are local units of storage in GCS.\n\nYou can register a GCS bucket as a volume through the following steps below.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"create\"></a>\n###1a: Create an IAM (Identity and Access Management) user\n\n1. Log into the [Google Cloud Platform console](https://console.cloud.google.com).\n2. Search for **IAM** in the top navigation bar. Select **IAM & Admin** from the drop-down menu.\n3. Click **Service accounts** in the left sidebar.\n4. Click **+ Create service account** below the search bar.\n5. Fill in account details, as shown below, making sure that:\n  * **Furnish a new private key** is checked\n  * A **Key type** of **JSON** is selected\n  * **Enable Google Apps Domain-wide Delegation** is unchecked \n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/d17cdde-Screen_Shot_2016-10-12_at_2.41.31_PM.png\",\n        \"Screen Shot 2016-10-12 at 2.41.31 PM.png\",\n        490,\n        442,\n        \"#d9dada\"\n      ]\n    }\n  ]\n}\n[/block]\n6. Click **Create** to finish.\nYour browser will download a JSON file containing the credentials for this user. Keep this file safe.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"authorize\"></a>\n###1b: Authorize this IAM user to access your bucket\n\n1. Using the dropdown menu next to Google Cloud Platform to navigate to **Storage**.\n2. Locate your bucket and click the three vertical dots to the far left of your bucket's name.\n3. Click **Edit bucket permissions**.\n4. Click **+ Add item**.\n5. From the drop-down menu, select **User** and enter the service account client's email in the next box. This email is located in the JSON downloaded in the previous section.\n6. From the drop-down menu, select **Reader** for Read Only (RO) access or **Writer** for Read-write (RW) access.\n7. Click **Save**.\n8. Select the three vertical dots to the far left of your bucket's name once more.\n9. Click **Edit object default permissions**.\n10. Click **+Add item**.\n11. From the drop-down menu, select **User** and enter the service account client's email in the next box.\n12. From the next drop-down menu, select **Reader**.\n13. Click **Save**.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"register-bucket\"></a>\n###1c: Register a bucket\n\nAt this point, you can associate the bucket with your CGC account by registering it as a volume.\n\nTo register your bucket as a volume, make the API request to [Create a volume](create-a-volume-v2), as shown in the HTTP request below. Be sure to paste in your authentication token for the X-SBG-Auth-Token key.\n\nThis request also requires a request body. Provide a `name` for your new volume, an optional description, and an object (`service`) containing the information in the table below. Specify the `access_mode` as `RO` for read-only permissions. Be sure to supply a `bucket` name and substitute in your own `credentials`.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Key\",\n    \"h-1\": \"Description of value\",\n    \"0-0\": \"`type`\\n_required_\",\n    \"0-1\": \"This must be set to `gcs`.\",\n    \"1-0\": \"`bucket`\\n_required_\",\n    \"1-1\": \"The name of your GCS bucket.\",\n    \"2-0\": \"`prefix`\\n_default: empty string_\",\n    \"3-0\": \"`client_email`\",\n    \"4-0\": \"`private_key`\",\n    \"2-1\": \"If provided, the value of this parameter will be used to modify any object key before an operation is performed on the bucket.\\n\\nEven though Google Cloud Platform is not truly a folder-based store and allows for almost arbitrarily named keys, the prefix is treated as a folder name. This means that after applying the prefix to the name of the object the resulting key will be normalized to conform to the standard path-based naming schema for files.\\n\\nFor example, if you set the `prefix` for a volume to `a10`, and import a file with `location` set to `test.fastq` from the volume to the CGC, then the object that will be referred to by the newly-created alias will be `a10/test.fast`\\\".\",\n    \"3-1\": \"The client email address for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials.\",\n    \"4-1\": \"The private key for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials.\"\n  },\n  \"cols\": 2,\n  \"rows\": 5\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"POST /v2/storage/volumes HTTP/1.1\\nHost: cgc-api.sbgenomics.com\\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\\ncontent-type: application/json\",\n      \"language\": \"http\",\n      \"name\": \"Create a volume\"\n    }\n  ]\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"name\\\": \\\"tutorial_volume\\\",\\n  \\\"description\\\": \\\"New google volume\\\",\\n  \\\"service\\\": {\\n    \\\"type\\\": \\\"gcs\\\",\\n    \\\"bucket\\\": \\\"test-bucket\\\",\\n    \\\"prefix\\\": \\\"\\\",\\n    \\\"credentials\\\": {\\n      \\\"client_email\\\": \\\"user:::at:::service.iam.gserviceaccount.com\\\",\\n      \\\"private_key\\\": \\\"-----BEGIN RSA PRIVATE KEY-----\\\\nMIIEpQIBAAKCAQEAsXj4E7swo97szcOrAcraSbsGnNuTU1b/4llyspDa0lltZIKL\\\\nfl5s3QoqbjUWqAZXkJexei85g49ULD8BGKH2r4EF+XyKcpoon4uIFcbmYcmsUXM\\\\nJ3ujgyL5DbWnQZ6GrqgFNRFVVz/PuvTZOd6KFCrjbbtCxfKoXQrmCwFC/4NlFR3v\\\\n1kavU81w201Mied3e+pxjfiQKAJOoy5I7kfuH20xfzHXWR2YHdQGbzOUZyPgmzZ6\\\\nH6Ry39b7bgLVbyk3++e13KrsTEf58rRzUHLzlcUDcGyf8iTO2vA2qzcbrbovwqJr\\\\n7H4ZfFllDMYQ/ISj4cmi+sz/hR43LUK86emrXwIDAQABAoIBADBr2fvAMbINsZm+\\\\njjTh/ObrAWXgvvSZIx3F2/Z+cUW9Ioyu1ZJ3/uncMTF6iKD1ggSwbqVQIq7zKaWP\\\\ndGNZ4sk62PEQSx8924iiNsGaIqyj5FmvuoD3SeiorR0hd+3+a67RpwIQpaE1ht7y\\\\nmSYh4riX7w9sbU6G44rnQ1azVG1UHvk5ieOD4OPvJopuc6D6ow1oJOnHE0k8v3HY\\\\n1FpLdWCL6nSERqXOI5w+tllG4NMUmTZ2jhaBSEM4PIJVO+24TM3XFCcvhZ7ipPMF\\\\nP5B8hV4hDA4Av1Ei7iuRZlJsH4sRrtHJE3/FZLgqHRRvt/7w4c1xnwirNghtTNMb\\\\nXVoaS/ECgYEA15vL3l22mIoePlcCxIgVCAxhKm6TVQZsAE2EaeVsJKDl0AgCtn/1\\\\nThMIPPGkO8jmjqHGgA+FhjoUQuCCdIuON00mUpmUxZlwI5+uknuK597/zAjd6W8s\\\\n7p9apvBUDfod0hwF9Jfw+aUtZm6EAUNR1Odbb+bpXp1luwfcesHe4QcCgYEA0rg8\\\\nZBBwh2DetU6wWh2JIejBH5SfRUqtEwo5WiEZhrEQLazcpX4w5uvESnT+xd7qx3yC\\\\n/vyzqmy+YwP92Ql0vZApdQoyKGHVntY/o3HYxZD3x+7BKThUs747WjdSo8SwBkSr\\\\nxEzLBgTqqcho6UXvYTTEAg11F5yNYzbvVf4vROkCgYEAh6XtTamIB9Bd1rrHcv5q\\\\nvPWM7DVFXGj96fLbLAS7VRAlhgyEKG2417YBqNYejb6Hz5TYXhll2F0SAkFd0hU7\\\\nFG/lfHJDt04hz0fXfTFc4yTZqnSpqQPZMQfw8LajK2gA+v/Gf2xYn7fcKGW/h0vj\\\\nYB9u16hfirdcGZ+Ih3MR1mECgYEAnq1b1KJIirlYm8FYrVOGe4FxRF2/ngdA05Ck\\\\nZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALJlQZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+CxZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MjZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALSi0sVSXpA=\\\\n-----END RSA PRIVATE KEY-----\\\"\\n    }\\n  },\\n  \\\"access_mode\\\": \\\"RO\\\"\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Create a volume request body\"\n    }\n  ]\n}\n[/block]\nYou'll see a response providing the details for your newly created volume, as shown below.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"href\\\": \\\"https://cgc-api.sbgenomics.com/v2/storage/volumes/rfranklin/tutorial_volume\\\",\\n  \\\"id\\\": \\\"rfranklin/tutorial_volume\\\",\\n  \\\"name\\\": \\\"tutorial_volume\\\",\\n  \\\"access_mode\\\": \\\"RO\\\",\\n  \\\"service\\\": {\\n    \\\"type\\\": \\\"gcs\\\",\\n    \\\"bucket\\\": \\\"rfranklin-test-bucket\\\",\\n    \\\"prefix\\\": \\\"\\\",\\n    \\\"endpoint\\\": \\\"https://www.googleapis.com\\\",\\n    \\\"credentials\\\": {\\n      \\\"client_email\\\": \\\"user@service.iam.gserviceaccount.com\\\",\\n      \\\"private_key\\\": \\\"-----BEGIN RSA PRIVATE KEY-----\\\\nMIIEpQIBAAKCAQEAsXj4E7swo97szcOrAcraSbsGnNuTU1b/4llyspDa0lltZIKL\\\\nfl5s3QoqbjUWqAZXkJexei85g49ULD8BGKH2r4EF+XyKcpoon4uIFcbmYcmsUXM\\\\nJ3ujgyL5DbWnQZ6GrqgFNRFVVz/PuvTZOd6KFCrjbbtCxfKoXQrmCwFC/4NlFR3v\\\\n1kavU81w201Mied3e+pxjfiQKAJOoy5I7kfuH20xfzHXWR2YHdQGbzOUZyPgmzZ6\\\\nH6Ry39b7bgLVbyk3++e13KrsTEf58rRzUHLzlcUDcGyf8iTO2vA2qzcbrbovwqJr\\\\n7H4ZfFllDMYQ/ISj4cmi+sz/hR43LUK86emrXwIDAQABAoIBADBr2fvAMbINsZm+\\\\njjTh/ObrAWXgvvSZIx3F2/Z+cUW9Ioyu1ZJ3/uncMTF6iKD1ggSwbqVQIq7zKaWP\\\\ndGNZ4sk62PEQSx8924iiNsGaIqyj5FmvuoD3SeiorR0hd+3+a67RpwIQpaE1ht7y\\\\nmSYh4riX7w9sbU6G44rnQ1azVG1UHvk5ieOD4OPvJopuc6D6ow1oJOnHE0k8v3HY\\\\n1FpLdWCL6nSERqXOI5w+tllG4NMUmTZ2jhaBSEM4PIJVO+24TM3XFCcvhZ7ipPMF\\\\nP5B8hV4hDA4Av1Ei7iuRZlJsH4sRrtHJE3/FZLgqHRRvt/7w4c1xnwirNghtTNMb\\\\nXVoaS/ECgYEA15vL3l22mIoePlcCxIgVCAxhKm6TVQZsAE2EaeVsJKDl0AgCtn/1\\\\nThMIPPGkO8jmjqHGgA+FhjoUQuCCdIuON00mUpmUxZlwI5+uknuK597/zAjd6W8s\\\\n7p9apvBUDfod0hwF9Jfw+aUtZm6EAUNR1Odbb+bpXp1luwfcesHe4QcCgYEA0rg8\\\\nZBBwh2DetU6wWh2JIejBH5SfRUqtEwo5WiEZhrEQLazcpX4w5uvESnT+xd7qx3yC\\\\n/vyzqmy+YwP92Ql0vZApdQoyKGHVntY/o3HYxZD3x+7BKThUs747WjdSo8SwBkSr\\\\nxEzLBgTqqcho6UXvYTTEAg11F5yNYzbvVf4vROkCgYEAh6XtTamIB9Bd1rrHcv5q\\\\nvPWM7DVFXGj96fLbLAS7VRAlhgyEKG2417YBqNYejb6Hz5TYXhll2F0SAkFd0hU7\\\\nFG/lfHJDt04hz0fXfTFc4yTZqnSpqQPZMQfw8LajK2gA+v/Gf2xYn7fcKGW/h0vj\\\\nYB9u16hfirdcGZ+Ih3MR1mECgYEAnq1b1KJIirlYm8FYrVOGe4FxRF2/ngdA05Ck\\\\nZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALJlQZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+CxZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MjZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALSi0sVSXpA=\\\\n-----END RSA PRIVATE KEY-----\\\"\\n    }\\n  },\\n  \\\"created_on\\\": \\\"2016-06-26T16:44:20Z\\\",\\n  \\\"modified_on\\\": \\\"2016-06-26T16:44:20Z\\\",\\n  \\\"active\\\": true\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Response body\"\n    }\n  ]\n}\n[/block]\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"import\"></a>\n##Step 2: Make an object from the bucket available on the CGC\n\nNow that we have a volume, we can make data objects from the bucket associated with the volume available as \"aliases\" on the CGC. Aliases point to files stored on your cloud storage bucket and can be copied, executed, and organized like normal files on the CGC. We call this operation \"importing\". Learn more about working with [aliases](doc:aliases).\n\nTo import a data object from your volume as an alias on the CGC, follow the steps below.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"launch-import\"></a>\n###2a: Launch an import job\n\nTo import a file, make the API request to [start an import job](start-an-import-job-v2) as shown below. In the body of the request, include the key-value pairs in the table below.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Key\",\n    \"h-1\": \"Description of value\",\n    \"0-0\": \"`volume_id`\\n_required_\",\n    \"0-1\": \"Volume ID from which to import the file. This consists of your username followed by the volume's name, such as `rfranklin/tutorial_volume`.\",\n    \"1-0\": \"`location`\\n_required_\",\n    \"2-0\": \"`destination`\\n_required_\",\n    \"3-0\": \"`project`\\n_required_\",\n    \"4-0\": \"`name`\",\n    \"5-0\": \"`overwrite`\",\n    \"5-1\": \"Specify as `true` to overwrite the file if the file with the same name already exists in the destination.\",\n    \"4-1\": \"The name of the alias to create. This name should be unique to the project. If the name is already in use in the project, you should use the overwrite query parameter in this call to force any file with that name to be deleted before the alias is created.\\n\\nIf name is omitted, the alias name will default to the last segment of the complete location (including the `prefix`) on the volume. Segments are considered to be separated with forward slashes ('/').\",\n    \"3-1\": \"The project in which to create the alias. This consists of your username followed by your project's short name, such as `rfranklin/my-project`.\",\n    \"2-1\": \"This object should describe the CGC destination for the imported file.\",\n    \"1-1\": \"Volume-specific location pointing to the file to import. This location should be recognizable to the underlying cloud service as a valid key or path to the file.\\n\\nPlease note that if this volume was configured with a `prefix` parameter when it was created, the `prefix` will be prepended to location before attempting to locate the file on the volume.\"\n  },\n  \"cols\": 2,\n  \"rows\": 6\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"POST /v2/storage/imports HTTP/1.1\\nHost: cgc-api.sbgenomics.com\\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\\ncontent-type: application/json\",\n      \"language\": \"http\",\n      \"name\": \"Start an import job\"\n    }\n  ]\n}\n[/block]\n\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{ \\n   \\\"source\\\":{ \\n      \\\"volume\\\":\\\"rfranklin/tutorial_volume\\\",\\n      \\\"location\\\":\\\"example_human_Illumina.pe_1.fastq\\\"\\n   },\\n   \\\"destination\\\":{ \\n      \\\"project\\\":\\\"rfranklin/my-project\\\",\\n      \\\"name\\\":\\\"my_imported_example_human_Illumina.pe_1.fastq\\\"\\n   },\\n   \\\"overwrite\\\": true\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Start an import job request body\"\n    }\n  ]\n}\n[/block]\nThe returned response details the status of your import, as shown below.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"{\\n  \\\"href\\\": \\\"https://cgc-api.sbgenomics.com/v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\\\",\\n  \\\"id\\\": \\\"SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\\\",\\n  \\\"state\\\": \\\"PENDING\\\",\\n  \\\"overwrite\\\": true,\\n  \\\"source\\\": {\\n    \\\"volume\\\": \\\"rfranklin/tutorial_volume\\\",\\n    \\\"location\\\": \\\"example_human_Illumina.pe_1.fastq\\\"\\n  },\\n  \\\"destination\\\": {\\n    \\\"project\\\": \\\"rfranklin/my-project\\\",\\n    \\\"name\\\": \\\"my_uploaded_example_human_Illumina.pe_1.fastq\\\"\\n  }\\n}\",\n      \"language\": \"json\",\n      \"name\": \"Response body\"\n    }\n  ]\n}\n[/block]\nLocate the `id` property in the response and copy this value to your clipboard. This `id` is an identifier for the import job, and we will need it in the following step.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n<a name=\"check-import\"></a>\n###2b: Check if the import job has completed\n\nTo check if the import job has completed, make the API request to [get details of an import job](get-details-of-an-import-job-v2), as shown below. Simply append the import job `id` obtained in the step above to the path.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"GET /v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN HTTP/1.1\\nHost: cgc-api.sbgenomics.com\\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\\ncontent-type: application/json\",\n      \"language\": \"http\",\n      \"name\": \"Get details of an import job\"\n    }\n  ]\n}\n[/block]\nThe returned response details the `state` of your import. If the state is `COMPLETED`, your import has successfully finished. If the `state` is `PENDING`, wait a few seconds and repeat this step.\n\nYou should now have a freshly-created alias in your project. To verify that a file has been imported, visit this project in your browser and look for a file with the same name as the key of the object in your bucket.\n\nCongratulations! You've now registered an GCS bucket as a volume and imported a file from the volume to the CGC. Learn more about [connecting your cloud storage from our Knowledge Center](connecting-cloud-storage-overview).\n\n<div align=\"right\"><a href=\"#top\">top</a></div>","excerpt":"","slug":"google-cloud-storage-tutorial","type":"basic","title":"Google Cloud Storage tutorial"}

Google Cloud Storage tutorial


[block:callout] { "type": "warning", "title": "On this page:", "body": "* [Overview](#overview)\n* [Procedure](#procedure)\n* [Prerequisites](#prerequisites)\n* [Step 1: Register an S3 bucket as a volume](#register)\n * [1a: Create an IAM (Identity and Access Management) user](#create)\n * [1b: Authorize this IAM user to access your bucket](#authorize)\n * [1c: Register a bucket](#register-bucket)\n* [Step 2: Make an object from the bucket available on the CGC](#import)\n * [2a: Launch an import job](#launch-import)\n * [2b: Check if the import job has completed](#check-import)" } [/block] <a name="overview"></a> ##Overview The Volumes API contains two types of calls: one to connect and manage cloud storage, and the other to import and export data to and from a connected cloud account. Before you can start working with your cloud storage via the CGC you need to authorize the CGC to access and query objects on that cloud storage on your behalf. This is done by creating a "volume". A volume enables you to treat the cloud repository associated with it as external storage for the CGC. You can 'import' files from the volume to the CGC to use them as inputs for computation. Similarly, you can write files from the CGC to your cloud storage by 'exporting' them to your volume. Learn more about [working with volumes](volumes). The CGC uses Amazon Web Services as a cloud infrastructure provider. This affects the cloud storage you can access and associate with your CGC account. For instance, you have full read-write access to your data stored in Amazon Web Services' S3 and read-only access to data stored in Google Cloud Storage. <div align="right"><a href="#top">top</a></div> <a name="procedure"></a> ##Procedure This short tutorial will guide you through setting up a volume for a GCS bucket. You'll register your GCS bucket as a volume and make an object from the GCS bucket available on the CGC. Note that as the CGC runs on AWS, you cannot export a file from the CGC to a GCS bucket, as this involves writing to that GCS bucket. Once a volume is created, you can issue import and export operations to make data appear on the CGC or to move your CGC files to the underlying cloud storage provider. [block:callout] { "type": "info", "body": "In this tutorial we assume you want to connect to an Google Cloud Storage bucket. The procedure will be slightly different for other cloud storage providers, such as a [Amazon S3 bucket](aws-cloud-storage-tutorial). For more information, please refer to our list of [supported cloud storage providers](supported-cloud-storage-providers)." } [/block] <div align="right"><a href="#top">top</a></div> <a name="prerequisites"></a> ##Prerequisites To complete this tutorial, you will need: 1.An [Google (GCP)](https://cloud.google.com/) account. 2. One or more buckets on this GCP account via [Google Cloud Storage (GCS)](https://cloud.google.com/storage/). 3. One or more objects (files) in your target bucket. 4. An authentication token for the CGC. Learn more about [getting your authentication token](get-your-authentication-token). Note that you cannot write to a GCS bucket from the CGC. <div align="right"><a href="#top">top</a></div> <a name="register"></a> ##Step 1: Register an GCS bucket as a volume To set up a volume, you have to first register a GCS bucket as a volume. Volumes mediate access between the CGC and your buckets, which are local units of storage in GCS. You can register a GCS bucket as a volume through the following steps below. <div align="right"><a href="#top">top</a></div> <a name="create"></a> ###1a: Create an IAM (Identity and Access Management) user 1. Log into the [Google Cloud Platform console](https://console.cloud.google.com). 2. Search for **IAM** in the top navigation bar. Select **IAM & Admin** from the drop-down menu. 3. Click **Service accounts** in the left sidebar. 4. Click **+ Create service account** below the search bar. 5. Fill in account details, as shown below, making sure that: * **Furnish a new private key** is checked * A **Key type** of **JSON** is selected * **Enable Google Apps Domain-wide Delegation** is unchecked [block:image] { "images": [ { "image": [ "https://files.readme.io/d17cdde-Screen_Shot_2016-10-12_at_2.41.31_PM.png", "Screen Shot 2016-10-12 at 2.41.31 PM.png", 490, 442, "#d9dada" ] } ] } [/block] 6. Click **Create** to finish. Your browser will download a JSON file containing the credentials for this user. Keep this file safe. <div align="right"><a href="#top">top</a></div> <a name="authorize"></a> ###1b: Authorize this IAM user to access your bucket 1. Using the dropdown menu next to Google Cloud Platform to navigate to **Storage**. 2. Locate your bucket and click the three vertical dots to the far left of your bucket's name. 3. Click **Edit bucket permissions**. 4. Click **+ Add item**. 5. From the drop-down menu, select **User** and enter the service account client's email in the next box. This email is located in the JSON downloaded in the previous section. 6. From the drop-down menu, select **Reader** for Read Only (RO) access or **Writer** for Read-write (RW) access. 7. Click **Save**. 8. Select the three vertical dots to the far left of your bucket's name once more. 9. Click **Edit object default permissions**. 10. Click **+Add item**. 11. From the drop-down menu, select **User** and enter the service account client's email in the next box. 12. From the next drop-down menu, select **Reader**. 13. Click **Save**. <div align="right"><a href="#top">top</a></div> <a name="register-bucket"></a> ###1c: Register a bucket At this point, you can associate the bucket with your CGC account by registering it as a volume. To register your bucket as a volume, make the API request to [Create a volume](create-a-volume-v2), as shown in the HTTP request below. Be sure to paste in your authentication token for the X-SBG-Auth-Token key. This request also requires a request body. Provide a `name` for your new volume, an optional description, and an object (`service`) containing the information in the table below. Specify the `access_mode` as `RO` for read-only permissions. Be sure to supply a `bucket` name and substitute in your own `credentials`. [block:parameters] { "data": { "h-0": "Key", "h-1": "Description of value", "0-0": "`type`\n_required_", "0-1": "This must be set to `gcs`.", "1-0": "`bucket`\n_required_", "1-1": "The name of your GCS bucket.", "2-0": "`prefix`\n_default: empty string_", "3-0": "`client_email`", "4-0": "`private_key`", "2-1": "If provided, the value of this parameter will be used to modify any object key before an operation is performed on the bucket.\n\nEven though Google Cloud Platform is not truly a folder-based store and allows for almost arbitrarily named keys, the prefix is treated as a folder name. This means that after applying the prefix to the name of the object the resulting key will be normalized to conform to the standard path-based naming schema for files.\n\nFor example, if you set the `prefix` for a volume to `a10`, and import a file with `location` set to `test.fastq` from the volume to the CGC, then the object that will be referred to by the newly-created alias will be `a10/test.fast`\".", "3-1": "The client email address for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials.", "4-1": "The private key for the Google Cloud service account to use for operations on this bucket. This can be found in the JSON containing your service account credentials." }, "cols": 2, "rows": 5 } [/block] [block:code] { "codes": [ { "code": "POST /v2/storage/volumes HTTP/1.1\nHost: cgc-api.sbgenomics.com\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\ncontent-type: application/json", "language": "http", "name": "Create a volume" } ] } [/block] [block:code] { "codes": [ { "code": "{\n \"name\": \"tutorial_volume\",\n \"description\": \"New google volume\",\n \"service\": {\n \"type\": \"gcs\",\n \"bucket\": \"test-bucket\",\n \"prefix\": \"\",\n \"credentials\": {\n \"client_email\": \"user@service.iam.gserviceaccount.com\",\n \"private_key\": \"-----BEGIN RSA PRIVATE KEY-----\\nMIIEpQIBAAKCAQEAsXj4E7swo97szcOrAcraSbsGnNuTU1b/4llyspDa0lltZIKL\\nfl5s3QoqbjUWqAZXkJexei85g49ULD8BGKH2r4EF+XyKcpoon4uIFcbmYcmsUXM\\nJ3ujgyL5DbWnQZ6GrqgFNRFVVz/PuvTZOd6KFCrjbbtCxfKoXQrmCwFC/4NlFR3v\\n1kavU81w201Mied3e+pxjfiQKAJOoy5I7kfuH20xfzHXWR2YHdQGbzOUZyPgmzZ6\\nH6Ry39b7bgLVbyk3++e13KrsTEf58rRzUHLzlcUDcGyf8iTO2vA2qzcbrbovwqJr\\n7H4ZfFllDMYQ/ISj4cmi+sz/hR43LUK86emrXwIDAQABAoIBADBr2fvAMbINsZm+\\njjTh/ObrAWXgvvSZIx3F2/Z+cUW9Ioyu1ZJ3/uncMTF6iKD1ggSwbqVQIq7zKaWP\\ndGNZ4sk62PEQSx8924iiNsGaIqyj5FmvuoD3SeiorR0hd+3+a67RpwIQpaE1ht7y\\nmSYh4riX7w9sbU6G44rnQ1azVG1UHvk5ieOD4OPvJopuc6D6ow1oJOnHE0k8v3HY\\n1FpLdWCL6nSERqXOI5w+tllG4NMUmTZ2jhaBSEM4PIJVO+24TM3XFCcvhZ7ipPMF\\nP5B8hV4hDA4Av1Ei7iuRZlJsH4sRrtHJE3/FZLgqHRRvt/7w4c1xnwirNghtTNMb\\nXVoaS/ECgYEA15vL3l22mIoePlcCxIgVCAxhKm6TVQZsAE2EaeVsJKDl0AgCtn/1\\nThMIPPGkO8jmjqHGgA+FhjoUQuCCdIuON00mUpmUxZlwI5+uknuK597/zAjd6W8s\\n7p9apvBUDfod0hwF9Jfw+aUtZm6EAUNR1Odbb+bpXp1luwfcesHe4QcCgYEA0rg8\\nZBBwh2DetU6wWh2JIejBH5SfRUqtEwo5WiEZhrEQLazcpX4w5uvESnT+xd7qx3yC\\n/vyzqmy+YwP92Ql0vZApdQoyKGHVntY/o3HYxZD3x+7BKThUs747WjdSo8SwBkSr\\nxEzLBgTqqcho6UXvYTTEAg11F5yNYzbvVf4vROkCgYEAh6XtTamIB9Bd1rrHcv5q\\nvPWM7DVFXGj96fLbLAS7VRAlhgyEKG2417YBqNYejb6Hz5TYXhll2F0SAkFd0hU7\\nFG/lfHJDt04hz0fXfTFc4yTZqnSpqQPZMQfw8LajK2gA+v/Gf2xYn7fcKGW/h0vj\\nYB9u16hfirdcGZ+Ih3MR1mECgYEAnq1b1KJIirlYm8FYrVOGe4FxRF2/ngdA05Ck\\nZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALJlQZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+CxZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MjZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALSi0sVSXpA=\\n-----END RSA PRIVATE KEY-----\"\n }\n },\n \"access_mode\": \"RO\"\n}", "language": "json", "name": "Create a volume request body" } ] } [/block] You'll see a response providing the details for your newly created volume, as shown below. [block:code] { "codes": [ { "code": "{\n \"href\": \"https://cgc-api.sbgenomics.com/v2/storage/volumes/rfranklin/tutorial_volume\",\n \"id\": \"rfranklin/tutorial_volume\",\n \"name\": \"tutorial_volume\",\n \"access_mode\": \"RO\",\n \"service\": {\n \"type\": \"gcs\",\n \"bucket\": \"rfranklin-test-bucket\",\n \"prefix\": \"\",\n \"endpoint\": \"https://www.googleapis.com\",\n \"credentials\": {\n \"client_email\": \"user@service.iam.gserviceaccount.com\",\n \"private_key\": \"-----BEGIN RSA PRIVATE KEY-----\\nMIIEpQIBAAKCAQEAsXj4E7swo97szcOrAcraSbsGnNuTU1b/4llyspDa0lltZIKL\\nfl5s3QoqbjUWqAZXkJexei85g49ULD8BGKH2r4EF+XyKcpoon4uIFcbmYcmsUXM\\nJ3ujgyL5DbWnQZ6GrqgFNRFVVz/PuvTZOd6KFCrjbbtCxfKoXQrmCwFC/4NlFR3v\\n1kavU81w201Mied3e+pxjfiQKAJOoy5I7kfuH20xfzHXWR2YHdQGbzOUZyPgmzZ6\\nH6Ry39b7bgLVbyk3++e13KrsTEf58rRzUHLzlcUDcGyf8iTO2vA2qzcbrbovwqJr\\n7H4ZfFllDMYQ/ISj4cmi+sz/hR43LUK86emrXwIDAQABAoIBADBr2fvAMbINsZm+\\njjTh/ObrAWXgvvSZIx3F2/Z+cUW9Ioyu1ZJ3/uncMTF6iKD1ggSwbqVQIq7zKaWP\\ndGNZ4sk62PEQSx8924iiNsGaIqyj5FmvuoD3SeiorR0hd+3+a67RpwIQpaE1ht7y\\nmSYh4riX7w9sbU6G44rnQ1azVG1UHvk5ieOD4OPvJopuc6D6ow1oJOnHE0k8v3HY\\n1FpLdWCL6nSERqXOI5w+tllG4NMUmTZ2jhaBSEM4PIJVO+24TM3XFCcvhZ7ipPMF\\nP5B8hV4hDA4Av1Ei7iuRZlJsH4sRrtHJE3/FZLgqHRRvt/7w4c1xnwirNghtTNMb\\nXVoaS/ECgYEA15vL3l22mIoePlcCxIgVCAxhKm6TVQZsAE2EaeVsJKDl0AgCtn/1\\nThMIPPGkO8jmjqHGgA+FhjoUQuCCdIuON00mUpmUxZlwI5+uknuK597/zAjd6W8s\\n7p9apvBUDfod0hwF9Jfw+aUtZm6EAUNR1Odbb+bpXp1luwfcesHe4QcCgYEA0rg8\\nZBBwh2DetU6wWh2JIejBH5SfRUqtEwo5WiEZhrEQLazcpX4w5uvESnT+xd7qx3yC\\n/vyzqmy+YwP92Ql0vZApdQoyKGHVntY/o3HYxZD3x+7BKThUs747WjdSo8SwBkSr\\nxEzLBgTqqcho6UXvYTTEAg11F5yNYzbvVf4vROkCgYEAh6XtTamIB9Bd1rrHcv5q\\nvPWM7DVFXGj96fLbLAS7VRAlhgyEKG2417YBqNYejb6Hz5TYXhll2F0SAkFd0hU7\\nFG/lfHJDt04hz0fXfTFc4yTZqnSpqQPZMQfw8LajK2gA+v/Gf2xYn7fcKGW/h0vj\\nYB9u16hfirdcGZ+Ih3MR1mECgYEAnq1b1KJIirlYm8FYrVOGe4FxRF2/ngdA05Ck\\nZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALJlQZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+CxZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvAL+MjZYl9Vl8pZqvAL+MZ4hpyYvs9CzX1KClL38XdaZ2ftKJB2tjzDZYl9Vl8pZqvALSi0sVSXpA=\\n-----END RSA PRIVATE KEY-----\"\n }\n },\n \"created_on\": \"2016-06-26T16:44:20Z\",\n \"modified_on\": \"2016-06-26T16:44:20Z\",\n \"active\": true\n}", "language": "json", "name": "Response body" } ] } [/block] <div align="right"><a href="#top">top</a></div> <a name="import"></a> ##Step 2: Make an object from the bucket available on the CGC Now that we have a volume, we can make data objects from the bucket associated with the volume available as "aliases" on the CGC. Aliases point to files stored on your cloud storage bucket and can be copied, executed, and organized like normal files on the CGC. We call this operation "importing". Learn more about working with [aliases](doc:aliases). To import a data object from your volume as an alias on the CGC, follow the steps below. <div align="right"><a href="#top">top</a></div> <a name="launch-import"></a> ###2a: Launch an import job To import a file, make the API request to [start an import job](start-an-import-job-v2) as shown below. In the body of the request, include the key-value pairs in the table below. [block:parameters] { "data": { "h-0": "Key", "h-1": "Description of value", "0-0": "`volume_id`\n_required_", "0-1": "Volume ID from which to import the file. This consists of your username followed by the volume's name, such as `rfranklin/tutorial_volume`.", "1-0": "`location`\n_required_", "2-0": "`destination`\n_required_", "3-0": "`project`\n_required_", "4-0": "`name`", "5-0": "`overwrite`", "5-1": "Specify as `true` to overwrite the file if the file with the same name already exists in the destination.", "4-1": "The name of the alias to create. This name should be unique to the project. If the name is already in use in the project, you should use the overwrite query parameter in this call to force any file with that name to be deleted before the alias is created.\n\nIf name is omitted, the alias name will default to the last segment of the complete location (including the `prefix`) on the volume. Segments are considered to be separated with forward slashes ('/').", "3-1": "The project in which to create the alias. This consists of your username followed by your project's short name, such as `rfranklin/my-project`.", "2-1": "This object should describe the CGC destination for the imported file.", "1-1": "Volume-specific location pointing to the file to import. This location should be recognizable to the underlying cloud service as a valid key or path to the file.\n\nPlease note that if this volume was configured with a `prefix` parameter when it was created, the `prefix` will be prepended to location before attempting to locate the file on the volume." }, "cols": 2, "rows": 6 } [/block] [block:code] { "codes": [ { "code": "POST /v2/storage/imports HTTP/1.1\nHost: cgc-api.sbgenomics.com\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\ncontent-type: application/json", "language": "http", "name": "Start an import job" } ] } [/block] [block:code] { "codes": [ { "code": "{ \n \"source\":{ \n \"volume\":\"rfranklin/tutorial_volume\",\n \"location\":\"example_human_Illumina.pe_1.fastq\"\n },\n \"destination\":{ \n \"project\":\"rfranklin/my-project\",\n \"name\":\"my_imported_example_human_Illumina.pe_1.fastq\"\n },\n \"overwrite\": true\n}", "language": "json", "name": "Start an import job request body" } ] } [/block] The returned response details the status of your import, as shown below. [block:code] { "codes": [ { "code": "{\n \"href\": \"https://cgc-api.sbgenomics.com/v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\",\n \"id\": \"SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN\",\n \"state\": \"PENDING\",\n \"overwrite\": true,\n \"source\": {\n \"volume\": \"rfranklin/tutorial_volume\",\n \"location\": \"example_human_Illumina.pe_1.fastq\"\n },\n \"destination\": {\n \"project\": \"rfranklin/my-project\",\n \"name\": \"my_uploaded_example_human_Illumina.pe_1.fastq\"\n }\n}", "language": "json", "name": "Response body" } ] } [/block] Locate the `id` property in the response and copy this value to your clipboard. This `id` is an identifier for the import job, and we will need it in the following step. <div align="right"><a href="#top">top</a></div> <a name="check-import"></a> ###2b: Check if the import job has completed To check if the import job has completed, make the API request to [get details of an import job](get-details-of-an-import-job-v2), as shown below. Simply append the import job `id` obtained in the step above to the path. [block:code] { "codes": [ { "code": "GET /v2/storage/imports/SbBQeWpfZ445jxgIBioYfnJ5oBl86nFN HTTP/1.1\nHost: cgc-api.sbgenomics.com\nX-SBG-Auth-Token: 3259c50e1ac5426ea8f1273259740f74\ncontent-type: application/json", "language": "http", "name": "Get details of an import job" } ] } [/block] The returned response details the `state` of your import. If the state is `COMPLETED`, your import has successfully finished. If the `state` is `PENDING`, wait a few seconds and repeat this step. You should now have a freshly-created alias in your project. To verify that a file has been imported, visit this project in your browser and look for a file with the same name as the key of the object in your bucket. Congratulations! You've now registered an GCS bucket as a volume and imported a file from the volume to the CGC. Learn more about [connecting your cloud storage from our Knowledge Center](connecting-cloud-storage-overview). <div align="right"><a href="#top">top</a></div>