{"_id":"578519e25ae9c20e00bc2611","version":{"_id":"55faf11ba62ba1170021a9aa","project":"55faf11ba62ba1170021a9a7","__v":37,"createdAt":"2015-09-17T16:58:03.490Z","releaseDate":"2015-09-17T16:58:03.490Z","categories":["55faf11ca62ba1170021a9ab","55faf8f4d0e22017005b8272","55faf91aa62ba1170021a9b5","55faf929a8a7770d00c2c0bd","55faf932a8a7770d00c2c0bf","55faf94b17b9d00d00969f47","55faf958d0e22017005b8274","55faf95fa8a7770d00c2c0c0","55faf96917b9d00d00969f48","55faf970a8a7770d00c2c0c1","55faf98c825d5f19001fa3a6","55faf99aa62ba1170021a9b8","55faf99fa62ba1170021a9b9","55faf9aa17b9d00d00969f49","55faf9b6a8a7770d00c2c0c3","55faf9bda62ba1170021a9ba","5604570090ee490d00440551","5637e8b2fbe1c50d008cb078","5649bb624fa1460d00780add","5671974d1b6b730d008b4823","5671979d60c8e70d006c9760","568e8eef70ca1f0d0035808e","56d0a2081ecc471500f1795e","56d4a0adde40c70b00823ea3","56d96b03dd90610b00270849","56fbb83d8f21c817002af880","573c811bee2b3b2200422be1","576bc92afb62dd20001cda85","5771811e27a5c20e00030dcd","5785191af3a10c0e009b75b0","57bdf84d5d48411900cd8dc0","57ff5c5dc135231700aed806","5804caf792398f0f00e77521","58458b4fba4f1c0f009692bb","586d3c287c6b5b2300c05055","58ef66d88646742f009a0216","58f5d52d7891630f00fe4e77"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"category":{"_id":"5785191af3a10c0e009b75b0","version":"55faf11ba62ba1170021a9aa","__v":0,"project":"55faf11ba62ba1170021a9a7","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-07-12T16:21:46.337Z","from_sync":false,"order":24,"slug":"connect-cloud-storage","title":"CONNECT CLOUD STORAGE"},"project":"55faf11ba62ba1170021a9a7","user":"5613e4f8fdd08f2b00437620","__v":4,"parentDoc":null,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2016-07-12T16:25:06.267Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":1,"body":"[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"On this page:\",\n  \"body\": \"* [What is a volume?](doc:volumes#section-what-is-a-volume-)\\n* [What information do I need to provide?](#section-what-information-should-i-provide-)\\n * [Volume name](#section-volume-name)\\n * [Access mode](#section-access-mode)\\n  * [Cloud service configuration](#section-cloud-service-configuration)\\n  * [(optional) Prefixes](#prefix)\\n* [What can I do with a volume?](doc:volumes#section-what-can-i-do-with-a-volume-)\\n   * [Operations on volumes](volumes#section-operations-on-volumes)\"\n}\n[/block]\n##What is a volume?\nCloud storage providers come with their own interfaces, features and terminology. At a certain level, though, they all view resources as data objects organized in repositories. Authentication and operations are commonly defined on those objects and repositories, and while each cloud provider might call these things different names and apply different parameters to them, their basic behavior is the same.\n\nThe CGC mediates access to these repositories using volumes. A volume is associated with a particular cloud storage repository that you have enabled the CGC to read from (and, optionally, to write to). Currently, volumes may be created using two types of cloud storage repository: Amazon Web Services' (AWS) S3 buckets and Google Cloud Storage (GCS) buckets. \n\nA volume enables you to treat the cloud repository associated with it as external storage for the CGC. You can 'export' files from the volume to the CGC to use them as inputs for computation. Similarly, you can write files from the CGC to your cloud storage by 'importing' them to your volume.\n\nThe following diagram shows the relationship between a user's AWS repositories, bucket1 and bucket2, and a volume, from which you can import and export the contents of the buckets to a CGC project.\n\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/X6uLl0tRwmwtV1ttBZcg_Volumes%20API%20diagram.png\",\n        \"Volumes API diagram.png\",\n        \"960\",\n        \"540\",\n        \"#b93734\",\n        \"\"\n      ],\n      \"caption\": \"An AWS S3 volume enables data import and export operations via a set of IAM credentials generated by the user.\"\n    }\n  ]\n}\n[/block]\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n##What information should I provide?\nTo create a volume on the CGC, you should provide the following information:\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n###Volume name\nA volume name consists of 3-32 letters of the English alphabet, numbers, and underscores (_). Each of your volumes must have a unique name.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n###Access mode\nThe access mode of a volume is either read-only (`RO`) or read-write (`RW`).\nEach volume will have its access mode set by default to read-only.\n\nRead-only volumes can only be used to make files available to the CGC for reading; for example, by importing files into projects. Read-write volumes' contents can be modified by operations performed with the Volumes API. Note that regardless of access mode, the owner of the volume is still the only CGC account that can issue operations on that volume. However, it is a good practice to avoid configuring your input buckets as read-write, as a safeguard against honest mistakes.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n###Cloud service configuration\nTo access the cloud storage provider on your behalf, the CGC needs to know the type of storage provider it should talk to, and be configured to interface with it.\n\nAs each cloud storage provider supports slightly different features and authorization schemas, the configuration will be different for each storage type. Depending on your selected storage provider and your intended use of the volume, there might be additional properties you may want to set to further modify the way the CGC accesses it.\n\nFor more details on each supported storage type and its corresponding configuration, see the following pages:\n\n[Amazon Web Services' Simple Storage Service (AWS S3) Volumes](doc:amazon-web-services-simple-storage-service-aws-s3-volumes) \n[Google Cloud Storage (GCS) Volumes](doc:google-cloud-storage-gcs-volumes) \n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n<a name=\"prefix\"></a>\n<h3>Prefix <span style=\"font-size:10px\">(optional)</span></h3>\n\nA prefix is a string that can be set on a volume to refer to a specific location on a single bucket. This feature can be used to replicate the folder or grouping structure inside a bucket in your cloud storage account. As such, a prefix is useful if you have grouped files devoted to distinct projects within a single bucket.\n\nFor example, on Amazon S3, folders can also be assigned specific access permissions in the bucket policy itself, or an IAM user can be configured to have read-only or read-write access per each folder. When you assign your volume a prefix as well as a bucket, you limit the volume's operations within the specified folder on the bucket.\n\nFor example, if you set the `prefix` for a volume to `\"a10\"`, and import a file with `location` set to `\"test.fastq\"` from the volume to the Platform, then the object that will be referred to by the newly-created alias will be `\"a10/test.fastq\"`.\n\nNote that our [AWS policy generator](http://docs.sevenbridges.com/docs/aws-policy-generator) allows you to set different access policies for groups of files with the same prefix in a volume.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n##What can I do with a volume?\nOnce a volume has been configured, you can make its data objects available for computation on the CGC, and, conversely, and copy files to the volume from the CGC. \n\nThese operations can be carried out using the Volumes API. All calls are listed in the Volumes section of the API documentation.\n\n<div align=\"right\"><a href=\"#top\">top</a></div>\n\n###Operations on volumes\n  * [Create a volume](doc:create-a-volume-v2) \n  * [List volumes](doc:list-volumes-v2) \n  * [Get details of a volume](doc:get-details-of-a-volume-v2)\n  * [Update a volume](doc:update-a-volume-v2) \n  * [Delete a volume](doc:delete-a-volume-v2) \n  * [Start an import job](doc:start-an-import-job-v2) \n  * [Get details of an import job](doc:get-details-of-an-import-job-v2) \n  * [Start an export job](doc:start-an-export-job-v2) \n  * [List export jobs](doc:list-export-jobs-v2) \n  * [List import jobs](doc:list-import-jobs-v2)\n\n<div align=\"right\"><a href=\"#top\">top</a></div>","excerpt":"","slug":"volumes","type":"basic","title":"Volumes"}
[block:callout] { "type": "warning", "title": "On this page:", "body": "* [What is a volume?](doc:volumes#section-what-is-a-volume-)\n* [What information do I need to provide?](#section-what-information-should-i-provide-)\n * [Volume name](#section-volume-name)\n * [Access mode](#section-access-mode)\n * [Cloud service configuration](#section-cloud-service-configuration)\n * [(optional) Prefixes](#prefix)\n* [What can I do with a volume?](doc:volumes#section-what-can-i-do-with-a-volume-)\n * [Operations on volumes](volumes#section-operations-on-volumes)" } [/block] ##What is a volume? Cloud storage providers come with their own interfaces, features and terminology. At a certain level, though, they all view resources as data objects organized in repositories. Authentication and operations are commonly defined on those objects and repositories, and while each cloud provider might call these things different names and apply different parameters to them, their basic behavior is the same. The CGC mediates access to these repositories using volumes. A volume is associated with a particular cloud storage repository that you have enabled the CGC to read from (and, optionally, to write to). Currently, volumes may be created using two types of cloud storage repository: Amazon Web Services' (AWS) S3 buckets and Google Cloud Storage (GCS) buckets. A volume enables you to treat the cloud repository associated with it as external storage for the CGC. You can 'export' files from the volume to the CGC to use them as inputs for computation. Similarly, you can write files from the CGC to your cloud storage by 'importing' them to your volume. The following diagram shows the relationship between a user's AWS repositories, bucket1 and bucket2, and a volume, from which you can import and export the contents of the buckets to a CGC project. [block:image] { "images": [ { "image": [ "https://files.readme.io/X6uLl0tRwmwtV1ttBZcg_Volumes%20API%20diagram.png", "Volumes API diagram.png", "960", "540", "#b93734", "" ], "caption": "An AWS S3 volume enables data import and export operations via a set of IAM credentials generated by the user." } ] } [/block] <div align="right"><a href="#top">top</a></div> ##What information should I provide? To create a volume on the CGC, you should provide the following information: <div align="right"><a href="#top">top</a></div> ###Volume name A volume name consists of 3-32 letters of the English alphabet, numbers, and underscores (_). Each of your volumes must have a unique name. <div align="right"><a href="#top">top</a></div> ###Access mode The access mode of a volume is either read-only (`RO`) or read-write (`RW`). Each volume will have its access mode set by default to read-only. Read-only volumes can only be used to make files available to the CGC for reading; for example, by importing files into projects. Read-write volumes' contents can be modified by operations performed with the Volumes API. Note that regardless of access mode, the owner of the volume is still the only CGC account that can issue operations on that volume. However, it is a good practice to avoid configuring your input buckets as read-write, as a safeguard against honest mistakes. <div align="right"><a href="#top">top</a></div> ###Cloud service configuration To access the cloud storage provider on your behalf, the CGC needs to know the type of storage provider it should talk to, and be configured to interface with it. As each cloud storage provider supports slightly different features and authorization schemas, the configuration will be different for each storage type. Depending on your selected storage provider and your intended use of the volume, there might be additional properties you may want to set to further modify the way the CGC accesses it. For more details on each supported storage type and its corresponding configuration, see the following pages: [Amazon Web Services' Simple Storage Service (AWS S3) Volumes](doc:amazon-web-services-simple-storage-service-aws-s3-volumes) [Google Cloud Storage (GCS) Volumes](doc:google-cloud-storage-gcs-volumes) <div align="right"><a href="#top">top</a></div> <a name="prefix"></a> <h3>Prefix <span style="font-size:10px">(optional)</span></h3> A prefix is a string that can be set on a volume to refer to a specific location on a single bucket. This feature can be used to replicate the folder or grouping structure inside a bucket in your cloud storage account. As such, a prefix is useful if you have grouped files devoted to distinct projects within a single bucket. For example, on Amazon S3, folders can also be assigned specific access permissions in the bucket policy itself, or an IAM user can be configured to have read-only or read-write access per each folder. When you assign your volume a prefix as well as a bucket, you limit the volume's operations within the specified folder on the bucket. For example, if you set the `prefix` for a volume to `"a10"`, and import a file with `location` set to `"test.fastq"` from the volume to the Platform, then the object that will be referred to by the newly-created alias will be `"a10/test.fastq"`. Note that our [AWS policy generator](http://docs.sevenbridges.com/docs/aws-policy-generator) allows you to set different access policies for groups of files with the same prefix in a volume. <div align="right"><a href="#top">top</a></div> ##What can I do with a volume? Once a volume has been configured, you can make its data objects available for computation on the CGC, and, conversely, and copy files to the volume from the CGC. These operations can be carried out using the Volumes API. All calls are listed in the Volumes section of the API documentation. <div align="right"><a href="#top">top</a></div> ###Operations on volumes * [Create a volume](doc:create-a-volume-v2) * [List volumes](doc:list-volumes-v2) * [Get details of a volume](doc:get-details-of-a-volume-v2) * [Update a volume](doc:update-a-volume-v2) * [Delete a volume](doc:delete-a-volume-v2) * [Start an import job](doc:start-an-import-job-v2) * [Get details of an import job](doc:get-details-of-an-import-job-v2) * [Start an export job](doc:start-an-export-job-v2) * [List export jobs](doc:list-export-jobs-v2) * [List import jobs](doc:list-import-jobs-v2) <div align="right"><a href="#top">top</a></div>