{"_id":"584fcb9e29fc1c1900d78370","project":"55faf11ba62ba1170021a9a7","version":{"_id":"55faf11ba62ba1170021a9aa","project":"55faf11ba62ba1170021a9a7","__v":37,"createdAt":"2015-09-17T16:58:03.490Z","releaseDate":"2015-09-17T16:58:03.490Z","categories":["55faf11ca62ba1170021a9ab","55faf8f4d0e22017005b8272","55faf91aa62ba1170021a9b5","55faf929a8a7770d00c2c0bd","55faf932a8a7770d00c2c0bf","55faf94b17b9d00d00969f47","55faf958d0e22017005b8274","55faf95fa8a7770d00c2c0c0","55faf96917b9d00d00969f48","55faf970a8a7770d00c2c0c1","55faf98c825d5f19001fa3a6","55faf99aa62ba1170021a9b8","55faf99fa62ba1170021a9b9","55faf9aa17b9d00d00969f49","55faf9b6a8a7770d00c2c0c3","55faf9bda62ba1170021a9ba","5604570090ee490d00440551","5637e8b2fbe1c50d008cb078","5649bb624fa1460d00780add","5671974d1b6b730d008b4823","5671979d60c8e70d006c9760","568e8eef70ca1f0d0035808e","56d0a2081ecc471500f1795e","56d4a0adde40c70b00823ea3","56d96b03dd90610b00270849","56fbb83d8f21c817002af880","573c811bee2b3b2200422be1","576bc92afb62dd20001cda85","5771811e27a5c20e00030dcd","5785191af3a10c0e009b75b0","57bdf84d5d48411900cd8dc0","57ff5c5dc135231700aed806","5804caf792398f0f00e77521","58458b4fba4f1c0f009692bb","586d3c287c6b5b2300c05055","58ef66d88646742f009a0216","58f5d52d7891630f00fe4e77"],"is_deprecated":false,"is_hidden":false,"is_beta":true,"is_stable":true,"codename":"","version_clean":"1.0.0","version":"1.0"},"category":{"_id":"56d0a2081ecc471500f1795e","project":"55faf11ba62ba1170021a9a7","__v":4,"pages":["56d0a2be376b040b005b304e","56d0a2e6376b040b005b3051","56d0ae447c2e100b000af980","56d0b36d40d36e1d00bc145a"],"version":"55faf11ba62ba1170021a9aa","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2016-02-26T19:05:44.797Z","from_sync":false,"order":9,"slug":"set-metadata-associated-with-a-private-file","title":"SET METADATA ASSOCIATED WITH A PRIVATE FILE"},"parentDoc":null,"__v":0,"user":"566590c83889610d0008a253","updates":[],"next":{"pages":[],"description":""},"createdAt":"2016-12-13T10:21:18.547Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":5,"body":"Two different manifest file formats are used on CGC. Which manifest file you should use depends on the functionality that you're using the manifest for.\n\n  * **CGC Uploader or Command Line Uploader** - use this format of manifest file in order to apply metadata when uploading files to the Platform.\n  * **Modifying metadata via the visual interface** - use this format of manifest file in order to modify the metadata for files that are already on the Platform in one of your projects. \n\n##CGC Uploader or Command Line Uploader\n\nWhen uploading files to the Platform using the [CGC Uploader](doc:upload-via-the-cgc-uploader#section-upload-files-using-a-manifest-file) or [Command Line Uploader](doc:set-metadata-using-the-command-line-uploader#section-set-metadata-for-multiple-files-using-a-manifest-file), a manifest file can be used to define metadata values for those files.\n\nThe supported file format for this manifest file is CSV, i.e. comma separated values. A CSV file contains a number of rows with columns which are separated with a comma.\nThe following rules apply:\n\n  * The lines are separated with a line break, while the columns are separated using a comma.\n  * The **first row** has to contain **column names** which are treated as metadata fields (e.g. “sample”, “library”).\n  * The **first column** has to contain the **names of the files** which will be uploaded. In case the files are not in the same directory as the manifest file, you should also include a path to the files (e.g. `../filename.fastq`).\n  * All subsequent columns should contain metadata fields which will be assigned to the specified files.\n  * Keys and values are case sensitive unless explicitly treated differently by a tool or a part of the Platform.\n  * Maximum number of key-value pairs per file is 1000, including null-value keys.\n  * Keys and values are UTF-8 encoded strings.\n  * Maximum length of a key is 100 bytes (UTF-8 encoding).\n  * Maximum length of a value is 300 bytes (UTF-8 encoding).\n  * Quotation marks are allowed. \n\nThe following example shows the content of the manifest for three files with three metadata fields.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"File name\",\n    \"h-1\": \"sample\",\n    \"h-2\": \"library\",\n    \"h-3\": \"paired_end\",\n    \"0-0\": \"file1.fastq\",\n    \"0-1\": \"sample1\",\n    \"0-2\": \"examplelibrary1\",\n    \"0-3\": \"1\",\n    \"1-0\": \"file2.fastq\",\n    \"1-1\": \"sample1\",\n    \"1-2\": \"examplelibrary1\",\n    \"1-3\": \"2\",\n    \"2-0\": \"file3.fastq\",\n    \"2-1\": \"sample2\",\n    \"2-2\": \"examplelibrary2\",\n    \"2-3\": \"1\"\n  },\n  \"cols\": 4,\n  \"rows\": 3\n}\n[/block]\nBelow is the same example in a comma separated format.\n\n`File name,sample,library,paired_end`\n`file1.fastq,sample1,examplelibrary1,1`\n`file2.fastq,sample1,examplelibrary1,2`\n`file3.fastq,sample2,examplelibrary2,1`\n\n##Modifying metadata via the visual interface\n\nThis manifest file is used In case you are modifying metadata for all of your project files via the visual interface. The metadata are stored in a manifest file which you can download, change values for the metadata fields and then upload it back to the Platform, thus applying changes to metadata in all of your project files.\n\nThe supported file format for this manifest file is CSV. A CSV file contains a number of rows with columns which are separated with a comma.\n\nThe following rules apply:\n\n  * The columns are separated using a comma while the lines are separated with a line break.\n  * The **first row** has to contain **column names**. The required columns are:\n   * `id` - specifies the file ID.\n   * `name` - specifies the file name.\n   * `project` - specifies the project name.\n   * `<metadata field name>` - at least one metadata field name should be specified.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"Columns which are specified after id, name and project will be treated as metadata fields.\"\n}\n[/block]\nThe following example shows a manifest file which contains the required columns and one metadata field (paired_end).\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"id\",\n    \"h-1\": \"name\",\n    \"h-2\": \"project\",\n    \"h-3\": \"paired_end\",\n    \"0-0\": \"581b298d20946e087b2ce503\",\n    \"0-1\": \"file1.fastq\",\n    \"0-2\": \"rfranklin/my-project\",\n    \"0-3\": \"1\",\n    \"1-0\": \"581b298d20946e087b2ce51f\",\n    \"1-1\": \"file2.fastq\",\n    \"1-2\": \"rfranklin/my-project\",\n    \"1-3\": \"2\",\n    \"2-0\": \"581b298d20946e087b2ce50b\",\n    \"2-1\": \"file3.fastq\",\n    \"2-2\": \"rfranklin/my-project\",\n    \"2-3\": \"1\"\n  },\n  \"cols\": 4,\n  \"rows\": 3\n}\n[/block]\nBelow is the same example in a comma separated format.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"id,name,project,paired_end\\n581b298d20946e087b2ce503,file1.fastq,rfranklin/my-project,1\\n581b298d20946e087b2ce51f,file2.fastq,rfranklin/my-project,2\\n581b298d20946e087b2ce50b,file3.fastq,rfranklin/my-project,1\",\n      \"language\": \"text\"\n    }\n  ]\n}\n[/block]","excerpt":"","slug":"format-of-a-manifest-file","type":"basic","title":"Format of a manifest file"}

Format of a manifest file


Two different manifest file formats are used on CGC. Which manifest file you should use depends on the functionality that you're using the manifest for. * **CGC Uploader or Command Line Uploader** - use this format of manifest file in order to apply metadata when uploading files to the Platform. * **Modifying metadata via the visual interface** - use this format of manifest file in order to modify the metadata for files that are already on the Platform in one of your projects. ##CGC Uploader or Command Line Uploader When uploading files to the Platform using the [CGC Uploader](doc:upload-via-the-cgc-uploader#section-upload-files-using-a-manifest-file) or [Command Line Uploader](doc:set-metadata-using-the-command-line-uploader#section-set-metadata-for-multiple-files-using-a-manifest-file), a manifest file can be used to define metadata values for those files. The supported file format for this manifest file is CSV, i.e. comma separated values. A CSV file contains a number of rows with columns which are separated with a comma. The following rules apply: * The lines are separated with a line break, while the columns are separated using a comma. * The **first row** has to contain **column names** which are treated as metadata fields (e.g. “sample”, “library”). * The **first column** has to contain the **names of the files** which will be uploaded. In case the files are not in the same directory as the manifest file, you should also include a path to the files (e.g. `../filename.fastq`). * All subsequent columns should contain metadata fields which will be assigned to the specified files. * Keys and values are case sensitive unless explicitly treated differently by a tool or a part of the Platform. * Maximum number of key-value pairs per file is 1000, including null-value keys. * Keys and values are UTF-8 encoded strings. * Maximum length of a key is 100 bytes (UTF-8 encoding). * Maximum length of a value is 300 bytes (UTF-8 encoding). * Quotation marks are allowed. The following example shows the content of the manifest for three files with three metadata fields. [block:parameters] { "data": { "h-0": "File name", "h-1": "sample", "h-2": "library", "h-3": "paired_end", "0-0": "file1.fastq", "0-1": "sample1", "0-2": "examplelibrary1", "0-3": "1", "1-0": "file2.fastq", "1-1": "sample1", "1-2": "examplelibrary1", "1-3": "2", "2-0": "file3.fastq", "2-1": "sample2", "2-2": "examplelibrary2", "2-3": "1" }, "cols": 4, "rows": 3 } [/block] Below is the same example in a comma separated format. `File name,sample,library,paired_end` `file1.fastq,sample1,examplelibrary1,1` `file2.fastq,sample1,examplelibrary1,2` `file3.fastq,sample2,examplelibrary2,1` ##Modifying metadata via the visual interface This manifest file is used In case you are modifying metadata for all of your project files via the visual interface. The metadata are stored in a manifest file which you can download, change values for the metadata fields and then upload it back to the Platform, thus applying changes to metadata in all of your project files. The supported file format for this manifest file is CSV. A CSV file contains a number of rows with columns which are separated with a comma. The following rules apply: * The columns are separated using a comma while the lines are separated with a line break. * The **first row** has to contain **column names**. The required columns are: * `id` - specifies the file ID. * `name` - specifies the file name. * `project` - specifies the project name. * `<metadata field name>` - at least one metadata field name should be specified. [block:callout] { "type": "info", "body": "Columns which are specified after id, name and project will be treated as metadata fields." } [/block] The following example shows a manifest file which contains the required columns and one metadata field (paired_end). [block:parameters] { "data": { "h-0": "id", "h-1": "name", "h-2": "project", "h-3": "paired_end", "0-0": "581b298d20946e087b2ce503", "0-1": "file1.fastq", "0-2": "rfranklin/my-project", "0-3": "1", "1-0": "581b298d20946e087b2ce51f", "1-1": "file2.fastq", "1-2": "rfranklin/my-project", "1-3": "2", "2-0": "581b298d20946e087b2ce50b", "2-1": "file3.fastq", "2-2": "rfranklin/my-project", "2-3": "1" }, "cols": 4, "rows": 3 } [/block] Below is the same example in a comma separated format. [block:code] { "codes": [ { "code": "id,name,project,paired_end\n581b298d20946e087b2ce503,file1.fastq,rfranklin/my-project,1\n581b298d20946e087b2ce51f,file2.fastq,rfranklin/my-project,2\n581b298d20946e087b2ce50b,file3.fastq,rfranklin/my-project,1", "language": "text" } ] } [/block]