{"metadata":{"image":[],"title":"","description":""},"api":{"url":"","auth":"required","settings":"","results":{"codes":[]},"params":[]},"next":{"description":"","pages":[]},"title":"About secondary (index) files","type":"basic","slug":"about-secondary-index-files","excerpt":"","body":"Some tools on the CGC that process user-specified input files may require complementary files in order to execute properly. These complementary files are commonly referred to as secondary files. A secondary file is usually an index file that allows tools to have indexed random access to the file which that secondary file accompanies. This means that the tool is able to access specific portions of the file directly, without having to iterate through its entire content.\n\n## Use secondary files in task executions\n\nIf you are executing an app where at least one input requires a secondary file to be used along with the file it accompanies, follow these guidelines to ensure successful task execution:\n1. **Secondary (index) files need to be added to the same project where the files they accompany are located and where tasks are being executed**. If a required index file is not available in the same project, this might result in task failure. \n2. **Secondary (index) files usually don't have to be set explicitly as task inputs**. Instead, they are pulled automatically when the task starts, if the files they accompany are set properly as task inputs and you added secondary files (indices) to the project where the task is being executed.\n\nFor example, if a file named `sample.bam` is used as an input and its index file is also required, all you need to do is make sure that its corresponding `sample.bam.bai` file is also present in the same project where the execution takes place.\n\n## Types of secondary files\n\nMost common secondary file types used on the CGC are:\n* **FAI** - Index file format for FASTA files.\n* **DICT** - Sequence dictionaries for FASTA files.\n* **BAI** - Index file format for BAM files.\n* **CRAI** - Index file format for CRAM files.\n* **TBI** - Index file format for tab-delimited files.\n* **IDX** - Index file format for VCF files.\n\n##FAI files\nFAI (**FA**STA **I**ndex) files enable indexed random access to FASTA files. They are structured in the same order of sequences as their corresponding FASTA files and have the same name as the FASTA file, with the **.fasta.fai** extension (for example, a FASTA file named **reference.fasta** would have a corresponding index file named **reference.fasta.fai**). A FAI file is essentially a text file consisting of lines with five tab-delimited columns in each line:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Column name\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"NAME\",\n    \"1-0\": \"LENGTH\",\n    \"2-0\": \"OFFSET\",\n    \"3-0\": \"LINEBASES\",\n    \"4-0\": \"LINEWIDTH\",\n    \"0-1\": \"Name of the reference sequence.\",\n    \"1-1\": \"Total length of this reference sequence, in bases.\",\n    \"2-1\": \"Offset (in terms of lines) within the FASTA file of the sequence's first base.\",\n    \"3-1\": \"The number of bases on each line.\",\n    \"4-1\": \"The number of bytes in each line, including the newline.\"\n  },\n  \"cols\": 2,\n  \"rows\": 5\n}\n[/block]\nA FAI file can be generated from a FASTA file using one of the suitable tools on the CGC, for example:\n* **SAMtools Index FASTA**\n* **SBG FASTA Indices** \n\n##DICT files\nA DICT file describes the content of the corresponding FASTA file. It contains information about contigs in the FASTA file and their sizes (in number of bases). A DICT file must have the same name as its corresponding FASTA file, but the extension must be **.dict** instead of **.fasta** (or **.fa**). For example, if there is a file named **reference.fasta**, the corresponding DICT file needs to be **reference.dict**. Tools on the CGC that can be used to generate a DICT file from a FASTA file are, for example:\n* **Picard CreateSequenceDictionary**\n* **SBG FASTA Indices**\n\n##BAI files\nBAM files can also be accompanied by an index file that has the same name as the BAM file, suffixed with **.bam.bai**. This file allows (analysis) tools to jump directly to specific alignment lines of the BAM file without starting from the first line and visiting all of the lines in between. A BAI file does not have any purpose without a corresponding BAM file since it does not actually contain any sequence data.\nIf you have a BAM file that does not have an accompanying index (BAI) file, the BAI file can be generated using one of the adequate tools from the CGC, for example:\n* **SAMtools Index BAM**\n* **Picard BuildBamIndex**\n* **Sambamba Index**\n\nHowever, the most important prerequisite for generating an index (BAI) is that **the BAM file is coordinate-sorted**.\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"body\": \"A BAI (index) file cannot be generated for an unsorted BAM file.\"\n}\n[/block]\nThe BAI file is structured so that it contains two coordinates for every compressed block of a BAM file.\n\n* One coordinate states precisely the offset of the location of the compressed block within the compressed BAM file\n* The other coordinate is the offset to the location from within the block to the particular alignment line that we want to access.\n\n##CRAI files\nCRAI files are external index files for files in the CRAM format (a compressed version of BAM that stores only reads different from the reference sequence). CRAI files follow the same naming convention as the aforementioned index file formats - they have the same name as the corresponding CRAM file, suffixed with **.cram.crai**.\n\n##TBI files\nTBI (tabix) format is the generic index file format for a lot of widely used tab-delimited formats such as GFF/GTF, BED, SAM, VCF etc. Tabix files have the **.tbi** extension, which is appended to the name of the file for which the TBI index is generated (for example, a file named **file.vcf.bgz** will have a tabix index file named **file.vcf.bgz.tbi**). A tabix file can be generated from a tab-delimited file that is **position-sorted** and **zipped by bgzip**. To zip a tab-delimited file by **bgzip** and generate a tabix file on the CGC, you can use tools such as:\n* **Tabix BGZIP** (compresses the tab-delimited file using bgzip).\n* **Tabix Index** (generates a tabix index file from a tab-delimited file that is position-sorted and has previously been zipped by bgzip).\n\n##IDX files\nIDX format is the index format used for indexing of VCF files. The purpose of IDX files is to allow tools to have indexed random access to VCF files. VCF index (IDX) files have the **.idx** extension which is appended to the name of the corresponding VCF file (a file named **sample.vcf** would have an index file named **sample.vcf.idx**).","updates":[],"order":6,"isReference":false,"hidden":false,"sync_unique":"","link_url":"","link_external":false,"_id":"6246f042b6a01c002e643dc5","createdAt":"2022-04-01T12:29:54.615Z","user":"5767bc73bb15f40e00a28777","category":{"sync":{"isSync":false,"url":""},"pages":["5626a5d644c87f0d00fe6396","5626a5e4e2ce610d004e3dd8","5626a89ce2ce610d004e3dde","56429a87f49bfa0d002f54e0"],"title":"RUN AN ANALYSIS","slug":"run-an-analysis","order":22,"from_sync":false,"reference":false,"_id":"55faf96917b9d00d00969f48","project":"55faf11ba62ba1170021a9a7","__v":4,"createdAt":"2015-09-17T17:33:29.016Z","version":"55faf11ba62ba1170021a9aa"},"version":{"version":"1.0","version_clean":"1.0.0","codename":"","is_stable":true,"is_beta":true,"is_hidden":false,"is_deprecated":false,"categories":["55faf11ca62ba1170021a9ab","55faf8f4d0e22017005b8272","55faf91aa62ba1170021a9b5","55faf929a8a7770d00c2c0bd","55faf932a8a7770d00c2c0bf","55faf94b17b9d00d00969f47","55faf958d0e22017005b8274","55faf95fa8a7770d00c2c0c0","55faf96917b9d00d00969f48","55faf970a8a7770d00c2c0c1","55faf98c825d5f19001fa3a6","55faf99aa62ba1170021a9b8","55faf99fa62ba1170021a9b9","55faf9aa17b9d00d00969f49","55faf9b6a8a7770d00c2c0c3","55faf9bda62ba1170021a9ba","5604570090ee490d00440551","5637e8b2fbe1c50d008cb078","5649bb624fa1460d00780add","5671974d1b6b730d008b4823","5671979d60c8e70d006c9760","568e8eef70ca1f0d0035808e","56d0a2081ecc471500f1795e","56d4a0adde40c70b00823ea3","56d96b03dd90610b00270849","56fbb83d8f21c817002af880","573c811bee2b3b2200422be1","576bc92afb62dd20001cda85","5771811e27a5c20e00030dcd","5785191af3a10c0e009b75b0","57bdf84d5d48411900cd8dc0","57ff5c5dc135231700aed806","5804caf792398f0f00e77521","58458b4fba4f1c0f009692bb","586d3c287c6b5b2300c05055","58ef66d88646742f009a0216","58f5d52d7891630f00fe4e77","59a555bccdbd85001bfb1442","5a2a81f688574d001e9934f5","5b080c8d7833b20003ddbb6f","5c222bed4bc358002f21459a","5c22412594a2a5005cc9e919","5c41ae1c33592700190a291e","5c8a525e2ba7b2003f9b153c","5cbf14d58c79c700ef2b502e","5db6f03a6e187c006f667fa4","5f894c7d3b0894006477ca01","6176d5bf8f59c6001038c2f7"],"_id":"55faf11ba62ba1170021a9aa","releaseDate":"2015-09-17T16:58:03.490Z","createdAt":"2015-09-17T16:58:03.490Z","project":"55faf11ba62ba1170021a9a7","__v":48},"project":"55faf11ba62ba1170021a9a7","__v":0,"parentDoc":null}

About secondary (index) files


Some tools on the CGC that process user-specified input files may require complementary files in order to execute properly. These complementary files are commonly referred to as secondary files. A secondary file is usually an index file that allows tools to have indexed random access to the file which that secondary file accompanies. This means that the tool is able to access specific portions of the file directly, without having to iterate through its entire content. ## Use secondary files in task executions If you are executing an app where at least one input requires a secondary file to be used along with the file it accompanies, follow these guidelines to ensure successful task execution: 1. **Secondary (index) files need to be added to the same project where the files they accompany are located and where tasks are being executed**. If a required index file is not available in the same project, this might result in task failure. 2. **Secondary (index) files usually don't have to be set explicitly as task inputs**. Instead, they are pulled automatically when the task starts, if the files they accompany are set properly as task inputs and you added secondary files (indices) to the project where the task is being executed. For example, if a file named `sample.bam` is used as an input and its index file is also required, all you need to do is make sure that its corresponding `sample.bam.bai` file is also present in the same project where the execution takes place. ## Types of secondary files Most common secondary file types used on the CGC are: * **FAI** - Index file format for FASTA files. * **DICT** - Sequence dictionaries for FASTA files. * **BAI** - Index file format for BAM files. * **CRAI** - Index file format for CRAM files. * **TBI** - Index file format for tab-delimited files. * **IDX** - Index file format for VCF files. ##FAI files FAI (**FA**STA **I**ndex) files enable indexed random access to FASTA files. They are structured in the same order of sequences as their corresponding FASTA files and have the same name as the FASTA file, with the **.fasta.fai** extension (for example, a FASTA file named **reference.fasta** would have a corresponding index file named **reference.fasta.fai**). A FAI file is essentially a text file consisting of lines with five tab-delimited columns in each line: [block:parameters] { "data": { "h-0": "Column name", "h-1": "Description", "0-0": "NAME", "1-0": "LENGTH", "2-0": "OFFSET", "3-0": "LINEBASES", "4-0": "LINEWIDTH", "0-1": "Name of the reference sequence.", "1-1": "Total length of this reference sequence, in bases.", "2-1": "Offset (in terms of lines) within the FASTA file of the sequence's first base.", "3-1": "The number of bases on each line.", "4-1": "The number of bytes in each line, including the newline." }, "cols": 2, "rows": 5 } [/block] A FAI file can be generated from a FASTA file using one of the suitable tools on the CGC, for example: * **SAMtools Index FASTA** * **SBG FASTA Indices** ##DICT files A DICT file describes the content of the corresponding FASTA file. It contains information about contigs in the FASTA file and their sizes (in number of bases). A DICT file must have the same name as its corresponding FASTA file, but the extension must be **.dict** instead of **.fasta** (or **.fa**). For example, if there is a file named **reference.fasta**, the corresponding DICT file needs to be **reference.dict**. Tools on the CGC that can be used to generate a DICT file from a FASTA file are, for example: * **Picard CreateSequenceDictionary** * **SBG FASTA Indices** ##BAI files BAM files can also be accompanied by an index file that has the same name as the BAM file, suffixed with **.bam.bai**. This file allows (analysis) tools to jump directly to specific alignment lines of the BAM file without starting from the first line and visiting all of the lines in between. A BAI file does not have any purpose without a corresponding BAM file since it does not actually contain any sequence data. If you have a BAM file that does not have an accompanying index (BAI) file, the BAI file can be generated using one of the adequate tools from the CGC, for example: * **SAMtools Index BAM** * **Picard BuildBamIndex** * **Sambamba Index** However, the most important prerequisite for generating an index (BAI) is that **the BAM file is coordinate-sorted**. [block:callout] { "type": "warning", "body": "A BAI (index) file cannot be generated for an unsorted BAM file." } [/block] The BAI file is structured so that it contains two coordinates for every compressed block of a BAM file. * One coordinate states precisely the offset of the location of the compressed block within the compressed BAM file * The other coordinate is the offset to the location from within the block to the particular alignment line that we want to access. ##CRAI files CRAI files are external index files for files in the CRAM format (a compressed version of BAM that stores only reads different from the reference sequence). CRAI files follow the same naming convention as the aforementioned index file formats - they have the same name as the corresponding CRAM file, suffixed with **.cram.crai**. ##TBI files TBI (tabix) format is the generic index file format for a lot of widely used tab-delimited formats such as GFF/GTF, BED, SAM, VCF etc. Tabix files have the **.tbi** extension, which is appended to the name of the file for which the TBI index is generated (for example, a file named **file.vcf.bgz** will have a tabix index file named **file.vcf.bgz.tbi**). A tabix file can be generated from a tab-delimited file that is **position-sorted** and **zipped by bgzip**. To zip a tab-delimited file by **bgzip** and generate a tabix file on the CGC, you can use tools such as: * **Tabix BGZIP** (compresses the tab-delimited file using bgzip). * **Tabix Index** (generates a tabix index file from a tab-delimited file that is position-sorted and has previously been zipped by bgzip). ##IDX files IDX format is the index format used for indexing of VCF files. The purpose of IDX files is to allow tools to have indexed random access to VCF files. VCF index (IDX) files have the **.idx** extension which is appended to the name of the corresponding VCF file (a file named **sample.vcf** would have an index file named **sample.vcf.idx**).