↳ Browse datasets via the Datasets API

[block:callout] { "type": "danger", "body": "This feature is in our advance access program. This means that, while it is fully operational, it is subject to change.", "title": "Advance Access" } [/block] [block:callout] { "type": "warning", "title": "On this page:", "body": "* [Overview](#section-overview)\n* [Return accessible datasets](#section-return-accessible-datasets)\n* [Return all entities within a dataset](#section-return-all-entities-within-a-dataset)\n* [Return all instances of an entity](#section-return-all-instances-of-an-entity)\n* [Return an entity's metadata schema](#section-return-an-entity-s-metadata-schema)\n* [Next step](#section-next-step)\n* [Resources](#section-resources)" } [/block] ##Overview Browse datasets via the Datasets API by issuing successive `GET` requests. Use these browsing requests individually or in conjunction with [querying](query-via-the-datasets-api). For instance, entities located through browsing are resources which can be the subject of a query. On this page, learn about requests to return: * [datasets you can access](#section-return-accessible-datasets) * [all the entities within a dataset](#section-return-all-entities-within-a-dataset) * [all instances of a single entity](#section-return-all-instances-of-an-entity) * [a single entity's metadata schema](#section-return-an-entity-s-metadata-schema) <div align="right"><a href="#top">top</a></div> ##Return accessible datasets Make the following `GET` request. Be sure to replace the [authentication token](get-your-authentication-token) with your own. [block:code] { "codes": [ { "code": "GET /datasets/ HTTP/1.1\nHost: cgc-datasets-api.sbgenomics.com\nX-SBG-Auth-Token: 7942f56901534434a054dafc3813bc96", "language": "http", "name": "Return accessible datasets" } ] } [/block] This returns a list of accessible datasets, as shown below. [block:code] { "codes": [ { "code": "{\n \"description\": \"Datasets API Advanced Access Program\",\n \"_links\": {\n \"self\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/\"\n },\n \"tcga_grch38\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/tcga_grch38/v0\",\n \"label\": \"TCGA GRCh38\"\n },\n \"ccle\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/ccle/v0\",\n \"label\": \"CCLE Legacy\"\n },\n \"tcga\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/tcga/v0\",\n \"label\": \"TCGA Legacy\"\n },\n \"target\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/target/v0\",\n \"label\": \"TARGET GRCh38\"\n },\n \"tcia\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/tcia/v0\",\n \"label\": \"TCIA\"\n },\n \"cptac\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/cptac/v0\",\n \"label\": \"CPTAC\"\n },\n \"icgc\": {\n \"href\": \"cgc-datasets-api.sbgenomics.com/datasets/icgc/v0\",\n \"label\": \"ICGC\" \n }\n }\n}", "language": "json", "name": "Response body" } ] } [/block] The `href` element lists the path for each dataset, such as `https://cgc-datasets-api.sbgenomics.com/datasets/ccle/v0` for CCLE. <div align="right"><a href="#top">top</a></div> ##Return all entities within a dataset Make a `GET` request to the `href` of a dataset to return all of its entities. Learn more about [each dataset's entities](about-metadata-for-datasets) from its metadata page. [block:code] { "codes": [ { "code": "GET /datasets/tcga/v0 HTTP/1.1\nHost: cgc-datasets-api.sbgenomics.com\nX-SBG-Auth-Token: 7942f56901534434a054dafc3813bc96", "language": "http", "name": "Return all entities within a dataset" } ] } [/block] The response contains a list of entities for the dataset you specified. [block:code] { "codes": [ { "code": "{\n \"_links\": {\n \"cases\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/cases/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/cases\"\n },\n \"analytes\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/analytes/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/analytes\"\n },\n \"radiation_therapies\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/radiation_therapies/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/radiation_therapies\"\n },\n \"drug_therapies\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/drug_therapies/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/drug_therapies\"\n },\n \"follow_ups\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/follow_ups/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/follow_ups\"\n },\n \"portions\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/portions/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/portions\"\n },\n \"aliquots\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/aliquots/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/aliquots\"\n },\n \"samples\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/samples/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/samples\"\n },\n \"slides\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/slides/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/slides\"\n },\n \"query\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/query\"\n },\n \"self\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0\"\n },\n \"files\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files\"\n },\n \"new_tumor_events\": {\n \"schema\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/new_tumor_events/schema\",\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/new_tumor_events\"\n }\n }\n}", "language": "json", "name": "Response body" } ] } [/block] Note that two items in the list returned, `query` and `self`, are not dataset entities: * `self` contains the property `href` which is set to the same path issued in making the query. * `query` contains the property `href` which is set to a path that can be used to issue a query into the entities returned, by making a `POST` request. Learn more about [querying with the Datasets API](query-via-the-datasets-api). <div align="right"><a href="#top">top</a></div> ##Return all instances of an entity Make a `GET` request to the `href` of an entity to return all instances of that entity within the specified dataset. For example, to see a list of all TCGA files, make the following request. [block:code] { "codes": [ { "code": "GET /datasets/tcga/v0/files HTTP/1.1\nHost: cgc-datasets-api.sbgenomics.com\nX-SBG-Auth-Token: 7942f56901534434a054dafc3813bc96", "language": "http", "name": "Return all instances of an entity" } ] } [/block] This returns the following response. As you can see, the response contains 100 results (count) per page. You can page through using the paths under `_links`. The resulting files are listed under the `_embedded` section. [block:code] { "codes": [ { "code": "{\n \"total\": 573241,\n \"count\": 100,\n \"_links\": {\n \"self\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files\"\n },\n \"next\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files?page=1\"\n },\n \"last\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files?page=5732\"\n },\n \"first\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files\"\n }\n },\n \"_embedded\": {\n \"files\": [\n {\n \"label\": \"LIGER_p_TCGA_166_180_SNP_N_GenomeWideSNP_6_G03_896632.ismpolish.data.txt\",\n \"id\": \"564a31a5e4b09c884b215ccb\",\n \"_links\": {\n \"self\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files/564a31a5e4b09c884b215ccb\"\n }\n }\n },\n {\n \"label\": \"6055424037_R05C01_Grn.idat\",\n \"id\": \"564a31a6e4b0298dd2c5492c\",\n \"_links\": {\n \"self\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files/564a31a6e4b0298dd2c5492c\"\n }\n }\n },\n {\n \"label\": \"mdanderson.org_SARC.MDA_RPPA_Core.SuperCurve.Level_2.B06C2A68-8B01-41AC-B3B2-20C7BBFA562B.txt\",\n \"id\": \"564a31a6e4b0298dd2c54934\",\n \"_links\": {\n \"self\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files/564a31a6e4b0298dd2c54934\"\n }\n }\n },\n {\n \"label\": \"PULED_p_TCGA_130_157_N_GenomeWideSNP_6_H10_831788.nocnv_hg19.seg.txt\",\n \"id\": \"564a31a6e4b0298dd2c54936\",\n \"_links\": {\n \"self\": {\n \"href\": \"https://cgc-datasets-api.sbgenomics.com/datasets/tcga/v0/files/564a31a6e4b0298dd2c54936\"\n }\n }\n },\n \n \n <snip>\n \n }\n ]\n }\n}", "language": "json", "name": "Response body" } ] } [/block] <div align="right"><a href="#top">top</a></div> ##Return an entity's metadata schema Make a `GET` request to an entity's href to obtain its schema. Each entity has its own metadata schema, a list of the metadata fields used to describe the entity and the permissible datatypes (strings, integers, etc) that each field takes. Each metadata field, such as `hasDataType` in the request below, is followed by an object which indicates the type of value for each field, such as `integer`, `string`, or `enum`. If the type is given as `enum`, the object also contains a list of all possible values for the given metadata field. Additionally, note that under `_links`, there is a list of connections to other entities in the dataset. In the example below, these connections include `"hasAliquot"`, `"hasCase"`, `"hasSample"`, `"hasPortion"`, and `"hasAnalyte"`. For example, to see the metadata schema for `files` send the request: [block:code] { "codes": [ { "code": "GET /datasets/tcga/v0/files/schema HTTP/1.1\nHost: cgc-datasets-api.sbgenomics.com\nX-SBG-Auth-Token: 7942f56901534434a054dafc3813bc96", "language": "http", "name": "Return an entity's metadata schema" } ] } [/block] This returns the following response which lists all the properties of the `file` entity: [block:code] { "codes": [ { "code": "{\n \"hasSize\": {\n \"type\": \"integer\"\n },\n \"hasDataType\": {\n \"values\": [\n \"Structural rearrangement\",\n \"Protein expression\",\n \"Gene expression\",\n \"Clinical\",\n \"Simple nucleotide variation\",\n \"DNA methylation\",\n \"Other\",\n \"Copy number variation\",\n \"Not available\",\n \"Raw sequencing data\",\n \"Raw microarray data\"\n ],\n \"type\": \"enum\"\n },\n \"id\": {\n \"type\": \"string\"\n },\n \"hasDataSubmittingCenter\": {\n \"values\": [\n \"MD Anderson - Institute for Applied Cancer Science\",\n \"Johns Hopkins / University of Southern California\",\n \"Baylor College of Medicine\",\n \"HudsonAlpha Institute for Biotechnology\",\n \"Harvard Medical School\",\n \"MD Anderson - RPPA Core Facility (Proteomics)\",\n \"Complete Genomics Inc.\",\n \"Washington University School of Medicine\",\n \"NCH BCR\",\n \"Broad Institute of MIT and Harvard\",\n \"Lawrence Berkeley National Laboratory\",\n \"Memorial Sloan-Kettering Cancer Center\",\n \"Wellcome Trust Sanger Institute\",\n \"University of North Carolina\",\n \"Canada's Michael Smith Genome Sciences Centre\",\n \"Not available\",\n \"Nationwide Children's Hospital BCR\",\n \"University of California, Santa Cruz\"\n ],\n \"type\": \"enum\"\n },\n \"hasDiseaseType\": {\n \"values\": [\n \"Adrenocortical Carcinoma\",\n \"Stomach Adenocarcinoma\",\n \"Colon Adenocarcinoma\",\n \"Cervical Squamous Cell Carcinoma and Endocervical Adenocarcinoma\",\n \"Brain Lower Grade Glioma\",\n \"Bladder Urothelial Carcinoma\",\n \"Sarcoma\",\n \"Lung Adenocarcinoma\",\n \"Testicular Germ Cell Tumors\",\n \"Cholangiocarcinoma\",\n \"Lymphoid Neoplasm Diffuse Large B-cell Lymphoma\",\n \"Thymoma\",\n \"Liver Hepatocellular Carcinoma\",\n \"Thyroid Carcinoma\",\n \"Uveal Melanoma\",\n \"Mesothelioma\",\n \"Kidney Renal Papillary Cell Carcinoma\",\n \"Kidney Renal Clear Cell Carcinoma\",\n \"Kidney Chromophobe\",\n \"Pheochromocytoma and Paraganglioma\",\n \"Acute Myeloid Leukemia\",\n \"Prostate Adenocarcinoma\",\n \"Uterine Corpus Endometrial Carcinoma\",\n \"Esophageal Carcinoma\",\n \"Breast Invasive Carcinoma\",\n \"Skin Cutaneous Melanoma\",\n \"Uterine Carcinosarcoma\",\n \"Pancreatic Adenocarcinoma\",\n \"Glioblastoma Multiforme\",\n \"Ovarian Serous Cystadenocarcinoma\",\n \"Head and Neck Squamous Cell Carcinoma\",\n \"Rectum Adenocarcinoma\",\n \"Lung Squamous Cell Carcinoma\"\n ],\n \"type\": \"enum\"\n },\n \"hasPlatform\": {\n \"values\": [\n \"HG-CGH-244A\",\n \"Ion Torrent PGM\",\n \"BCR Record\",\n \"Not available\",\n \"HG-CGH-415K_G4124A\",\n \"Illumina DNA Methylation OMA003 CPI\",\n \"Illumina HiSeq\",\n \"AgilentG4502A_07_2\",\n \"Complete Genomics\",\n \"AgilentG4502A_07_1\",\n \"H-miRNA_8x15Kv2\",\n \"Hospital Record\",\n \"HiSeq X Ten\",\n \"Illumina DNA Methylation OMA002 CPI\",\n \"LS 454\",\n \"Affymetrix U133 Plus 2\",\n \"Illumina GA\",\n \"AgilentG4502A_07_3\",\n \"MDA_RPPA_Core\",\n \"Illumina HumanHap550\",\n \"ABI capillary sequencer\",\n \"CGH-1x1M_G4447A\",\n \"ABI SOLiD\",\n \"Affymetrix SNP Array 6.0\",\n \"Mixed platforms\",\n \"Illumina MiSeq\",\n \"HT_HG-U133A\",\n \"H-miRNA_8x15K\",\n \"HG-U133_Plus_2\",\n \"HuEx-1_0-st-v2\",\n \"Illumina Human Methylation 27\",\n \"Illumina Human 1M Duo\",\n \"Illumina Human Methylation 450\"\n ],\n \"type\": \"enum\"\n },\n \"hasReferenceGenome\": {\n \"values\": [\n \"GRCh37-lite_WUGSC_variant_2\",\n \"GRCh37\",\n \"HG19\",\n \"GRCh37-lite_WUGSC_variant_1\",\n \"HG18\",\n \"HS37D5\",\n \"GRCh37-lite-+-HPV_Redux-build\",\n \"HG18_Broad_variant\",\n \"NCBI36_BCCAGSC_variant\",\n \"NCBI-human-build36\",\n \"GRCh37_BI_Variant\",\n \"NCBI36_BCM_variant\",\n \"HG19_Broad_variant\",\n \"NCBI36_WUGSC_variant\",\n \"GRCh37-lite\",\n \"Not available\"\n ],\n \"type\": \"enum\"\n },\n \"publishedDate\": {\n \"type\": \"dateTime\"\n },\n \"hasDataSubtype\": {\n \"values\": [\n \"Bisulfite sequence alignment\",\n \"Sequencing tag counts\",\n \"Sequencing tag\",\n \"Isoform expression quantification\",\n \"Genotypes\",\n \"miRNA quantification\",\n \"Copy number variation\",\n \"Simple somatic mutation\",\n \"Raw intensities\",\n \"Protein expression quantification\",\n \"Gene expression summary\",\n \"Methylation percentage\",\n \"Gene expression quantification\",\n \"Structural variation\",\n \"Copy number segmentation\",\n \"Microsattelite instability\",\n \"Not available\",\n \"Intensities Log2Ratio\",\n \"Exon quantification\",\n \"Biospecimen data\",\n \"Probeset summary\",\n \"Simple nucleotide variation\",\n \"Methylation beta value\",\n \"Intensities\",\n \"Exon junction quantification\",\n \"Unaligned reads\",\n \"Copy number estimate\",\n \"LOH\",\n \"Aligned reads\",\n \"Normalized copy numbers\",\n \"Clinical data\",\n \"Normalized intensities\"\n ],\n \"type\": \"enum\"\n },\n \"lastModifiedDate\": {\n \"type\": \"dateTime\"\n },\n \"_links\": {\n \"hasAliquot\": \"anyURI\",\n \"self\": {\n \"href\": \"metadata-api-vayu.sbgenomics.com:9989/datasets/v0/tcga/files/schema\"\n },\n \"hasSample\": \"anyURI\",\n \"hasPortion\": \"anyURI\",\n \"hasAnalyte\": \"anyURI\",\n \"hasCase\": \"anyURI\"\n },\n \"hasAccessLevel\": {\n \"values\": [\n \"Controlled\",\n \"Open\"\n ],\n \"type\": \"enum\"\n },\n \"hasDataFormat\": {\n \"values\": [\n \"BED\",\n \"FA\",\n \"XLSX\",\n \"DGE-TAG\",\n \"IDAT\",\n \"XML\",\n \"TXT\",\n \"BAM\",\n \"BAI\",\n \"DAT\",\n \"CHP\",\n \"MAF\",\n \"CEL\",\n \"TIF\",\n \"TAR\",\n \"FSA\",\n \"Not available\",\n \"GCT\",\n \"VCF\",\n \"TARGZ\"\n ],\n \"type\": \"enum\"\n },\n \"hasInvestigation\": {\n \"values\": [\n \"TCGA-THYM\",\n \"TCGA-KIRC\",\n \"TCGA-SARC\",\n \"TCGA-ESCA\",\n \"TCGA-PCPG\",\n \"TCGA-PRAD\",\n \"TCGA-UCEC\",\n \"TCGA-ACC\",\n \"TCGA-READ\",\n \"TCGA-UVM\",\n \"TCGA-CESC\",\n \"TCGA-COAD\",\n \"Not available\",\n \"TCGA-TGCT\",\n \"TCGA-DLBC\",\n \"TCGA-KICH\",\n \"TCGA-THCA\",\n \"TCGA-HNSC\",\n \"TCGA-UCS\",\n \"TCGA-CHOL\",\n \"TCGA-BLCA\",\n \"TCGA-GBM\",\n \"TCGA-SKCM\",\n \"TCGA-LUSC\",\n \"TCGA-STAD\",\n \"TCGA-LUAD\",\n \"TCGA-LIHC\",\n \"TCGA-KIRP\",\n \"TCGA-BRCA\",\n \"TCGA-MESO\",\n \"TCGA-PAAD\",\n \"TCGA-LAML\",\n \"TCGA-OV\",\n \"TCGA-LGG\"\n ],\n \"type\": \"enum\"\n },\n \"hasStoragePath\": {\n \"type\": \"string\"\n },\n \"hasExperimentalStrategy\": {\n \"values\": [\n \"Exon array\",\n \"CGH array\",\n \"RNA-Seq\",\n \"miRNA-Seq\",\n \"AMPLICON\",\n \"Not available\",\n \"VALIDATION\",\n \"WXS\",\n \"DNA-Seq\",\n \"Total RNA-Seq\",\n \"Gene expression array\",\n \"Protein expression array\",\n \"miRNA expression array\",\n \"Genotyping array\",\n \"Bisulfite-Seq\",\n \"Methylation array\",\n \"WGS\",\n \"MSI-Mono-Dinucleotide Assay\"\n ],\n \"type\": \"enum\"\n },\n \"label\": {\n \"type\": \"string\"\n },\n \"hasSubmitterId\": {\n \"type\": \"string\"\n },\n \"uploadDate\": {\n \"type\": \"dateTime\"\n },\n \"hasGDCFileUUID\": {\n \"type\": \"string\"\n }\n}", "language": "json", "name": "Response body" } ] } [/block] <div align="right"><a href="#top">top</a></div> ##Next step Browsing requests can be used in conjunction with querying. For instance, entities located through browsing are resources which can be the subject of a query. Learn more about [querying via the Datasets API](doc:query-via-the-datasets-api). <div align="right"><a href="#top">top</a></div> ##Resources * [Query via the Datasets API](doc:query-via-the-datasets-api) * [About the Datasets API](doc:about-the-datasets-api) * [About metadata for datasets](doc:about-metadata-for-datasets) <div align="right"><a href="#top">top</a></div>

The CGC Knowledge Center

GET STARTED

TUTORIALS

ACCESS THE CGC

MANAGE AN ANALYSIS ENVIRONMENT

COLLABORATE SECURELY

DATASETS HUB

PUBLIC PROJECTS

BRING YOUR PRIVATE DATA

SET METADATA ASSOCIATED WITH A PRIVATE FILE

SEARCH FILES

ADD FILES TO A PROJECT

ORGANIZE FILES

ARCHIVE FILES

FIND APPS

BRING YOUR TOOLS

DESCRIBE YOUR TOOL

TOOL WRAPPING TIPS AND TRICKS

EDIT AN APP

RUN AN ANALYSIS

TASK EXECUTION

REVIEW A TASK

VIEW ANALYSIS RESULTS

INTERACTIVE ANALYSIS

DOWNLOAD RESULTS

DEVELOPER TOOLS

CONNECT CLOUD STORAGE

API Hub

API Reference

FOLDERS API (Advance Access)

BULK API ACTIONS

WATCH AND LEARN

DATA CRUNCHER

SBFS (Beta)

SBFS Command Reference

GLOSSARY