Load all GEO GDS datasets

[JTFouquier]

We need to build a tool to load all GEO GDS datasets (GDS datasets contain curated metadata) It should support incremental update.

Useful resources:

• http://www.ncbi.nlm.nih.gov/geo/info/geo_paccess.html
• http://www.bioconductor.org/packages/release/bioc/html/GEOquery.html

---

• Bitbucket: https://bitbucket.org/sulab/biogps_core/issue/37
• Originally reported by: Chunlei Wu
• Originally created at: 2013-11-20T00:39:55.422

[JTFouquier]

An alternative is to load datasets from ArrayExpress (they imports dataset from GEO (plus other resources) and did the manual curations).

Related code examples:

http://pythonhosted.org/bioservices/references.html#bioservices.arrayexpress.ArrayExpress

---

Original comment by: Chunlei Wu

[JTFouquier]

Ref: http://www.ebi.ac.uk/arrayexpress/help/programmatic_access.html

To get a list of Experiments:

• by array type id
  http://www.ebi.ac.uk/arrayexpress/json/v2/files?array=A-AFFY-33

full list of array types can be obtained here:
ftp://ftp.ebi.ac.uk/pub/databases/microarray/data/array/

• by species
  http://www.ebi.ac.uk/arrayexpress/json/v2/experiments?species="homo sapiens"

For a given Experiment (using the id returned from above queries):

• get the list of files associated with this Experiment
  http://www.ebi.ac.uk/arrayexpress/json/v2/files/E-MEXP-31

  *.sdrf.txt file is the sample description file   
  *.processed.*.zip file is the processed data file.   
  

• get the metadata
  http://www.ebi.ac.uk/arrayexpress/json/v2/experiments/E-MEXP-31

---

Original comment by: Chunlei Wu

[JTFouquier]

ArrayExpress experiment loaded from GEO has the ID pattern like this one:

E-GEOD-32474 <--> GSE32474

---

Original comment by: Chunlei Wu

[JTFouquier]

When making web-service calls, consider using a library like httplib2, with the support of local caching (so that avoid hitting web services too much during the development).

---

Original comment by: Chunlei Wu