Hosting Mongo server for data for this week challenge

[ianks]

Due to the size of the data in this challenge, I figured some people would have some issues getting it indexed/hosted in Mongo in a reasonable time. I went ahead and started a Mongo server with all of the data hosted.

You can access it like so:
$ mongo --host 198.199.113.194

[ianks]

cc: @CSCI-4830-002-2014/students

[BrianNewsom]

Thanks Ian! This is great. Also I just used

$ mongo 198.199.113.194

to access it, the -h flag brought up help for my windows machine.

[ianks]

Good call! Updating post now.

[indiesquidge]

It was working moments ago just as mongo 198.199.113.194, but now it won't work at all. I get the error:

MongoDB shell version: 2.6.5
connecting to: 198.199.113.194/test
2014-11-14T13:51:32.511-0700 warning: Failed to connect to 198.199.113.194:27017, reason: errno:61 Connection refused
2014-11-14T13:51:32.511-0700 Error: couldn't connect to server 198.199.113.194:27017
(198.199.113.194), connection attempt failed at src/mongo/shell/mongo.js:148
exception: connect failed

I think the update must have changed something. Are you sure the IP stayed the same?

[ianks]

I just upgraded mongo, for some reason the Debian package was at 2.0, now we at 2.6. Should be good now. Can you confirm it works?

[indiesquidge]

👍 Thanks!

[antsankov]

Awesome stuff!

[JoshFerge]

you da best

[alne4294]

Thank you!

[mynameisfiber]

👍

[dawsbot]

Can't...

> db.reddit.find({},{}).sort({ups:1})
error: {
        "$err" : "Runner error: Overflow sort stage buffered data usage of 33555127 bytes exceeds internal limit of 33554432 bytes",
        "code" : 17144

        "code" : 17144
}

[ianks]

@dawsonbotsford Try throwing a .limit(10) on there.

[mynameisfiber]

You could always limit it to comments from one particular day in the dataset. This'll add potential biases that you can think about and find ways to get around (maybe redo the calculation for different days and check that the conclusions are approximately the same)

[dawsbot]

Where do I find the weather data on this server?

[ianks]

I don't believe anyone had uploaded that yet. Feel free to do so in a new collection, though.

[dawsbot]

Peyman added it in JSON format in collection weather

[dawsbot]

How are we supposed to convert the csv to json? Peyman and I are dead in the water

[BrianNewsom]

Import it as CSV using --type CSV and --headerline
On Nov 16, 2014 6:58 PM, "Dawson Botsford" [email protected] wrote:

How are we supposed to convert the csv to json? Peyman and I are dead in
the water

—
Reply to this email directly or view it on GitHub
#3 (comment)
.

[dawsbot]

Do we need to build our own headerline?

[BrianNewsom]

If you look at the file, the first line is a header...

http://docs.mongodb.org/manual/reference/program/mongoimport/

is what I referenced.

[ianks]

So I ended up booting an instance with 32CPUs and 32GB of memory to deal with these queries....

$ mongo --host 104.236.191.166