Cursor with results size > 1Mb

[ikarius]

I'm trying to fetch a result that apparently exceeds the 1Mb default size of 'maxBatchBytes' run command option.

Apparently trying to operate a cursor with datasets with a size > 1mb, is blocking ('with-open' may have closed connection and batch becomes unavailable?)

And by the way, is there any way to pass options to the 'run' command
like the ones described at: http://www.rethinkdb.com/api/javascript/run/ ?

Am I trying to do this in a wrong way ?

Here's a sample which works with limit of 400 (<1mb, plain records) and does not with limit >= 480 (>1mb, cursor returned, result stalled forever):

(map identity
     (with-open [conn (rt/connect :db "ee_classifier")]
       (-> (rt/db "ee_classifier")
           (rt/table "text_to_translate" )
           (rt/limit 500 )
           (rt/pluck :ner )
           (rt/run conn ))))

[ikarius]

If I add:
(rt/coerce-to "array")
it's ok, I get a full array

But apparently, the cursor mechanism hangs in the creation of the lazy-seq...

[danielcompton]

This looks like the same issue as #69? What happens is that RethinkDB returns different result types based on the size of the results. We should be returning a cursor interface to all of these results, but currently small results are returned as objects and big results are returned as cursors. This is not ideal and will be changing. As far as the cursor hanging in lazy-seq, can you try running your query and enabling DEBUG logging on rethinkdb.net in logback.xml? You should see whether results are coming over the wire or not.

Cursors need to be consumed to request more results, what were you doing when you found it hanging?

[ikarius]

For instance, a 'map' or a 'first' operation on the cursor will block.

I'll activate logging, and will keep you posted...