Add option to salesforce for asyncMode: true/false on bulk(...)

[taylordowns2000]

Right here, if the user has specified asyncMode in their bulk options, we should exit, returning batchInfo to state.references.

https://github.com/OpenFn/adaptors/blob/main/packages/salesforce/src/Adaptor.js#L236

If they have not provided this extra asyncMode option, we should continue with the .then() and return all record info to state as we currently do. (No change there.)

[mtuchi]

// Check for asyncMode
if (resolvedOptions.asyncMode === false) {
  // Only fetch batch information and return
  return batch.on('queue', function (batchInfo) {
    console.info('Batch queued:', batchInfo);
    resolve(batchInfo); // Resolve with batch information
  });
} else {
   // Process batch results in async mode
  return batch
    .on('queue', function (batchInfo) {
      console.info('Batch queued:', batchInfo);
      const batchId = batchInfo.id;
      var batch = job.batch(batchId);
      batch.poll(pollInterval, pollTimeout);
    })
    .then(async (res) => {})

[josephjclark]

I don't know if this is practical for large batches of data which get chunked into multiple batches in salesforce.

So far as I can tell, Jsforce (and maybe even salesforce?) does't really give us any notification that all batches have been queued.

So we know when the batch has finished. We know when a batch is added to the queue. But we don't know when all items have been queued.

That means that in async mode, all we can do is upload the data and exit without reporting any batch inf. Which means the user will never be able to check the status of a batch, which is terrible.

The only thing I can suggest is:

• We get or set the size of batch the salesforce will use. Say 1k records per batch
• We listen for each queue event and count them. For 10k records, that's 10 batches
• When we count the expected number of queued batches, and when they all come in, we return them

Maybe @mtuchi you can check: can we set the batch size in jsforce? If we can do that, maybe we can press forward.

Saying that, I do think that updating jsforce has to be the priority and all other fixes can wait