Adding draft extension for host-provided scratch memory

[jatinchowdhury18]

No description provided.

[abique]

Hi @jatinchowdhury18

I have some questions with the interface.

1. Is this scratch memory thread-local?
2. Can you get the scratch memory pointer from the thread pool? If so is it shared amongst the all the tasks, or each one will use the current thread local one?

[defiantnerd]

Hi @jatinchowdhury18

I have some questions with the interface.

1. Is this scratch memory thread-local?
2. Can you get the scratch memory pointer from the thread pool? If so is it shared amongst the all the tasks, or each one will use the current thread local one?

The only constraint is that the memory is valid for a plugin during it's process call. Like the event queue or the audio buffers. No further lifetime guarantees are given.

[abique]

Hi @jatinchowdhury18
I have some questions with the interface.

1. Is this scratch memory thread-local?
2. Can you get the scratch memory pointer from the thread pool? If so is it shared amongst the all the tasks, or each one will use the current thread local one?

The only constraint is that the memory is valid for a plugin during it's process call. Like the event queue or the audio buffers. No further lifetime guarantees are given.

Then consider this:

• each voice needs 10K of scratch buffer
• you process 32 voices in parallel

If you have a single scratch buffer for process(), you can't process your voice in parallel, or you need 32 * 10K.
If you have a thread local scratch that you can query from the thread pool, then you can process your voices in parallel with a scratch buffer of 10K.

I believe this needs to be clarified in the spec.

[defiantnerd]

Then consider this:

• each voice needs 10K of scratch buffer
• you process 32 voices in parallel

If you have a single scratch buffer for process(), you can't process your voice in parallel, or you need 32 * 10K. If you have a thread local scratch that you can query from the thread pool, then you can process your voices in parallel with a scratch buffer of 10K.

I believe this needs to be clarified in the spec.

You can only request/register one buffer per instance with a predefined size during activation. The host signals in the return value of request_size if you can access that.

In the process call you just call auto *scratchmem = ext_scratch_memory->access(host); and if scratchmem is not a null you can just use it during the process call.

But there is only one buffer and the size does not change as long as the plugin is active.

[baconpaul]

Then consider this:

• each voice needs 10K of scratch buffer
• you process 32 voices in parallel

If you have a single scratch buffer for process(), you can't process your voice in parallel, or you need 32 * 10K. If you have a thread local scratch that you can query from the thread pool, then you can process your voices in parallel with a scratch buffer of 10K.
I believe this needs to be clarified in the spec.

You can only request/register one buffer per instance with a predefined size during activation. The host signals in the return value of request_size if you can access that.

In the process call you just call auto *scratchmem = ext_scratch_memory->access(host); and if scratchmem is not a null you can just use it during the process call.

But there is only one buffer and the size does not change as long as the plugin is active.

I think Alex's point is if you use the thread pool extension to schedule jobs, those thread pool local jobs will be effectively parallel and running under process.

Can a thread-pool extension job access memory? If so is it distinct per thread or is it a single memory location?

My guess is: The thread pool and memory scratch extensions need some careful co-consideration. And the patterns where people use the scratch memory will also require scratch-per-thread-voice not scratch-per-process-block in those cases.

[jatinchowdhury18]

Yeah, thanks for bringing this up, I hadn't considered the inter-operation of scratch-memory and the thread-pool extension.

I think the simplest solution would be to have the plugin request the total amount of scratch memory that it needs across all possible threads. Since thread_pool.request_exec is blocking, I think it should be safe for the thread pool jobs to use the scratch memory, provided the plugin makes sure that each thread is using an independent "chunk" of the memory. So a simple example would look like:

struct My_Plugin
{
    size_t scratch_mem_per_voice = 10'000;
    size_t num_voices = 32;
    char* scratch_memory_data = nullptr;

    void activate(...) {
        scratch_memory_ext.pre_reserve(host, scratch_mem_per_voice * num_voices);
    }

    void process(...) {
        // Get all the scratch memory here
        scratch_memory_data = scratch_memory_ext.access(host);
        thread_pool_ext.request_exec(host, num_voices);
    }

    void thread_pool_callback(uint32_t task_index) {
        // Get a partition of the scratch memory for this voice to use
        char* this_voice_scratch_mem = scratch_memory_data + scratch_mem_per_voice * task_index;
        // do the actual DSP work here...
    }
};

However, I can see a few reasons why this might not be an ideal solution... For example, if the host thread pool only has 8 threads, then reserving enough scratch memory for 32 voices to be processed in parallel is a bit wasteful. Maybe the best solution is to have two scratch memory mechanisms... one scratch buffer that is intended to be accessed during the process() callback, and another to be accessed during the thread_pool_exec() callback?

typedef struct clap_host_scratch_memory {
    bool(CLAP_ABI *pre_reserve_process_scratch)(const clap_host_t *host, size_t scratch_size_bytes);
    void*(CLAP_ABI *access_process_scratch)(const clap_host_t *host);

    bool(CLAP_ABI *pre_reserve_thread_pool_scratch)(const clap_host_t *host, size_t scratch_size_per_thread_bytes);
    void*(CLAP_ABI *access_thread_pool_scratch)(const clap_host_t *host, uint32_t task_index);
} clap_host_scratch_memory_t;

All that said, I don't have much experience with the thread-pool extension, so I'll defer to your more informed opinions :).

[abique]

Here is my proposal:

The scratch is a thread local pointer, so if you retrieve it from the process call, you'll get a pointer that you can share with all the jobs. If you retrieve it from the thread pool, you get a pointer that is only for the current job.

I think this is the correct direction because it corresponds to how the host will implement this feature: each audio thread will have a single scratch buffer (thread local) whose size is greater or equal to the max requested size of all plugin instances.

The total scratch memory is: num_threads * max_scratch_size.
You definitelly don't want max_scratch_size = max(plug_num_voices * plug_voice_scratch_size), but we want instead max_scratch_size = max(plug_voice_scratch_size).

[Trinitou]

Do we ever use size_t in any other extension? -> uint?_t ?

[abique]

Good point, uint32_t would do the job I think.

[baconpaul]

Why not use uint64_t which is size_t in most of our systems? The host has the option to return no for values out of bounds

[jatinchowdhury18]

I don't have a preference for uint32_t vs uint64_t... but Trinitou is right that CLAP doesn't use size_t anywhere else, so I don't think we should use it here.

[abique]

A scratch size bigger than 4 GB would be problematic I suppose, remember nthreads * max_scratch_size.
Anyway regardless of the type, many host will likely have their own threshold.
uint32_t seems sufficient to me, but I'm happy with uint64_t as well.

[jatinchowdhury18]

The scratch is a thread local pointer, so if you retrieve it from the process call, you'll get a pointer that you can share with all the jobs. If you retrieve it from the thread pool, you get a pointer that is only for the current job.

This makes sense! I've added some comments intended to clarify this point, but please let me know if there are ways I can improve my explanation :).

[Trinitou]

It looks pretty good to me now. Any further details can be discussed on next IMO.

[abique]

"but it's content isn't persistent between callbacks."

It could be confused with it's allocation that isn't persistent.

[abique]

Both optimization: space and CPU cache are valid regardless of the available memory on the system.

[abique]

then this method must not be called and will return NULL.

[p--b]

suggestion: I think [being-activated] should be documented somewhere, because I believe it is necessary for correctness here (assuming it means that the plugin may only call this whilst it is in the call stack of a call to clap_plugin.activate by the host): otherwise there could be a race between calls to reserve and access?

[p--b]

question: does this imply that if a plugin transitions activated -> deactivated -> being-activated, in the final being-activated state it must re-request scratch memory via a call to reserve? Or is the prior reservation still valid?

[p--b]

thought: If I understand correctly, I think this could be challenging for a host to implement efficiently, as I'm not sure the host necessarily knows during the call to reserve how many concurrent jobs a plugin is going to request during process, and therefore how much memory needs to be allocated if it wants to be able to run every job in parallel. Consequently, the host would be forced to allocate the worst-case amount of requested memory for every thread pool thread it creates, which might be much more memory than is actually required.

As an extreme example, if the plugin knows it is never going to create thread pool jobs, then all of the per-thread scratch buffers allocated by the host for that plugin are wasted.

I wonder if it would be worth augmenting reserve to have a uint32_t concurrent_buffers argument, which;

• If 0, means that all tasks receive the same buffer from access and it is up to the plugin to deal with that as it wishes (perhaps even a feature)
• If n > 0 means that task indices up to n - 1 will successfully receive a buffer pointer upon calling access, and task indices n and upwards may receive NULL?