Asynchronicity and user APIs

[tmedwards]

One of SugarCube v3's, aspirational, goals is to enable the use of IndexedDB as a storage provider. Unfortunately, since IndexedDB only offers an asynchronous API, this has consequences for any SugarCube API that provides or depends on storage.

Internally, this isn't an issue. For user facing APIs, however, this could have serious consequences. For example, the recall() API would be seriously changed by this:

// Current synchronous usage.
<<set $foo to recall('foo', 'bar')>>

// Asynchronous usage directly using the returned `Promise`.
<<set recall('foo', 'bar').then(value => $foo to value)>>

That's not a change that I think most users would be happy to see—the non-technical ones, at the very least.

Try to find a solution or workaround for this that's more user friendly.

EDIT: Addendum

There's also the possibility of using await, however, that's only slightly less bad for users. Assuming it could be made to work in the context it would execute within in the first place.

// Asynchronous usage `await`'ing the returned `Promise`.
<<set $foo to await recall('foo', 'bar')>>

[HiEv]

It seems like the solution would be to add an asynchronous version of those calls (or make it a new parameter in the function call to make it asynchronous), and make the current synchronous calls just "await" until the event completes to resume executing that code.

That would preserve backwards compatibility and keep coding simple, while the existence of the asynchronous versions would allow the option for more advanced coders to further optimize their code.

[tmedwards]

[…] make the current synchronous calls just "await" until the event completes to resume executing that code.

Perhaps I'm misunderstanding what you meant, because I don't see how that helps.

The await operator is only valid within async scopes—i.e., the top-level scope of either an async function or ES module. Attempting to use it outside of that context is a syntax error.

Changing recall()'s signature to async function recall(…), so that the asynchronous APIs used within can be await'd, only punts where the Promise eventually comes from. Instead of being explicitly returned from within the function, it would be implicitly returned because the function itself is now async.

E.g., Assuming that State.metadata is an asynchronous API, the following are functionally equivalent, both returning a Promise.

function recall(key, defaultValue) {
	return State.metadata.has(key)
		.then(hasKey => hasKey ? State.metadata.get(key) : defaultValue);
}

async function recall(key, defaultValue) {
	const hasKey = await State.metadata.has(key);
	return hasKey ? await State.metadata.get(key) : defaultValue;
}

[HiEv]

Possibly you're taking me a bit too literally?

I just mean that, if you have code like this:

<<set $foo = "unset">>
<<set $foo = recall("foo", "bar")>>
<<= $foo>>

then that synchronous version of recall() would simply not continue on to the next line of code (a.k.a. it would "await") until that recall() resolves, thus that code would actually output the correct value at the end.

On the other hand, lets say that recallA() is the asynchronous version of recall(), if you did this:

<<set $foo = "unset">>
<<set $foo = recallA("foo", "bar")>>
<<= $foo>>

then that would display "unset", though $foo would eventually be set to the correct value when that recallA() resolves.

As for handling asynchronous functions in SugarCube, you might want to have an <<asynch>> macro which works something like this:

<<asynch recallA("foo", "bar")>>
	<<set $foo = $asynchValue>>  /* $asynchValue = "fulfilled" value */
<<rejected>>
	<<set $foo = $asynchValue>>  /* $asynchValue = "rejected" value */ 
<<catch>>
	<<set $foo = $asynchValue>>  /* $asynchValue = "error" value */
<<finally>>
	<<replace "#foo">>$foo<</replace>>
<</asynch>>

That should allow developers to handle the full range of Promise results without having to resort to JavaScript. You could give it an "await" parameter if you wanted that macro to resolve before going on to the next section of code, otherwise it would proceed immediately to the next macro or line of text/markup in that code.

[tmedwards]

What synchronous version? If the data layer becomes asynchronous, then anything depending on it becomes asynchronous. There would be no synchronous version in that scenario.

If that wasn't the case there'd be no issue in the first place.

[HiEv]

For review, I suggested adding an asynchronous version of the calls, so that there would be both synchronous and asynchronous versions of those calls.

And no, just because the calls become asynchronous internally does not mean that the code has to act asynchronously externally, i.e. from the point of view of someone developing their own code using SugarCube.

Obviously producing the appearance of synchronous code would be somewhat difficult, since you'd have to wait to resume processing the code only after each internally asynchronous call resolves when using the "synchronous" version of a function. However, if you do that, then, from the perspective of a person developing with SugarCube code, it would effectively be synchronous code.

I mean, I assume that the code is run through your own interpreter, correct? So the interpreter would have to modify it so that the code would act as though it was synchronous.

Do you understand what I'm getting at now?

[tmedwards]

For review, I suggested adding an asynchronous version of the calls, so that there would be both synchronous and asynchronous versions of those calls.

I know what you suggested. Again, there would be no synchronous version. If a synchronous version were possible, then this, again, would not be an issue.

And no, just because the calls become asynchronous internally does not mean that the code has to act asynchronously externally, i.e. from the point of view of someone developing their own code using SugarCube.

Please explain, in detail, how that is supposed to work.

I mean, here's what I know. You cannot simply make an asynchronous function appear to be synchronous. You have to deal with the returned Promise in one way or another, either directly via its methods or via await. There's no magic wand to wave that requirement away. If a function has to return something from an asynchronous data layer, then you cannot decouple that asynchronicity from the returned value.

I mean, I assume that the code is run through your own interpreter, correct? So the interpreter would have to modify it so that the code would act as though it was synchronous.

No. The only interpreter is JavaScript. SugarCube's TwineScript has never been more than a thin layer of sugar on top of it—not to mention that recall() could be called from pure JavaScript anyway; e.g., Story JavaScript, <<script>> macros, etc.

Do you understand what I'm getting at now?

Not really, no.

[HiEv]

Please explain, in detail, how that is supposed to work.

OK, let's say that the user has this code in their passage:

<<set $foo = "unset">>
<<set $foo = recall("foo", "bar")>>
<<= $foo>>

What you would do once you got to the synchronous version of the recall() function there, would be to wait for the return of the promise internally to get that data, and then, once you got that, resume processing of that code. So it would execute the code up to the point of the recall(), do the recall(), then once the recall() completed, resume running the code by setting the value of $foo and running the rest of the code after that. Thus $foo would be set to the same value as it would have been set to using the current version of recall().

Yes, internally the code would be asynchronous, but as far as the average Twine developer goes, it would be synchronous.

Basically, you just need to hold off on executing any other code at the point where an internally asynchronous function like that is called until after that asynchronous function completes, and then you can resume executing the rest of the code, thus allowing it to act synchronously, at least as far as the developer using SugarCube is concerned.

Internally you could represent that as running the rest of the code from inside the .then() part of the Promise. How easy that is for you to do, I admit that I don't know, but it would only need to be done for a handful of uncommonly used cases.

Do you get what I'm suggesting now?