More Finite Generation System

[datacrystals]

I've been thinking of editing the generation pipeline to be a bit more finite (as in scene by scene, rather than chapter by chapter). Curious to get any thoughts or opinions on that.

[LoggeL]

Will definitely increase the quality of single scenes but might cause a disjoint regarding the chapter. Some detail information doesn't get carried along and gets made up by the LLM. This this causes it to sometimes produce conflicting details or to repeat the same details over multiple chapters.

[datacrystals]

That's true - I've never worked with RAG before myself, so this might take some tinkering to get working. On the flip side, I think it'll yield much better results since there will be probably longer output, with more opportunity for world-building. I'd also like to have it set the tone for each scene, so that they have some differences between scenes.

For now, I want to fix the per-chapter generation issues, and get this version mostly working, then I'll dive into attempting to implement this.

[LoggeL]

Another more brute force method would be to fill the input context window with previous chapters which could help the model maintain the writing style and carry information along a bit better.
On the downside this will increase cost, decrease speed and might cause context loss problems.

[datacrystals]

I've tried this with abysmal results unfortunately - In previous iterations of this project, I just fed it the whole story in the context window, but it just didn't really add anything. It seems that even if models have '128k' token context window, they don't actually pay attention to all of it. It seems like they really only listen to the last 5-10k (ish) tokens.

Could just be the models that I'm using though. I had to copy a lot of them off HF to OLLAMA, and I'm not sure if they have been setup right for OLLAMA, since I don't truthfully know how to use OLLAMA very well. I might've screwed up and accidentally limited the context window to some short amount, which would explain a lot.

[LoggeL]

Don't worry. Most LLMs show a loss in the middle effect https://arxiv.org/abs/2307.03172
So I guess feeding the last chapter back might help but more than that is probably counter productive.

[datacrystals]

Very interesting! Thanks for sharing. Yeah we should probably focus on implementing RAG eventually but that's a later problem :)