Persist storage save individual state items in a store as a new key-value pair (which in turn is serialized/deserialized) instead of entire state serialized/deserialized

[liamgui]

Hi, this perhaps might not be such a great idea, so I'm open to discussion about how its not feasible, or doesn't make sense.

Question:
Why are we serializing/deserializing and writing the entire state object from a store when we persist as opposed to saving them individually to separate key-value records in the storage?

I understand it is far less complex to just simply serialize the entire state (+ version) and throw that into localStorage. However, I have adopted using IndexedDB with a custom storage (a version where i don't serialize or deserialize, but rather filter out any functions from the state and then save the entire state to IndexedDB.

However, I am using this in a situation where I will have multiple writes of data per second for an increasingly large data set (think procedural generation of a tile map in a game with a game store). Putting aside that I could optimize so that it doesn't write multiple times a second... In my mind, I would really like to have each state property be its own IndexedDB key-value record.

This would enable me to only write updates that pertain to the tile map, instead of the entire state object for that store. Now I understand I probably should have a separation of concerns and move the map stuff to its own store (and now that I think about it, that's not a terrible idea). However, this question still somewhat applies as even then I would have to put just the tilemap in its own store in order to only write to that item when it gets updated.

I haven't done any benchmarking on my custom storage solution, so what I'm doing might be pretty slow and I may just opt for saving the specific state items to the IndexedDB in a set item in an action just for those ones and persist the rest.. 🤷‍♂️

Also, if I were to use the slices pattern where I add it all to one global store, this seems the like problem would compound (I haven't actually tested that yet) since you are now persisting everything together.

Thoughts? (and sorry for the ramble)

[dai-shi]

Custom storage option may help.

[liamgui]

I am using a custom storage. However, by default, it seems that when setItem is called, the "newValue" that gets passed in is the entire state of that store and not just the one property of the state that has changed. That makes it unfeasible to compare what property has actually changed and only write that update.

[liamgui]

I should note that I modeled my storage after the createJsonStorage that comes with zustand.

export function createNonSerializedJSONStorage<S>(
	getStorage: () => StateStorage,
): PersistStorage<S> | undefined {
	let storage: StateStorage | undefined;
	try {
		storage = getStorage();
	} catch {
		// prevent error if the storage is not defined (e.g. when server side rendering a page)
		return;
	}
	const persistStorage: PersistStorage<S> = {
		getItem: (name) => {
			return (storage as StateStorage).getItem(name) ?? null;
		},
		setItem: (name, newValue) => {
			return (storage as StateStorage).setItem(name, newValue);
		},
		removeItem: (name) => (storage as StateStorage).removeItem(name),
	};
	return persistStorage;
}

Then beyond that, I create a storage that gets used as the getStorage() (and uses "idb-keyval") where I use setItem to set from that library.

[liamgui]

Also, I didn't want to serialize/deserialize to/from string for IndexedDB, which is why i created this (since the createJsonStorage automatically serializes/deserializes)

[liamgui]

Just to finish showing what my storage looks like: (this uses lodash's pick to filter out any action/getter functions in the state)

import { del, get, set } from "idb-keyval";
import { pick } from "lodash";
import { PersistStorage, StorageValue } from "zustand/middleware";

export const storage: StateStorage = {
	getItem: async (name: string): Promise<any | null> => {
		return (await get(name)) || null;
	},
	setItem: async (
		name: string,
		value: StorageValue<Record<string, any>>,
	): Promise<void> => {
		const state = pick(
			value.state,
			Object.keys(value.state).filter(
				(key) => typeof value.state[key] !== "function",
			),
		);
		await set(name, { version: value.version, state });
	},
	removeItem: async (name: string): Promise<void> => {
		await del(name);
	},
};

// to create a new StateStorage that will allow for non-string values when using setItem
export interface StateStorage {
	getItem: (name: string) => any | null | Promise<any | null>;
	setItem: (name: string, value: any) => unknown | Promise<unknown>;
	removeItem: (name: string) => unknown | Promise<unknown>;
}

[liamgui]

I guess it would be nice to have setItem to only include the state values that change. Was looking at persist.ts - seems like if we could change setItem to pass in the state (which appears to only be the state items that are changed) here:

zustand/src/middleware/persist.ts

Line 218 in 15eaeb7

void setItem()

Perhaps it could default to the way it is, but with an extra option in the persist middleware, it only sends through the updatedValue(s) from the state

This would allow fine grained control of how we want to handle each state item update individually as opposed to just lumping it all together.

[dominictobias-bullish]

💯 really surprised this is the default and only way I can see to do it since the entire state is passed to the getter and setter of storage. It would be great if we can store on an individual level, because:

1. Scalability

Imagine a ChatGPT and you have to write every chat state on any chat change. Unfeasibly inefficient.

2. Secure vs unsecure storage

We may want to store some items in local/session/indexedDB storage, or secure/insecure in the case of mobile. Secure storage on mobile also has a limit of 2048kb per key. Splitting a store up arbitrarily for this requirement isn't ideal.

I think it stems from redux style state being immutable - if something changes, everything up the tree changes and the whole store is considered changed. It's a bit of a sledgehammer in this age of fine grained reactivity but more in line with React's philosophy. I guess something like Legend State is more suited to this kind of behavior.