Adding option to compress store before saving it to localstorage? #2575
Replies: 2 comments
-
That's exactly one of the reasons that we deprecated zustand/src/middleware/persist.ts Lines 97 to 105 in 018358c So, create such a storage to match the API and pass it. zustand/src/middleware/persist.ts Lines 18 to 24 in 018358c |
Beta Was this translation helpful? Give feedback.
-
Thanks for the suggestion. I'm doing it like the following now, any suggestions? In the end the compression is worse than what is being logged, because the result is stringified for the storage. But still better than stringifying the ArrayBuffer for storage, which blows it up even more. import type { PersistStorage, StateStorage, StorageValue } from 'zustand/middleware/persist';
const readAllChunks = async (readableStream: ReadableStream): Promise<string> => {
const reader = readableStream.getReader();
let result = '';
const pump = async (): Promise<string> => {
const { value, done } = await reader.read();
if (done) {
return result;
}
result = value.reduce((acc: string, code: number) => (
`${acc}${String.fromCharCode(code)}`
), result);
return pump();
};
return pump();
};
const compress = async (value: ArrayBuffer): Promise<string> => {
// eslint-disable-next-line no-undef
const cs = new CompressionStream('gzip');
const writer = cs.writable.getWriter();
writer.write(value);
writer.close();
return readAllChunks(cs.readable);
};
const decompress = async (data: string): Promise<ArrayBuffer> => {
const byteArray = new Uint8Array(data.split('').map((char) => char.charCodeAt(0)));
// eslint-disable-next-line no-undef
const cs = new DecompressionStream('gzip');
const writer = cs.writable.getWriter();
writer.write(byteArray);
writer.close();
return new Response(cs.readable).arrayBuffer();
};
export interface CompressedJSONStorageOptions<S> {
arrayBufferFields: (keyof S)[],
}
export function createCompressedJSONStorage<S>(
getStorage: () => StateStorage,
options: CompressedJSONStorageOptions<S>,
): PersistStorage<S> | undefined {
let storage: StateStorage | undefined;
try {
storage = getStorage();
} catch (error) {
// prevent error if the storage is not defined (e.g. when server side rendering a page)
return undefined;
}
const persistStorage: PersistStorage<S> = {
getItem: async (name) => {
const parse = async (str: string | null) => {
if (str === null) {
return null;
}
const raw = JSON.parse(str);
const uncompressedFields = (await Promise.all(
options.arrayBufferFields.map(async (fieldKey: keyof S): Promise<[keyof S, ArrayBuffer]> => ([
fieldKey,
await decompress(raw.state[fieldKey]),
])),
));
const state = {
...raw.state,
...Object.fromEntries(uncompressedFields),
};
return {
...raw,
state,
} as StorageValue<S>;
};
const str = await (storage as StateStorage).getItem(name) ?? null;
return parse(str);
},
setItem: async (name, newValue) => {
const compressedFields = (await Promise.all(
options.arrayBufferFields.map(async (fieldKey: keyof S): Promise<[keyof S, string]> => {
const compressed = await compress(newValue.state[fieldKey] as ArrayBuffer);
const from = (newValue.state[fieldKey] as ArrayBuffer).byteLength;
const to = compressed.length;
console.log(`"${fieldKey as string}" compressed from ${from}B to ${to}B. (reduced size by ${(100 - (to / from * 100)).toFixed(2)}%)`);
return [
fieldKey,
compressed,
];
}),
));
const state = {
...newValue.state,
...Object.fromEntries(compressedFields),
};
return (storage as StateStorage).setItem(
name,
JSON.stringify({
...newValue,
state,
}),
);
},
removeItem: (name) => (storage as StateStorage).removeItem(name),
};
return persistStorage;
} it's being called like // options for persist
{
// ...
storage: createCompressedJSONStorage(() => localStorage, { arrayBufferFields: ['fieldName1', 'fieldName2'] }),
} |
Beta Was this translation helpful? Give feedback.
-
Storytime
In recent applications I often stored huge amounts of data in localstorage, and occasionally chrome decided for some of my users who do not use the application very often to clear these items.
For an older application, I could mitigate this issue partially by compressing the data before storing it.
Right now I'm building an application where the usere needs to store a 1MB binary file in the store, which also needs to be persisted. (currently using zustand@4.5.2)
This value is an arraybuffer created from
file.arraybuffer()
Using persist without any options causes serialization to turn the arraybuffer into an empty object, so my current "workaround" was to use the deprecated
serialize
/deserialize
functions to convert it into a regular array and back to Arraybuffer when deserializing.This works but causes the item in localstorage to blow up to over 3MB. 😅
So from there I went a bit further and decided to use
Compressionstream
/Decompressionstream
with gzip to reduce the size of the store. But the compressed data is still an array, so this brings the size back down to (at least) ~1.2MB...I tried a bit reducing the size further by converting the compressed numeric data to a string, but was not really successfull.
Question:
Do you think it makes sense to add an option like
compress: true
to the options of the persist middleware?It would clearly make it more difficult to analyze the store data using devtools, so it should be optional, but I think some weird users (like me) might benefit from it.
If you think it's too niche, maybe you can provide an example which is not using
serialize
/deserialize
to achieve the same thing?Beta Was this translation helpful? Give feedback.
All reactions