Skip to content

Polkadot state JSON not parsable #87

Description

@ggwpez

When I run npm start with a Polkadot archive node locally, it results in:

node:buffer:593
    slice: (buf, start, end) => buf.utf8Slice(start, end),
                                    ^
Error: Cannot create a string longer than 0x1fffffe8 characters
    at Object.slice (node:buffer:593:37)
    at Buffer.toString (node:buffer:811:14)
    at Object.readFileSync (node:fs:497:41)
    at main (/home/vados/fork-off-substrate/index.js:123:31)
    at processTicksAndRejections (node:internal/process/task_queues:96:5) {
  code: 'ERR_STRING_TOO_LONG'
}

NodeJS only allows for strings to be 512 MiB, currently the state of Polkadot seems to be larger that that.
Example file.

There are some async parsing libs, but they are slow and I dont know how to write the state back to disk with them either...

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions