Parsing Large JSON with NodeJS

Recently I was tasked with parsing a very large JSON file with Node.js   Typically when wanting to parse JSON in Node its fairly simple. In the past I would do something like the following.

const fs = require('fs');
const rawdata = fs.readFileSync('file.json');
const data = JSON.parse(rawdata);

Or even simpler with a require statement like this

const data = require('./file.json');

Both of these work great with small or even moderate size files, but what if you need to parse a really large JSON file, one with millions of lines, reading the entire file into memory is no longer a great option.

Because of this I needed a way to “Stream” the JSON and process as it went.  There is a nice module named ‘stream-json’ that does exactly what I wanted.

With stream-json, we can use the NodeJS file stream to process our large data file in chucks. 

const StreamArray = require( 'stream-json/streamers/StreamArray');
const fs = require('fs');

const jsonStream = StreamArray.withParser();

//internal Node readable stream option, pipe to stream-json to convert it for us
fs.createReadStream('file.json').pipe(jsonStream.input);

//You'll get json objects here
//Key is the array-index here
jsonStream.on('data', ({key, value}) => {
    console.log(key, value);
});

jsonStream.on('end', ({key, value}) => {
    console.log('All Done');
});

Now our data can process without running out of memory, however in the use case I was working on, inside the stream I had an asynchronous process.  Because of this, I still was consuming huge amounts of memory as this just up a very large amount of unresolved promises to keep in memory until they completed.

To solve this I had to also use a custom Writeable stream like this.

const StreamArray = require( 'stream-json/streamers/StreamArray');
const {Writable} = require('stream');
const fs = require('fs');

const fileStream = fs.createReadStream('file.json');
const jsonStream = StreamArray.withParser();
const processingStream = new Writable({
	write({key, value}, encoding, callback) {

		//some async operations
		setTimeout(() => {
			console.log(key,value);
			//Runs one at a time, need to use a callback for that part to work
			callback();
		}, 1000);
	},
	//Don't skip this, as we need to operate with objects, not buffers
	objectMode: true
});
//Pipe the streams as follows
fileStream.pipe(jsonStream.input);
jsonStream.pipe(processingStream);
//So we're waiting for the 'finish' event when everything is done.
processingStream.on('finish', () => console.log('All done' ));

The Writeable stream also allows each asynchronous process to complete and the promises to resolve before continuing on to the next, thus avoiding the memory backup. 

This stack overflow is where I got the examples for this post.

Also note another thing I learned in this process is if you want to start Node with more than the default amount of RAM you can use the following command.

node --max-old-space-size=4096 file.js

By default the memory limit in Node.js is 512 mb, to solve this issue you need to increase the memory limit using command –max-old-space-size.  This can be used to avoid the memory limits within node. The command above would give Node 4GB of RAM to use.