write and read streams everything you need to know?

node js Streams are collections of data — just like arrays or strings. The difference is that streams might not be available all at once, and they don’t have to fit in memory. This makes streams really powerful when working with large amounts of data, or data that’s coming from an external source one chunk at a time.

1
2
3
4
5
6
7
8


const fs = require('fs');
const file = fs.createWriteStream('./file.txt');
//1e6 = 1000000
for(let i=0; i<= 1e6; i++) {
  file.write('We can Change\n');
}

file.end();

fs.createReadStream() allows you to open up a readable stream in a very simple manner. All you have to do is pass the path of the file to start streaming in. It turns out that the response (as well as the request) objects are streams.

1
2
3
4
5


We can Change
We can Change
We can Change
x1000000 times of line

this will consume you pc memory.

1
2
3
4
5
6
7


const fs = require('fs');

fs.readFile('./file.txt', (err, data) => {
    if (err) throw err;
    console.log(data.toString());
});

The fs.readFile() function buffers the entire file. To minimize memory costs, when possible prefer streaming via fs.createReadStream().

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


const fs = require('fs');
const csv = require('fast-csv');

fs.createReadStream('./file.txt')
    .pipe(csv.parse({ headers: false }))
    .on('error', error => console.error(error))
    .on('data', (data) => {
        console.log(data)
    });

this will read the file.txt

 ['We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
[ 'We want change.' ]
 for 1 million time

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


const fs = require('fs');
const server = require('http').createServer();

server.on('request', (req, res) => {
  fs.readFile('./big.file', (err, data) => {
    if (err) throw err;

    res.end(data);
  });
});

server.listen(8000);

now the above code is not in a stream and would not recommend it in an istance where your dealling with over a million lines or lage file as wi have .

But insteady we can create createReadStream() to create a http server that streams the files to the client. Since the code is simple enough, it is pretty easy just to read through it and comment why each line is necessary

1
2
3
4
5
6
7
8
9


const fs = require('fs');
const server = require('http').createServer();

server.on('request', (req, res) => {
  const src = fs.createReadStream('./file.text');
  src.pipe(res);
});

server.listen(8000);

check out this post from node for more

write and read streams everything you need to know?

See Also