Reading and Writing to CSV Files in Node.js

A comma-separated values (CSV) file is a plain text file that stores tabular data. The values are separated by commas. They can be used to collect or store information for various purposes. One of the ways I've used it was to collect user data from clients and then do a bulk update to the database. You may have a different use-case but want to know how to read data from a CSV file and how to output tabular data to a CSV file. In this post, I'll show you how to achieve those objectives with JavaScript, in Node.js.

Getting started

I'll start with initializing a new Node.js project by running npm init -y command which will generate a package.json file. We will be working with the csv-parser package to read CSV files.

The example you will work on in this post will collect some data about users and generate a random password for them. We'll use the random-words package to generate random words to use as a password.

After initializing your package.json, let’s install our dependencies:

npm install csv-parser random-words

Reading CSV files

In order to read a CSV file, we will use the csv() function from the csv-parser library. The function takes a single argument which can either be an options object or an array of strings to use as headers. The first line of a CSV file usually contains the headers, so in most cases you won’t need to specify headers yourself. Passing an options object allows you to configure how the data is parsed. Some of the options you can specify are:

newline: Specifies the character to denote the end of a line in a CSV file. The default is \n.
quote: Specifies a character to denote a quoted string in the file. The default is ”.
separator: Specifies a character to use as the column separator for each row. The default is a comma.
skipLines: Specifies the number of lines at the beginning of the file that the parser should skip over, prior to parsing headers. The default is 0.

We will work with a CSV file that contains names and roles. In the project directory, add a file named input.csv and put the content below in it.

Firstname,Surname,Roles
Dammy,Greg,admin
Bad,Boggie,account
Jean,Lotus,account
Sandy,Begar,sales

Add another file index.js and copy and paste the code below in it.

const fs = require('fs')
const csv = require('csv-parser')
const randomWords = require('random-words')
const users = [];
function generateUsername(firstname, surname) {
    return `${firstname[0]}-${surname}`.toLowerCase();
}
fs.createReadStream('input.csv')
  .pipe(csv())
  .on('data', function (row) {
    const username = generateUsername(row.Firstname, row.Surname);
    const password = randomWords(3).join("-");
    
    const user = {
        username,
        firstname: row.Firstname,
        surname: row.Surname,
        roles: row.Roles,
        password
    }
    users.push(user)
  })
  .on('end', function () {
      console.table(users)
      // TODO: SAVE users data to another file
    })

In the code above, we're referencing the fs module so we can use it to interact with the file system and read data from the file. We are also referencing the csv-parser module which will be used to parse the data in the CSV file, and the random-words module, which is only needed to generate a string of words that’ll be used as password.

The users variable is declared and will be used to hold an array of user objects.

To read the data from the file, we call fs.createReadStream('input.csv') with the path to the file. This will create a stream for which we can read data from and then the data is piped to the csv() function created by the csv-parser module. We didn’t pass any arguments to it, which means that it’ll use the default options when reading and parsing data. The data event is emitted when each row of data is parsed, except for the header row. To read the header row when it’s parsed, you can listen for the headers event. We’re only interested in the row data in our example, so we listen for the data event with a callback function which will:

Receive the parsed row as an argument.
Generate a password by calling randomWords(3).join("-").
Generate a username by calling generateUsername(row.Firstname, row.Surname).
Create a user object with the generated password, username, and row data. Then, add that user object to the users array by calling users.push(user).

The randomWords() function takes an argument that indicates the number of words to generate. It generates words based on that specification and returns an array of strings. We specified that we want three words. This returns an array with three elements. We then call join(“-”) to concatenate them into a single string, with each word separated by -.

When all the data is parsed, the end event is emitted. We listen for this event, and when it’s emitted, it calls our callback function which prints the data to the console.

If you run the code, you should see the generated data printed to the console. Your passwords will be different from this, since they are being randomly generated when the code is run.

(index) username firstname surname roles password 0 'd-greg' 'Dammy' 'Greg' 'admin' 'yet-kids-sides' 1 'b-boggie' 'Bad' 'Boggie' 'account' 'action-exchange-unusual' 2 'j-lotus' 'Jean' 'Lotus' 'account' 'between-hat-dig' 3 's-begar' 'Sandy' 'Begar' 'sales' 'create-no-remain'

Writing CSV data to file system

You've seen how reading CSV file data can be accomplished with the help of the csv-parser library. There are times you would want to process some data and output the result to a CSV file. This action can be initiated by the user from the application, by clicking an export to CSV or Download button. They will want to download the tabular data they see in the app for later use or for sharing through a different medium. You can do this in Node.js and we can apply it to the example from the previous section, by outputting the data logged to the console to a file.

We will now add another function for saving data to a CSV file. Open index.js and add the functions below to it:

function writeToCSVFile(users) {
  const filename = 'output.csv';
  fs.writeFile(filename, extractAsCSV(users), err => {
    if (err) {
      console.log('Error writing to csv file', err);
    } else {
      console.log(`saved as ${filename}`);
    }
  });
}

function extractAsCSV(users) {
  const header = ["Username, Password, Roles"];
  const rows = users.map(user =>
     `${user.username}, ${user.password}, ${user.roles}`
  );
  return header.concat(rows).join("\n");
}

In the code above, we defined two functions. The writeToCSVFile() function takes an array of users as an argument. It calls the fs.writeFile() with the name of the file as the first argument, and the data to write to the file as the second argument, and a callback function to run after the write completes or an error occurs.

The extractAsCSV() function takes the users array as an argument, then it generates another array where each element in the array is a string containing comma-separated values for a row, and the result of this map function assigned to the rows variable. We then concatenate the rows and header arrays and call .join(“\n”), which will return a string of each element in the array separated by a newline character (“\n”). This will result in a string that contains comma-separated values. We return that string as the result of the function call. The resulting string should look like the data below (your passwords will be randomized and different than these):

Username, Password, Roles
d-greg, yet-kids-sides, admin
b-boggie, action-exchange-unusual, account
j-lotus, between-hat-dig, account
s-begar, create-no-remain, sales

We will call the writeToCSVFile() function when all the data is parsed from the input.csv file. We will do this in the callback function of the end event handler.

Go to the end event handler for the read stream on line 29 and add a function call to writeToCSVFile. The end event is emitted once all the csv parser has finished parsing the data it received.

.on('end', function () {
    writeToCSVFile(users)
})

If you run the command node index.js, it should read the input.csv file and write the output to output.csv.

Recap

A CSV file is a comma-separated text file that can be used to store tabular data. In this post, I showed you how to use the fs and csv-parser modules to read the content of a CSV file and to also write data to a CSV file. To read the data, you open a readable stream with fs.createReadStream() and pipe it to the csv() function from the csv-parser library. To read the data, you listen for the data event which is emitted for each row of data parsed. I also explained some of the options you can pass to the csv() function in order to configure how you want the data to be parsed.

The code shown in this post can be found on GitHub.

Learn more

Learn more about using an Extract, Transform, and Load (ETL) pipeline to process large CSV files with Node.js in our free tutorials: