Use Streams to Extract, Transform, and Load CSV Data

Using the fs module and streams, we can take a local file and perform ETL steps on it to transform the file into JSON and write it to disk. We can use a helpful node module from npm, csvtojson, to convert the csv data for us. With these tools, we can:

• Create a readstream from a local csv file
• Pipe the csv data to csvtojson to transform it to JSON
• Alter the data using a Transform stream
• Write the transformed JSON data to a file using a WriteStream
• Handle error events

By the end of this tutorial, you should be able to:

• Explain how to use the fs module to create read/write streams
• Implement Transform streams to modify data as it passes through a stream
• Understand how to handle error events with streams
• Convert CSV data to JSON using streams