Data Brokering with Node.js: Process data at the speed of technology

Free instant access to our full course.

Start learning now
People waiting at a bus stop in front of a sign that says Node.js

What you'll learn

In this full course from Hey Node, learn how to harness the power of Node.js to process big data including:

  • How to install Node.js and manage a local development environment
  • How to read, and write data via both the file system and Node.js streams
  • The inner workings of Node's event loop, promises, and asynchronous APIs
  • Using npm to manage 3rd party libraries, and keeping custom code organized with modules
  • Setting up an API proxy server using Express
  • Common design patterns for applications that process large amounts of data with Node.js

What you'll build

When it comes to learning Node.js, there's no substitute for tutorials that build real, production-ready, applications. Some of the things you'll build include:

Two different Node applications that use an Extract, Transform, Load (ETL) pipeline to process data from NASA's exoplanet API: one using the `fs` module, and the other using streams

An Express-based API proxy server that allows us to customize data from the Exoplanet API and return a subset of the data customized for our needs, plus performance optimizations like caching and data compression.


This course contains approximately 12 hours of written and video content, code samples, and exercises covering:

Processing Large Amounts of Data with Node.js

Learn to process a large amount of data with Node.js, (called data brokering), using the ETL (Extract, Transform, Load) pattern. With Node, you can share data between services without tightly coupling applications together or rewriting them. We often have data locked away in a legacy API, third party system, or somewhere else. When we need to access that data from a new frontend application, a GraphQL server, (or anywhere in our stack where we weren't accessing data before), Node.js is an excellent tool.

In this Node tutorial series, learn about using the Node ETL pattern to retrieve, move, and process data anywhere.


Node’s Module System

A module is any file or directory in the node_modules directory that can be loaded by the Node.js require() function. As building blocks of code structure, Node.js modules allow developers to better structure, reuse, and share code. Node.js applications use Node package manager (npm) or yarn and apackage.json file to keep track of the modules they depend on. Learn what npm is, how to create a package.json file, and how to install or update specific packages using npm. These sections are full of best practices for using npm for both beginners and experienced Node.js developers.


Write Asynchronous Node.js Code With Async, Await, and Promises

In this tutorial series, learn how to write clean, non-blocking Node.js code using async, await, and promises. Understanding asynchronous code is an important part of understanding how Node.js works.


Node.js for Reading and Writing Files

Learn about the Node.js core fs module’s fs.readFile andfs.writeFile. Use Node.js to parse JSON data from files, and save a Node.js object to JSON.


What Are Streams in Node.js?

Like Unix pipes, Node.js streams allow you to transfer large amounts of data in real on mobile devices or websites by connecting the data source and consumer. Node.js streams can transfer large amounts of data quickly and efficiently by breaking the data into chunks instead of reading it all into memory at once. In this Node.js tutorial, learn to use fs.createReadStream,fs.createWriteStream, and pipe to process large CSV and JSON files.


Install and Upgrade Node.js

Learn how to download and install Node.js, update to the latest version of Node, and manage multiple versions of Node.js on the same environment using nvm (the Node Version Manager). These tutorials also explain the Node.js version release cycle, security updates, and LTS releases.


Manage Application Configuration with Environment Variables in Node.js

Learn what environment variables are and they can be used to configure a Node.js application for both development and production. Then install the popular dotenv package with npm and create a .env file to define and document your variables.


User Authentication in Node.js Applications

Authentication, and authorization, allow us to know more about the individuals who are interacting with our applications. Who are they, why are they here, and perhaps most importantly what are they allowed to do? This set of tutorials explores the fundamentals of building secure Express applications, handling sensitive user input, and communicating authentication status via sessions, cookies, and JSON Web Tokens between a Node.js back-end and a front-end application of your choice.


Explore Node's Event Loop

The event loop is a fundamental concept of Node.js. It opens the door to understanding Node’s asynchronous processes and non-blocking I/O. It outlines the mechanisms that make Node a successful, powerful, and popular modern framework. These tutorials are useful for Node.js developers who want a deeper understanding of what's happening under the hood of every application, and those who want to take full control of every step of its running cycle.


See the complete list of tutorials →
Join now

Join now and get instant access to all our tutorials.

Jon says "Hi!"

Meet Jon Church, your Node.js teacher. He writes, researches, codes, and records courses that teach people the infinite possibilities of Node.js. He's also an enthusiastic supporter and contributor to the Node.js project.

Meet the rest of our team

"The Osio Labs team has done an incredible job listening to the needs of the Node.js community, working closely with a diverse contingent of members from across the globe, to craft a training program that fits the needs of today's Node.js developer."

Dan Shaw, Global Head of Developer Relations, PayPal

"Your videos have been awesome! Been watching the free ones off and on for the past few years. Bought a subscription recently and am getting a lot out of it!"

Andrew Rivera, Drupal Developer

"Great Job @drupalizeme. U are doing great guides 4 #Drupal. I personally used their site to prepare for @acquia certified program and I won."

Ildephonse Bikino, Deputy Director for ICT in Education Projects

Who is this for?

Data Brokering with Node.js aims to help JavaScripters who need a better understanding of back-end or application-layer concepts, those with a specific Node problem to solve, and those levelling up their tech skills.

Who is this by?

Hey Node is a labor of love from Osio Labs. We’re Open Source Inside and Out, and we support and lead projects that foster and grow open source communities and the technology they create.

Up next

Be the first to know when we add new content. Sign up for our newsletter.

Join our mailing list