Data Brokering with Node.js: Process data at the speed of technology
Free instant access to our full course.
Start learning nowWhat you'll learn
In this full course from Hey Node, learn how to harness the power of Node.js to process big data including:
- How to install Node.js and manage a local development environment
- How to read, and write data via both the file system and Node.js streams
- The inner workings of Node's event loop, promises, and asynchronous APIs
- Using npm to manage 3rd party libraries, and keeping custom code organized with modules
- Setting up an API proxy server using Express
- Common design patterns for applications that process large amounts of data with Node.js
What you'll build
When it comes to learning Node.js, there's no substitute for tutorials that build real, production-ready, applications. Some of the things you'll build include:
Two different Node applications that use an Extract, Transform, Load (ETL) pipeline to process data from NASA's exoplanet API: one using the `fs` module, and the other using streams
An Express-based API proxy server that allows us to customize data from the Exoplanet API and return a subset of the data customized for our needs, plus performance optimizations like caching and data compression.
Tutorials
This course contains approximately 12 hours of written and video content, code samples, and exercises covering:
- Processing Large Amounts of Data with Node.js
- Node’s Module System
- Write Asynchronous Node.js Code With Async, Await, and Promises
- Node.js for Reading and Writing Files
- What Are Streams in Node.js?
- Install and Upgrade Node.js
- Manage Application Configuration with Environment Variables in Node.js
- Set up API proxy middleware with Node.js Express
- User Authentication in Node.js Applications
- Explore Node's Event Loop
Processing Large Amounts of Data with Node.js
Learn to process a large amount of data with Node.js, (called data brokering), using the ETL (Extract, Transform, Load) pattern. With Node, you can share data between services without tightly coupling applications together or rewriting them. We often have data locked away in a legacy API, third party system, or somewhere else. When we need to access that data from a new frontend application, a GraphQL server, (or anywhere in our stack where we weren't accessing data before), Node.js is an excellent tool.
In this Node tutorial series, learn about using the Node ETL pattern to retrieve, move, and process data anywhere.
Tutorials:
Node’s Module System
A module is any file or directory in the node_modules directory that can be loaded by the Node.js require()
function. As building blocks of code structure, Node.js modules allow developers to better structure, reuse, and share code. Node.js applications use Node package manager (npm) or yarn and apackage.json file to keep track of the modules they depend on. Learn what npm is, how to create a package.json file, and how to install or update specific packages using npm. These sections are full of best practices for using npm for both beginners and experienced Node.js developers.
Tutorials:
- How the Node Module System Works
- Introduction to the Node Package Manager (NPM)
- What Is package.json?
- Create a package.json File
- What Is package-lock.json?
- How to Install NPM Packages
- Uninstall Packages from a Project
- How to Use Semantic Versioning in NPM
- How to Update a Node Dependency with NPM
- What Are NPM Scripts?
- Restart a Node.js Application upon Changing a File
- Organize Your Node.js Code into Modules
Write Asynchronous Node.js Code With Async, Await, and Promises
In this tutorial series, learn how to write clean, non-blocking Node.js code using async
, await
, and promises. Understanding asynchronous code is an important part of understanding how Node.js works.
Tutorials:
Node.js for Reading and Writing Files
Learn about the Node.js core fs module’s fs.readFile
andfs.writeFile
. Use Node.js to parse JSON data from files, and save a Node.js object to JSON.
Tutorials:
What Are Streams in Node.js?
Like Unix pipes, Node.js streams allow you to transfer large amounts of data in real on mobile devices or websites by connecting the data source and consumer. Node.js streams can transfer large amounts of data quickly and efficiently by breaking the data into chunks instead of reading it all into memory at once. In this Node.js tutorial, learn to use fs.createReadStream
,fs.createWriteStream
, and pipe
to process large CSV and JSON files.
Tutorials:
Install and Upgrade Node.js
Learn how to download and install Node.js, update to the latest version of Node, and manage multiple versions of Node.js on the same environment using nvm (the Node Version Manager). These tutorials also explain the Node.js version release cycle, security updates, and LTS releases.
Tutorials:
Manage Application Configuration with Environment Variables in Node.js
Learn what environment variables are and they can be used to configure a Node.js application for both development and production. Then install the popular dotenv package with npm and create a .env file to define and document your variables.
Tutorials:
Set up API proxy middleware with Node.js Express
Learn what a proxy server is, and why you might need a proxy for your API. Then build one using the popular Node.js Express server.
Tutorials:
- What Is an API Proxy?
- What Is the Express Node.js Framework?
- How to Add a Route to an Express Server in Node.js
- Express Middleware in Node.js
- Use Express to Create an API Proxy Server in Node.js
- How to Set up an Express.js Server in Node.js
- Set up Routes for Your API in Node.js
- Optimize an Express Server in Node.js
- Add Compression to Express in Node.js
- Add Response Caching to a Node.js Express Server
User Authentication in Node.js Applications
Authentication, and authorization, allow us to know more about the individuals who are interacting with our applications. Who are they, why are they here, and perhaps most importantly what are they allowed to do? This set of tutorials explores the fundamentals of building secure Express applications, handling sensitive user input, and communicating authentication status via sessions, cookies, and JSON Web Tokens between a Node.js back-end and a front-end application of your choice.
Tutorials:
- What Is the Difference Between Authorization and Authentication?
- What Are Form Validation and Sanitization?
- Process User Login Form with ExpressJS
- How to Validate and Sanitize an ExpressJS Form
- What Is the Difference Between Sessions and JSON Web Tokens (JWT) Authentication?
- Set Up ExpressJS Session Authentication for Node Applications
- Authenticate Users with Node ExpressJS and Passport.js
Explore Node's Event Loop
The event loop is a fundamental concept of Node.js. It opens the door to understanding Node’s asynchronous processes and non-blocking I/O. It outlines the mechanisms that make Node a successful, powerful, and popular modern framework. These tutorials are useful for Node.js developers who want a deeper understanding of what's happening under the hood of every application, and those who want to take full control of every step of its running cycle.
Tutorials:
Join now and get instant access to all our tutorials.
Jon says "Hi!"
Meet Jon Church, your Node.js teacher. He writes, researches, codes, and records courses that teach people the infinite possibilities of Node.js. He's also an enthusiastic supporter and contributor to the Node.js project.
"The Osio Labs team has done an incredible job listening to the needs of the Node.js community, working closely with a diverse contingent of members from across the globe, to craft a training program that fits the needs of today's Node.js developer."
"Your videos have been awesome! Been watching the free ones off and on for the past few years. Bought a subscription recently and am getting a lot out of it!"
"Great Job @drupalizeme. U are doing great guides 4 #Drupal. I personally used their site to prepare for @acquia certified program and I won."
Who is this for?
Data Brokering with Node.js aims to help JavaScripters who need a better understanding of back-end or application-layer concepts, those with a specific Node problem to solve, and those levelling up their tech skills.
Who is this by?
Hey Node is a labor of love from Osio Labs. We’re Open Source Inside and Out, and we support and lead projects that foster and grow open source communities and the technology they create.
Up next
Be the first to know when we add new content. Sign up for our newsletter.
Join our mailing list