Data Brokering with Node.js: Process data at the speed of technology
Free instant access to our full course.Start learning now
What you'll learn
In this full course from Hey Node, learn how to harness the power of Node.js to process big data including:
- How to install Node.js and manage a local development environment
- How to read, and write data via both the file system and Node.js streams
- The inner workings of Node's event loop, promises, and asynchronous APIs
- Using npm to manage 3rd party libraries, and keeping custom code organized with modules
- Setting up an API proxy server using Express
- Common design patterns for applications that process large amounts of data with Node.js
What you'll build
When it comes to learning Node.js, there's no substitute for tutorials that build real, production-ready, applications. Some of the things you'll build include:
Two different Node applications that use an Extract, Transform, Load (ETL) pipeline to process data from NASA's exoplanet API: one using the `fs` module, and the other using streams
An Express-based API proxy server that allows us to customize data from the Exoplanet API and return a subset of the data customized for our needs, plus performance optimizations like caching and data compression.
This course contains approximately 12 hours of written and video content, code samples, and exercises covering:
- Processing Large Amounts of Data with Node.js
- Node’s Module System
- Write Asynchronous Node.js Code With Async, Await, and Promises
- Node.js for Reading and Writing Files
- What Are Streams in Node.js?
- Install and Upgrade Node.js
- Manage Application Configuration with Environment Variables in Node.js
- Set up API proxy middleware with Node.js Express
- Explore Node's Event Loop
Processing Large Amounts of Data with Node.js
Learn to process a large amount of data with Node.js, (called data brokering), using the ETL (Extract, Transform, Load) pattern. With Node, you can share data between services without tightly coupling applications together or rewriting them. We often have data locked away in a legacy API, third party system, or somewhere else. When we need to access that data from a new frontend application, a GraphQL server, (or anywhere in our stack where we weren't accessing data before), Node.js is an excellent tool.
In this Node tutorial series, learn about using the Node ETL pattern to retrieve, move, and process data anywhere.
Node’s Module System
A module is any file or directory in the node_modules directory that can be loaded by the Node.js
require() function. As building blocks of code structure, Node.js modules allow developers to better structure, reuse, and share code. Node.js applications use Node package manager (npm) or yarn and apackage.json file to keep track of the modules they depend on. Learn what npm is, how to create a package.json file, and how to install or update specific packages using npm. These sections are full of best practices for using npm for both beginners and experienced Node.js developers.
- How the Node Module System Works
- Introduction to the Node Package Manager (NPM)
- What Is package.json? free
- Create a package.json File free
- What Is package-lock.json?
- How to Install NPM Packages
- Uninstall Packages from a Project
- How to Use Semantic Versioning in NPM free
- How to Update a Node Dependency with NPM
- What Are NPM Scripts?
- Restart a Node.js Application upon Changing a File
- Organize Your Node.js Code into Modules
Write Asynchronous Node.js Code With Async, Await, and Promises
In this tutorial series, learn how to write clean, non-blocking Node.js code using
await, and promises. Understanding asynchronous code is an important part of understanding how Node.js works.
Node.js for Reading and Writing Files
Learn about the Node.js core fs module’s
fs.writeFile. Use Node.js to parse JSON data from files, and save a Node.js object to JSON.
What Are Streams in Node.js?
Like Unix pipes, Node.js streams allow you to transfer large amounts of data in real on mobile devices or websites by connecting the data source and consumer. Node.js streams can transfer large amounts of data quickly and efficiently by breaking the data into chunks instead of reading it all into memory at once. In this Node.js tutorial, learn to use
pipe to process large CSV and JSON files.
Install and Upgrade Node.js
Learn how to download and install Node.js, update to the latest version of Node, and manage multiple versions of Node.js on the same environment using nvm (the Node Version Manager). These tutorials also explain the Node.js version release cycle, security updates, and LTS releases.
Manage Application Configuration with Environment Variables in Node.js
Learn what environment variables are and they can be used to configure a Node.js application for both development and production. Then install the popular dotenv package with npm and create a .env file to define and document your variables.
Set up API proxy middleware with Node.js Express
Learn what a proxy server is, and why you might need a proxy for your API. Then build one using the popular Node.js Express server.
- What Is an API Proxy? free
- What Is the Express Node.js Framework? free
- How to Add a Route to an Express Server in Node.js
- Express Middleware in Node.js
- Use Express to Create an API Proxy Server in Node.js
- How to Set up an Express.js Server in Node.js
- Set up Routes for Your API in Node.js
- Optimize an Express Server in Node.js
- Add Compression to Express in Node.js
- Add Response Caching to a Node.js Express Server
Explore Node's Event Loop
The event loop is a fundamental concept of Node.js. It opens the door to understanding Node’s asynchronous processes and non-blocking I/O. It outlines the mechanisms that make Node a successful, powerful, and popular modern framework. These tutorials are useful for Node.js developers who want a deeper understanding of what's happening under the hood of every application, and those who want to take full control of every step of its running cycle.
Join now and get instant access to all our tutorials.
"The Osio Labs team has done an incredible job listening to the needs of the Node.js community, working closely with a diverse contingent of members from across the globe, to craft a training program that fits the needs of today's Node.js developer."
"Your videos have been awesome! Been watching the free ones off and on for the past few years. Bought a subscription recently and am getting a lot out of it!"
"Great Job @drupalizeme. U are doing great guides 4 #Drupal. I personally used their site to prepare for @acquia certified program and I won."
Who is this for?
Who is this by?
Hey Node is a labor of love from Osio Labs. We’re Open Source Inside and Out, and we support and lead projects that foster and grow open source communities and the technology they create.
Be the first to know when we add new content. Sign up for our newsletter.Join our mailing list