Data Brokering with Node.js: Process data at the speed of technology

Processing Large Amounts of Data with Node.js

Learn to process a large amount of data with Node.js, (called data brokering), using the ETL (Extract, Transform, Load) pattern. With Node, you can share data between services without tightly coupling applications together or rewriting them. We often have data locked away in a legacy API, third party system, or somewhere else. When we need to access that data from a new frontend application, a GraphQL server, (or anywhere in our stack where we weren't accessing data before), Node.js is an excellent tool.

In this Node tutorial series, learn about using the Node ETL pattern to retrieve, move, and process data anywhere.

Tutorials:

• Overview of Data Brokering With Node.js
• Node.js ETL (Extract, Transform, Load) Pipeline: What Are We Building?
• What Is the Node.js ETL (Extract, Transform, Load) Pipeline?
• ETL: Extract Data with Node.js
• ETL: Transform Data with Node.js
• ETL: Load Data to Destination with Node.js

Node’s Module System

A module is any file or directory in the node_modules directory that can be loaded by the Node.js require() function. As building blocks of code structure, Node.js modules allow developers to better structure, reuse, and share code. Node.js applications use Node package manager (npm) or yarn and apackage.json file to keep track of the modules they depend on. Learn what npm is, how to create a package.json file, and how to install or update specific packages using npm. These sections are full of best practices for using npm for both beginners and experienced Node.js developers.

Tutorials:

• How the Node Module System Works
• Introduction to the Node Package Manager (NPM)
• What Is package.json?
• Create a package.json File
• What Is package-lock.json?
• How to Install NPM Packages
• Uninstall Packages from a Project
• How to Use Semantic Versioning in NPM
• How to Update a Node Dependency with NPM
• What Are NPM Scripts?
• Restart a Node.js Application upon Changing a File
• Organize Your Node.js Code into Modules

Write Asynchronous Node.js Code With Async, Await, and Promises

In this tutorial series, learn how to write clean, non-blocking Node.js code using async, await, and promises. Understanding asynchronous code is an important part of understanding how Node.js works.

Tutorials:

• Understanding Promises in Node.js
• Use Promise.all to Wait for Multiple Promises in Node.js
• Use JavaScript's Async/Await with Promises
• How to Make API Requests with Request-Promise in Node.js

Node.js for Reading and Writing Files

Learn about the Node.js core fs module’s fs.readFile andfs.writeFile. Use Node.js to parse JSON data from files, and save a Node.js object to JSON.

Tutorials:

• What is the fs (File System) Module?
• Read/Write JSON Files with Node.js

What Are Streams in Node.js?

Like Unix pipes, Node.js streams allow you to transfer large amounts of data in real on mobile devices or websites by connecting the data source and consumer. Node.js streams can transfer large amounts of data quickly and efficiently by breaking the data into chunks instead of reading it all into memory at once. In this Node.js tutorial, learn to use fs.createReadStream,fs.createWriteStream, and pipe to process large CSV and JSON files.

Tutorials:

• What Is a Node.js Stream?
• Use Streams to Extract, Transform, and Load CSV Data
• Stream to an HTTP Response with Node.js

Install and Upgrade Node.js

Learn how to download and install Node.js, update to the latest version of Node, and manage multiple versions of Node.js on the same environment using nvm (the Node Version Manager). These tutorials also explain the Node.js version release cycle, security updates, and LTS releases.

Tutorials:

• Overview: How to Manage Node.js Locally
• Install Node.js Locally with Node Version Manager (nvm)

Manage Application Configuration with Environment Variables in Node.js

Learn what environment variables are and they can be used to configure a Node.js application for both development and production. Then install the popular dotenv package with npm and create a .env file to define and document your variables.

Tutorials:

• How to Use Environment Variables in Node.js
• Set up and Test a Dot Env (.env) File in Node

Set up API proxy middleware with Node.js Express

Learn what a proxy server is, and why you might need a proxy for your API. Then build one using the popular Node.js Express server.

Tutorials:

• What Is an API Proxy?
• What Is the Express Node.js Framework?
• How to Add a Route to an Express Server in Node.js
• Express Middleware in Node.js
• Use Express to Create an API Proxy Server in Node.js
• How to Set up an Express.js Server in Node.js
• Set up Routes for Your API in Node.js
• Optimize an Express Server in Node.js
• Add Compression to Express in Node.js
• Add Response Caching to a Node.js Express Server

User Authentication in Node.js Applications

Authentication, and authorization, allow us to know more about the individuals who are interacting with our applications. Who are they, why are they here, and perhaps most importantly what are they allowed to do? This set of tutorials explores the fundamentals of building secure Express applications, handling sensitive user input, and communicating authentication status via sessions, cookies, and JSON Web Tokens between a Node.js back-end and a front-end application of your choice.

Tutorials:

• What Is the Difference Between Authorization and Authentication?
• What Are Form Validation and Sanitization?
• Process User Login Form with ExpressJS
• How to Validate and Sanitize an ExpressJS Form
• What Is the Difference Between Sessions and JSON Web Tokens (JWT) Authentication?
• Set Up ExpressJS Session Authentication for Node Applications
• Authenticate Users with Node ExpressJS and Passport.js

Explore Node's Event Loop

The event loop is a fundamental concept of Node.js. It opens the door to understanding Node’s asynchronous processes and non-blocking I/O. It outlines the mechanisms that make Node a successful, powerful, and popular modern framework. These tutorials are useful for Node.js developers who want a deeper understanding of what's happening under the hood of every application, and those who want to take full control of every step of its running cycle.

Tutorials:

• How the Event Loop Works in Node.js
• Explore the Timers Phase of Node's Event Loop
• Explore the I/O Callbacks Phase of the Node.js Event Loop
• Explore the Immediate Callbacks Phase of Node's Event Loop