Your First Node API

Async

A Complete Server: Getting Started

A Complete Server: Persistence

A Complete Server: Authentication

A Complete Server: Deployment

Command Line Interfaces

Testing Node.js Applications

Bonus: Persistence with PostgreSQL

Changelog

Enter your name and email address below, then click the **"Send my free chapter"** button to get started

The Complete Guide to Building Production Apps with Node.js

Fullstack Node.js

Our server is off to the races. We have a functioning API that can service a web front-end and other clients.

This is a great start, but as of right now our data is static. While it's true that we can dynamically return data to a client with paging and filtering, the source of the data can never change.

If we were building a company on top of this service, it wouldn't be very useful. What we need now is persistence. A client should be able to add, remove, and modify the data that our API serves.

Currently our data is stored on disk. Our app uses the filesystem to retrieve data. We 

 continue to use the filesystem when we add persistence. We would do this by using 

 methods, but that's not a good idea for a production app.

When we run apps in production, we want to have multiple servers running at the same time. We need this for a variety of reasons: redundancy, scale, zero-downtime deploys, and handling server maintenance. If we continue to use the filesystem for persistence, we'll run into a problem when we try to run on multiple servers.

Without special setup, each app would only have access to its local filesystem. This means that if a new product is created on server A, any client connected to server B wouldn't be able to access it. Effectively, we'd have multiple copies of the app, but they would all have different data.

We can use a database to get around this issue. We can have as many instances of our apps as we'd like, and they can all connect to the same database, ensuring they all have access to the same data.

Databases allow us to separate the data from our running code, so that our app does not have to run at the same location where our data is stored. This allows us to run instances of our apps wherever we'd like. We could even run a local copy that is identical to the production version (assuming our local version has access to the production database).

Databases do add additional complexity to an app. This is why we started with the filesystem -- it's the simplest way to handle data. However, the small increase in complexity comes with a lot more power and functionality. In the previous chapter, our code was inefficient. When we wanted to find products matching specific criteria, we had to iterate over our full collection to find them. We'll use database indexes so that these lookups are much more efficient.

There are many different databases to choose from when creating a production app, and they each have their own tradeoffs and are popular for different reasons. When the LAMP stack was popular, MySQL was the go-to. Over time, developers gravitated to other databases that did not use SQL like MongoDB, Redis, DynamoDB, and Cassandra.

SQL databases (MySQL, PostgreSQL, MariaDB, SQLite, Microsoft SQL Server, etc...) store data with tables and rows. If we were to think of the closest analog in JavaScript terms, each table would be an array of rows, and each row would be an array of values. Each table would have an associated schema that defines what kinds of values belong in each index of its rows. By storing data this way, SQL databases can be very efficient at indexing and retrieving data.

Additionally, SQL databases are easy for people to work with directly. By using SQL (structured query language), one can write mostly human readable commands to store and retrieve data with a direct interface to the database server.

While these two attributes have advantages, they also come with some tradeoffs. Namely, to store a JavaScript object in a SQL database we need code to translate that object into a SQL insert command string. Similarly, when we want to retrieve data, we would need to create a SQL query string, and then for each row we'd need to use the table's schema to map fields to key/value pairs. For example, our product objects each have an array of tags. The traditional way to model this in a SQL database is to maintain a separate table for tags and yet another table where each row represents a connection between a product and a tag. To retrieve a product and its tags we'd construct a query that can pull data from all three tables and join it together in a way that makes sense.

These tradeoffs are by no means a deal-breaker, most production apps will use an ORM (Object Relational Mapping) like 

 to handle these translation steps automatically. However, these additional conversion steps highlight that while very mature, capable, and efficient, SQL databases were not created with JavaScript apps in mind.

Conversely, MongoDB was created to closely match how we work with data in JavaScript and Node.js. Rather than using a table and row model, MongoDB has collections of documents. Each document in a collection is essentially a JavaScript object. This means that any data we use in our code will be represented very similarly in the database.

This has the distinctive property of allowing us to be flexible with what data we decide to persist in the database. Unlike with SQL, we would not need to first update the database's schema before adding new documents to a collection. MongoDB does not require each document in a collection to be uniform, nor is it required that object properties need to be defined in a schema before being added.

Of course, this additional freedom is itself a tradeoff. It is often very nice to have this flexibility. When creating an app, it is useful to evolve the data as changes are made. Unfortunately, if we aren't careful, we can wind up with a collection of documents that are inconsistent, and our app will need to contend with collections of objects that vary in subtle (or not so subtle) ways. For this reason, we will use Mongoose to get the best of both worlds.

Over time, developers have come to rely on SQL databases to enforce data validation. If a row is inserted with the wrong types of data in its fields, the database can refuse with an error. The app doesn't need to worry about handling that type of validation logic. MongoDB has no opinions on what types of data belongs in a document, and therefore data validation needs to be done in the app.

Additionally when persisting data with a SQL database, its common to have store relationships between rows. When retrieving data, the database can automatically "join" related rows to conveniently return all necessary at once. MongoDB does not handle relationships, and this means that an app using MongoDB would need its own logic to fetch related documents.

Mongoose is an ODM (object data modeling) library that can easily provide both of these features (and more) to our app. Later in this chapter, we'll show how to add custom validation logic that goes well beyond what a SQL database can provide and how to store and retrieve related documents with ease.

We're now ready to convert our app to use MongoDB to store our data. Luckily, we've built our app so that switching the persistence layer will be very easy.

We've built our app using modules with specific responsibilities:

 because the only difference is where data is stored and how it is loaded. 

 can continue to call the same exact methods from 

. To get to the same functionality we have using the file system, the only thing we need to do is change the functionality of the 

 methods so that they load data from MongoDB instead of the filesystem.

That said, if we start by making those modifications, it will be difficult to check if they're working correctly. For 

 to pull data from MongoDB, we'll need to make sure there are documents in the database. It will be more helpful to first add the 

---
code: https://gitlab.com/fullstackio/books/newline-course-apps/david-guttman-fullstack-node.js-book-app/-/tree/main/code/src/04-persistence?ref_type=heads
---

# A Complete Server: Persistence

Our server is off to the races. We have a functioning API that can service a web front-end and other clients.

This is a great start, but as of right now our data is static. While it's true that we can dynamically return data to a client with paging and filtering, the source of the data can never change.

If we were building a company on top of this service, it wouldn't be very useful. What we need now is persistence. A client should be able to add, remove, and modify the data that our API serves.

Currently our data is stored on disk. Our app uses the filesystem to retrieve data. We _could_ continue to use the filesystem when we add persistence. We would do this by using `fs.writeFile()` and other `fs` methods, but that's not a good idea for a production app.

When we run apps in production, we want to have multiple servers running at the same time. We need this for a variety of reasons: redundancy, scale, zero-downtime deploys, and handling server maintenance. If we continue to use the filesystem for persistence, we'll run into a problem when we try to run on multiple servers.

Without special setup, each app would only have access to its local filesystem. This means that if a new product is created on server A, any client connected to server B wouldn't be able to access it. Effectively, we'd have multiple copies of the app, but they would all have different data.

We can use a database to get around this issue. We can have as many instances of our apps as we'd like, and they can all connect to the same database, ensuring they all have access to the same data.

Databases allow us to separate the data from our running code, so that our app does not have to run at the same location where our data is stored. This allows us to run instances of our apps wherever we'd like. We could even run a local copy that is identical to the production version (assuming our local version has access to the production database).

Databases do add additional complexity to an app. This is why we started with the filesystem -- it's the simplest way to handle data. However, the small increase in complexity comes with a lot more power and functionality. In the previous chapter, our code was inefficient. When we wanted to find products matching specific criteria, we had to iterate over our full collection to find them. We'll use database indexes so that these lookups are much more efficient.

There are many different databases to choose from when creating a production app, and they each have their own tradeoffs and are popular for different reasons. When the LAMP stack was popular, MySQL was the go-to. Over time, developers gravitated to other databases that did not use SQL like MongoDB, Redis, DynamoDB, and Cassandra.

SQL databases (MySQL, PostgreSQL, MariaDB, SQLite, Microsoft SQL Server, etc...) store data with tables and rows. If we were to think of the closest analog in JavaScript terms, each table would be an array of rows, and each row would be an array of values. Each table would have an associated schema that defines what kinds of values belong in each index of its rows. By storing data this way, SQL databases can be very efficient at indexing and retrieving data.

Additionally, SQL databases are easy for people to work with directly. By using SQL (structured query language), one can write mostly human readable commands to store and retrieve data with a direct interface to the database server.

While these two attributes have advantages, they also come with some tradeoffs. Namely, to store a JavaScript object in a SQL database we need code to translate that object into a SQL insert command string. Similarly, when we want to retrieve data, we would need to create a SQL query string, and then for each row we'd need to use the table's schema to map fields to key/value pairs. For example, our product objects each have an array of tags. The traditional way to model this in a SQL database is to maintain a separate table for tags and yet another table where each row represents a connection between a product and a tag. To retrieve a product and its tags we'd construct a query that can pull data from all three tables and join it together in a way that makes sense.

These tradeoffs are by no means a deal-breaker, most production apps will use an ORM (Object Relational Mapping) like [Sequelize](http://docs.sequelizejs.com/) to handle these translation steps automatically. However, these additional conversion steps highlight that while very mature, capable, and efficient, SQL databases were not created with JavaScript apps in mind.

Conversely, MongoDB was created to closely match how we work with data in JavaScript and Node.js. Rather than using a table and row model, MongoDB has collections of documents. Each document in a collection is essentially a JavaScript object. This means that any data we use in our code will be represented very similarly in the database.

This has the distinctive property of allowing us to be flexible with what data we decide to persist in the database. Unlike with SQL, we would not need to first update the database's schema before adding new documents to a collection. MongoDB does not require each document in a collection to be uniform, nor is it required that object properties need to be defined in a schema before being added.

Of course, this additional freedom is itself a tradeoff. It is often very nice to have this flexibility. When creating an app, it is useful to evolve the data as changes are made. Unfortunately, if we aren't careful, we can wind up with a collection of documents that are inconsistent, and our app will need to contend with collections of objects that vary in subtle (or not so subtle) ways. For this reason, we will use Mongoose to get the best of both worlds.

Over time, developers have come to rely on SQL databases to enforce data validation. If a row is inserted with the wrong types of data in its fields, the database can refuse with an error. The app doesn't need to worry about handling that type of validation logic. MongoDB has no opinions on what types of data belongs in a document, and therefore data validation needs to be done in the app.

Additionally when persisting data with a SQL database, its common to have store relationships between rows. When retrieving data, the database can automatically "join" related rows to conveniently return all necessary at once. MongoDB does not handle relationships, and this means that an app using MongoDB would need its own logic to fetch related documents.

Mongoose is an ODM (object data modeling) library that can easily provide both of these features (and more) to our app. Later in this chapter, we'll show how to add custom validation logic that goes well beyond what a SQL database can provide and how to store and retrieve related documents with ease.

## Getting Started

We're now ready to convert our app to use MongoDB to store our data. Luckily, we've built our app so that switching the persistence layer will be very easy.

We've built our app using modules with specific responsibilities:

-   `server.js` creates the server instance and connects endpoints to route handlers.
-   `api.js` contains functions are responsible for converting HTTP requests into HTTP responses using our model module, `products.js`
-   `products.js` is responsible for loading and manipulating data.

We don't need to change anything about `server.js` or `api.js` because the only difference is where data is stored and how it is loaded. `api.js` can continue to call the same exact methods from `products.js`. To get to the same functionality we have using the file system, the only thing we need to do is change the functionality of the `Products.list()` and `Products.get()` methods so that they load data from MongoDB instead of the filesystem.

That said, if we start by making those modifications, it will be difficult to check if they're working correctly. For `Products.list()` and `Products.get()` to pull data from MongoDB, we'll need to make sure there are documents in the database. It will be more helpful to first add the `Products.create()` method to make that easier.