Course Introduction

What You'll Learn in This Course

Golang Useful Resources and Tools

The authors

The history of Go

Installing Go

Golang: An Overview of Fundamental Concepts

Introduction

A Quick Primer on Key-Value Servers and Hash Tables

How to Create a Basic hello world Webserver with Go

Build a Basic hello world JSON API Server with Go + Chi

How to Create an In-Memory Hash Table API with Go + Chi

Conclusion

A key-value server

Testing a key-value server

What is End-to-End Testing, Integration Testing, and Unit Testing?

Writing Unit Tests for JSON Functions in Go

Parallel Testing in Golang: How to Use t.Parallel()

A Brief Overview of Golang Benchmarks

Integration testing

Further testing types

Distributed key-value server

What is Distributed Consensus and How Does it Work?

An Introduction to Using Raft for Distributed Consensus

Create a Distributed Webserver using Golang and Raft

Connecting Golang Servers With a Handler and Middleware

Building a Store Package to Sync Data in Go with Raft's FSM

Testing Distributed Golang Webservers in Docker Containers

Distributed Key Value Server

Monitoring

White Box Monitoring vs Black Box Monitoring

A Introduction to Golang Logging

Defining and Implementing Distributed System Tracing with Go

Alerting

Understanding the Basics of Canary Deployments

Wait, what's a load balancer?

What aren't we doing?

So what will we be doing?

How to Create a Basic Random-per-request Load Balancer

What is Weighted Round Robin Load Balancing?

Streamlining Configuration Updates for Canary Deployments

Other considerations

Three Open-Source Load Balancers Written in Go

A canary deployment server

Chat server

Chatting about chat servers

What features don't we want (yet)?

What features do we want?

Using a SQLite database in Golang

How to Create an API for a Golang Chat Server

Creating a Frontend Interface in Go for a Chat Server

How to Deploy a Golang Webserver With Docker

Future improvements

Future thinking

Project ideas

Areas of more research

In this book we will show you the some of the concerns of backend web development through the lens of Go.

It is taught by Nat Welch (formerly an SRE at Google) and Steve McCarthy (Etsy), two Go developers who have been writing software for more than 25 years combined.

In the course, we'll be covering the creation and concerns of a number of fundamental backend building blocks or larger applications:

- A key value server, including the use of distributed consensus algorithms
- A canary deployment server
- A real time chat service
- Testing and monitoring

Regardless of your background, Go can make you a _better backend developer_, but learning Go is challenging because the language looks simple on the outside, but you won't get its full benefit unless you know what it can _really_ do. 

**By the end of the first chapter, you'll already _have built a key-value server reminiscent of similar systems like Redis and memcached._** The subsequent chapters will dive into more hands-on projects and Go concepts including:

- How to test your Go systems with unit tests and integration tests
- **Project**: create a **distributed version of the key-value server** in order to solve the typical problems of server systems such as _node failure_
- How to solve the **problem of "consensus"** in distributed Go systems
- How to write code to **keep data in sync** when running a system for servers located around the globe
- Implementing **black box and white box monitoring** systems for Go servers
- **Project**: building a **load balancer server** from scratch
- Why and how to implement a **canary deployment server** for bug-free gradual deployments
- How to build a **configuration server** to allow zero-downtime deploys 
- Understanding the various tradeoffs in the **design decisions** for Go server systems
- **Project**: build a Go chat server system from the ground up 
- **Database modeling** for backend development as part of your chat server project
- **and a lot more**...

## A Key-Value Server Like Redis And Memcached

n the early chapters of the book we learn how to build a Golang version of a key-value server that works in a manner similar to Redis and Memcached. You'll implement each of these features in Go:

- set and retrieve values by key
- adding persistent storage
- using data encodings such as `JSON` and `base64`
- testing the server with unit tests and integration tests

In these chapters, we also learn how to build a more advanced solution for the problem: **a distributed version of the key-value server**. You will learn how to use **consensus algorithms** in Go to solve the problem of keeping distributed servers in sync. The solid grounding you get in programming Go in these first chapters readies you for more complex Go projects.

## A Real-Time Chat Service

A real-time chat server implemented in Go gives you the opportunity to bring together systems thinking as well as concerns common for web apps:

- relational data modeling with SQL
- authentication
- implementing RESTful resources

Working through this comprehensive project will give you **a high degree of confidence about using Go to implement a huge project from scratch**. You will feel ready to use Go with relational databases such as PostgreSQL or MySQL, and databases in general.

## A Canary Deployment Server And Load Balancer

The work you do in the chapters covering building a canary deployment server and load balancer will open your eyes to the full power of Go. Gradual deployments are a commonly used strategy for code deployments, and the projects you build in these chapters get you familiar with building this kind of **infrastructure software**. 

Your load balancer will implement _random-per-request_ as well as _round-robin_ load balancing. These projects will give you a thorough immersion in systems problem solving with Go. You'll be able to show off your skills on bold projects of your own, in job interviews, and on the job when collaborating with colleagues.

## This book is the _fastest_ way to learn modern backend Go development

With _The newline Guide to Reliable Webservers with Go_, you won't waste time reading shallow blog posts that can't go beyond showing you the simplest Go examples or Todo apps. Instead, **within the first chapter** you'll be well on your way to creating a **Redis-like key-value server**, and the complexity of the systems you're able to build only accelerates from there.

This book opens the door to **more effective backend development** and utilizing the full power of Golang for **microservices**. As Go transforms the systems development landscape, you can use this book and the skills it teaches you to contribute to a faster, more effective systems development experience for your team.

With _The newline Guide to Reliable Webservers with Go_ you can **go from novice to proficiency in Go in a predictable way, without wasting time, and without picking up bad Go coding habits**.

So grab a copy now to advance your journey with Go.

In this book we show important concepts in backend web development through the lens of Go.

Reliable Webservers with Go

The importance of testing your web servers

How to apply distributed consensus algorithms to your key value server

How do we want to store our chat history?

As is common with every web application that wants to store data, we need a database. But there are many kinds of databases which differ based on the storage needs of an application. The main database types fall within 

 categories, also sometimes known as SQL and NoSQL.

The primary requirements for our database are:

So we are going to opt for a relational database.

We could have chosen some variety of NoSQL database, but not all of them meet both our requirements. Key-value stores, for example, often don't allow one to access data in an ordered fashion. Their main purpose is to provide specific pieces of data as fast as possible, rather than letting you search for a collection of data that meets certain criteria.

Document stores, another type of NoSQL database (the most popular of which is probably MongoDB) do allow for the ordered retrieval of documents, but don't always have the ability to retrieve data relating to other data in the database. We could certainly use a document store for our chat server, but due to the way our data will be stored, a SQL database matches our needs best.

Among SQL databases, we have a number of options: PostgreSQL, MySQL and SQL Server are the most common server variants of SQL, while SQLite is the most common embedded variant. Any of them would be fine choices, but because we are erring on the side of simplicity here, we will use 

Having an embedded database will make the setup easier. You will not need to set up a database server, or an admin account, or ensure that there is a user with the proper privileges available. You only need to know what file you want SQLite to use.

Now that we know what database we will be using, we can talk about how we will be using it.

Object-relational mappings, or ORMs, are an incredibly useful tool in the web development kit. They create a common interface to interact with any flavor of SQL, and set up a mapping between data objects as they exist in the program and data objects as they exist in the database.
One could still use SQL to do this mapping, and many ORMs even have a RAW SQL concept, allowing the user to write a SQL query that will be directly executed on the database, rather than generated from the ORM constructs. But more typically that option is only used for more complicated SQL queries, and most of the common cases are very well covered by the ORM.

ORMs are also useful for other features they offer for the data storage lifecycle. You can generally add hooks to various points when changing data, for example:

Hooks around lifecycle events are a great reason to use an ORM. In this chapter, we will be using 

. It is one of the most popular Go ORMs, with all of the features we need. It's also relatively straightforward and easy to work with.

Although there are a large number of solid Go ORMs, another very popular one that deserves a specific mention is 

. It's definitely worth taking a look at if you find that GORM is missing a feature you particularly like, or if you just want a slightly different take on what an ORM written in Go can look like.

Now that we have our general feature set and know what we're building, it's time to put together most of our database-related code.

 struct. It will store every piece of state that we will need in our handlers. The reason for setting up this struct is to avoid needing to have global state. For the moment, the only thing we need to store is a database connection, making our 

 for our ORM, so the database connection is a 

 struct. Because GORM will use this struct to create the database table that users will be stored in, we want this definition to include everything that the table does, including relationships to other tables. In this case, the only relationship will be to 

, because all messages are sent by users, so a 

type User struct {
	gorm.Model
	Name     string `gorm:"uniqueIndex"`
	Password string
	Messages []Message `gorm:"foreignKey:UserName"`
}

Let's go through the user model. As mentioned, it has a foreign key relationship to 

 (a struct we will write next). It also uses 

, which automatically includes an ID, and CreatedAt, UpdatedAt, and DeletedAt fields. The ID is a 

. GORM explicitly shows the definition of 

 field. That way, we ensure all names are unique and we won't run into any user name confusion in our chat app.

. GORM will recognize this as a foreign key relationship, as defined in the 

 section of their documentation. By default it will use a 

 struct. However, because it is important to know who sends any particular message, and not just their ID, we can set the foreign key to be the 

---
title: "Using a SQLite database in Golang"
# description: desc
# privateVideoUrl: url
---

# How do we want to store our chat history?

As is common with every web application that wants to store data, we need a database. But there are many kinds of databases which differ based on the storage needs of an application. The main database types fall within _relational_ and _non-relational_ categories, also sometimes known as SQL and NoSQL.

The primary requirements for our database are:

-   retrieving data in an ordered fashion
-   retrieving data that relates to other data we are storing

So we are going to opt for a relational database.

We could have chosen some variety of NoSQL database, but not all of them meet both our requirements. Key-value stores, for example, often don't allow one to access data in an ordered fashion. Their main purpose is to provide specific pieces of data as fast as possible, rather than letting you search for a collection of data that meets certain criteria.

Document stores, another type of NoSQL database (the most popular of which is probably MongoDB) do allow for the ordered retrieval of documents, but don't always have the ability to retrieve data relating to other data in the database. We could certainly use a document store for our chat server, but due to the way our data will be stored, a SQL database matches our needs best.

Among SQL databases, we have a number of options: PostgreSQL, MySQL and SQL Server are the most common server variants of SQL, while SQLite is the most common embedded variant. Any of them would be fine choices, but because we are erring on the side of simplicity here, we will use **[SQLite](https://sqlite.org/index.html)**.

Having an embedded database will make the setup easier. You will not need to set up a database server, or an admin account, or ensure that there is a user with the proper privileges available. You only need to know what file you want SQLite to use.

Now that we know what database we will be using, we can talk about how we will be using it.

### Object-relational mapping (ORM)

Object-relational mappings, or ORMs, are an incredibly useful tool in the web development kit. They create a common interface to interact with any flavor of SQL, and set up a mapping between data objects as they exist in the program and data objects as they exist in the database.
One could still use SQL to do this mapping, and many ORMs even have a RAW SQL concept, allowing the user to write a SQL query that will be directly executed on the database, rather than generated from the ORM constructs. But more typically that option is only used for more complicated SQL queries, and most of the common cases are very well covered by the ORM.

ORMs are also useful for other features they offer for the data storage lifecycle. You can generally add hooks to various points when changing data, for example:

-   you could set up some code to run directly after a user has been created to complete other parts of your user setup flow
-   you could run a check before a user is saved to ensure that the data being added passes validation that is outside of the database

Hooks around lifecycle events are a great reason to use an ORM. In this chapter, we will be using [GORM](https://gorm.io/). It is one of the most popular Go ORMs, with all of the features we need. It's also relatively straightforward and easy to work with.

Although there are a large number of solid Go ORMs, another very popular one that deserves a specific mention is [Beego](https://beego.me/). It's definitely worth taking a look at if you find that GORM is missing a feature you particularly like, or if you just want a slightly different take on what an ORM written in Go can look like.

### Writing our database code

Now that we have our general feature set and know what we're building, it's time to put together most of our database-related code.

Start with a `database.go` file.

The first thing we need is a general `Config` struct. It will store every piece of state that we will need in our handlers. The reason for setting up this struct is to avoid needing to have global state. For the moment, the only thing we need to store is a database connection, making our `Config` struct rather simple:

```go
type Config struct {
	DB *gorm.DB
}
```

As mentioned, we will be using [GORM](https://gorm.io/) for our ORM, so the database connection is a `*gorm.DB`, and we need to add `gorm.io/gorm` to our imports at the top of the file.

#### Our user model

Moving on, we need to define our `User` struct. Because GORM will use this struct to create the database table that users will be stored in, we want this definition to include everything that the table does, including relationships to other tables. In this case, the only relationship will be to `Message`, because all messages are sent by users, so a `User` connects to every `Message`.

```go
type User struct {
	gorm.Model
	Name     string `gorm:"uniqueIndex"`
	Password string
	Messages []Message `gorm:"foreignKey:UserName"`
}
```

Let's go through the user model. As mentioned, it has a foreign key relationship to `Message` (a struct we will write next). It also uses `gorm.Model`, which automatically includes an ID, and CreatedAt, UpdatedAt, and DeletedAt fields. The ID is a `uint`, DeletedAt is a `sql.NullTime`, and the remaining fields are `time.Time`. GORM explicitly shows the definition of `gorm.Model` in [their documentation](https://gorm.io/docs/models.html#gorm-Model).

We are adding a `gorm:"uniqueIndex"` on the `Name` field. That way, we ensure all names are unique and we won't run into any user name confusion in our chat app.

It also includes `Messages`, which is a slice of `Message`. GORM will recognize this as a foreign key relationship, as defined in the [has many](https://gorm.io/docs/has_many.html) section of their documentation. By default it will use a `UserID` field included in the `Message` struct. However, because it is important to know who sends any particular message, and not just their ID, we can set the foreign key to be the `UserName` instead.

##### An aside on the foreign key decision

Having a string foreign key is uncommon because of the size of the index that is created for the foreign key. Typically foreign keys are numbers which allows the index on the key to be smaller, because the key itself will typically be smaller. Strings are quite variable in size and frequently larger than integers.

However, this decision is convenient because our `Message` model will be able to have all the necessary information to display a message within itself, so we won't need to do a SQL JOIN in order to retrieve the contents of the message. That could be nice for a potential future where the database is sharded and JOINs become much more complicated.