Tech Stack Overview: Homechart's Go API Server

August 10, 2022 Mike

In part one of Homechart’s tech stack overview, we’ll be exploring Homechart’s HTTP API server component. This component provides access to Homechart’s data, performs background tasks on the data, and serves the Homechart UI assets. All from a single binary written in Go/Golang.

Codebase Layout

The Homechart API component is a HTTP web server written in Go. Within the codebase, we have broken out the code into modules, all compiled into a single binary. Each module covers an independent piece of functionality, here are a few of the important ones:

  • config - Homechart configuration management
  • controllers - HTTP handlers
  • logger - Global loggers and tracing
  • models - Data models managing the various object types in Homechart’s database
  • postgresql - Functions for interacting with PostgreSQL
  • tasks - Background task runner

config

There are many ways to configure Homechart, and all of them happen within this package. It contains the Homechart config schema, which can be modified using environment variables, a config JSON file, or even HashiCorp Vault. You can read more about how our config works and the schema for it on our docs website.

controllers

Controllers in Homechart represent the API endpoint handlers. These handlers are responsible for validating permissions, inputs, and orchestrating model changes. An API request URL typically maps to a single handler, such as a POST to /api/v1/plan/tasks uses the PlanTaskCreate handler. We use the Chi library for our router.

Chi routing example

Middleware

Before most requests hit their designated handler, they are intercepted by middleware functions. These middleware functions run validations such as authentication and authorization checks, add contexts, or increment metrics.

OpenAPI

The API controller code is also annotated with comments to generate OpenAPI/Swagger documentation. This process uses a tool called swag. The API documentation is generated dynamically as part of each release, allowing us to keep our API documentation always up-to-date. You can find it under /api/docs on every Homechart instance.

OpenAPI annotations

Response Wrapping

To allow our UI and other API clients to have a well-known schema for parsing responses and reduces the amount of code needed to do it, most of the responses from Homechart are wrapped in a common object schema:

Homechart's HTTP response

SSE

The Homechart UI displays data changes in real time, so that if your partner updates the shopping list you’ll see those changes immediately on your end, too. The technology behind this is Server Sent Events (SSE).

The Homechart UI starts listening for SSE notifications as soon as it’s opened, and these notifications tell the UI when to pull down new data or remove deleted items.

Why not use WebSockets? Mostly because SSE is simpler, automatically reconnects, and we don’t need bi-directional communication (the UI/API clients can just perform REST operations).

Static Files

A key part of Homechart is having everything in a single binary. This includes our UI files (HTML/JS/CSS). Go gives us the flexibility to embed the UI files (via embed), serve the files from a local directory, or proxy to a dev server (like Vite):

Serving embedded files

logger

The Homechart logger module has two main responsibilities: print logs to the console and send traces to a telemetry service like Jaeger. It exposes one simple function, logger.Log(ctx context.Context, err error, msg ...string), which checks the error and other context data to extract log data and determine the log level:

  • DEBUG - Traces and session debugging
  • NOTICE - Security or authentication events
  • ERROR - Server-side errors that probably should not happen

Every function in Homechart logs at the end of its completion, regardless of if it succeeds or not. This lets us step through events in the logs or in traces very easily, as seen in this log output from a DELETE request to /api/v1/health/items/:id:

Verbose console logs

Or use Jaeger and see the output even easier:

Clean Jaeger traces

models

Models contains all of the data objects for Homechart and methods for create/read/update/delete (CRUD).

Organization

Because of its vast functionality, Homechart has a lot of unique data types. The top level data types are either an AuthAccount (representing a user’s account) or an AuthHousehold (representing a a user’s household). Users can create or join multiple Households, and those relationships are captured via an AuthAccountAuthHousehold. Almost all objects in Homechart are owned by either an AuthAccountID or an AuthHouseholdID.

An example data type

Interfaces

Over the course of developing Homechart, a number of abstractions were invented as similar functions were written to handle CRUD operations. These abstractions, known as interfaces in Go, enforce a contract for our data types and help keep things consistent.

The Model interface

You may be wondering why our public interface doesn’t expose any useful public methods (in Go, if the name of something doesn’t StartWithACapital, it won’t be usable by other packages). We call a few helper functions before most CRUD operations (like permission checks, or setting IDs), and so export wrapper functions instead:

Create wrapper function

Converting our models to this format wiped a ton of redundant code and uncovered some inconsistent behavior with how models were being used.

ReadAll

One of the main requirements for Homechart is the ability to work offline. Internet can be spotty, and the data in Homechart needs to be available anywhere. To facilitate this, the Homechart UI reads everything when requesting data for an object, like PlanTasks.

Using sophisticated caching headers, we can deliver per-table, per-user differential updates and cache the result. This lets the UI work with all available data, always, and dramatically reduces the load to the Homechart API server.

postgresql

PostgreSQL is the real hero behind the scenes for Homechart. It’s extremely powerful and flexible, and there has yet to be a data query or manipulation that PostgreSQL couldn’t perform.

Homechart performs direct queries against PostgreSQL via a combination of pq (soon to be pgx) and sqlx. This allows us complete control over indexing, triggers, and functions, at the cost of having to write all of the SQL queries ourselves.

A simple SQL query

Listeners

Using LISTEN/NOTIFY, every Homechart API server connected to PostgreSQL can relay messages between them via PostgreSQL and receive messages from PostgreSQL. We use this extensively to generate SSE notifications–broadcast a message to all Homechart API instances via PostgreSQL so they can relay the messages to their SSE clients.

Locks

The Homechart API server is also a task runner. Due to the nature of tasks, there needs to be a coordinator to perform or delegate tasks to avoid multiple instances running the same task (and our users getting repeat emails about a task being due or an event happening).

Instead of trying to add distributed consensus to Homechart or adding another technology for coordination, Homechart uses PostgreSQL Advisory Locks to control who is the task runner. These locks will last as long as the instance is online, so having the other instances periodically attempt to claim the lock allows us to rapidly failover this role if that instances goes offline.

Caching

Homechart originally used your typical Redis/in-memory key/value store for caching. This required our users to have Redis or sacrifice memory for cache purposes, and it imposed a constraint on Homechart to keep the cache fresh and aggressively clear it if there’s any chance that it’s stale.

This was a lot of work for not a lot of gain, so we moved caching to an UNLOGGED PostgreSQL table a few releases back. This reduced a lot of moving parts for the API server, as we no longer had to run a separate cache busting Goroutine and maintain code to cache bust individual data types. Some other benefits:

  • No Redis or other caching tier needed
  • PostgreSQL does a great job caching data already, so any memory we add to it will be shared equally between the cache and other database functions
  • We can use database triggers to clear stale caches natively
The cache table

tasks

Homechart’s API server not only processes REST transactions, but it also performs a number of background tasks on a schedule: clean up old data, send notifications, calculate metrics, etc. Using Goroutines, we can trivially farm out this work on demand within the API server process instead of needing a separate utility or app to run them.

The task scheduler

Summary

Writing Homechart in Go helped us iterate fast and avoid pulling in a lot of dependencies:

  • The stdlib is extremely comprehensive and frees us from dependency hell
  • Goroutines allow us to ffortlessly distribute work
  • Embed lets us deliver our software as a single binary

We hope you enjoyed this overview of Homechart’s API. In part two, we’ll explore the Homechart UI architecture, and in part three, we’ll checkout our tooling and CI/CD processes.