October Recap

11/13/202412:00:00 AM

I've always wanted to do some sort of "what I'm working on" or personal changelog esq post, so I'm gonna split the difference and try something new.

jorb

jorb is a small Go library that implements a basic job queue on top of Postgres. Internally I use this for a few web services that need background job scheduling, and it was built with 10+ years of dealing with redis/rabbitmq bullshit in mind. While I've had jorb floating around for a while October marks the public open-source release.

yamon

yamon is another repo I've had kicking around in personal use for a while, and is just now reaching maturity for open-source release. yamon is a simple metrics/logs/events collector for observability data, similar to a otel or datadog agent/collector. yamon aggregates all data to central servers and then inserts it into ClickHouse for long term storage and processing.

saint

saint screenshot showing visualizations of system host performance

saint is a new project I'm working on for a few prospective clients. It builds on top of some previous work and implements a browser-based observability solution that lets you write Deno-based TypeScript to produce rich and interactive dashboards. All the heavy data querying and processing logic in your TypeScript is ran on the backend within a Deno sandbox, and data is sent to the browser where it can be rendered via a variety of panels (graphs, tables, etc).

For the moment I have saint plugged into my personal ClickHouse instance (we do have a few demos that read sqlite3/etc files, but those are just demos for the moment) which stores monitoring data generated via yamon for my personal infrastructure. At the time of writing I'm storing 55 billion metrics, 78 million log lines, and around a million generic "events".

TypeScript/Deno - Scripting Is Love

It's no shock I've built another tool that implements scripting with Deno, but I'd argue it really is the perfect scripting runtime:

Fast V8 is really good and while Deno is still maturing much of its surface area is more than performant enough for 99% of use cases.
Types strictly define and document your API making it trivial for users to integrate and build.
Tooling is perversive and flexible due to the lack of solid standard TypeScript use
Libraries Deno's wide support for npm and jsr modules lets you pick pretty much anything off the shelf that you need. I've been impressed with how robust the Node support is, seemingly any npm module you want just works (with some small pains and rare caveats).
Simple often scripting environments are designed to be used by inexperienced users, who may just need the simplicity of JavaScript.

With that tangent aside here is a snippet of a code that renders a dashboard on saint, demonstrating how easy it is to create powerful abstractions that let you wip up informative dashboards with rich visualizations in seconds:

import { dashboard } from "@saint/sdk/mod.ts";
import { host, interval, makeGraph } from "../common.ts";

const dataset = makeGraph("zfs.dataset", {
  z: (i) => `${i.name} ${i.dataset}`,
  extraFields: {
    "tags['dataset']": "dataset",
  },
});
const zfetch = makeGraph("zfs.zfetch");

export const example = dashboard("zfs", [[dataset, zfetch]], {
  title: "ZFS",
  vars: [interval, host],
  tags: ["infra", "host"],
  refreshInterval: "15 seconds",
});

The combination of TypeScript and ClickHouse is pretty lethal, as 30 days of data render on this dashboard in 370ms, and it looks pretty good too!

image showing a demonstration of what the above example code produces, two charts with ZFS metrics displayed

Long Term