Posts on Very Good Software, Not Virus

AI as a means of evading accountability

Tue, 08 Apr 2025 11:04:56 -0500

There’s no doubt in my mind that the unifying characteristic of modern leadership is evasion of accountability and dereliction of responsibility. We’ve seen it in politics for probably as long as the sport has existed, but the recent Signal debacle was really a shining example of this trend. If I, as an employee of a private S-corp, discussed anything nearly as sensitive or detailed about my company’s inner dealings on a Signal group that had a lurking journalist, I’d be fired and at best have to take some lesser position for the rest of my career, but if you’re willing to fellate a dementia-addled billionaire, you will face zero consequences. Lack of accountability is hot right now, and it’s only getting hotter.

Speaking of work, recently the company I work for hired a new CTO, whose introductory statement to the company included a commitment to “jump start our usage of AI” at the company. They set a humble goal of merely having AI write most of our tests as well as implement new features by the end of the year. Setting goals we won’t meet is old hat for the company, but its fetishization of AI is comparatively new, though not uncommon. AI has become the hot topic that every company must acknowledge, lest they be regarded as ignorant. Every CEO that might emphasize how their company is meeting the needs of its customers in order to have those customers delightfully return to you in the future is met with a “who gives a fuck, how are you shoehorning LLMs into every layer of your operation?”

I find the whole thing laughable. The reason features are slow to develop at the company I work at is, simply put, not because of implementation speed. An LLM is going to have just as hard a time gettings accurate requirements out of an offshored product manager as a human being will.

The lack of accountability trend is older, but AI allows it to propagate, and in a way enforces it. Let’s say my company did somehow manage to write a feature with AI, who is going to be in the Zoom call when it doesn’t work as expected? Who is going to be held accountable when it’s not delivered on time because the LLM didn’t account for edge case X or stipulation Y? Who will be in the postmortems when the service it builds goes down? Will the AI be the incident commander?

I saw a post on BlueSky today about a new-to-me command of the docker CLI: docker ai.

god save us from ourselves

[image or embed]
— Xe (@xeiaso.net) April 7, 2025 at 10:41 AM

You can do things like docker ai run redis, and it will run commands for you, without telling you the output:

Tried it on my work machine and it was able to create a Redis deployment. The prompt I gave was "set the password to hunter2" and it did not. The more horrifying part is that when I asked it to clean up the deployment it asked if it could remove a deployment WITHOUT SHOWING THE ARGUMENTS TO THE TOOL

[image or embed]
— Xe (@xeiaso.net) April 8, 2025 at 10:04 AM

Who is going to be fired when the company discovers straggler Kubernetes clusters running myriad abandoned services that don’t even serve traffic, but do consume compute? AI allows the C-suite to take credit for pace of innovation, and mandates that human beings pay the prices for their failings.

For whatever it’s worth, I actually love writing code with the assistance of LLMs, and the “thinking” models fascinate me more than I care to admit. They’re really useful assistants (keyword), and they may even one day be capable of doing the things the aforementioned CTO insisted were possible today, in simpler cases, but every experience I’ve ever had professionally tells me that the vast majority of productivity failures at companies are simply not because developers don’t write code fast enough.

And what of the Juniors? I got my start in this industry by learning how to code during every free bit of time I had, and then taking a relatively low-paying position at a startup doing precisely the kinds of tasks we’re delegating to soulless robots today. Are programmers destined to be the factory workers of tomorrow, in that we’ll be replaced by machines, and only a few very experienced workers will be necessary to fill in the gaps those machines leave behind?

Johnny Boursiquot summarizes my thoughts on this succinctly:

If you're a Senior+ Software Engineer today, you're among the last of your kind.
Think about that for a minute.
— Johnny Boursiquot (@jboursiquot.com) March 6, 2025 at 3:36 PM

How I write tests in Go

Wed, 03 Apr 2024 17:38:19 -0600

How I write unit tests in Go

One of my favorite features of Go is that unlike many popular languages, it comes with it’s own testing framework, the testing package. Let’s say we have this trivial function in a file called numbers.go:

package numbers
func addNumbers(numbers ...int) int {
sum := 0
for _, number := range numbers {
sum += number
}
return sum
}

The idiomatic way to test this function is to create a second file called numbers_test.go and test it using the aforementioned testing package:

package numbers
import (
"testing"
)
func Test_addNumbers(t *testing.T) {
if addNumbers(3, 7) != 10 {
t.FailNow()
}
}

Then I can run go test and see the output of the test:

ok github.com/example/package/numbers

Alternatively, if the test fails (i.e. if I expect the result to be 11 instead), I get an appropriate error report:

--- FAIL: Test_addNumbers (0.00s)
FAIL
FAIL github.com/example/package/numbers 0.255s
FAIL

This works pretty well, overall. When I need to find the tests for a given file, I can usually assume they’re in its corresponding _test.go file. The testing framework is built into the standard library, and does exactly what you’d expect with this code.

We have the technology

This isn’t how I actually like to write Go unit tests, however. There are techniques and libraries I use to make unit testing in Go more effective to write and understand when things go (heh) awry.

The test command

You can run go test in a given directory and see the output of your test, but that leaves a lot of functionality on the table that can and will be useful. The full command I typically use is:

go test -cover -shuffle=on -race -vet=all -failfast <$RELEVANT_PACKAGE(S)>

Here’s what each of these flags do:

-cover instructs the test runner to determine coverage for a given package.
-shuffle=on runs the tests in a random order, instead of in declared order.
-race activates the data race detector.
-vet=all “runs go vet on the package and its test source files to identify significant problems. If go vet finds any problems, go test reports those and does not run the test binary.”
-failfast stops test execution when a given unit test fails. It allows tests executed in parallel (more on that in a bit) to finish.

Table-driven tests

I’m going to say something sort of controversial in the Go zeitgeist: I find table tests very rarely useful. For something like our addNumbers function, I might actually do it, because the inputs and outputs are simple. Here’s what that might look like:

package numbers
import (
"testing"
)
type addNumbersTestCase struct {
inputs []int
expectedOutput int
}
func Test_addNumbers_table(t *testing.T) {
testCases := []addNumbersTestCase{
{
inputs: []int{4, 4},
expectedOutput: 8,
},
{
inputs: []int{10, 0},
expectedOutput: 10,
},
{
inputs: []int{-1, 1},
expectedOutput: 0,
},
}
for _, testCase := range testCases {
actual := addNumbers(testCase.inputs...)
if testCase.expectedOutput != actual {
t.FailNow()
}
}
}

That said, it’s pretty easy to end up in a situation where your table tests need to be quite complicated and gnarly to get full coverage.

Goland lets you generate tests for functions using a default template that attempts to adhere to the tradition of table-driven tests. Here’s what the generated test function looks like for a method in a side project. This method fetches a webhook object from the database, and accepts 3 total parameters — a context, a webhook ID, and an accountID:

package whatever
func TestQuerier_GetWebhook(t *testing.T) {
type fields struct {
tracer tracing.Tracer
logger logging.Logger
secretGenerator random.Generator
oauth2ClientTokenEncDec encryption.EncryptorDecryptor
generatedQuerier generated.Querier
timeFunc func() time.Time
config *dbconfig.Config
db *sql.DB
migrateOnce sync.Once
}
type args struct {
ctx context.Context
webhookID string
accountID string
}
tests := []struct {
name string
fields fields
args args
want *types.Webhook
wantErr assert.ErrorAssertionFunc
}{
// TODO: Add test cases.
 }
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
q := &Querier{
tracer: tt.fields.tracer,
logger: tt.fields.logger,
secretGenerator: tt.fields.secretGenerator,
oauth2ClientTokenEncDec: tt.fields.oauth2ClientTokenEncDec,
generatedQuerier: tt.fields.generatedQuerier,
timeFunc: tt.fields.timeFunc,
config: tt.fields.config,
db: tt.fields.db,
migrateOnce: tt.fields.migrateOnce,
}
got, err := q.GetWebhook(tt.args.ctx, tt.args.webhookID, tt.args.accountID)
if !tt.wantErr(t, err, fmt.Sprintf("GetWebhook(%v, %v, %v)", tt.args.ctx, tt.args.webhookID, tt.args.accountID)) {
return
}
assert.Equalf(t, tt.want, got, "GetWebhook(%v, %v, %v)", tt.args.ctx, tt.args.webhookID, tt.args.accountID)
})
}
}

I think this is kind of a mess, and it doesn’t even include test cases yet. I’d rather have the querier built by an only-available-to-tests constructor that puts sensible defaults for all these values.

The point of this is not to dunk on Goland (I am a happily paying customer and have been for many years), it’s just to illustrate how even a non-trivial function can really blow up the readability of a table-driven test.

Parallel

Go tests can be run in parallel, by declaring the test as parallel in the top level. This behavior is on by default, and can only be disabled by setting the -parallel=1 flag when invoking go test (which can be useful when debugging flaky tests).

I’m of the opinion that tests should be parallel unless they absolutely cannot be, and even then, I’d rather fix the flaw behind why they can’t be run in parallel than force them to be run sequentially.

Running your tests in parallel generally allows you to suss out concurrency bugs earlier, even if you’re not using the race detector (which you should), and will eventually lead to you developing concurrency-safe habits when writing code in the first place. So generally, when I write a test, the very first line is the parallel declaration:

package numbers
import (
"testing"
)
func Test_addNumbers(t *testing.T) {
t.Parallel()
expected := 10
actual := addNumbers(3, 7)
if expected != actual {
t.FailNow()
}
}

Subtests

Subtests are a feature of Go’s testing library that allow you to have smaller tests for specific circumstances. These allow you to get some of the benefits of a behavior-driven testing style, without having to invoke an external dependency. I usually write subtests even when there’s only one test case I care about, because it incurs no real overhead to do so, and allows you to easily test a new case in the event the tested function gets more complex.

When I write subtests, I usually name the top-level *testing.T object a capitalized T, and use the conventional lower-cased t for subtests only. This way, the subtest logic always looks like an idiomatic Go test, and there’s no confusion of what variable is which or what shadows what. Calls to .Parallel() will have to be made for both the big and little *T variables.

Here’s a demonstration that adapts our earlier example for the addNumbers function:

package numbers
import (
"testing"
)
func Test_addNumbers(T *testing.T) {
T.Parallel()
T.Run("standard", func(t *testing.T) {
t.Parallel()
expected := 10
actual := addNumbers(3, 7)
if expected != actual {
t.FailNow()
}
})
}

Now if I decide to add another test case, it’s simply a matter of copying the subtest block, changing the name, and then making the test match that description:

package numbers
import (
"testing"
)
func Test_addNumbers(T *testing.T) {
T.Parallel()
T.Run("standard", func(t *testing.T) {
t.Parallel()
expected := 10
actual := addNumbers(3, 7)
if expected != actual {
t.FailNow()
}
})
T.Run("with many numbers", func(t *testing.T) {
t.Parallel()
expected := 15
actual := addNumbers(1, 2, 3, 4, 5)
if expected != actual {
t.FailNow()
}
})
T.Run("with negative numbers", func(t *testing.T) {
t.Parallel()
expected := 3
actual := addNumbers(4, -1)
if expected != actual {
t.FailNow()
}
})
}

Additionally, if there are test fixtures or constants that are relevant to all subtests, they can live in the space between the big-T call to .Parallel() and the first declared subtest.

Libraries

While the standard library testing package is pretty great, there have been some great contributions by the community to the ecosystem that make testing in Go even better.

Fake Values

Typically, when I need to interact with an object in a test, I want to have unpredictable values to the extent possible for that object, so that I’m discouraged from relying on those conventions when writing tests.

So for instance, if I need a *User object for a test, I want to have no idea what it’s username might be, as opposed to a common static value that I might copy/paste from place to place, which would make many tests fail if it was inadvertently changed.

I like to use brianvoe/gofakeit for this. It comes with a ton of sensible functions with which to build fake instances of different types, and I’ve never had a problem with it. Here’s what that fake user builder might look like:

package whatever
import (
"github.com/example/project/internal/authorization"
"github.com/example/project/internal/pkg/pointer"
"github.com/example/project/pkg/types"
fake "github.com/brianvoe/gofakeit/v7"
)
func buildFakeTime() time.Time {
return fake.Date().Add(0).Truncate(time.Second).UTC()
}
func BuildFakeUser() *types.User {
fakeDate := buildFakeTime()
return &types.User{
ID: identifiers.New(),
FirstName: fake.FirstName(),
LastName: fake.LastName(),
EmailAddress: fake.Email(),
Username: fmt.Sprintf("%s_%d_%s", fake.Username(), fake.Uint8(), fake.Username()),
Birthday: pointer.To(buildFakeTime()),
AccountStatus: string(types.UnverifiedHouseholdStatus),
TwoFactorSecret: base32.StdEncoding.EncodeToString([]byte(fake.Password(false, true, true, false, false, 32))),
TwoFactorSecretVerifiedAt: &fakeDate,
ServiceRole: authorization.ServiceUserRole.String(),
CreatedAt: buildFakeTime(),
}
}

Testify

I’m a huge fan of the (assert|require|mock) libraries in testify. In our example test, using testify would look like:

package numbers
import (
"testing"
"github.com/stretchr/testify/assert"
)
func Test_addNumbers(T *testing.T) {
T.Parallel()
T.Run("standard", func(t *testing.T) {
t.Parallel()
expected := 10
actual := addNumbers(3, 7)
assert.Equal(t, expected, actual)
})
}

I think assert.Equal is fairly obvious in its function for users who are new to a code base. testify is probably the only library I can think of that I would wholeheartedly support being absorbed into the standard library.

You can use require instead if you want the test to stop in the event of a failure. This is usually good for dependencies of a test, i.e. if I need to create a file and pass it to a function, I should probably require that no errors happen when creating it:

package whatever
import (
"os"
"testing"
"github.com/stretchr/testify/require"
)
func TestSomeFunction(T *testing.T) {
T.Parallel()
T.Run("standard", func(t *testing.T) {
t.Parallel()
f, err := os.Create("some_path.txt")
require.NoError(t, err)
require.NotNil(t, f)
})
}

mock is also useful for defining mock implementations of structures. Say I have the following interface:

package something
type UserRetriever interface {
GetUser(ctx context.Context, userID string) (*User, error)
}

Implementing a mock for this interface becomes a matter of:

package something
import (
"github.com/stretchr/testify/mock"
)
type mockUserRetriever struct {
mock.Mock
}
func (m *mockUserRetriever) GetUser(ctx context.Context, userID string) (*User, error) {
returnValues := m.Called(ctx, userID)
return returnValues.Get(0).(*User), returnValues.Error(1)
}

Using the above mock in a test looks like this:

package something
import (
"testing"
"github.com/stretchr/testify/mock"
)
func TestUserRetrieval(T *testing.T) {
T.Parallel()
T.Run("standard", func(t *testing.T) {
t.Parallel()
expected := &User{ID: "something"}
mur := &mockUserRetriever{}
mur.On("GetUser", mock.Anything, expected.ID).Return(expected, nil)
// pass our mock to whatever uses it and test it's functionality

mock.AssertExpectationsForObjects(t, mur)
})
}

I won’t lie, I’m not deeply impressed with the way that mock.Mock determines which variables to return in what order, and when things go wrong (i.e. you copy and paste something in the wrong order or have the wrong indices on your return values) it’s VERY hard to suss out, but by and large, testify/mock does an incredible job of what it needs to do.

You can avoid having to put mock.Anything in an expectation call by writing a type matcher. For context.Context, it would look like this:

mock.MatchedBy(func(context.Context) bool { return true })

There’s also testify/suite, which I have used, but wouldn’t recommend. It’s useful when you have a ton of common prerequisites for tests that need to be spun up ahead of time, but I still think it makes more sense to just make those prerequisites the product of a common function, instead of writing non-standard test code.

Testcontainers

This is a neat tool I’ve only recently started using, but to great effect. It’s very common to have to write code that interfaces with a database, or a k/v store, or an in-memory cache, but before testcontainers, you either had to write a suite of integration tests that used that code to verify it did what you wanted, or otherwise fly blind in production. With testcontainers, I can spin up ephemeral containers that run a given piece of software and have my code interact with that to verify functionality.

In practice, it’s not perfect. These tests generally run great on my machine, and in Github Actions, but you must remember that you’re asking your computer to make a fake computer within itself, spin up a piece of software on that fake computer, and handle all the networking shenanigans that go along with interfacing with it, so there’s plenty of opportunity for mishaps and errors to occur.

Sometimes a testcontainer invocation will just fail, and the reason behind it can be summarized up as: computers are hard. Since they can be a big more error prone than standard tests, I don’t typically write subtests for these. Instead, I tend to write one big function that uses a single instance of the container to test a bunch of related functions.

Here’s an example of the test for that aforementioned webhook retrieval function:

package whatever
import (
"context"
"database/sql"
"fmt"
"testing"
"github.com/example/project/pkg/types"
"github.com/example/project/pkg/types/converters"
"github.com/example/project/pkg/types/fakes"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"gopkg.in/matryer/try.v1"
)
func buildDatabaseClientForTest(t *testing.T, ctx context.Context) (*Querier, *postgres.PostgresContainer) {
t.Helper()
dbUsername := fmt.Sprintf("%d", hashStringToNumber(t.Name()))
testcontainers.Logger = log.New(io.Discard, "", log.LstdFlags)
var container *postgres.PostgresContainer
err := try.Do(func(attempt int) (bool, error) {
var containerErr error
container, containerErr = postgres.RunContainer(
ctx,
testcontainers.WithImage(defaultImage),
postgres.WithDatabase(splitReverseConcat(dbUsername)),
postgres.WithUsername(dbUsername),
postgres.WithPassword(reverseString(dbUsername)),
testcontainers.WithWaitStrategyAndDeadline(2*time.Minute, wait.ForLog("database system is ready to accept connections").WithOccurrence(2)),
)
return attempt < 5, containerErr
})
require.NoError(t, err)
require.NotNil(t, container)
connStr, err := container.ConnectionString(ctx, "sslmode=disable")
require.NoError(t, err)
dbc, err := ProvideDatabaseClient(ctx, logging.NewNoopLogger(), tracing.NewNoopTracerProvider(), &config.Config{ConnectionDetails: connStr, RunMigrations: true, OAuth2TokenEncryptionKey: "blahblahblahblahblahblahblahblah"})
require.NoError(t, err)
require.NotNil(t, dbc)
return dbc, container
}
func createWebhookForTest(t *testing.T, ctx context.Context, exampleWebhook *types.Webhook, dbc *Querier) *types.Webhook {
t.Helper()
// create
 if exampleWebhook == nil {
exampleWebhook = fakes.BuildFakeWebhook()
}
dbInput := converters.ConvertWebhookToWebhookDatabaseCreationInput(exampleWebhook)
created, err := dbc.CreateWebhook(ctx, dbInput)
assert.NoError(t, err)
require.NotNil(t, created)
assert.Equal(t, exampleWebhook, created)
webhook, err := dbc.GetWebhook(ctx, created.ID, created.BelongsToAccount)
assert.NoError(t, err)
assert.Equal(t, webhook, exampleWebhook)
return created
}
func TestQuerier_Integration_Webhooks(t *testing.T) {
ctx := context.Background()
dbc, container := buildDatabaseClientForTest(t, ctx)
databaseURI, err := container.ConnectionString(ctx)
require.NoError(t, err)
require.NotEmpty(t, databaseURI)
defer func(t *testing.T) {
t.Helper()
assert.NoError(t, container.Terminate(ctx))
}(t)
user := createUserForTest(t, ctx, nil, dbc)
accountID, err := dbc.GetDefaultAccountIDForUser(ctx, user.ID)
require.NoError(t, err)
require.NotEmpty(t, accountID)
exampleWebhook := fakes.BuildFakeWebhook()
exampleWebhook.BelongsToAccount = accountID
createdWebhooks := []*types.Webhook{}
// create
 createdWebhooks = append(createdWebhooks, createWebhookForTest(t, ctx, exampleWebhook, dbc))
// other tests would go here

// delete
 for _, webhook := range createdWebhooks {
assert.NoError(t, dbc.ArchiveWebhook(ctx, webhook.ID, accountID))
var exists bool
exists, err = dbc.WebhookExists(ctx, webhook.ID, accountID)
assert.NoError(t, err)
assert.False(t, exists)
var y *types.Webhook
y, err = dbc.GetWebhook(ctx, webhook.ID, accountID)
assert.Nil(t, y)
assert.Error(t, err)
assert.ErrorIs(t, err, sql.ErrNoRows)
}
}

(I acknowledge that this code is missing some function definitions, but it’s meant for you to get an idea, not copy/paste)

Failure Cases

Some functions in the Go standard library return errors that you might surface to higher-level call sites, but in order to test that those errors are caught and returned, Here are some ways I manage to achieve failures for very specific circumstances in Go.

JSON Encoding

It’s a pretty common operation in Go code to take an instance of a given struct and render it as JSON. This function is capable of returning an error in the event the source struct is unrepresentable, but it’s actually pretty hard to make a struct unrepresentable in Go. You basically have to do it on purpose.

The way I achieve this is by using json.Number, a string alias meant to represent numbers, and giving it a non-numerical value. Here’s what that looks like:

type BreakableStruct struct {
Thing json.Number
}
func TestRenderToJSON(T *testing.T) {
T.Parallel()
T.Run("with invalid structure", func(t *testing.T) {
t.Parallel()
x := &BreakableStruct{Thing: "stuff"}
actual, err := json.Marshal(x)
assert.Nil(t, actual)
assert.Error(t, err)
})
}

URL Parsing

Parsing a URL in Go returns an error, but trying to force this error to occur was very confusing to me. I had to dive into the actual test cases in the standard library for net/url. Here are some inputs you might think return an error from url.Parse, but actually don’t:

an empty string
a URL with double dots in the path (i.e. https://google.com/../../var/data)
just a scheme (i.e. https://)
a URL with emojis in it (i.e. https://😘.🔥)
a URL with an invalid scheme (i.e. farts://)

All of these actually don’t return an error. The only way I was able to force it to occur by using:

fmt.Sprintf(`%s://`, string(byte(127)))

For whatever reason, this causes url.Parse to fail.

Conclusion

In many other languages, you have to not only evaluate testing libraries, but also write your tests in a style that complies with that library’s expectations. Gophers are blessed to have a thoroughly adequate solution out-of-the-box, and even further blessed to have an active ecosystem where folks are making in-depth testing a walk in the park.

Go’s Error Handling Is Perfect, Actually

Mon, 12 Feb 2024 21:34:37 -0600

Spend any amount of time in programming circles, and just as the sun rises and falls, you are certain to hear someone complain about error handling in Go. These complaints are, anecdotally, rarely well thought out suggestions on what error handling could or should be like in a language like Go, but often merely boil down to “I don’t like having to look at it”.

You’ll mainly see people complain about having to litter their apps with if err != nil, how they feel it makes the code verbose, and how they miss just writing ten very consequential lines one after the other in more dynamic languages like Python or Javascript.

Less often, you’ll see folks bemoan the lack of a stdlib-defined Result type, which would have either a value or an error, like in Rust. Every now and then you’ll see someone bemoan how error handling works in highly concurrent Go programs, or how some errors can be nil and not-nil simultaneously.

In defense of `if err != nil`

I have a spicy hot take on the matter: Go’s error handling is not just fine, not just great, but actually perfect for 99% of programs. (I will readily admit the last thing about error nilness is annoying, but in the nearly decade of time I’ve spent writing Go, I can think of perhaps a handful of times where it actually reared its head. I also ascribe that issue to the subject of nil in Go, which is a whole other topic, this article is about errors.)

I regard the matter like this: do you want to know when something bad can happen in your program, or don’t you? I asked ChatGPT to Please write some python code for me that goes through every row of a CSV file, performs a get request for the value in the first column, and parses the responses as a JSON map of strings to strings:

import csv
import requests
def process_csv(csv_file):
with open(csv_file, 'r') as file:
csv_reader = csv.reader(file)
next(csv_reader)
for row in csv_reader:
url = row[0]
data = requests.get(url).json()
print(data)
if __name__ == "__main__":
csv_file = "your_csv_file.csv"
process_csv(csv_file)

There’s basically no error handling here, but there’s a ton that can go wrong. What happens if the file doesn’t exist, or is corrupted? What happens if you don’t have permissions to read it? What happens if the GET request fails? What happens if the response body isn’t valid JSON, or doesn’t match the expected shape? The answer, in the case of Python, is an exception gets thrown, and since there’s no code to catch it, it’s handled by the broader runtime, printing a stack trace.

Traceback (most recent call last):
File "main.py", line 15, in <module>
process_csv(csv_file)
File "main.py", line 5, in process_csv
with open(csv_file, 'r') as file:
^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'your_csv_file.csv'

Note that for any sufficiently complex program that invokes many dependencies, this stack trace will be so far down the chain that you may not even see where you’re making the call that causes it.

I asked ChatGPT to write the same code, but in Go:

package main
import (
"encoding/csv"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
)
func fetchData(url string) (map[string]string, error) {
response, err := http.Get(url)
if err != nil {
return nil, err
}
defer response.Body.Close()
var data map[string]string
if err := json.NewDecoder(response.Body).Decode(&data); err != nil {
return nil, err
}
return data, nil
}
func processCSV(csvFile string) error {
file, err := os.Open(csvFile)
if err != nil {
return err
}
defer file.Close()
reader := csv.NewReader(file)
if _, err = reader.Read(); err != nil && err.Error() != "EOF" {
return err
}
for {
row, err := reader.Read()
if err != nil {
if err.Error() == io.EOF {
break
}
return err
}
url := row[0]
data, err := fetchData(url)
if err != nil {
fmt.Printf("Failed to fetch data from %s: %v\n", url, err)
continue
}
fmt.Printf("Data from %s:\n", url)
for key, value := range data {
fmt.Printf("%s: %s\n", key, value)
}
fmt.Println()
}
return nil
}
func main() {
csvFile := "your_csv_file.csv"
if err := processCSV(csvFile); err != nil {
fmt.Printf("Error processing CSV: %v\n", err)
}
}

Wouldn’t you know it, we have all the aforementioned errors handled! If there’s something wrong with the file, that will get surfaced. If there’s something wrong with the GET request, that will be surfaced. If the response doesn’t contain valid JSON, that will get surfaced. Is there more error handling code in the Go version? Yes, because that’s how Go is idiomatically written.

The Python I posted above, while it certainly could be written a better way, doesn’t look meaningfully different from 90%+ of the Python I’ve ever had to work with professionally. My only major gripe with the Go variant is that it doesn’t check the length of row before accessing that index, but that’s ChatGPT for you.

There are folks who will say that the Go version is less readable than the Python version. For me, this depends on how you measure readability. If you measure it from the time you first see the code to when you understand what it’s trying to accomplish, I could entertain the suggestion that Python wins. If you measure it from the perspective of how long it takes to suss out what the different execution paths or outcomes could possibly be from a given chunk of code, I think Go wins.

Even if you didn’t know that file reads could fail, or network requests could fail, you would understand that both are possible after reading the Go code, but not from the Python code.

I’d rather see a billion if err != nil statements in my code than have an error occur that I cannot quickly and effectively diagnose because it comes with a bunch of unrelated noise.

Result types in Go

I’ve seen the RIIR crew suggest that one of the things that would make Go tolerable to them would be some equivalent of the Result type in Rust. The Result type allows for you to return one value that can contain either the anticipated value, or an error, but not both. So you sort of get the chance to collapse the standard res, err := someOperation() pattern you see in a lot of Go into just res := someOperation().

I asked ChatGPT to Please write some rust code that returns a Result container with a string, have it check the current time and return an error if the unix timestamp is even or a positive result if the unix timestamp is odd:

use std::time::{SystemTime, UNIX_EPOCH};
fn check_current_time() -> Result<String, String> {
let current_time = SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs();
if current_time % 2 == 0 {
Err("Even timestamp".to_string())
} else {
Ok("Odd timestamp".to_string())
}
}
fn main() {
match check_current_time() {
Ok(msg) => println!("Result: {}", msg),
Err(err) => println!("Error: {}", err),
}
}

This is, frankly, neat, and not a bad idea. There’s an alternative universe where Go had generics from the jump and the Result type was implemented/utilized in the standard library, and it’s not the worst outcome I could think of. There are libraries you can use now, post-generics, that do just this, and an old and long-closed proposal to add it to Go specifically cites the Rust variant.

The only opposition I have to the Result type in Go is that we wouldn’t be able to make use of it in the standard library without either breaking backwards compatibility, writing Result variants of existing API calls (so NewRequest, NewRequestWithContext, and NewRequestWithContextAndResult), or issuing new /v2 variants of existing packages (like the recently-released math/rand/v2 package), which then means we’ll have some libraries and programs that use the old style with one return value, some with the new style, and many instances of confused programmers using the wrong one. It would be as close to a Go equivalent of the Python 2/3 transition debacle as I think we could manage.

I also don’t really think it meaningfully improves readability. Compare the above Rust code to the Go equivalent:

package main
import (
"errors"
"fmt"
"time"
)
func checkCurrentTime() (string, error) {
if time.Now().Unix()%2 == 0 {
return "", errors.New("even timestamp")
}
return time.Now().Format(time.Kitchen), nil
}
func main() {
result, err := checkCurrentTime()
if err != nil {
fmt.Println(err)
} else {
fmt.Println(result)
}
}

Our main function is 6 lines, compared to Rust’s 4. I suppose that adds up over time and with a larger project, but I still just don’t think it’s the massive win for readability that some folks proclaim it to be.

Conclusion

None of this post was meant to denigrate Python, Rust, Javascript, or any other language, or its fans, or indeed anything at all. I just think a lot of the criticism around this particular element of the Go programming language is missing the forest for the trees.

Preventing generated files from causing problems

Tue, 29 Aug 2023 23:40:19 -0600

The premise

Generating files is an inevitable consequence of writing most software, but in particular Go. For a very long time, the collective wisdom when asked about support for generics could be summarized as “just generate files for the types you need, that’s what generics support in the compiler would amount to,” and the ethos stuck with the community.

In my primary side project, for instance, I generate service configs from Config literals, so that I know they will properly unmarshal when the service tries to load them. I also use wire for dependency injection, which as you may have guessed, generates a big function with a bunch of constructors invoked in the correct order. A not-unpopular choice of library for interacting with SQL databases is the beloved sqlc, which generates Go code from properly annotated SQL query files.

The problem

The problem with generated files is that they can become out of data. Someone on the team changes a query in a PR, but doesn’t re-run the sqlc compiler to produce fresh output. Someone changes a config, which causes subsequent deployments to fail because the name of a JSON field no longer matches. Someone changes the order of parameters in a constructor, and all of its tests, but doesn’t re-run wire to ensure the build step fails. (The last one is actually fairly interceptable, assuming you’re doing the bare minimum of trying to build your binary in CI, but I’m not here to judge anybody’s shortcomings).

THe solution

How do I prevent these inevitabilities from ruining any given evening? I require that when myself or a team member introduces a new class of generated files, we also introduce a CI step that runs the required generation command, and fails if git detects changes afterwards. Here’s my sqlc setup, for instance:

on:
pull_request:
paths:
- cmd/**
- pkg/**
- internal/**
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
name: generated_files
jobs:
queries:
timeout-minutes: 10
strategy:
matrix:
go-version: [ '1.21.x' ]
runs-on: ubuntu-latest
steps:
- name: Install Go
uses: actions/setup-go@v4
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v3
- name: Ensure configs can be generated
run: make queries
- name: Check for changes
run: git diff --exit-code

This installs Go, checks out the code, runs the query generation command, and asks Git for any diffs and to fail if they’re present. Note that, in my case, I’m using make and Docker to compile the sqlc queries with docker run --rm --volume $(shell pwd):/src --workdir /src --user $(shell id -u):$(shell id -g) sqlc/sqlc:1.22.0 compile --no-database --no-remote, but in the event I couldn’t or weren’t, you’d also then be able to document exactly what tools are necessary for generated which files, which helps newcomers to the repository.

Bring your own interface

Tue, 29 Aug 2023 23:04:56 -0600

I have a side project which, like most good software, uses a structured logger. Initially I used logrus, then I used zap, and then I found zerolog, which I’ve used now for a number of years.

How did I switch loggers as many times without causing myself a headache? Easy, I maintained a simple Logger interface:

type Logger interface {
Info(string)
Debug(string)
Error(err error, whatWasHappeningWhenTheErrorOccurred string)
WithValue(string, any) Logger
WithValues(map[string]any) Logger
WithRequest(*http.Request) Logger
WithResponse(response *http.Response) Logger
WithError(error) Logger
WithSpan(span trace.Span) Logger
}

At every place in the code that has even the slight inclination of needing to log, I use this Logger type. This stands in contrast with most of the professional code I’ve written which makes use of a dependency-specific *zap.Logger everywhere. When I wanted to use zap instead of logrus, I very simply wrote a zap implementation of the above interface, and made use of it instead of the logrus one. They’re very shallow wrappers; here’s what WithValue looks like for the zap implementation, for example:

func (l *zapLogger) WithValue(key string, value any) logging.Logger {
return &zapLogger{logger: l.logger.With(zap.Any(key, value))}
}

The way the app conjures a Logger instance is from a logging-specific Config object, which has a method for providing a logger:

func (cfg *Config) ProvideLogger() logging.Logger {
if cfg == nil {
return logging.NewNoopLogger()
}
switch cfg.Provider {
case ProviderZerolog:
return zerolog.NewZerologLogger(cfg.Level)
case ProviderZap:
return zap.NewZapLogger(cfg.Level)
default:
return logging.NewNoopLogger()
}
}

Having things configured this way makes it trivial to switch between logging instances. I can change one line of config and have the entire application’s logging behavior change in response.

New Toys

Recently Go 1.21 introduced the slog package, which is the standard library implementation of a structured logger like those I mentioned above. I wanted to use it, and I was able to start doing so in my app by:

writing a slog-compatible implementation of the logging.Logger interface and
changing the config to specify that the slog logging provider should be used.
adding a case to the above switch statement to account for the new ProviderSlog option.

One quick PR and the whole app uses slog now.

It doesn’t stop at logging. I have similar interfaces for:

message queues (Redis, SQS, Pub/Sub)
search indices (Algolia, Elasticsearch)
talking to analytics providers (Segment, PostHog)
feature flag checking (LaunchDarkly, PostHog, Split)
sending emails through a service (Segment, Mailjet, Mailgun)
object storage (S3, GCS, local disk)

At any moment, the primary implementation provider for these core functions can be changed with a simple config update.

What makes a good use case for this pattern?

The functionality should have a core purpose that probably won’t meaningfully change. For instance, the message queue code will only ever deal with reading from or writing to message queues. At no point will it suddenly also be responsible for sending emails. Logging packages will only ever be responsible for logging values, etc.
There should also be a number of reasonable implementations. There are many logging libraries, email service providers, and there will be more of them in the future, too.
Little meaningful impact on switching between said providers. For example, I have an authentication package, which is responsible for verifying TOTP codes and checking submitted passwords against their hashes. I could hypothetically have a bcrypt implementation, and an scrypt implementation, but instead I only have an argon2id implementation, because if I were to suddenly switch from that to scrypt, nothing stored in the database would ever work properly barring a huge migration I wouldn’t want to account for in my implementation.

Adopting this pattern gives you flexibility at the cost of having to write a bit of glue code (like the Config stuff I showed above). Another benefit of this is that you get to define what you really need that interface for. Notice, for instance, that I don’t have a Warn method in my Logger interface. That’s because I basically never log at that level in practice. This is just a personal quirk of mine, I don’t know when you’d Warn in the API server code I write, so I just never found a need to implement it.

Sylvia's Law

Sun, 09 Jul 2023 00:45:12 -0500

The Life and Death of Sylvia Joyce

My aunt Sylvia was my grandmother’s first child. My grandmother had married my biological grandfather against her guardians' wishes, he was in the military, so she was entirely alone when Sylvia came. My aunt Sylvia would live a full life of ups, downs, and the myriad things in between. She got married, divorced, had children, found love again, vacationed, sang, sewed, and more than anything, loved her family. In 2018, she was diagnosed with metastatic liver cancer.

She spent her last days in a hospice care facility in San Antonio. We visited her and she remarked how the hospice care facility didn’t have good food, and all she wanted was some decent italian. It’s San Antonio, so they don’t have any, but I looked up the best-rated local italian place and had them deliver a whole lasagna and slice of chocolate cake to her room. It was the first and last kindness I ever did for her.

At Aunt Sylvia’s funeral, we were making casual conversation when her daughter mentioned the nightmare she had just had to deal with when closing out her mom’s affairs. Aunt Sylvia had been renting an apartment on her meager Social Security checks when she got her diagnosis. When my cousin went to fetch family heirlooms from her mom’s last dwelling, the landlord stood between her and the doorway and demanded a lease termination fee be paid before letting my cousin in to fetch her family’s heirlooms. As we were driving home, I told my wife “that shit should be fucking illegal.”

Legislating

My wife, at the time, was Chief of Staff for the state representative who presided over the district containing the hospice center and apartment complex. I don’t think I could have declared that it should have been illegal to a person more capable of making it so.

My wife had legislation banning lease termination fees for deceased tenants drafted very quickly. When you draft law at the Texas House, you invoke the Lege Council, who are a team of lawyers who know where laws pertaining to given matters are kept, and how to change it in such a way that would both achieve some goal and (ideally) survive judicial scrutiny down the line. Drafting law involves telling them in plain English the effects you’d like the law to have, and on the other end of it, they give you a piece of paper that you can bring to the House floor to propose as law. My wife walked in and said “make it against the law to charge a survivor of a deceased tenant a lease termination fee.” It seems so simple, maybe even a bit dystopian in restrospect.

Lobbying Pushback

I can only recall this story second hand, but I recall that the bill actually didn’t receive much pushback from landlords or apartment company lobbies. When the subject of that law came up with one such lobbyist, he remarked something to my wife along the lines of “yeah, that’s the sort of bill we’d normally fight, but if we do that, we look like monsters, so congrats on that one.”

When the bill finally passed, it was done so with explicit acknowledgement of my aunt. May she rest in peace.

What I Learned at Founders Academy

Fri, 29 Jul 2022 12:45:12 -0500

Context

Recently I attended a 3-day seminar offered by a local venture capital firm called “Founder’s Academy,” hosted by Gordon Dougherty. I enjoyed my time in the lectures, and learned quite a bit, which I wanted to document here.

I’m grateful to Mr. Dougherty for offering this free series of lectures, and was glad to have had the chance to attend. I’m also grateful to my managers at Allma, who gave me ample time off from work in the middle of the day to attend, and for being very encouraging of me when I was honest with them about what I was attending and why.

Spoiler Alert

I’m the sort of learner who obsessively watches YouTube videos for weeks on end about a given topic until I absorb just enough details via osmosis to appear outwardly competent. I have taught myself this way:

how to drive a stick shift
how to regularly win Dark Souls encounters
how to write code
etc.

So naturally, I’ve been watching a ton of Startup School / This Week in Startups / miscellaneous other channels content on various things about pitching, product development, timing of fundraising, etc. I want to emphasize that this is not me bragging, merely taking inventory of the little nest of artifacts I’ve assembled in the corner of my brain labelled “business.”

Observation: much of early startup advice is geared towards B2B SaaS companies

Every company idea I’ve ever had has been intended to serve normal, working-class folks. If what I build ends up serving folks who simply rentier capitalism their way to an identity, then that’s fine too, but only if it happens to serve them well because it primarily serves normal people well.

There were large portions of the Founder’s Academy curriculum (and anecdotally, most startup economy curriculum) that simply do not apply to B2C businesses. There was a large portion of the lecture that talked about strategic partnerships that involved two software vendors with symbiotic functions collaborating to make large money sales to big enterprise businesses. It was hard for me to extrapolate how that might apply to a B2C, but I wouldn’t rule out that I’m merely not creative a thinker enough.

I’d go as far as to say much of the prevailing wisdom about how to start a tech company tacitly expects you to be building a B2B SaaS of some kind. I have a couple theories as to why, but I would still very much describe myself as “not knowing anything” about business, so I’ll keep them to myself. :)

Validating Product Assumptions

Let’s compare a tech product to a cheeseburger. If I were to endeavor to develop a burger preparation process to produce a burger worth paying for, my personal approach would be to arrive at and end result where I make everything on the burger. That means I’m deciding what the recipe for our buns are, what the meat to fat ratio is in the patty, what temperature it’s cooked at, what time of day I need to make sure the mustard is prepared by, how much salt is going into my pickling brine, etc.

The conventional startup entrepreneurial wisdom seems instead to be: pick one small part of the overall problem, and get your attempt at solving it out the door as fast as possible. Figure out how to make buns OR a patty OR some condiments, and buy the rest from other, established providers, and get feedback on the end result.

That’s all well and good, but what if my thesis is that the whole burger is awful and needs change? If I make just the patty, and then customer feedback amounts to “well, I mean…it’s a burger,” is it because my thesis (that replacing the whole burger is what the customer wants) is wrong, or because I haven’t done it yet?

This seems to be the essential risk of entrepreneurship, and the very first risk that most investors seek to avoid. You just sort of silently acknowledge before getting real customers that you could be wasting a lot of your own time (and possibly your friends and family’s money, more on that later) until you have some objective data that convinces you otherwise.

Privilege

Before this event, all my research around starting a company had this weird undertone of privilege. It’s not uncommon to hear founders talking about raising six-figure friends & family rounds.

I’m happy for those folks, that their loved ones can facilitate their endeavors like that, but I simply do not have friends or family capable of doing something like that. In the event they were, it would probably be only barely so, and would consequently have so much guilt associated with any potential loss that I wouldn’t be able to, in good conscience, ask them for that much money in the first place.

I’ve deleted two fairly personal paragraphs attempting to detail my situation, but suffice it to say: if I aim to build a company, there’s no nest egg somewhere for my family to rely upon while I build it. I’m simply going to have to work a day job, and build it in my (increasingly rare) spare time. This is a reality I’ve been aware of the entire time I’ve had entrepreneurial aspirations, but I assumed I’d have to keep it a dirty little secret, like, I have a company, and I play as a full-time CEO while still going to my day job’s standup at 9:30AM or something.

I was surprised to see my kind of situation was not only explicitly acknowledged, but apparently common. One major question I had going into this was “is it okay to pay myself enough to make ends meet, even if that amount is actually quite large because I’ve been bad with money for most of my life?” Apparently the answer is “yes, but, y’know, talk to your investors about it ahead of time.” I came away feeling much more comfortable about ever having to approach a VC to ask for funding.

You can screw over your company’s future and not realize it

The lecturer detailed a situation where a company had agreed to a pair of early contracts with very large customers that had wholly untenable contract positions. That is to say, whoever attempted to acquire the company would have a very sincere interest in not servicing those parts of the contracts, and there’s basically no way to guarantee that the contracted party would agree to amending it.

So even though these contracts had been in place for multiple years, and were set to continue for years to come, and the company had been operating normally prior to seeking acquisition, it was nevertheless operating comfortably from the dressed-up confines of a death trap. I would have assumed something like that would at least have some level of obviousness.

Investor management is a crucial aspect of running a growing company

My prior understanding of the relationship between investor and entrepreneur was akin to client and contractor. I know investors meet with company leadership at some regular interval to see how things are going, and obviously more of that happens during fundraising periods.

What I learned at Founder’s Academy is that actually investors lose focus and interest in your company, and will simply forget you exist if you don’t keep them excited with updates. Say one of your investors is an expert in a particular niche, and you’ve just had a decent win in that arm of the company, that’s a situation I would have imagined simply writing down to bring up in those routine meetings, but the prevailing wisdom is to go out of your way to email that investor. “Look, we just had in <some arena we’ve discussed previously>! ”

I guess I imagined it went more like “hey, it’s your money, if you want to check in on what we’re doing with it, the onus is on you to reach out”, but it somehow makes much more sense that the person who provides the capital not only doesn’t feel obligated to check up on it, but in fact feels entitled to updates from those who received it.

Founder Compositions vary and matter

When I think of all the companies I’ve worked for that were started by people I could talk to, there are one or maybe two total early founders who actually put work into a product that ends up playing a role in the company’s formation. I once worked for a company that had 4 founders, but it definitely felt like the odd one out.

At Founder’s Academy, they taught us that even when you have just two founders, it’s a very very good idea to just give 1% of your company away to a trusted advisor for the sake of splitting tie votes. Apparently two founders with 50/50 equity is a warning sign to some investors.

Additionally, “cofounder” is really more of a vibe than an artifact of equity distribution. The lecturer said what investors really want out of a founder relationship is a “partner-in-crime” aspect. That is to say, you are both, without needing to acknowledge it, working towards the same goal as a natural consequence of your day-to-day impulses.

You can have any number of “founders”, but some may be founders only in the sense that they’ve been involved in the company during its inception, not because they own any significant portion of equity.

Greener Cloud Pastures

Tue, 29 Mar 2022 23:04:56 -0600

Preface

I want to take the time before writing to note that I have great respect and appreciation for the folks who work at any of the companies mentioned and/or on any of the products mentioned. Computers are hard; making stuff that is primarily meant to be consumed by them is even harder. My goals here are similar to those of that post. I hope that there can be something constructive that comes out of this post, but at the very worst, I hope it simply doesn’t offend or disparage anybody.

Background

In my last article, I wrote about my endeavor to get a side project deployed to a professional cloud, specifically AWS. To briefly summarize, it didn’t end up going to plan, and it proved to be quite expensive to host a very simple application there. I still want to host this application somewhere, so I started to look at alternatives to AWS for deployment.

After all was said and done, I was left with a lot of indecision and fatigue. I liked the level of service quality I got from AWS; I just really didn’t like paying for it. When I thought about all the moving bits and pieces of the application, it just didn’t feel like this thing should cost more than, say, $40 at most to host per month. I was hoping I could land somewhere that would serve my app basically just as well for about 1/3 of the price if at all possible.

What I wanted from a provider on the technical side was simply a way to ship an app container and have the provider put it on the public internet. Let me connect that container to a database, maybe some object storage, and a queue, and I’ll be good to go.

First Candidate: DOAP

I started to look at DigitalOcean’s App Platform, which seemed to touch a lot of these bases. I could define a containerized app, explicitly associate a database with it, and leave the rest to them.

After playing with this for about two days, I ended up deciding against it. It seemed to work well enough, but I also felt like I was gracing the outer limits of when it was appropriate to use the platform. It felt as though if my app decided to do even one more thing, I wouldn’t be able to host it that way anymore. That I’d be back to the same drawing board I was sitting at in no time.

I was left with the impression that the target audience for App Platform is agencies ñ folks approached by non-technical parties who need a specific kind of application to be made and hosted. I could see a workflow where you basically just have one App Platform definition you use to deploy any number of hyper-similar WordPress sites or whatever. If the client came back and asked for something that exceeded the capabilities of DOAP, the agency could just quote a wild price, but I’m trying to avoid wild prices.

DigitalOcean Shortcomings

One thing I made very heavy use of on AWS was managed services, specifically RDS and SQS. Being able to have a queue without having to worry about that queue’s infrastructure was very nice, but DigitalOcean has no analog for these. You could probably write up an App Platform template for deploying a containerized version of NATS or whatever, and they do have a managed Redis offering, but it’s much more expensive ($15/month at the time of writing) than SQS (consumption-based, with a generous free tier ñ one of the few things that didn’t cost me money on AWS). I also just sort of liked not having to think about it as anything other than a generic queue.

Kubernetes

One thing DigitalOcean does have that is actually priced fairly decently is their managed Kubernetes offering. Previously, I have stated that using Kubernetes is a mistake until proven otherwise. I violated these tenets and explored using Kubernetes for this app in DigitalOcean for about three days before I gave up. I should have trusted my gut.

I could probably write an article just on why Kubernetes didn’t work out, but I’ll spare us all the misery. I got as far as having a local Kubernetes setup for the app that worked, but I could never manage to get the one in DigitalOcean connected to the public internet.

Where to Now?

So, I’ve fully ruled out DigitalOcean, App Platform, and the Kubernetes offering, which leaves only Azure and Google Cloud Platform (GCP).

Azure has a number of container hosting solutions, but the one that seems most akin to ECS is called Azure Container Apps. It’s in preview and was launched very recently, so while I’m sure it’s probably fine, I don’t really have a lot of confidence using it just yet. They also have AKS ñ a managed Kubernetes offering ñ but I’m not doing that either, for the aforementioned reasons.

I’ve used GCP before in my career and for side projects. This very blog is hosted there, as a static site, and I pay something like $20/year all said and done. When I worked at WP Engine, we made very heavy use of GCP for all of our internal software and a not insignificant chunk of our actual business offering as well, so I had a vague impression of its production reliability. Somewhat like my AWS experience, I’d only really bothered writing code that deployed there, not actually deploying it. (I did have a fairly successful endeavor a few years ago to rebuild and redeploy an old application to Cloud Run that went very well.)

I decided to give GCP a shot.

App Recap

To briefly recall the app we’re talking about and its needs, it’s a meal management app. You put in recipes, create meal plans for the week from those recipes, and allow others in your household to participate. From a technical perspective, it needs:

a database
a server container
an event queue
something to trigger code execution when events arrive on the queue

I decided to use Cloud Run to host the server container, Cloud SQL for the database, Cloud Pub/Sub for the event queue, and Cloud Functions for the event responders. GCP supports assigning Cloud Functions as responders to Pub/Sub events, like how you can configure Lambda functions to respond to SQS events.

GCP Terraform Obstacles

One goal I maintained from the prior effort was to have it so that the full environment could be spun up from nothing. That I could take a bare project with no resources and, after one CI task run, have the full app running and available. This was beneficial last time in reducing costs, since I could use Terraform to destroy everything in one run if I met this condition.

AWS allowed me to accomplish this particular goal. I could go from having nothing to having the full environment up in 30 minutes, with no preconfiguration in my AWS account other than the Terraform Cloud user stuff (which I’m not counting).

GCP, however, had a bunch of hurdles in the way for me. For one, every API needs explicit authorization.

Want to create a Container Registry? You must first enable the Container Registry API.
Want to administer Cloud SQL databases? You must first enable the Cloud SQL Admin API.
Want to administer Cloud Run instances? You must first enable the Cloud Run Admin API.

After a while of playing API authorization whack-a-mole, I discovered what I thought would be a shortcut to this step:

Enable the Cloud Resource Manager API
Use Terraform to enable all the stuff I’d need with google_project_service resources.

This didn’t work. In fact, it did the opposite of working. Because I (foolishly) decided to add a bunch of the aforementioned resources that I had already enabled by hand, and hadn’t granted the lesser Terraform Cloud user rights to disable these things, it really messed with my Terraform state. Thankfully, cleaning that up is easy, but I learned so that you don’t have to, hopefully.

Additionally, for things like Domain verification (ugh), there didn’t immediately appear to be a good way of doing that automatically, which I’m not exactly against.

I will say, the GCP Terraform provider does a better job of catching errors in the “terraform validate” process. Many times with the AWS work, I would have something pass “validate” but fail to “apply” for something that could have totally been caught in “validate”. I very much appreciate whoever is responsible for this improvement.

IAM Woes

IAM is also pretty confusing. In the console, users are called “Principals”, but in some old docs and guides, they seem to just be called “Users”? Users can have roles and service accounts, and those service accounts can have a subset of roles. I think I’ve recalled all this correctly, but I’m not confident I have!

Another thing that threw me for a loop was how to create a user for the Cloud Run API service. The Terraform definition has space for a Service Account ID, so I created a service account in Terraform and assigned it the “Secret Manager Secret Accessor” role. Only, Terraform gave me an error:

| Error: Error setting IAM policy for service account 'projects/xxxxxxxxxxx/serviceAccounts/api-server@xxxxxxxxxxx.iam.gserviceaccount.com': googleapi: Error 400: Role roles/secretmanager.secretAccessor is not supported for this resource., badRequest

_{(Yes, the ., badRequest part is really in there.)} So, I needed a Principal, and it took me a while to figure out the right incantation of Terraform to produce it, because they are not named kindly:

resource "google_service_account" "api_server_account" {
account_id = "api-server"
display_name = "API Server"
}
resource "google_project_iam_member" "api_server_user" {
project = local.project_id
role = "roles/viewer"
member = format("serviceAccount:%s", google_service_account.api_server_account.email)
}

From there, you can grant that Principal specific roled permissions:

resource "google_project_iam_binding" "api_user_secret_accessor" {
project = local.project_id
role = "roles/secretmanager.secretAccessor"
members = [
google_project_iam_member.api_server_user.member,
]
}

and then assign it to your Cloud Run app:

resource "google_cloud_run_service" "api_server" {
name = "api-server"
location = "us-central1"
traffic {
percent = 100
latest_revision = true
}
autogenerate_revision_name = true
template {
spec {
service_account_name = google_service_account.api_server_account.email
}
}
# yadda yadda yadda
}

This, combined with some new code to handle environment-mounted secrets, led to my first “successful” deployment to Cloud Run. Only, I couldn’t access it because I hadn’t granted “allUsers” the permission to invoke this particular Cloud Run application. Nice default, I think, but requires more terraform IAM finagling:

data "google_iam_policy" "public_access" {
binding {
role = "roles/run.invoker"
members = [
"allUsers",
]
}
}
resource "google_cloud_run_service_iam_policy" "public_access" {
location = google_cloud_run_service.api_server.location
project = google_cloud_run_service.api_server.project
service = google_cloud_run_service.api_server.name
policy_data = data.google_iam_policy.public_access.policy_data
}

This is called out in the Terraform docs for GCP’s Cloud Run resource, but it’s called “noauth”, which I somehow failed to parse as “make public”.

Cloud Run DNS Woes

Another thing I couldn’t manage to automate my way out of was the explicit domain name association that GCP requires for Cloud Run. When you create a Cloud Run service, you get a URL like “https://app-name-blah-blah-blah.run.app”, which you can’t just CNAME. So, you instead have to create a “Domain Mapping” that seems to tell Google’s routers, “It’s okay to respect this other domain.” The IAM around this is really funky, though. The Cloud Run IAM Roles documentation page explicitly calls out that:

Roles only apply to Cloud Run services, they do not apply to Cloud Run domain mappings. The Project > Editor role is needed to create or update domain mappings.

“So much for my preciously curated list of permissions; stupid Terraform has to be an editor!” I cursed. Only it turned out that even when I granted it that permission, Terraform still couldn’t manage the domain mapping. I kept getting the same error over and over again:

| Error: Error waiting to create DomainMapping: resource is in failed state "Ready:False", message: Caller is not authorized to administer the domain 'web.site'. If you own 'web.site', you can obtain authorization by verifying ownership of the domain, or any of its parent domains, via the Webmaster Central portal: https://www.google.com/webmasters/verification/verification?domain=web.site. We recommend verifying ownership of the largest scope you wish to use with subdomains (e.g. verify 'example.com' if you wish to map 'subdomain.example.com').

That link would take me to a page that gleefully told me I had already verified it. I think it was just the case that the Terraform user isn’t the explicit owner of the Domain like I am? Honestly not sure what happened here. I resolved to manage the damn mapping manually myself, which worked.

Cloud Functions vs. Lambda

When I used Lambda, I quite liked that I could provide it a compiled binary in a zip folder. It felt a little old fashioned, but it definitely worked. Cloud Functions works with a little more magic ñ the bad kind. Rather than provide a binary, you upload code to a Google Cloud Storage bucket, and that sets up a Cloud Build trigger, which runs a preconfigured script against the code in question depending on your chosen runtime. Getting even a simple Cloud Function to upload via Terraform proved very difficult for me.

While nearly all the provided example Cloud Function code samples work, they do so because they don’t use anything but the standard library. If you want to use any of your own private code as libraries in your Cloud Function, you basically have to have a “go.mod” file per Cloud Function. This is pretty different from how I write Go code day to day, where I generally have exactly one go.mod file per repository. No big whoop.

Building Cloud Function artifacts can be done with the “pack” CLI for local testing, but you can’t ship those artifacts to Cloud Functions, you can only ship raw code to be built by Cloud Build. Maybe it’s documented somewhere, but this whole build process felt very opaque and yields confusing errors that didn’t aid me in diagnosis. Some examples:

Info 2022-02-08 19:48:39.274 CST Step #1 - "build": ERROR: No buildpack groups passed detection.
Info 2022-02-08 19:48:39.274 CST Step #1 - "build": ERROR: Please check that you are running against the correct path.
Info 2022-02-08 19:48:39.274 CST Step #1 - "build": ERROR: failed to detect: no buildpacks participating.

(I was uploading a .zip that had an empty folder in it.)

Info 2022-02-08 22:26:55.090 CST Step #1 - "build": Running "go run /cnb/buildpacks/google.go.functions-framework/0.9.4/converter/get_package/main.go -dir /workspace/serverless_function_source_code (GOCACHE=/layers/google.go.functions-framework/gcpbuildpack-tmp/app)"
Info 2022-02-08 22:26:55.270 CST Step #1 - "build": 2022/02/09 04:26:55 Unable to extract package name and imports: unable to find Go package in /workspace/serverless_function_source_code.
Info 2022-02-08 22:26:55.272 CST Step #1 - "build": exit status 1
Info 2022-02-08 22:26:55.278 CST Step #1 - "build": Done "go run /cnb/buildpacks/google.go.functions-framework/0.9.4/c..." (187.47553ms)
Info 2022-02-08 22:26:55.278 CST Step #1 - "build": Failure: (ID: 7a420ccf) 2022/02/09 04:26:55 Unable to extract package name and imports: unable to find Go package in /workspace/serverless_function_source_code.
Info 2022-02-08 22:26:55.278 CST Step #1 - "build": exit status 1

(No root-level Go file.)

Info 2022-02-09 23:25:32.516 CST Step #1 - "build": Done "go list -m" (4.498213ms)
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": Failure: (ID: 03a1e2f7) ...ry.io/otel/exporters/otlp/internal/retry@v1.3.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": go.opentelemetry.io/otel/exporters/otlp/otlpmetric@v0.26.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc@v0.26.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": go.opentelemetry.io/otel/exporters/otlp/otlptrace@v1.3.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": go.opentelemetry.io/otel/internal/metric@v0.26.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
...A few dozen more of those.
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": To ignore the vendor directory, use -mod=readonly or -mod=mod.
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": To sync the vendor directory, run:
Info 2022-02-09 23:25:32.516 CST Step #1 - "build": go mod vendor

(This was from trying to not have more than one go.mod file. I’d never seen this error in any Go I’ve ever written before.)

 - "build": Failure: (ID: 7a966edd) vendored dependencies must include "github.com/GoogleCloudPlatform/functions-framework-go"; if your function does not depend on the module, please add a blank import: `_ "github.com/GoogleCloudPlatform/functions-framework-go/funcframework"`

(The why behind this is never really explained, but I’m not worried about a random empty import.)

All said and done, it took me two evenings staying up to 1:30AM or so to get all the lights to turn green.

Cloud Function Woes

I kept trying to diagnose why the function that finalizes meal plans was failing, and it didn’t seem to matter what I changed in code, the subsequent logs would never show up. I was eventually able to suss out that Terraform wasn’t updating my Cloud Function at all. It was still on version one, despite a dozen or so “successful” deploys. It turned out I wasn’t the only person who had experienced this, and after applying that solution, was finally able to start debugging in proper form.

After some normal user errors, I encountered an issue getting my now-running Cloud Function to connect to the database.

Cloud SQL connection failed. Please see https://cloud.google.com/sql/docs/mysql/connect-overview for additional details: ensure that the account has access to "xxxxxxxx-dev:us-central1:dev" (and make sure there's no typo in that name). Error during generateEphemeral for xxxxxxxx-dev:us-central1:dev: googleapi: Error 403: The client is not authorized to make this request., notAuthorized

(Again, the “., notAuthorized” part was actually present in the output. Funnily enough, every time I get errors related to this, they link to the MySQL docs even though I’m using the Postgres version of CloudSQL. I let it pass because you can easily navigate to the Postgres version from those docs.)

Cloud Run Woes

Cloud Run was not without its own, similar, issues. I’d be updating code and a new revision wouldn’t get deployed. I realized I was probably asking too much of Terraform here, so I tried to go about it via the official Github Action for deploying Cloud Run services. Immediately, I encountered an error:

ERROR: (gcloud.run.deploy) PERMISSION_DENIED: Permission `iam.serviceaccounts.actAs` denied on service account api-server@service-dev.iam.gserviceaccount.com (or it may not exist).

After some searching, I happened upon this StackOverflow answer for precisely this problem and realized I needed to add the “Service Account User” permission to my GitHub Deployer IAM Principal. This caused my next deploy to work, but the one after that failed with the familiar error message. I discovered that somehow the Service Account User role was being removed from the Google Actions user after each deploy. So I put the relevant permission (“iam.serviceaccounts.actAs”) in a custom role and gave that role to the Actions user. That worked, and I could continue deploying without interruption.

Database Connectivity

On AWS, whenever I needed to talk to the database, I employed this awful trick where I would spin up a Cloud 9 instance and use the terminal for that instance to connect to the RDS instance. Since the RDS instance wasn’t configured for public connectivity, this was the only way I could manage to run raw queries against it.

For GCP, this is even harder because, by default, the Postgres instances are configured to reject connections that don’t have TLS enabled, and the certs you can download from the GCP interface don’t seem to work on localhost or with an app like Beekeeper. You can connect via Cloud Shell, but you can’t use the certs there either.

The only way I’ve found to consistently run raw queries against the database is to temporarily disable the TLS requirement and then connect from the Cloud Shell while remembering to reactivate it. Kind of painful, but not the world’s biggest inconvenience.

Google Cloud Console

Just a quick note, but I quite like it, actually. It’s just as easy to find things on GCP as it is on AWS. One complaint is that the interface, at times, completely fails to work if you disable common NoScript domains like “googletagmanager.com”.

Static Site Woes

This project also makes use of Google Cloud Storage for the static page that interacts with the API. When I tried to create a bucket with the appropriate name for my domain in GCP, I encountered this error:

The bucket you tried to create is a domain name owned by another user.

This ended up being that my Terraform service account user wasn’t listed as an owner of the domain in the Webmaster controls. Adding it was easy enough, but the form didn’t trim whitespace so complained that the service account I provided it with (which has a “.iam.gserviceaccount.com” domain name) was not a valid Google account. Deleting the very hard to notice leading space was the ticket, though.

Monitoring Improvements

One of the worst things about AWS was feeling like I basically could not have metrics or decent traces, feeling like I had to use X-Ray because of the state of the observability product market. Happy to report that Cloud Trace is leagues better than X-Ray, in my opinion, as is the log manager for GCP. This service collects 100% of traces the service produces.

Additionally, GCP has these things called Uptime Checks, which, uh, make HTTP requests to your hosted services and report on the latency encountered. I was able to set one up for both the user-facing static webapp and the API server being hosted in Cloud Run.

As you can see, aside from the occasional outlier, it’s been pretty reliable. One thing I wasn’t able to accomplish in either AWS or GCP was app-level metrics, specifically Go runtime metrics. Locally, I can visualize how much time is spent, for instance, collecting garbage, but I cannot do this on GCP. Rather, I probably could, but my understanding of how Cloud Run operates is that the application itself is spun up and down depending on inbound traffic, so I’m not sure that collecting them would even be useful under that scheme.

The webapp for this service is a static site, which talks to the API server over HTTP, authenticated by a cookie. The API server issues a cookie for the “www.” part of the app, but the browser will not include this cookie in requests to the API server because it has a different subdomain. Something needs to watch for requests to a given set of path prefixes on the webapp and forward it to the API service.

On AWS, I was able to use a CloudFront Distribution to do this:

ordered_cache_behavior {
path_pattern = "/api/v1/*"
allowed_methods = ["GET", "HEAD", "OPTIONS", "PUT", "POST", "PATCH", "DELETE"]
cached_methods = ["GET", "HEAD", "OPTIONS"]
target_origin_id = local.api_origin_id
forwarded_values {
query_string = true
headers = ["Origin"]
cookies {
forward = "whitelist"
whitelisted_names = ["servicecookie"]
}
}
min_ttl = 0
default_ttl = 0
max_ttl = 0
compress = true
viewer_protocol_policy = "redirect-to-https"
}

It’s been long enough that I can’t explain to you how that worked, but I can testify that it did work. GCP’s Cloud Storage doesn’t, as far as I can discern, have a comparable feature. Locally, I was solving this problem by having a Caddy instance that proxied requests, so I thought maybe deploying a Caddy instance would be the most effortless solution, but I’d just use Cloud Run to host it, which would mean I’m effectively paying twice for any request made to the API service.

Cloudflare Workers ended up being the solution that worked best for me. They have an example that I was able to very easily adapt to my usecase:

addEventListener('fetch', event => {
event.respondWith(handleRequest(event.request));
});
async function handleRequest(request) {
var url = new URL(request.url);
if (url.pathname.startsWith('/api/') || url.pathname.startsWith('/users/')) {
url.hostname = 'api.service.dev';
}
return await fetch(url, request);
}

And the corresponding Terraform:

resource "cloudflare_worker_script" "dev_reverse_proxy" {
name = "dev_reverse_proxy"
content = file("reverse_proxy.js")
}
resource "cloudflare_worker_route" "users_route" {
zone_id = var.CLOUDFLARE_ZONE_ID
pattern = "https://www.service.dev/users/*"
script_name = cloudflare_worker_script.dev_reverse_proxy.name
}
resource "cloudflare_worker_route" "api_route" {
zone_id = var.CLOUDFLARE_ZONE_ID
pattern = "https://www.service.dev/api/*"
script_name = cloudflare_worker_script.dev_reverse_proxy.name
}

Cloudflare Workers come with a quite generous free tier, which I’m not worried about really ever exhausting.

Pricing Outcomes

Since the impetus to all this was the price for AWS, it seems fair to evaluate the price of running on GCP. Just so we’re all up to date, here’s what the bill is for:

A Cloud SQL Postgres 13 instance running 24/7
A Cloud Run API server running when traffic is received
A Pub/Sub topic that triggers Cloud Functions
Cloud Functions that run on a cron to manipulate data
Prober Cloud Functions that run every five minutes and actually make use of the hosted service.

The prober function basically signs up a four-person household, creates some recipes, creates a meal plan, and votes on it for all members. It then verifies that the finalizer is running by waiting until the meal plan is finalized after all votes are in. So, I’m actually getting regular traffic to this service.

Here’s the bill I received for running the service in March. I’ve left out details specific to my account like the ID of the project, about $5 in credits towards my account, as well as things like SKU IDs which are probably not unique but are definitely not helpful in understanding the cost.

Service description	SKU description	Cost type	Usage start date	Usage end date	Usage amount	Usage unit	Unrounded Cost ($)	Cost ($)
Cloud SQL	Cloud SQL for PostgreSQL: Zonal - Micro instance in Americas	Usage	2022-03-01	2022-03-31	742.765	hour	7.799017	7.8
Stackdriver Trace	Spans ingested	Usage	2022-03-01	2022-03-31	“36,903,907”	count	6.877799	6.88
Cloud SQL	Cloud SQL for PostgreSQL: Zonal - Standard storage in Americas	Usage	2022-03-01	2022-03-31	19.994	gibibyte month	3.398864	3.4
Secret Manager	Secret access operations	Usage	2022-03-01	2022-03-31	“1,004,162”	count	2.982484	2.98
Cloud Run	CPU Allocation Time	Usage	2022-03-01	2022-03-31	“124,279.60”	vCPU-second	2.981886	2.98
Cloud Functions	CPU Time	Usage	2022-03-01	2022-03-31	“197,184.96”	GHz-second	1.968551	1.97
Cloud Functions	Memory Time	Usage	2022-03-01	2022-03-31	“123,073.24”	gibibyte second	0.304218	0.3
Secret Manager	Secret version replica storage	Usage	2022-03-01	2022-03-31	8.997	month	0.179528	0.18
Cloud Run	Memory Allocation Time	Usage	2022-03-01	2022-03-31	“62,138.05”	GiB-second	0.154453	0.15
Cloud Storage	Standard Storage US Multi-region	Usage	2022-03-01	2022-03-31	5.635	gibibyte month	0.146461	0.15
Cloud Storage	Multi-Region Standard Class B Operations	Usage	2022-03-01	2022-03-31	“84,984”	count	0.033944	0.03
Cloud Run	Cloud Run Network Egress via Carrier Peering Network - North America Based	Usage	2022-03-01	2022-03-31	1.049	gibibyte	0.01781	0.02
Cloud Storage	NA-based Storage egress via peered/interconnect network	Usage	2022-03-01	2022-03-31	0.231	gibibyte	0.009187	0.01
Cloud Storage	Multi-Region Standard Class A Operations	Usage	2022-03-01	2022-03-29	194	count	0.00097	0
Cloud SQL	Network Internet Egress from Americas to Americas	Usage	2022-03-01	2022-03-31	0.005	gibibyte	0.000932	0
Cloud SQL	Network Internet Egress from Americas to EMEA	Usage	2022-03-01	2022-03-31	0.001	gibibyte	0.000133	0
Cloud SQL	Network Internet Egress from Americas to China	Usage	2022-03-01	2022-03-30	0	gibibyte	0.000003	0
Cloud SQL	Network Internet Egress from Americas to APAC	Usage	2022-03-06	2022-03-26	0	gibibyte	0.000002	0
Cloud Run	Memory Allocation Time	Usage	2022-03-06	2022-03-31	1.75	GiB-second	0	0
Cloud Run	Requests	Usage	2022-03-01	2022-03-31	“1,049,960”	Requests	0	0
Cloud Run	Cloud Run Network Intra Region Egress	Usage	2022-03-01	2022-03-31	2.496	gibibyte	0	0
Cloud Scheduler	Jobs	Usage	2022-03-01	2022-03-31	62	Job-days	0	0
Cloud Functions	Invocations	Usage	2022-03-01	2022-03-31	“1,011,934”	invocations	0	0
Cloud Functions	Network Egress from us-central1	Usage	2022-03-01	2022-03-31	1.549	gibibyte	0	0
Cloud Functions	CPU Time	Usage	2022-03-01	2022-03-31	13.06	GHz-second	0	0
Cloud Functions	Memory Time	Usage	2022-03-01	2022-03-31	175.525	gibibyte second	0	0
Cloud Logging	Log Volume	Usage	2022-03-01	2022-03-31	4.507	gibibyte	0	0
Cloud Build	Build time	Usage	2022-03-15	2022-03-29	32.583	minutes of build time	0	0
Cloud Storage	Download Worldwide Destinations (excluding Asia & Australia)	Usage	2022-03-01	2022-03-31	0.042	gibibyte	0	0
Cloud Storage	Regional Standard Class B Operations	Usage	2022-03-15	2022-03-29	107	count	0	0
Cloud Storage	Standard Storage US Regional	Usage	2022-03-01	2022-03-31	0.099	gibibyte month	0	0
Cloud SQL	Network Google Egress from Americas to Americas	Usage	2022-02-28	2022-03-31	5.633	gibibyte	0	0
Cloud Pub/Sub	Message Delivery Basic	Usage	2022-03-01	2022-03-31	0.003	tebibyte	0	0
Cloud Pub/Sub	Intra-region data delivery	Usage	2022-03-01	2022-03-31	1.102	gibibyte	0	0

Conclusion

I knew going into this I could probably hit a hosting target price of about $40, but hitting $27 was just very sweet. This puts the GCP cost at about $100 less than the equivalent AWS hosting. Even with the recent changes to pricing that everybody seemed to freak out about, to the best of my ability, I think this would raise my cost by maybe $5/month in the worst circumstances?

I’m quite happy with the level of service quality I get, I feel confident in the application I’ve deployed, and I can easily pay for it without even having to let my spouse know ahead of time. I think GCP will be the platform I opt for going forward.

Baby's First AWS Deployment

Sat, 29 Jan 2022 23:34:37 -0600

Preface

I intend for this post to serve as an experience report from a newbie AWS user. Hopefully you enjoy it and don’t feel like you’ve wasted your time at the end.

I want to take the time to note that I have great respect for all the folks who make these companies work and who build these products. None of my opinions are directed at a human being who has a name and/or respirates. If I couldn’t figure something out, I assume (and expect you to assume as well) that at the very least, I am more responsible for that shortcoming than the service providers or their documentation. In the extremely unlikely event anybody but me and my proofreader (hi Caroline!) ever even reads this, I hope that I will have provided enough information on whatever confusions I experienced that maybe something actionable can come of it.

The App in Question

The app itself is pretty simple. It’s a meal plan manager, where you can organize and propose a meal plan for your household for the week. So, you can say, for example, “For dinner on Wednesday, we can do spaghetti and meatballs, or egg fried rice or baba ganoush,” and then every member of your household can vote on which one they’d prefer to eat. An async worker tallies the rankings using the Schulze method and finalizes the meal plan. Thus far, it has lived as a local-only project I work on in my spare time. I’ve sort of over-engineered it on purpose in the spirit of learning.

A worker process checks the database every minute for meal plan proposals that are still awaiting votes but have expired voting deadlines so it can tally up the winner and finalize the meal plan. This is done in an external worker so that I don’t bog down any of the HTTP traffic the server handles with unrelated compute burden.

Why a data changes queue? I don’t have any concrete product plans for this yet, as it were, but I think you could do something like email the meal plan creator if a vote is received and the meal plan is consequently finalized, for instance. You could produce audit logs for a more serious application. It’s also where any customer data analytics collection should occur, so you don’t inherit that latency during user requests.

This might seem like a lot, but it’s actually quite pared down from a previous iteration of the system where there were separate queues for writes, updates, deletes, the meal plan finalization worker (which I had set up as merely the sole component of a broader “chores” worker), and the data changes queue.

Goals

I had a few things I really wanted to make sure I accomplished:

I want to run it in AWS. To call it cargo culting is likely fair, but every single company I’ve ever worked for has used AWS. I’ve never been on the team that actually, y’know, uses it, though. Plus, it’s bound to be a valuable skill for a software engineer to have.
Since it’s going to be coming out of my own wallet, I want to – as best I can and where appropriate – avoid expensive pitfalls.
- I follow Corey Quinn on Twitter (and, because of my recent foray into AWS, the Last Week in AWS newsletter he also organizes), so I know about the pitfalls of Managed NAT Gateways and such. So I’d like to avoid that, if possible.
- In the AWS Pricing Calculator, I ran what I thought I would need as a bare minimum, what I thought I’d really need, and what I would recommend if I was spending someone else’s money. The estimates came out to be something like $80, $120, and $210 each (they also ended up being meaningfully inaccurate). I resolved to be willing to tolerate up to $200/month on this service in the event unforeseen costs occurred for this first go-around.
No Kubernetes, if possible.
- Invariably someone is going to read this and say, “What about Kubernetes?”. Kubernetes is a great solution for the right problems, but I don’t think a simple CRUD site connecting to a database is a worthy candidate. Also, since I’m on AWS, their offering, EKS, costs $75 without any compute to even start using, which I think will put me over my acceptable budget threshold.
- If I switch to Kubernetes, then I have to run it locally to build the configuration and to validate that it works the way it should. Last time I had to do this for work, it involved Vagrant and Minikube, and well…please don’t make me do this.
I want in on the Serverless hype.
- That is to say, ideally, I end up with absolutely no machines that I, or anybody else, could SSH into.
- Ideally, I could define my infrastructure in terms of how little and how many resources it should receive, and have AWS do the plumbing to make it all function properly.
I want it to scale.
- If you’re groaning, well, same. “Scale” has many meanings, and I certainly can’t profess that I’ve built something that can handle most of those meanings.
- Ultimately, what I would like is to be at least positioned such that my little service could use as few resources as possible when (in the most likely case) nobody is using it, but also be able to handle a sudden spike in popularity without completely dying.
- As someone who has had services eat shit before, that’s “scaling” to me.
I want to maintain as close to 100% of this infrastructure in code form as possible.
- To satisfy this, I chose Terraform. I’ve never gotten a chance to write Terraform professionally, but I have seen it used frequently at my workplaces over the last few years, so I feel confident in using it. I was afraid of screwing up state management, and was glad to discover that Hashicorp (who make Terraform) have an official offering in the form of Terraform Cloud that can manage the state for you.
- I considered using Pulumi, which seems like a really cool project (shout out to all the folks building Pulumi!). That said, their pricing model for their cloud offering is based on hours in which they’re responsible for managing your infrastructure, measured in their own little credit currency, and I just didn’t like that. I don’t like the idea of wondering if I have enough funny money to get through the month, and I think it’s incorrect to suggest that Pulumi is “actively managing” anything when it isn’t running.
As close to 100% of the infrastructure has to be able to be spun up from start to finish in as few steps as possible, and ideally, just one.
- I want to be able to go from having some AWS credentials attached to an account with no resources to a fully functioning website, ideally with a single CI run.
- I don’t actually know how much all of this is going to cost me. If I can spin it up and down on demand, then I can iterate on it when I have the time and save myself the money when I am just too swamped to touch it.
I want to be able to view logs, traces, and metrics. The three pillars, as it were.
Ideally, no vendor-specific code written.
- Let me clarify: I don’t want to have to end up writing some glue code that is only relevant in AWS and is significant in scope. I make an exception for Lambdas themselves, since the actual wrapper stuff you’d have to implement is not significant. I could very easily take a Go Lambda and turn it into a Cloud Function. I’m talking about code that you simply couldn’t port over in any fashion to another environment. I don’t want to write, for instance, API Gateway Authorization Lambdas.

Translating to the Cloud

For AWS, I went with ECS Fargate for the main API server hosting. There are a few ways of hosting web servers in AWS, but Fargate seems to be the newest and most container-oriented. It seemed to be the closest thing to a “give me a container and some operational specifications, and I’ll take it from here” service, and the pricing didn’t seem too bad. When I did the calculation, the bare minimum you could spend (which was plenty for my little server) was just shy of $8/month.

For the data changes queue, I used SQS, and for the changes worker that listens on that queue, I went with Lambda. I’ve used Lambda before, though never triggered by an SQS queue. I decided to use Lambda for the meal plan finalization worker as well.

Database

I opted for Aurora Serverless in my first go around, because I was really sold on the “this is THE serverless database option” hype. I imagined all these infinite scaling possibilities, and I assumed a low-tier RDS instance would get something paltry, like 10 or 15 connections per instance. “I’m gonna have so much traffic!” I told myself. “I’m gonna end up firing dozens if not hundreds of Lambdas at once and easily overwhelm whatever RDS instance I can afford!”

This was very wrong. According to the relevant AWS documentation page, I discovered that a db.t2.micro should have 1000000000 ÷ 9531392 = 104.9 connections, if I’m doing that math right (I may very well not be). I also learned that having lots of traffic doesn’t mean you will spin up lots of Lambdas, as they execute in batches, and you have to be very explicit to set it up to react in a one-to-one manner.

Aurora Serverless proved to be much too expensive, and came with some, in my opinion, major caveats. For one, they only let you use version 10.14 of Postgres, which came out on the same day as version 12.4.

Additionally, the AWS calculator lets you configure a *sigh* “Aurora Serverless PostgreSQL-Compatible Edition” (they must get paid by the syllable at AWS) with only one ACU (Aurora Capacity Unit – nine syllable bonus points for you), and it will tell you that one ACU is $43.80, at the time of this writing. You actually cannot configure a Serverless Postgres Aurora instance with less than two ACUs, which means the least you can spend is nearly $90, but there is no indication of this “gotcha” in the billing calculator. I ended up using a small RDS instance for the database.

Diving In

I got started by creating a Terraform Cloud account. I played around with Terraform first by setting up a version controlled workspace and wrote some minor code to ensure all the organization’s repositories have access to the same base set of issue labels:

variable "issue_label_configs" {
type = list(any)
default = [
{
Name = "blocked",
Description = "blocked by other issues",
Color = "B60205",
},
{
Name = "nice-to-have"
Description = "things we should maybe do. maybe not."
Color = "E5E4D3"
},
{
Name = "tech debt"
Description = "paying back technical debt."
Color = "9AFE3B"
},
# etc...
]
}
variable "repositories" {
type = list(string)
default = [
"api_server",
"webapp",
# etc...
]
}
locals {
all_labels = setproduct(var.repositories, var.issue_label_configs)
}
resource "github_issue_label" "labels" {
count = length(local.all_labels)
repository = local.all_labels[count.index][0]
name = local.all_labels[count.index][1].Name
description = local.all_labels[count.index][1].Description
color = local.all_labels[count.index][1].Color
}

This allowed me to get a little comfortable with HCL and Terraform Cloud before writing anything that would spend money. Highly recommended.

I already had an AWS account, and I briefly tried to set up something that made sense for this little organization, but I wasn’t left, after about a day of searching, with a clear picture of how I should manage that. I could use AWS Organizations to link multiple companies and have one account per environment, but that seems like overkill and a lot to set up for a little app, so I just used the account I already had and created a key for Terraform and GitHub Actions. Terraform needs the key to create the infrastructure; GitHub Actions needs the key to push container images on merge.

While I used the VCS configuration for the global org Terraform workspace, I opted to use the remote configuration for the actual project repositories. This is because I want to be able to control when the actual Terraform apply happens, whereas if it’s VCS-backed, it fires off as soon as the commit happens.

Configuration

I chose to use AWS Systems Manager Parameter Store (more syllable bonus points for you!) to store things like the address of the SQS queues and database credentials. That way, I can have Terraform create those values and update them when appropriate, and my deployed software can have a stable place from which to fetch configuration values. This helps me keep deployment to one step, as opposed to a method where I’d have to first create the resources and then put their respective values into a config in a second step.

There are a LOT of ways to host configuration values in AWS. You could use Parameter Store (and, from within that, you can use a standard or advanced parameter, the only difference of which seems to be a larger payload limit), you could use AWS AppConfig if you need really fancy features like schema validation, you could plop it into an S3 bucket, or you could probably use Dynamo if you really wanted. There are no consistent guidelines for which of these products is advisable for any given situation, and I ended up using Parameter Store because it’s what we do at work.

Observability

Figuring out an observability stack that worked ended up feeling impossible. Locally, I use OpenTelemetry reporting straight to a Jaeger instance for traces, a Prometheus instance for metrics, and a Grafana instance to visualize all of it. I feel it’s important to note the Collector’s absence here, as it is the main way OpenTelemetry advises getting data out to various providers, via a collector instance or sidecar. I have experience using the Collector, but the OpenTelemetry code I wrote for this repository comes from an era where the best way to get this done was to use the Contrib libraries.

In AWS, I just used X-Ray for traces and CloudWatch for logs, but CloudWatch metrics are pretty expensive. At $0.30/metric (as of the time of this writing), the standard suite of Go runtime metrics would cost $8.70/month. I just assume that blows up if I have multiple services.

AWS offers a managed Grafana service and a managed Prometheus service. However, only the Prometheus service is set up in the Terraform provider, which means I couldn’t provision them together. I had half a mind to add the provider myself, but even if we ignore that I know nothing about how either product works or how to stitch the two of them together, am I going to spend my precious free weekend time adding features to not one but TWO publicly traded companies’ products?

I tried to implement Grafana Cloud, since they offer a pretty competent free tier, and I very much enjoy using Grafana in my local stack. They require you run their own Grafana Agent sidecar, but I was never quite able to get it all working correctly. The team itself was very communicative and invited me to troubleshoot with them on GitHub. This was how I discovered a lot of my failings were user error. I still like Grafana Cloud quite a bit, and would like to get to use it in the future. The team inspired great confidence in their product for me.

I took a gander at DataDog, which I’ve also used at nearly every job I’ve ever had, but it is very difficult to determine how much it would cost me. I know it’s $15/host, but does the OpenTelemetry sidecar I’d have to run in ECS count as a host? Each container? Each service? Each Lambda invocation? The best I could gather was that it would cost me anything from $50 to $400 to use, and either side of that huge range still puts me outside my budget.

Honeycomb is an attractive offering and was easy to get working. It wasn’t until I learned that they only support logs and traces on the free plan, and metrics aren’t achievable without calling a sales rep, that I felt dismayed. Every indication I could find for how much this would cost me (to be fair, there were few) looked like it started at $1k.

There are a ton of other providers in this space, but they all feel either dated compared to DataDog and Honeycomb or geared exclusively towards large Enterprise contracts. I don’t begrudge these companies for pricing the way they do, because they’re not thinking of folks like me when they make these decisions, and it probably makes sense for them not to. Doesn’t help me out all that much, though.

I ended up using X-Ray for tracing and CloudWatch for logging. I never managed to implement the Go runtime metrics I wanted because I couldn’t manage to get the OpenTelemetry code working in AWS, despite a few days of my best efforts.

😎 pic.twitter.com/8cD6wT0S0x
— footloose and fancy-free (@vgsnv) December 15, 2021

Elasticsearch Woes

Locally, the service used Elasticsearch to allow users to search through ingredient names and such. At first, I tried to use Elastic.co’s Cloud offering, which is about $17/month. I found that the base Cloud offering seems to be geared towards folks using the ELK stack, but I didn’t need the LK part of it. Also, VPC restrictions meant I couldn’t get the Lambdas to write to it.

I tried to use AWS' managed Elasticsearch offering, but it came with some serious restrictions. “The t2.micro.search instance type supports only Elasticsearch 1.5 and 2.3." (a version that came out at the end of March 2016, almost six years ago). I also wasn’t ever able to get the code I’d written (that worked with Elastic Cloud) to communicate with it, though in hindsight, it must have been some combination of IAM and VPC failures on my part. I spent the better part of a week trying to get this to work, and it was literally so frustrating that I rewrote the search code to just do ILIKE queries to the database and removed Elasticsearch from the project altogether.

if you have the nerve, gall, the unmitigated audacity to try and run an Elasticsearch machine in AWS on something approachable like a https://t.co/8eNdwUy2NB, they punish you by making you use a version of Elasticsearch from nearly six years ago. This is customer obsession, folks.
— footloose and fancy-free (@vgsnv) November 27, 2021

Aspiration Shopping

A phenomenon I noticed when I was looking through the (sincerely, epic) list of services that AWS offers is that it’s easy to start adding things to your mental cart that might be premature for your needs. Around the time I started planning all this stuff, AWS announced Cloudwatch Evidently, and I got really into the idea that I would end up writing some Evidently tests after all was said and done. AppConfig was another thing like that, a moment where I said, “OoooOOOOOoooo, that would be fancy!”. I don’t have any good advice for avoiding this phenomenon but feel nevertheless obligated to call attention to it.

Ordering is Important

One of the first obstacles I encountered was maintaining my desired operational order. I wanted to not only be able to spin everything up from scratch, but also have all these Lambdas set up before pushing code. The deploy job has two phases that operate in tandem: “build”, which compiles binaries for the Lambdas, and “scaffold”, which effectively runs “Terraform apply”. After those, the “deploy” job runs, building and shipping containers and the artifacts from “build”. I haven’t managed to find time to figure out how to build the containers and download them in later jobs.

The problem comes from Terraform requiring some artifact to set up a Lambda function. I could make the “scaffold” step have “build” as a prerequisite, and have “scaffold” download those artifacts, but I don’t like that. Instead, a neat trick I learned was that you can give the Lambda an empty zip folder by using the “hashicorp/archive” provider to create an empty zip folder:

data "archive_file" "dummy_zip" {
type = "zip"
output_path = "${path.module}/data_changes_lambda.zip"
source {
content = "hello"
filename = "dummy.txt"
}
}

Then, you can reference it in your Lambda definitions:

resource "aws_lambda_function" "example" {
# required things here
filename = data.archive_file.dummy_zip.output_path
}

This allowed my desired order to work correctly, with the caveat that if your Lambdas manage to get invoked before the “deploy” job can finish deploying the relevant Lambda artifacts, they will fail with a spectacular error.

You could probably also use containers, since the Lambda runtime supports those as well, but I haven’t tried it, so I don’t know what happens when you set it to use a Lambda that has no images pushed to it yet.

VPC Headaches

Whenever I’d worked in AWS before, I invariably ran into IAM issues, which led me to believe that IAM would be the thing I fought with most in this effort, but far and away, it was VPC. There are basically no good tutorials for how to provision this correctly; you kind of just have to copy the same IP ranges everybody else has copy/pasted and hope you get there. I had at least two moments where something I was pretty sure I’d already tried a few times that didn’t work (but just wanted to check one last time) worked all of a sudden, and I was too frustrated and tired to even be happy about it.

Troubleshooting security groups proved to be quite difficult. There’s no “hey, can an ECS cluster with security group X talk to database Y?” button, but oh boy, did I need one. There is the AWS VPC Reachability Analyzer (fat syllable bonuses all around), but it is not useful for analyzing security groups, as far as I can tell.

Speaking of security groups, I was particularly confused by what counts as ingress. If I have a service that makes an outbound request on port 443, do I also have to allow inbound traffic on 443 for the response? Maybe revealing I had this question makes me look stupid or something, but when all you have to go on is that your service can’t make HTTP requests (because the security group is misconfigured), it’s very hard to rule that out without trying it.

Avoiding Managed NAT Gateways

This proved to be more difficult than I thought it would be. For instance, I found this very, very good tutorial on how to not only use ECS and Terraform, but also specifically for a Go project like mine, which I followed as close to the letter as I could. Everything worked after applying it, too, only I noticed too late that resource "aws_nat_gateway" "ngw" was the notorious, wretched beast I’d feared all along. I was able to find a way to remove it, but it became clear that this is a pattern. Other tutorials I looked at for how to troubleshoot problems asked to use a Managed NAT Gateway as well, because it’s an easy answer to these problems, and most AWS users don’t seem to have to pay the bill.

When I tried to get my workers, uhh…working, they were failing to access SSM Parameter Store because I had put them in a private subnet so they could access the database, which was also in the private subnet. It turns out doing this means they don’t have access to the public internet, and consequently, to access any of a number of AWS services, you have to spin up VPC Endpoints. They are priced in a way that is very much similar to Managed NAT Gateways, and they serve the same functional purpose, only they’re restricted to a singular service, which I took to mean they’re the same service with a different name and maybe simpler configuration.

Later, when I tried to move the meal plan finalizer worker into ECS to try and save on these costs (thinking it would be easier for them to access SSM from ECS than from Lambda), I found I had the exact same problem: the worker couldn’t connect to any AWS services unless it was running in the same task that ran the server (because that server container has an Internet Gateway to receive traffic).

ECS/ALB Headaches

It took me a few days to get my IAM set up so that the ECS service could actually, you know, talk to SSM or RDS or anything. Once I had gotten that up and I could see that my logs were getting to the “server’s up, serving traffic” phase, they were still being systematically killed every 30 seconds or so. They were in this state for a couple of days before I eventually realized that when you set up an ALB to talk to a web service hosted in ECS, it comes with a health check, and this defaults to “/”. This server doesn’t serve anything at the root, so it was failing the health check even though it was healthy. That was pretty confusing, but also pretty easy to fix. Soon enough, the server was up and enjoying life without premature destruction.

SQS Headaches

For about a day, I had a Lambda deployed to AWS that, because of a minor bug, couldn’t successfully execute. By default, a successful Lambda execution is required to acknowledge a message and get it off of the queue. So, if you don’t configure a dead letter queue (I hadn’t), and your code will never succeed (aforementioned), then the message will just get re-placed on the queue, and you’ve got yourself an infinite loop that eats through your budgets:

i want to know how pic.twitter.com/K7yxo9WN8F
— footloose and fancy-free (@vgsnv) December 22, 2021

_{(I figured out how)}

Outcome

I was able to get the service spun up in AWS as I had designed it and able to confirm it worked as I suspected. I also set up a service prober Lambda that would run every five minutes, create a household with four members, propose a meal plan, have some users vote for some options, wait for the meal plan to get finalized, and then validate the outcome. This prober ran successfully for a month or so, and the service handled everything like a champ.

That said, it was VERY expensive. When I was building this in December, I would start my evening at about 7:30PM by deploying and having Terraform spin everything up, and then I would have Terraform Cloud destroy whatever inventory was present by 1:30AM. Nonetheless, my bill for December was $130-ish. I realized the Aurora problem, switched to RDS, and got rid of a lot of the Lambdas to simplify the application and consequent bill. I let the new iteration run for almost a month and was headed towards another $120 bill, which I think means the old system would have cost something like $190/month if I hadn’t swapped databases.

so this seems bad pic.twitter.com/3eu3kM2Nna
— footloose and fancy-free (@vgsnv) January 7, 2022

January’s bill was about $130. I started to move the worker into ECS then, to save costs, but ultimately felt burned by the whole thing. The networking charges aren’t even really that much money in the grand scheme of it all, but the principle of having to pay AWS to connect infrastructure it runs to infrastructure it runs just didn’t sit right with me. I spun down all my infrastructure, and will be taking my code and service elsewhere.

What’s Next?

I don’t want to keep building on AWS, because it feels like going to a sleepover at the house of a girl who has told everyone she hates you. I don’t know where I’ll go next with all of this. Azure doesn’t feel ready yet. DigitalOcean has an app platform now that I might try out. GCP was okay, but has positioned itself of late as the enterprise-y provider of choice. Oh no, am I going to have to use Kubernetes?! I’ll figure it out and make another post.

Conclusions

AWS is a really solid platform that will reliably serve your applications. Once you get through the IAM/VPC hurdles, everything works well and feels sturdy. There are lots of solutions to the various problems one can encounter when building services or tackling new problems. The quality of service you get is truly great, but it is not offered with any benevolence or charity. You will get very high quality service, and you absolutely will pay a pretty penny for it. It’s not without its quirks, but obviously serviceable.

The biggest lesson I’ve taken away from this endeavor is that AWS assumes you have access to some VC-backed bank account, and thus litter their documentation with traps where you have no choice but to give them more money. The broader community aids them in this goal by never mentioning the specter of cost in tutorial blogs. It’s not enough that you’d write your congressional representative about; nobody writes their rep about a $35 business expense. It is more than enough, however, to make me feel like I’m not working with a partner, I’m inflicting myself with a parasite.

I’m really happy with both GitHub Actions and Terraform. There were a few times where Terraform screwed up and left me in a state where I had to manually intervene, but considering the nature of what it does and how well it normally does it, I’m more than willing to tolerate that. Cloudflare was awesome, too, and I’ve long been a fan of the service they operate.

Porktrack: how I turned a goofy idea into a real career

Thu, 01 Aug 2019 11:11:13 -0600

A time gone by

I’d like to tell the story of how I got into the software engineering field, but like any post I make on this website, I feel the need to justify the endeavor first. I’m sharing because whenever I’ve shared it with folks in the past, they seem to have enjoyed it and almost always ask clarifying questions in disbelief, which I think is a safe indicator that the story is maybe good. I’m also interested in simply documenting this story while the details are still immediately recallable.

To set up some context, in 2012, I took CS101 and was primarily left with a fondness for programming. I loved figuring out how to make the computer do things, and I loved finding new things to make the computer do. I kept writing trivial programs and exploring programming in my spare time even after I left school. I wrote some math quizzers, a random color generator, and some other miscellaneous things that were useful to nobody, myself included. I remember contemplating building something like a Twitter client next, but going from generating a random color to authenticating over OAuth when you don’t understand how websites work is a daunting task.

I was working as a security guard for a local condominium complex. I made $11 an hour, and I had just gotten married. I remember sitting next to my new bride, in awe of the responsibility that she had entrusted me by marrying me, and feeling wholly unworthy of the privilege. I was, in colloquial terms, a scrub, but I was at least a self-aware scrub. I wanted a way to turn my fondness of programming into a career, so as to rectify my scrub stature.

What to build

The thing that is hardest about learning how to write code is knowing what to work on. You can be the best and brightest student our planet has ever known, and you could spend hours absorbing the knowledge of how conditionals work, where and when to attach a method to a class, etc., but if you don’t know to what end it’s for, you will never apply it meaningfully, and your understanding will dwindle. I recognized this problem early on and sort of just dealt with it by coming up with (and getting myself hyped about) contrived applications to build. I was never really motivated by games programming either, which is not to cast aspersions at anybody, just a note that I didn’t have an interest in that vehicle for developing programming skills.

Seeking guidance

Thankfully, the building I worked at had lots of software engineers as tenants, and I was their middleman to receiving their packages. One day, I asked a resident for guidance around what I should build next. He responded by suggesting I create a website, which I initially rejected categorically. “I don’t want to be a web developer!” I whined. Web developers, I’d surmised, spent their days changing div colors and filling websites with ads. I wanted to do Real Programming™. “The web is dumb! I want to be a Real Programmer™!” I whined further. What he said next was a pivotal moment for me:

"The web's not dumb, you're dumb."

Okay, so he didn’t actually say that, but he did walk me through the logical inconsistencies of wanting to do “real programming” that somehow didn’t involve the web in a meaningful way. He made it clear that I couldn’t do the larger things I wanted to do without some foundational knowledge best acquired by building a website.

I’d had a really dumb idea for a website I was actually interested in building. The idea was to present a user with an interface for putting in their birthday, then calculating the day they were conceived and looking up what the #1 song on the Billboard Hot 100 was that week, the implication being that if any song was most likely to be on the radio when your parents did the deed to bear your existence, it would likely be the most popular one. I also thought to include an interface for noting if you were a late or early delivery, as that obviously impacts the results.

I explained it to the man (who really just wanted to pick up his new headphones or whatever) and worried aloud that I didn’t think I could put such a silly thing on a résumé, and asked if he had any better ideas. He said something to the tune of “in this field, and in this city, you can put whatever the fuck you want on a résumé and they’ll probably bring you in.”

Actually building Porktrack

I knew I had to start by collecting the relevant data first. I chose (and learned) Python in order to fetch the data because I’d read in Steven Levy’s In The Plex that Google had chosen Python for their web crawling tasks, and I didn’t know what cargo culting was. My first stab was to take the data from the Wikipedia pages for the Hot 100 chart by year (1959 as an example). Initially, this meant reading the HTML of the page as a string and using Python’s string find() and replace() methods on various HTML properties to extract the actual data. I say HTML properties and not, like, XPath because I definitely did not know what XPath was (and performed an actual facepalm when I finally discovered them).

This obviously didn’t scale well, as some of these pages are formatted wildly differently from each other. I briefly contemplated “standardizing” the formatting by being a particularly pedantic member of the Wikipedia community, but it didn’t feel appropriate to let this project have any lasting impacts on Wikipedia. In the oldest version of the source code that I can find, I’d already realized how poorly this scaled and ended up switching to a modified HTMLParser object. I probably went that way because a Stack Overflow post suggested that was how you scraped the web with Python.

The parser would traverse the elements on the page, and incremented state based on the current state. So, the data would come in the form of three table cells in a row, the first with the date, the second with the artist, and the third with the song title. Once the script got to where it could semi-reliably ascertain who had what hits when, it spat them out as SQL queries to stdout and saved the output as what I would learn is called a migration script. I would then execute the file’s query in PHPMyAdmin, which meant I had a database. Eventually, I modified the script so that it would also fetch the top YouTube video ID from a search for “{artist} {song_name}”.

My understanding of the LAMP stack was that it was literally[^1] the only way to make a website where the response was dynamic. I just dove straight into the Codecademy course for PHP. If you had told me you could write web servers in Python, I would have rejoiced, because building the scraper made me really fall in love with the language.

Development environment

I didn’t know how to set up a local development environment and, indeed, was using Windows mostly, where these things tend to be harder than they should. My workflow ended up being that I would write PHP files on my local machine, transfer them to the appropriate place on the machine via FileZilla, and then hit the web page in order to see my changes. It is still, to this day, one of the best development environments I’ve ever had. I’m not even being kinda sarcastic. I love instant feedback, and had I gone through the process of setting up WAMP or whatever, I likely would have doubted that whatever was working on my machine would actually work on production. When your dev environment is production, though, no such doubt can exist. (Note: I am not recommending this workflow for anything meaningful.)

One of the main (only?) benefits to being a security guard is the downtime. Most of the time, your job is just to be present, but only physically. Most security guards don’t have access to a computer, and many times if they do, they are subject to rigorous IT net nanny policies because their computers are just a workstation in a warehouse full of workstations, and they aren’t going to provision them differently because the security guard is bored. We’re all bored. I was lucky, though, because I worked for a condo complex, so there were no IT firms, and the computer was a Dell shitbox I bought from the Office Depot myself on the HOA credit card, if I recall correctly. I was able to install Python on it and able to use IDLE during lulls in the workday in order to write the scripts that fetched data for the database. I was also able to install FileZilla and change the actual production website.

Infrastructure

The only infrastructure provider I felt comfortable using was DigitalOcean, who I still think really deserve a lot of credit for building an interface that someone who knows nothing about programming can still use and feel like they haven’t screwed themselves out of a paycheck. I used the one-click-setup LAMP stack droplet. I don’t remember if this configuration came with PHPMyAdmin, but I didn’t know it existed before this experience, so it wouldn’t surprise me at all. I probably could have swung the $10/month droplet, but I stuck with $5 because of how little I made. I used DigitalOcean’s DNS tool and nameservers as well.

I eventually cooked up enough crappy PHP that the site did what I thought it should do. I was also learning HTML/CSS/JavaScript/SQL in order to accomplish this task, and I really enjoyed myself. I put the files onto the droplet via FTP, and the site seemed to work as I expected. I did some basic testing around awful input, but I didn’t understand what unit tests were or how to write them, so I didn’t do any of that.

15 minutes of fame

What happened next is still one of the wildest things to happen to me in my entire life. Once I felt like the site was in decent shape, I posted it onto /r/music, and it took off like a rocket. My post received thousands of upvotes and made it to the front page, which led Porktrack to receive just over 2.1M hits in the first 10 days it was live. I know this because I had the foresight to include Google Analytics on the page. The site didn’t have ads on it for almost all of it, if I recall correctly (eventually did place ads on the site via AdSense).

Most of the traffic fizzled out and died by mid June, but I kept checking the AdSense results because I had vowed to shut down Porktrack the month it stopped making enough to pay for its own droplet. Almost half a year later, I was running an errand and mindlessly checked AdSense to find that I had made $2,500 that day. It was the day before Thanksgiving, and someone had posted it to a different subreddit (that I can’t manage to find at the moment), where it was gaining a similar influx of traffic from bored Americans waiting for planes. I ended up making $3,300 after all was said and done, which was cool.

One of my favorite parts about this experience was other folks' reactions. People genuinely seemed to think the idea was amusing, and nobody seemed to be particularly offended. One of the other folks who lived at those condos was a guy who was part of a morning radio trio. He came up to the guardhouse the day after it went viral telling us about this crazy website his broadcasting company had approved as a topic of morning conversation. He explained to me that radio stations all over the country get the exact same list, and it’s up to the individual hosts to decide what they’ll talk about. He said they hadn’t opted to talk about it that morning, but he said it had been a close contender. The look on his face when he found out it was my website is something I don’t think I’ll ever forget, if I’m lucky.

Indeed, I was reached out to by a few different radio stations who let me know they spoke about my website on air. Some larger websites also wrote about the project, which I interpret not as a sign of incoming fame or fortune, but more as a sign that content folks are desperate to talk about anything at all.

Your browser does not support the audio element.

^{^{edited audio clip of FM96 in London, ON, CA talking about Porktrack}}

Your browser does not support the audio element.

^{^{edited audio clip of FM94.5 The Buzz in Houston, TX, US talking about Porktrack}}

Encouragement and detraction

I mentioned in the comments on the Reddit post that it was my first website and I’d spent almost a month writing Python “scraping” code to get the database, and then built a website around it, yadda yadda. I thought I’d include this screenshot, as it was really motivating, and I think I even made it my desktop wallpaper for a hot second.

Some folks thought the idea was good and tried their own hands at building the app. I got contacted by somebody via Reddit message who said that someone had “copied” my website and made a Dutch (I think?) version of it. This is the only evidence I have that the website existed:

Another person had seen from the comments/tweets/looking at my code on GitHub that I was a beginner and took it upon themselves to implement it their own way and send it to me. It certainly looked nicer than any of the Python I’d written up to that point, and, for some reason, I kept this screenshot, likely smitten that anybody else would write code after something I’d written:

It is still the Internet, though, so there was, of course, the usual pettifoggery.

Database debacles

Crucial to remember in all this is that for me, the request-response lifecycle, the notion of a CRUD app, was becoming obviously clear to me, and its power was palpable. “This is how the Internet works!” I rejoiced, with the realization that if I wanted to build a website for keeping track of cats I saw on the street, I was just a little schema and SQL away from having exactly that!

I was on top of the world and wanted to put my new knowledge to great use, so I did what any hapless fool in that circumstance would do: I reinvented analytics. I created a table in PHPMyAdmin that had columns for some basic stuff around timestamp, source IP, referring URL, etc. I queried this new database table the way you might expect of someone who doesn’t know what they’re doing: by opening a connection to it. Every time someone hit the website, the page would make two MySQL connections to the exact same database, one for the tracks, one for the analytics data.

Shortly after launching the site, folks objected to the process by which their Porktracks were determined, noting that their parents were immigrants/die-hard country fans/otherwise would never have listened to the Hot 100. So, I very quickly adapted my scripts to account for the country and Latin charts, created new tables for them, and queried them the way I had done thus far. Yup, another two connections.

This very quickly unraveled and resulted in my $5 DigitalOcean droplet soliciting union membership interest from the other droplets. I think it might actually be impossible to be stupid enough to look at a degraded web page initiating four connections to the same database and not figure out the solution, because I’m certain I would have just speechlessly accepted my plight if it were. After switching to just the one connection per request and eliminating data gathering in its entirety, I never had issues with the site going down ever again.

Even though I knew nothing, I knew that “real” websites did load tests and other validation work before shipping software. I allowed myself to be convinced that I shouldn’t put in similar efforts for this website because of how goofy it was. I didn’t want to be the person who spent a great deal of time making a gimmick website reliable, but as a consequence of that attitude, I suffered the failing defeat of having my site go down. I learned the very valuable lesson that if something is worth doing, it’s probably worth treating it like it’s going to be the sort of runaway success that would give you uptime problems. If you treat everything like it’s going to go gangbusters, then you’ll make very different decisions around how your application is designed/hosted/maintained.

The real benefit

Ultimately, the consequence of building the site that I’m most grateful about was that it gave me something to do. I’d heard that single-page applications were a thing, and I didn’t know how they worked, but I had a working site receiving steady traffic. I would wait until late at night and cobble together some awkward vanilla JS, a results-specific route that spat out HTML, and accomplished a “single-page app” by replacing the .innerHTML field. Then, I heard that nobody writes JavaScript, everybody writes jQuery, so I rewrote my crappy vanilla JS into crappy jQuery JS. Then, I wanted to create a mobile app for it, so I started down that process too (but never actually finished it, probably to everybody’s benefit).

I got to where I was so comfortable talking shop with other software developers at meetups and such that they started to inquire where I was working and would react with disbelief upon learning that I was a security guard. I got a lot of encouragement from the Austin tech community in general, before or after feeling acclimated to the task. I eventually mustered up the courage to put together a résumé and started applying to junior developer jobs. I was told twice after being rejected that the novelty of Porktrack and the implied chutzpah required to put something like that on my résumé were too alluring for some folks to pass on at least an interview.

It took me a solid eight months of applying every day, going to interviews where the expectations were completely out of line with what I’d advertised on my résumé, and failing to get those jobs but getting better at and more comfortable with the process with each step. Eventually, I was hired at a startup as a junior web developer, and the rest of my career is what I’m in the middle of now.

Luck

The thing I always try to emphasize when I recount this whole thing is that much of my “success” came in the form of good fortune. I was/am fortunate in so many ways:

I was fortunate that I didn’t have any children or sick relatives to tend to, and a supportive spouse, so I had all the time and space I needed to try and accomplish my goal. I routinely had 4+ hours of free time on weekdays and virtually the entire weekend to read tutorials and run little experiments. From what I gather, that would never fly in a house with a greater than zero amount of children.
I was also lucky to work at a place that was more than fine with me doing extracurricular activities while “on the job.” My bosses at the time both knew that I was working on Porktrack and that I wanted to find a way to be a software developer. They always allowed me to spend time at work reading about code or writing dumb programs, provided I took care of my actual work obligations (which I always did). They helped test Porktrack for me a few times, and when the time came to interview, they were very generous and understanding about the necessary away time I needed to attend them. If I had worked somewhere where resources were more scarce or had otherwise made myself intractably necessary somewhere, I don’t think I could have done it.
I mentioned this earlier, but it’s worth repeating that I am lucky that the friendly folks who lived at the condos indulged my endless pestering for suggestions on what to learn or work on next. I had actually managed to finagle an internship from one of them as a QA intern but got hired at the startup shortly after starting, so I left for that.
I was lucky that I lived in a city with an active tech economy/community. Most cities in the United States cannot facilitate the sort of access I took advantage of to learn from professionals. Many cities with good tech economies have pretty pitiful meetup scenes, too.

I worked really hard, to be certain, but all the hard work in the world would have been for naught if my situation had been like that of the average. I had a blast, got to bring folks a very small amount of joy, and it never once felt onerous. I feel sometimes as though I found a cheat code by accident. It’s not supposed to be that you can have a lot of fun and end up making good money. It’s supposed to be that you spend at least three decades doing menial tasks for people who don’t care about you and remark upon the expense of your wage with disgust.

Porktrack redux

Recently, I saw this neat video where a man recreated his very first flipbook from 30 years ago:

I felt inspired, and obviously thought about my very first website and realized that I’ve learned so much in the last five years. Almost nothing about how I build personal projects is the same as it was. In fact, quite a few options for hosting such a website have come about in the intervening years. I resolved to rebuild the application and take the time to address some of the complaints I had as a maintainer back in the day. The main things I want to tackle:

Scaling: Porktrack didn’t scale well at all, either up or down. I want to rebuild it in a way where it can be hosted basically forever, without costing me a tremendous amount of money to do so. Simply put, it’s not important enough to me to warrant $60/year to host. To give you an idea, if I can’t pay for this site’s hosting costs with a dime per month and have change in return, I will consider myself unsuccessful.
Language: Shortly after starting my first job, I discovered Go and have not really written much Python outside of work since then. I want to rewrite the core components of Porktrack in Go.
Data completeness: When I last ran Porktrack, I had a reminder every month to run the script again and add things to the database so that I wouldn’t lose track of that. I’d like the new version of the site to update itself as much as possible, without human intervention. Also, the YouTube video IDs I’d save would very frequently get axed for having copyrighted content, so I want to minimize that consequence to whatever extent I can.

Sketch out a plan

The first order of business was to build the thing that fetched the database. In the old application, I had Python scripts that spat out SQL migrations for all the Porktracks. This was handy because it took a week or more before I finally felt like the script did the thing it was supposed to do every time. This way, in the event that the script got 95% of the entries correct but maybe futzed up a single quote here or there, I could just manually fix it in a text editor without having to add a conditional to my code or otherwise write a bunch of safeguards.

This time around, I’m doing something similar, only I’m writing to a local SQLite database that doesn’t get added to version control. This accomplishes a few things:

It gives me a similar sort of in-between inspection scheme where I can validate that the data looks the way it’s supposed to.
It makes it so that I don’t have to constantly fetch from Billboard and risk detection/IP blocking.
Lots of tools exist for querying a local SQLite database.

Architectural choices

I decided to host the application on Cloud Run. Cloud Run is a managed serverless application platform, announced this year at Google Next, where you publish a container to a private registry and then tell GCP to associate that container with an HTTPS-enabled URL, which will execute your server when it receives a request. Like most serverless schemes, you are only charged for the time your application is alive. Thankfully, you just write a server. Standard routing frameworks work just fine.

There is one main consideration for serverless applications, however, which is the database. Cloud SQL is completely viable for Cloud Run use, (and as a matter of fact, the Cloud Run interface provides a convenient mechanism for pre-defining Cloud SQL connections) but a Cloud SQL instance costs ~$8/month minimum, which, as we all know, is greater than a dime. Cloud Firestore, however, supports tons of simultaneous connections for queries and gives you (as of the time of this writing) 50,000 free queries per day.

I ended up using the Spotify API to provide sound samples of the songs instead of YouTube. In the years since I last built Porktrack, YouTube has started requiring JavaScript to load search results, and their API limitations are, in my humble opinion, disrespectful. There basically wasn’t a way I felt comfortable retrieving that data and validating that it was still relevant, whereas from what I can tell, the Spotify IDs are more permanent.

So, I’ve rewritten Porktrack as an entirely serverless application and made a serverless data store its primary source of truth. I have code that initializes the database locally, and after the local SQLite database is loaded and I feel confident that data is collected well, I run a simple function that will iterate over every entry in this database and save it to a Cloud Firestore collection.

I also have a Cloud Function being activated by a Cloud Scheduler job once a week on Sundays to check the Billboard site and run the same function that gets run when we build the SQLite database, and then save that data into Cloud Firestore.

I got to handle the scaling problems by waiting for infrastructure providers to get WAY better at what they did. I got to port it from a good language to a better one, and I don’t have to set a reminder on my phone to run a script and execute some SQL, because managed infrastructure will handle that for me. ~~You can check out the current version of the site by going to https://www.porktrack.com.~~

Update 2/1/20

The new app has been running for a few months now, I wanted to demonstrate the actual monthly costs of the app.

Here’s October 2019’s bill:

Here’s November 2019’s bill:

Here’s December 2019’s bill:

I’ve shut down the site now, since I’ve proven my point, and really, this site shouldn’t exist. :)

[^1] I mean literally literally, not figuratively literally.

Weapons and Tools

Tue, 26 Feb 2019 19:04:35 -0600

A hammer wielded by a worker trained to hit things is a tool. A hammer wielded by someone intent on doing harm is a weapon. The hammer requires no metamorphosis yet starts as a tool and ends as a weapon. So, there must be a threshold where the tool becomes a weapon, what is it?

When I write software, I don’t build things that do things, I build tools for doing things with. If I’m tasked with communicating with an external API, I don’t just write the code for executing raw requests. I write a client for interacting with that external API, and give it an either straightforward or thought-out abstraction. This way, when the time arises for another person to interact with that same API, it’s however much easier to do than it would have been.

Lately, though, I’ve been asking myself how I could weaponize whatever tool I’m building. That external API client, in the hands of someone who knows how to use concurrency well, could be a very precise DDoS tool, for instance. Most of the time my answers are some awkward mental stretch like that, because I’m fortunate to work for a WordPress hosting company. I suppose if your evil scheme involved spinning up thousands of WordPress instances, we’d be your huckleberry.

My first stab at answering this question was that misuse transforms tools into weapons. After all, the hammer was not meant to harm anyone, yet there are an infinite number of ways to misuse a hammer without hurting anybody at all. In our premise, however, the hammer wasn’t wielded by a fool who didn’t know how to use a hammer properly; it was wielded by someone intent on doing harm. Maybe what transforms a tool into a weapon is motivated misuse, which is to say that turning a tool into a weapon requires not only its deliberate misuse (weaponization), but also the desire to do so.

That seems satisfactory, but it instantly begs the question in my mind: Can you design a tool that is impossible to weaponize¹?

Since there are two hard dependencies for the weaponization of tools (motive and misuse), can we design tools so that they aren’t capable of being misused, thus eliminating the second factor? I’d posit that the statement “some tools are more weaponizable than others” is non-controversially true. A hammer is weaponizable, I think, in part because its purpose is relatively generic. Its purpose is to hit things², ostensibly nails, but not necessarily. There’s nothing nail-exclusive about the shape of your run-of-the-mill hammer’s head.

Can we make the hammer unweaponizable? I think we can get pretty close, but what we’d have wouldn’t be called a hammer because nobody would recognize it as such³. Nevertheless, it doesn’t take a particularly vivid imagination to conjure up a horror movie where someone befalls a fatality at the hands of a particularly motivated user of the marketed-as-non-weaponizable hammer.

I beat this dead horse only to illustrate that the line between weapon and tool is quickly and unremarkably crossed. It should be noted that a weaponized hammer has the obvious implication of a consequential murder, but you could make the case that tools are weaponized to nobody’s harm, or even for pure benefit, all the time. Chemotherapy is weaponized against cancer cells, scissors are weaponized against excess paper, caffeine is weaponized against poor decision making the night prior.

Our world is simultaneously riddled with tools generic enough to be weaponized and people with the motivation to misuse them. Even more remarkable, to me, is that I’m fortunate enough to be able to spend the majority of my day with the most fascinating and generic tool humanity has ever devised: the computer!

The computer is a bit like the hammer, only several orders of magnitude deeper. It’s a tool so generic it can make other tools. It can make tools that help it make tools. It can make tools that help the tools that allow it to make the tools that the tools it makes are trying to make for it. That’s cool as heck, but as you’ll recall, we’ve assumed the more generic the tool, the more weaponizable it is (computers are, like, super weaponizable, ya’ll). Furthermore, there is an entire industry chock-full of folks like me who use these very generic tools to make tools, which we’ve assumed are also potential weapons.

I’m certain that there is no easy, memorable catchphrase that will bestow upon anyone who hears it the knowledge to make unweaponizable tools. Acknowledging that doesn’t absolve our profession of the obligation to ensure that we build safeguards into tools we think might be particularly weaponizable. Engineers at Google and Microsoft have had to take the extraordinary step of publicly registering their fervent opposition to the weaponization of the tools they’ve developed. That’s a great first step, and the braver comrades of those groups have put their resignation letters where their mouth is. More of that, please.

These engineers' protests may be some of the first we’ve seen, but they won’t be the last. I won’t work at the hosting company forever, and students won’t be in a position of writing relatively harmless applications under the critical gaze of an academic forever either. At some point, more of us than we’re comfortable acknowledging will be asked to build something that has truly horrifying and obvious weaponization opportunities, or even be asked to build literal weapons. If you’re a professional software engineer at any level, that is a scenario you need to think about and reconcile with your morality now, so that you may be firm in your decision later, to whatever consequence.

It’s worthwhile to take the time to point out that regulation in itself is both a tool and weapon. Regulations aiming to prevent the production of meaningful weapons could be weaponized against the developers of tools that either occupy the grey area between tool and weapon or are otherwise capable of marketing to folks as such in order to justify their regulation. ↩︎
The hammer is such a generic tool that someone smart, long (enough) ago, decided, “This thing doesn’t do enough,” and added a nail puller to the other side. So even though the hammer came with two sides of its primary function, someone decided that was too much hammer by half and invented another thing to replace it. ↩︎
In my amateur science fiction imagination, I can picture something the size of a shot glass that would have some form of smaller-diameter beater inside it that would try to ensure its own surface was as level with the surface of the outer lid as it could manage, applying linearly greater force to the nail head until it determined that progress would not be made. ↩︎

Using Docker to compile Golang plugins on OS X

Thu, 11 Jan 2018 20:23:35 -0600

Okay so this might be old news to some of ya’ll, but I just got it figured out, and any time you figure something out that didn’t have an immediate answer online, you should blog about it, right?

The Problem

I’ve been building a side project for a while that has an API server component. It’s built with Postgres in mind, but being that it’s something meant to be self-hosted in the future, I wanted to provide any potential users with the option of using whatever database they chose.

The server in question is written in Go, and the main problem I was worried about facing was related to imports. Sure, I could write a postgres package, and a mysql package, and I could even import them both in my main.go file and instantiate the appropriate struct based on a user config, but where does that madness stop? Could my precious main.go one day look like this?

package main
import(
"database/sql"
"github.com/org/project/api/storage/db/postgres"
"github.com/org/project/api/storage/db/mysql"
"github.com/org/project/api/storage/db/mongodb"
"github.com/org/project/api/storage/db/sqlite"
"github.com/org/project/api/storage/db/cockroachdb"
"github.com/org/project/api/storage/db/esoteric_database_you_have_never_heard_of"
"github.com/org/project/api/storage/db/plain_old_txt_files_lol"
_ "github.com/lib/pq"
_ "github.com/lib/repeat"
_ "github.com/lib/ad"
_ "github.com/lib/infinitum"
)

That obviously won’t do. I remember reading about Go’s plugins when they came out a while back and decided that might be an interesting way to tackle this problem. I could break my postgres package out into its own repo, import it as the default option, and accept plugins that were built by developers for their favorite databases, so long as they satisfied the interface I had defined for what constituted a database storer.

So I built a tiny demo app that executed a very simple query, and used some code from the existing repo to create a tiny package that I could then compile into a plugin to test my theory. I ran go build -buildmode=plugin, and received this output:

-buildmode=plugin not supported on darwin/amd64

Note that Go returns this error even if there’s a syntax error in the code. It won’t even bother trying to see if your code is okay if you’re trying to use -buildmode=plugin on an unsupported platform.

Rats! In all my excitement and fervor, I never thought to check if plugins were even supported on my platform. It even says on the release notes that I linked above! Plugin support is currently only available on Linux. It was a nice reminder that because I sort of naturally gravitated towards being a Go developer on OS X, I’m much more frequently in a position to beta test new things than be shut out of them.

But what to do about my problem?

Docker to the rescue

I wasn’t worried about the API being able to use the generated files, because they run entirely in Docker containers anyway. I run debug builds locally and integration tests everywhere using Docker compose. So I wondered if I could set up a Dockerfile that would add my package, build the .so file I needed, and spit it back out to my host machine so I could hand it over to my other package and get all the things I wanted out of plugins without having to futz with partitioning drives or picking a distro out and otherwise completely reconfiguring how I do my work.

Here’s what I came up with:

FROM golang:latest
WORKDIR /go/src/github.com/verygoodsoftwarenotvirus/plugintest

ADD . .

CMD go build -buildmode=plugin -o /output/result.so

and the corresponding bash script:

mkdir -p output
docker build -t plugins .
docker run --volume=$(pwd)/output:/output --rm -t plugins

Run your bash script, and you’ll have output/$PKG_NAME.so in your folder in relatively no time. Note that you still have to run the program that intends to use the plugin in a linux-based Docker container, but if you’ve made it this far, you probably know how to do that.

Hopefully this helps somebody else who faced the same predicament I did. :)

Posts on Very Good Software, Not Virus

AI as a means of evading accountability

How I write tests in Go

How I write unit tests in Go

We have the technology

The test command

Table-driven tests

Parallel

Subtests

Libraries

Fake Values

Testify

Testcontainers

Failure Cases

JSON Encoding

URL Parsing

Conclusion

Go’s Error Handling Is Perfect, Actually

In defense of if err != nil

Result types in Go

Conclusion

Preventing generated files from causing problems

The premise

The problem

THe solution

Bring your own interface

New Toys

Sylvia's Law

The Life and Death of Sylvia Joyce

Legislating

Lobbying Pushback

What I Learned at Founders Academy

Context

Spoiler Alert

Observation: much of early startup advice is geared towards B2B SaaS companies

Validating Product Assumptions

Privilege

You can screw over your company’s future and not realize it

Investor management is a crucial aspect of running a growing company

Founder Compositions vary and matter

Greener Cloud Pastures

Preface

Background

First Candidate: DOAP

DigitalOcean Shortcomings

Kubernetes

Where to Now?

App Recap

GCP Terraform Obstacles

IAM Woes

Cloud Run DNS Woes

Cloud Functions vs. Lambda

Cloud Function Woes

Cloud Run Woes

Database Connectivity

Google Cloud Console

Static Site Woes

Monitoring Improvements

Frontend Cookie Frustrations

Pricing Outcomes

Conclusion

Baby's First AWS Deployment

Preface

The App in Question

Goals

Translating to the Cloud

Database

Diving In

Configuration

Observability

Elasticsearch Woes

Aspiration Shopping

Ordering is Important

VPC Headaches

Avoiding Managed NAT Gateways

ECS/ALB Headaches

SQS Headaches

Outcome

What’s Next?

Conclusions

In defense of `if err != nil`