vaderlab-infra — Phase 1

Provisions two always-free OCI ARM instances, each on its own network, plus the tenancy's spend tripwire. One box will run a self-hosted NetBird control plane; the other will serve public sites. Written in Go to double as Go+Pulumi practice that transfers directly to your work stack.

What this builds

Three Pulumi projects in one Go module, sharing builders under internal/:

netbird/                                    foundation/
  VCN 10.0.0.0/16                             Budget ($1/month, root compartment)
  ├─ Internet Gateway                         ├─ Alert: ACTUAL   ≥1%
  ├─ Route Table    (0.0.0.0/0 → IGW)         └─ Alert: FORECAST ≥100%
  ├─ Security List  (22, 443, 80, UDP 3478)
  ├─ Subnet 10.0.1.0/24
  │  └─ A1.Flex  1 OCPU / 6 GB, Ubuntu 24.04 arm64
  │     └─ cloud-init installs Docker + compose
  ├─ Reserved Public IP
  └─ Route 53 A record → that IP    ← optional; zone is looked up, NOT managed

sites/
  VCN 10.1.0.0/16          ← deliberately non-overlapping with netbird
  ├─ Internet Gateway
  ├─ Route Table    (0.0.0.0/0 → IGW)
  ├─ Security List  (22, 443, 80 — no UDP 3478; coturn lives on netbird)
  ├─ Subnet 10.1.1.0/24
  │  └─ A1.Flex  1 OCPU / 6 GB, Ubuntu 24.04 arm64
  │     └─ cloud-init installs Docker + compose
  └─ Reserved Public IP

Eight resources each for the two workload projects, three for foundation. The Ubuntu release is configurable (ubuntuVersion, default 24.04); Oracle publishes no 26.04 image family as of Aug 2026. Instance size is configurable per project (ocpus, memoryGb), capped in code at that project's declared share of the tenancy allowance.

Why two boxes rather than one. The always-free Ampere allowance is a divisible tenancy-wide pool, so one 2 OCPU / 12 GB box and two 1 OCPU / 6 GB boxes cost the same: nothing. Splitting it buys separate kernels and separate Docker daemons, so a compromised or misbehaving site container cannot reach the service that authorizes access to the whole overlay — and breaking a site can never cost you overlay access. Isolation you get structurally beats isolation you have to maintain.

Why separate VCNs. The two boxes talk as overlay peers over the public internet, so neither needs private access to the other. Separate networks let either project move to a different provider without orphaning the other's infrastructure. The non-overlapping CIDRs matter because NetBird can advertise network routes, and overlapping ranges break routing in a way that is painful to unwind once peers are enrolled.

Deadline: 18 Aug 2026. Oracle halved the always-free Ampere A1 allowance from 4 OCPU / 24 GB to 2 OCPU / 12 GB on 15 Jun 2026, and from 18 Aug 2026 it auto-terminates instances over the new limit. The two boxes sum to exactly 2 OCPU / 12 GB, and each project's main.go refuses to build anything larger than its declared share, returning an error rather than silently generating a bill.

That sum is an invariant no single program can check — each project sees only its own config. internal/freetier holds the tenancy totals in one place and each project declares its slice; keeping them consistent is a human job.

One-time prerequisites

1. OCI account + CLI auth. Install the OCI CLI and run oci setup config (creates ~/.oci/config). The Pulumi OCI provider reads this automatically — no keys are stored in this repo.
2. Pulumi CLI + Go 1.26 installed.
3. An SSH keypair (ssh-keygen -t ed25519 if you don't have one).
4. A domain you control. Not needed for Phase 1's pulumi up, but the NetBird control plane that follows needs a public FQDN with real TLS: the dashboard, management API, signal, and relay are all served over HTTPS behind a reverse proxy, and Let's Encrypt validates over HTTP on port 80. Plan on pointing a subdomain's A record at this instance's public IP.

Projects and stacks

One repo, one Go module, several Pulumi projects. Each has its own directory, its own state, and its own config:

Directory	Project	Stack	Owns
netbird/	vaderlab-netbird	oci	control-plane box + its VCN
sites/	vaderlab-sites	oci	public-sites box + its VCN
foundation/	vaderlab-foundation	oci	spend budget + alert rules
dns/	vaderlab-dns	aws	professional domain's zone + records
dns-personal/	vaderlab-dns-personal	aws	personal domain's zone + records
dns-defensive/	vaderlab-dns-defensive	aws	defensive .com zone + its registration
site-partner/	vaderlab-site-partner	aws	partner's site: zone, bucket, CDN, certificate

Stacks are named after the provider, not an environment:

prlakhani / vaderlab-netbird / oci
   org          project         stack

The provider is the axis this repo actually varies (OCI now, OVH or another VPS later); there is no second environment. Naming this way lets a future ovh stack sit beside oci with both configs retained, since Pulumi keeps stack config whether or not resources are up. It also means either box can move providers independently — which is the point of separating them.

A stack is named for what it creates, not what it reads. vaderlab-dns/aws builds Route 53 records whose value comes from an OCI stack, via a StackReference to the netbird box's IP. It is still an aws stack. Getting this backwards is easy: these were originally all named oci on the reasoning that the repo is "an OCI project", which inverts the rule the convention exists to express.

Why several projects and not one. A Pulumi project is a blast-radius boundary: pulumi destroy operates on a whole stack, and a mistake in one program cannot propose changes to resources it doesn't own. foundation is separate for a sharper reason — governance should outlive the thing it governs, and a budget destroyed alongside a workload is missing precisely when a teardown-and-rebuild is generating unexpected spend. The DNS projects are split per domain so that an error in one cannot reach another's records, which matters most for the one that isn't yours.

Projects sharing a stack name share a config filename — every oci project has a Pulumi.oci.yaml, every aws project a Pulumi.aws.yaml. Consequences in Gotchas; one of them cost an hour.

Setup

Each project is configured and deployed from its own directory. Run go mod tidy once at the repo root first — the module is shared.

Do foundation first. It is the cheapest to create and the one you want watching while everything else is built.

cd foundation
pulumi stack init prlakhani/vaderlab-foundation/oci
# tenancyOcid is the ROOT compartment, deliberately different from
# compartmentOcid: OCI requires every budget to be created in the root,
# whatever compartment it measures.
pulumi config set --secret tenancyOcid       "ocid1.tenancy.oc1..xxxxx"
pulumi config set --secret compartmentOcid   "ocid1.compartment.oc1..xxxxx"
pulumi config set --secret budgetAlertEmail  "you@example.com"
pulumi preview && pulumi up

Then either box — they are independent and the order doesn't matter:

cd ../netbird        # or ../sites
pulumi stack init prlakhani/vaderlab-netbird/oci
pulumi config set --secret compartmentOcid    "ocid1.compartment.oc1..xxxxx"
pulumi config set --secret sshPublicKey       "$(cat ~/.ssh/id_ed25519.pub)"
pulumi config set --secret sshIngressCidr     "$(curl -4 -s ifconfig.me)/32"
pulumi config set          availabilityDomain "AD-1"
pulumi config set          ocpus              1
pulumi config set          memoryGb           6

pulumi preview       # read the plan — nothing is created yet
pulumi up            # or ../retry-up.sh netbird, if capacity is scarce

The netbird project can also create its own DNS record, in an existing AWS Route 53 hosted zone. Both keys are optional — omit them and no DNS resource is created and no AWS credentials are needed:

pulumi config set --secret dnsZoneName  "example.com"
pulumi config set --secret netbirdFqdn  "netbird.example.com"

Choose that FQDN carefully — it is a one-way door. Every enrolled NetBird peer stores the management URL, so changing the hostname later means re-enrolling every device rather than editing a record.

Set ocpus and memoryGb explicitly even though they have defaults. The fallback is the project's share in main.go, which is a policy ceiling recording how much of the tenancy pool the project may claim — not a size request. Letting one value serve both purposes means editing a Go constant silently resizes a live box, and makes "1 OCPU because I chose it" indistinguishable from "1 OCPU because I forgot".

Config keys need no project prefix — each program calls config.New(ctx, ""), which defaults to its own project namespace. That default is also why config cannot be moved between projects by copying YAML: the namespace is part of the key.

compartmentOcid and sshPublicKey are duplicated across netbird and sites because Pulumi config is per-stack with no shared layer. Pulumi ESC is the eventual fix; three copies of two values is not yet worth a new dependency.

Secrets and state

Pulumi.oci.yaml is committed on purpose. Values set with --secret are stored as secure: ciphertext encrypted with a key held by Pulumi Cloud, which never appears in this repo. Committing it keeps config versioned alongside the code that reads it, and a leaked repo yields nothing usable.

compartmentOcid, sshPublicKey, and sshIngressCidr are marked secret for privacy, not secrecy — none is a credential, but the OCID identifies your tenancy, the CIDR is your home IP, and the public key is a stable identity fingerprint. In a public repo that is worth encrypting; it costs nothing.

Verified caveat — do not trust the [secret] display. cfg.Require() returns secret config as an ordinary plaintext Go string. Pulumi has no taint tracking through your code: the [secret] shown during pulumi up is a literal find-and-replace of the secret's exact bytes in the progress output only. Transforming the value (uppercasing, slicing, interpolating) defeats the masking silently, and pulumi stack output prints every derived value in plaintext — it is stored unencrypted in state, backups, and history.

Rule: never ctx.Export() a value derived from secret config. The current exports (publicIp, instanceId, sshCommand) are safe — none carries one.

Note that cfg.Require is required (not a style choice) for compartmentOcid: OCI data-source args like GetImagesArgs.CompartmentId take a plain string, not a pulumi.StringInput, so RequireSecret would not type-check there.

State is far more sensitive than config. pulumi stack export writes every resolved attribute of every resource — the SSH public key, the home IP in the security list, every OCID — as plaintext JSON. backup.json and edited.json are gitignored for that reason, but the safer habit is to delete them as soon as whatever needed them is finished.

Verify (Phase 1 success test)

Per box, from netbird/ or sites/:

pulumi stack output sshCommand      # prints: ssh ubuntu@<public-ip>
ssh ubuntu@$(pulumi stack output publicIp)
# on the box:
docker --version && docker compose version   # cloud-init should have installed both
cloud-init status                            # want: status: done
nproc && free -g                             # want: 1 and ~6 GB

cloud-init status is worth checking explicitly rather than inferring from Docker being present. It reports error or degraded for failures that leave a usable-looking box — which is exactly what happened to the sites box on its first boot, for reasons in Gotchas below.

If those succeed on both boxes, Phase 1's foundation is done. Next: deploy the NetBird self-hosted compose stack on the netbird box and enroll your first peers.

Do not "prove it works" by tethering to your phone. An earlier version of this file suggested that, and it contradicts the whole point of sshIngressCidr: once you have narrowed ingress to your home /32, SSH from any other network is supposed to time out. A failure there means the firewall is working.

The test also proved nothing to begin with. This box lives in OCI, not on your LAN — the only route to it from home wifi is the public internet. A successful SSH from home already exercises the internet gateway, route table, security list, and public IP. There is no second network that tests anything more.

If a connection from an allowed network hangs, distinguish "blocked" from "broken" by comparing pulumi config get sshIngressCidr against curl -4 -s ifconfig.me on the client. OCI's security list drops non-matching packets silently rather than sending a RST, so a blocked source and a dead host look identical from the client side.

Gotchas

• Building needs ~7.5 GB of RAM. go build ./... compiles pulumi-oci/.../oci/core — a single package of ~230 generated files — as one unit, and the default 4-way parallelism will get the compiler OOM-killed (exit 137) on a 14 GB desktop. Build serialized:

  GOMAXPROCS=2 go build -p 1 -gcflags=all=-c=1 ./...
  

  Use the same flags for go vet and go test — build flags are part of the cache key, so dropping them silently recompiles everything and OOMs again.

• "Out of host capacity" on pulumi up — by far the biggest obstacle here, so this is what actually worked (us-ashburn-1, Aug 2026):

  Upgrade to Pay-As-You-Go. That is the fix. Everything else is a coping strategy. On a free tenancy the first box failed 165 consecutive attempts across all three ADs over several hours, with no time-of-day pattern, and shrinking to 1 OCPU / 6 GB did not help either. It placed shortly after the PAYG upgrade. Free and PAYG requests draw from differently-contended pools, and a free tenancy in a busy region may simply never get A1.

  Expect a propagation delay. The first round of attempts after upgrading still failed; it placed roughly an hour later. Don't conclude the upgrade didn't work.

  PAYG costs nothing while you stay inside the always-free allowance — but it removes the guardrail that made overspending impossible, so set the budget config keys above at the same time.

  Secondary tactics, useful while you wait:

  1. Ask for less. Capacity is fragmented, so a host with 1 OCPU / 6 GB free is easier to find than one with 2 / 12, and A1.Flex resizes up later (see below). A smaller box is a foothold, not a dead end.

     pulumi config set ocpus 1
     pulumi config set memoryGb 6
     

  2. Retry persistently, cycling ADs. ./retry-up.sh <project-dir> does this unattended — e.g. ./retry-up.sh netbird. It is safe to leave running, since the network resources already exist and each retry only re-attempts the instance.
  3. Ask OCI directly whether capacity exists before burning attempts. This is read-only and free:

     oci compute compute-capacity-report create \
       --compartment-id <compartment-ocid> \
       --availability-domain "DZxh:US-ASHBURN-AD-3" \
       --shape-availabilities '[{"instanceShape":"VM.Standard.A1.Flex","instanceShapeConfig":{"ocpus":2,"memoryInGBs":12}}]'
     

     Note it answers at availability-domain granularity (available-count is normally null), and it's advisory — nothing reserves the capacity between the report and your pulumi up.

  Check oci iam availability-domain list first — many regions have only one AD, and the AD-1..3 numbering is scrambled per tenancy.

• Resizing up is safe, but slow — and only while the instance runs. Once you have a box, pulumi config set ocpus 2 && pulumi config set memoryGb 12 && pulumi up updates it in place: shape_config is an updatable field, and public IP, private IP, and volume attachments all survive. It reboots. Budget real time — a 1/6 → 2/12 resize took over 1,000 seconds here.

  pulumi preview first and confirm it says ~ update. If it ever says +-replace, stop: that would destroy your instance and put you back in the capacity queue. Protect(true) on the instance is there to make that failure loud rather than silent.

  Never stop the instance to resize it. A running instance holds its host slot; a stopped one must be re-placed on start, which is exactly where people lose the box permanently.

• cloud-init timing: Docker install runs on first boot and takes a minute or two after the instance shows as running. If docker isn't there yet, wait and re-check; see /var/log/cloud-init-output.log on the box to watch it.

• Locking down SSH: the default is 0.0.0.0/0 so you cannot lock yourself out before you know your IP. Once a box is reachable, set sshIngressCidr to <your-ip>/32 and pulumi up again — in each project separately. The two stacks hold independent copies, so narrowing one leaves the other open. The sites box drops public SSH entirely once it joins the overlay; the control plane keeps it as the recovery path of last resort.

• Use curl -4 to find your IP. Bare curl ifconfig.me returns whichever protocol curl connected over, and on an IPv6-capable machine that is often an IPv6 address. /32 is an IPv4 netmask (a single IPv6 host is /128), and the VCN is IPv4-only, so an IPv6 source CIDR is both wrong and rejected by OCI.

• Residential IPs rotate. If SSH stops working after an ISP lease change, re-run the sshIngressCidr config line and pulumi up. The security list updates in place in seconds — the instance is untouched.

• A box with a reserved IP has no internet while it boots. This one produced a box that came up looking fine and had installed nothing.

  An OCI internet gateway only routes for a VNIC that already holds a public IP, and there is no NAT gateway here. A reserved IP is a separate resource Pulumi attaches ~30–60s after the instance reports RUNNING, so a box created with EphemeralPublicIP: false is briefly ahead of its own networking. Every apt, curl, and snap call in early cloud-init fails with "Temporary failure resolving", and pulumi up reports success long before cloud-init status reports error.

  The fix is in internal/cloudinit: a bounded wait for DNS in bootcmd, not runcmd — package_update and package_upgrade run in the config stage, before runcmd, and need DNS just as much. It gives up after 5 minutes and continues, on the reasoning that a hung boot is worse than a failed install and the log will say which happened.

  Converting an existing box has no such problem: it is already booted, and the VNIC is rewritten under it. Greenfield is the hard case here, because a machine that has never booted cannot tell you its network isn't ready.

• Ephemeral vs reserved public IPs. An ephemeral IP is returned to Oracle when the instance stops — it survives reboots, so this is easy to miss until the day it isn't. Anything with DNS pointing at it should hold a reserved address.

  Converting is safe: create_vnic_details.assign_public_ip is updatable, so the VNIC changes in place rather than replacing the instance (verified on a live box — uptime was continuous across the change). Do it before DNS exists, because the address does change. A private IP may hold only one public IP, so a greenfield box should be created with EphemeralPublicIP: false rather than having an ephemeral removed later.

  There is no published charge for holding one — see Cost.

• DNS: what is and isn't dangerous. Worth stating precisely, because the intuitive fear here is misplaced and the real risks are elsewhere.

  Not a risk: managing a zone does not put its existing records under Pulumi's control. Zone and record are separate resources, and pulumi up leaves undeclared records alone — the standing complaint about the AWS provider is that it won't clean them up. Destroying a zone even fails with HostedZoneNotEmpty unless you set force_destroy.

  The real risks, in order:

  1. force_destroy = true on a zone resource. That is the flag that makes pulumi destroy take every record with it, including email. Don't set it on a zone carrying anything you did not create.
  2. Declaring a zone instead of importing it. Route 53 assigns new nameservers to every new zone. Declare a domain you already own without importing and you get a second, parallel zone; the registrar still delegates to the original, so the new one is inert and every record you "manage" goes nowhere. Silent, and confusing to debug — dig keeps returning the old answers, correctly.
  3. Getting an MX or SPF record wrong. Mail failures are delayed and quiet, unlike a web outage. This is the argument for reviewing DNS diffs carefully, and for keeping record values in plaintext so a diff is readable at all.

  internal/awsdns currently looks the zone up read-only and manages only individual records — the minimum footprint, appropriate while only one record is codified.

  AllowOverwrite is left at its default of false, so creating a record that already exists fails rather than silently taking ownership of it.

• Projects sharing a stack name share a config filename. Every oci project has a Pulumi.oci.yaml, every aws project a Pulumi.aws.yaml — the file is named for the stack, not the project, so it collides across projects in the same group. Two consequences:

  Do not use pulumi config copy --dest between projects. --dest names a stack, but the config file path is derived from the working directory plus the stack name, and nothing checks that those agree — project names are cloud-side identifiers with no filesystem mapping. Run from one project, it decrypts with the source stack's key, re-encrypts with the destination's, and writes back over the file it just read. Exit code 0, no output. The symptom is could not decrypt secret at index N on the source stack, plus no config file in the destination directory — which looks normal, since pulumi stack init alone never creates one.

  If it has already happened, the overwritten file is already encrypted under the destination's key. Rescue it rather than retyping: re-namespace it into the destination directory with sed, then git checkout -- the original. Renaming the config namespace does not break decryption — Pulumi Cloud encrypts with a per-stack key and the key name is not associated data.

  pulumi stack rm deletes the config file too, along with the stack. It is still in git; the working tree just loses it.

Cost

These stacks are designed to bill $0. Together they sit exactly at the always-free ceiling: 2 OCPU / 12 GB of Ampere A1 across both boxes, boot volumes well inside the 200 GB free allowance, and 10 TB/month of egress shared. There is no published charge for public IPv4: Oracle's release announcement for reserved public IPs says "There is no charge to use this feature."

Read that with its date attached — 26 January 2018, the feature's launch post rather than a current price list. The reference documentation is silent on IP pricing either way. AWS is the cautionary case: public IPv4 was free there for years too, until February 2024 introduced $0.005/hr whether attached or not, about $44/year for an idle address. The budget alarm exists for exactly this class of change.

Two things are worth knowing anyway.

Headroom is thin, and it is tenancy-wide. The monthly allowance is 1,500 OCPU-hours and 9,000 GB-hours for the whole tenancy, not per instance. A 31-day month is 744 hours, so two boxes at 1 OCPU / 6 GB consume 1,488 and 8,928 between them — 99.2% of both. The split changes nothing about the total; it buys isolation, not room.

The practical consequence: there is no headroom for a third Ampere instance, however small, and none for briefly resizing a box "just to test something". Both tip you into billing. Overage is cheap (cents, at $0.01/OCPU-hr + $0.0015/GB-hr), but it won't be $0 — this repo ran a few days of it while both boxes were temporarily oversized.

On PAYG, nothing stops you spending. A free account cannot create billable resources at all — the account type is the guardrail. Upgrading removes it: the same API call that used to fail now succeeds and invoices you. OCI has no spend cap, so the foundation project's budget is the best available control — it turns a silent bill into an email. Deploy it first and leave it alone.

What this would cost if it weren't free

Normalized to a 730-hour month; Oracle bills per second. us-ashburn-1 list prices, verified 2026-08-12.

Shape	family	per OCPU-hr	per GB-hr	at 2 OCPU / 12 GB
A1.Flex	Ampere Altra (ARM)	$0.010	$0.0015	$27.74
A2.Flex	AmpereOne (ARM)	$0.014	$0.0020	$37.96
A4	AmpereOne M (ARM)	$0.0138	$0.0027	$43.80
E4.Flex	AMD EPYC Milan	$0.025	$0.0015	$49.64
E5/E6.Flex	AMD EPYC Genoa/Turin	$0.030	$0.0020	$61.32

Plus $0.0425/GB-month for a Balanced (10 VPU) boot volume — $2.13 for 50 GB. Egress beyond 10 TB is $0.0085/GB. All-in for this exact box: ~$29.87/month.

Watch the OCPU units. On AMD/Intel one OCPU is a hyperthreaded core = 2 vCPU; on Ampere one OCPU is 1 physical core, no SMT. Comparing raw OCPU counts against a VPS provider's "vCores" understates OCI's x86 cost by 2×.

Versus OVH US (the likely migration target)

OVH US is Intel, unmetered traffic, daily backups included. "From" prices are promotional; the bold figure is the standing rate and the right planning number.

Spec	OVH US	from / standing	OCI E5 equiv.	OCI $/mo	premium
2 vCPU / 4 GB / 40 GB	VPS-1	$4.54 / $6.46	1 OCPU, 4 GB	$29.44	4.6×
4 vCPU / 8 GB / 75 GB	VPS-2	$8.50 / $9.99	2 OCPU, 8 GB	$58.67	5.9×
6 vCPU / 12 GB / 100 GB	VPS-3	$12.32 / $19.97	3 OCPU, 12 GB	$87.47	4.4×
8 vCPU / 24 GB / 200 GB	VPS-4	$23.37 / $36.98	4 OCPU, 24 GB	$131.14	3.5×

The takeaway: OCI's paid tier is not a fallback. At 3–6× OVH it is never the rational choice for this workload — the entire value is the $0 price point, so paying Oracle anything is the signal to leave rather than to upsize. A1 is Oracle's only price-competitive silicon; x86 on OCI costs roughly twice what ARM does for equivalent work.

Put positively: ~$10/month is the real cost of Oracle withdrawing the free tier. Cheap enough that the free tier is a nice-to-have, not a dependency.

Moving to OVH later ("go live")

The network + firewall concepts port directly. The plan is one Pulumi project that branches on a provider config key, rather than a project per provider — chosen deliberately so it forces the VPS concepts to be abstracted provider-agnostically.

The seam to hold: a shared, provider-agnostic spec (vCPU/RAM, CPU architecture, OS family and version, SSH key, ingress rules, cloud-init) with per-provider implementations behind it. Provider-specific knobs — compartmentOcid, availabilityDomain, VM.Standard.A1.Flex — stay inside the OCI implementation and must never leak into the shared spec, or it becomes an OCI interface with an OVH adapter bolted on. Pulumi.oci.example.yaml is already grouped along that seam.

Architecture is on that list for a concrete reason. Cheap ARM doesn't exist in the US market — Scaleway has no US region at all, and Hetzner's CAX ARM line is Germany/Finland only (its Ashburn and Hillsboro locations are AMD-only). So the realistic migration target is x86, and this codebase currently hardcodes ARM in two places that look unrelated: main.go filters images on aarch64 via the A1 shape, and cloudinit.go pins an arm64 apt line for the Docker repo. Both were written as OCI details; neither is. On an OVH stack both would be wrong on day one, in different files — and the cloud-init one wouldn't surface until first boot, long after pulumi up reported success.

That's the easiest kind of leak to miss, because right now everything works and nothing points at the assumption.

Worth building when the second provider is real, not before: with one implementation the abstraction is unvalidated and will come out OCI-shaped.