NetClawGoogle CloudAINetwork automation

A Risk in the Clouds: NetClaw on GCP

NetClaw on GCP, from one Border to a distributed Risk: milestones R01 to R09C

By John Capobianco and Claude.

Night-side globe with three glowing Border nodes in Montréal, Belgium and Singapore joined by teal arcs, a home-lab house linked to all three, and a lobster mascot in a hard hat holding a clipboard.
Three Borders, three regions, one Risk, with a home-lab Member attached to all of them.

On November 1 we had one NetClaw Border, a Mac on a desk and a roadmap with twenty items on it. By November 11 we had run three Borders in Montréal, Belgium and Singapore as a single Risk, with Members on a home lab answering through any of them. We had also run a version of that mesh inside OpenShell sandboxes, with DefenseClaw sitting in the model path. We built each piece, broke it on purpose, wrote down what broke by accident, and tore it all down again.

This post walks through those milestones, R01 to R09C of the roadmap behind issue #310, in order. For each one we cover what we set out to do, what we built, what surprised us, the evidence, and where the spec and pull request live. Every number here comes from a spec, a verification record or a merge commit in the public repository.

Here is the short version, before the long one.

It works. It is not production-ready.

You can run NetClaw Borders and Members in GCP today, on-prem, or as a mix of both. You can give a Risk high availability with more than one Border, and you can spread a Risk across regions. We have run cloud, hybrid, HA and three-region Risks live, against lab devices in CML, with real models and signed results.

There is no authentication in front of it. There is no sign-in, no user accounts, no SSO and no RBAC. Anyone who has a Border’s URL can open its HUD, use Chat and spend the model API key behind it. Keep the URLs private. Users, SSO and RBAC are roadmap phases R15 to R17.

Please experiment. Use a throwaway GCP project, set a budget alert before you create anything, and tear it all down when you are done.

In this article

The words we use

A Risk is the whole deployment. The name is real: a risk is the collective noun for a group of lobsters, and NetClaw is a lobster. Instead of one NetClaw carrying every skill, a Risk is a group of focused claws behind a single front door. A classic standalone NetClaw is a Risk of one.

A Border fronts a Risk. It is the coordination and policy boundary. It runs the owner service, enrolls Members, presents approvals, dispatches work, collects results and decides what is exported to other organizations. It has its own identity. From R08 onward, several Borders can front one Risk and take part in its HA. A Border does not need to run a model at all, and in R03 ours didn’t.

A Member attaches to a Border and provides a capability: pyATS and SSH access to devices, the CML API, NetBox, a hosted model or a local Ollama model. Each Member has its own identity, runtime, model profile and tools, and holds its own integration credentials. Members dial out to their Borders. They never listen for each other, and member-to-member connections are refused.

iN2N is the internal channel: an enrolled Member’s link to its own organization’s Border. eN2N, carried by the NCFED protocol, is external federation between the Borders of independently administered organizations, with bilateral consent and restricted exports. An external Border gets no vote and no path to your Members. NCFED is written up as an IETF Internet-Draft, draft-capobianco-ncfed-00. R08 adds a third relationship, the cluster link between Borders of the same Risk, which carries HA membership.

Diagram of a Risk: a Border hexagon above five Member tiles (pyATS/SSH, CML, NetBox, Hosted model, Ollama) connected upward by iN2N outbound arrows, red X marks between Members, and an eN2N/NCFED link to another organization’s Border.
A Risk: one Border in front, Members dialing out over iN2N, eN2N/NCFED to other organizations. Members never talk to each other.

The roadmap at a glance

Issue #310 asked for multi-user NetClaw: sign-in, SSO, RBAC, personal memory and a shared knowledge base. The roadmap that answers it starts somewhere else, with transport, execution safety and cloud placement. Users and roles are only as trustworthy as the mesh they ride on. Each milestone is a full spec-driven cycle (specify, clarify, plan, tasks, analyze, implement) with its own verification record and a human acceptance gate.

ID Milestone Phase Status
R01 Direct cloud transport contract and TLS identity fixes prerequisite Accepted (spec 168)
R02 Execution parity on NetClaw production mode prerequisite Accepted (spec 171)
R03 GCP Border with on-prem members (pyATS/SSH, CML) 1 Accepted (spec 172)
R04 GCP cloud member: NetBox and subnet, hosted model 2 Accepted (spec 173)
R04a Cloud Border client integration: VS Code and Mobile client gate Accepted (spec 174)
R04b Cloud HUD and Mobile QR enrollment client follow-up Accepted (spec 175)
R05 Ollama members: cloud IPv6 and on-prem IPv4 3 Accepted (spec 176)
R06 External Border over direct eN2N/NCFED eN2N Accepted (spec 177)
R07 Single-region recovery and tunnel lifecycle gate gate Accepted (spec 178)
R08 Border HA authority and protected quorum transport prereq. for 4 Accepted with limits (spec 179)
R09 GCP multi-region: three meshed Borders 4 Live evidence, torn down (spec 182)
R09A GCP Borders and HA mesh in OpenShell + DefenseClaw 4 Live evidence, torn down three times (spec 185)
R09B GCP completeness survey 4 Complete, reference (spec 192, not pushed)
R09C GCP deployment docs, setup and guides refresh closes GCP In progress (spec 193, not pushed)
R10 Azure single region 5 Planned
R11 Azure multi-region 5 Planned
R12 AWS single region 6 Planned
R13 AWS multi-region 6 Planned
R14 Multi-cloud: nine Borders, three providers 7 Planned
R15 Multi-user foundation 8 Planned
R16 Bring-your-own SSO 9 Planned
R17 RBAC, delegated authority, approvals and audit 10 Planned
R18 Personal conversations and memory 11 Planned
R19 Shared network knowledge base 11 Planned
R20 Issue #310 closure: isolation suite and UX 11 Planned

This article covers R01 through R09C. Everything in it is read-only work: show commands, NetBox reads and subnet calculations. That is a rule, not a coincidence. Global invariant G11 holds device and system writes back until R17 brings authority, approval and audit.

Transit-map roadmap: R01 and R02 merge into a coral line through R03 to R09C with a We are here pin at R09C, then gray branches to Azure R10–R11, AWS R12–R13, Multi-cloud R14, Users·SSO·RBAC R15–R17 and Memory·Knowledge base·Closure R18–R20.
Where we are. R04a and R04b, the client-integration steps, sit between R04 and R05.

Foundations

R01: direct cloud transport and TLS identity

Goal. Give a cloud Border a stable, verifiable place on the network. Members at home need to reach it and other organizations need to federate with it, and nothing in the middle should read or rewrite the traffic.

What we built. R01 started life as the ngrok transport contract, and the spec directory still carries that name. On the first day we confirmed that our ngrok account was on the Free plan with no reserved TCP addresses, and we pivoted. A cloud Border now gets a stable provider address and two separate TCP ports: one for iN2N (its own Members) and one for eN2N/NCFED (other organizations’ Borders). TLS 1.3 terminates in NetClaw itself, never in a proxy or a tunnel. Members stay outbound-only. ngrok remains a supported option for Borders stuck behind NAT, but nothing requires it. The registry records which transport each endpoint uses. The routing address, the verified TLS name and the enrolled application identity stay three separate things.

What surprised us. A new test caught accepted direct channels still carrying the legacy ngrok transport label. It was a small bug, but exactly the kind of mislabel that makes later evidence lie. We also moved the pilot Border to a new public address mid-stage. Stale updates to the address record were refused, there was no silent fallback to the old address, and both peers reconnected with the same enrolled key.

Evidence. 40 of 40 tasks complete. A real Mac Member enrolled over iN2N to a GCP Border in Montréal, and an independent external Border was admitted over TLS NCFED with bilateral consent. The negative suite refused wrong TLS names and CAs, altered exporters, forged possession, unknown Members, expired and replayed tokens, and revoked channels. The cloud iN2N port rejected cleartext, PROXY headers and TLS 1.2. The NCFED port rejected raw BGP, HTTP and PROXY. Both Member entry points had zero TCP listeners. Live ngrok is still unqualified.

One more thing belongs here, because it explains the warning at the top. We chose to publish the R01 lab’s full HUD, Chat, Canvas and RAG included, without a login. That was the first open door, and every later stage kept it open.

Spec and PR. specs/168-ngrok-transport-contract · PR #319, merged November 2.

Cloud Border with separate iN2N and eN2N/NCFED ports, a padlock inside labeled TLS 1.3 ends here, an on-prem Member laptop sending an outbound-only arrow to the iN2N port, an External Border on the eN2N port, and a dashed optional ngrok (NAT) path.
R01: separate iN2N and eN2N/NCFED ports, TLS ending inside NetClaw, Members dialing out. ngrok stays optional.

R02: execution parity on production mode

Goal. Make a delegated operation as safe as a local one. The model may propose, but only an exact, approved, one-time operation executes, and only on the Member that owns it.

What we built. Principal-bound execution. The owner reviews the operation, the Member, the model profile, the device, the tool and the exact arguments, then approves one bounded read. The broker validates exactly what was approved, and the Member enforces it. Approvals are exact, scoped, expiring and single-use. If the device, tool, arguments, Member, model, principal, expiry or signature changes, the Member refuses. Every request now carries a principal, and until R15 the only principal is the owner.

What surprised us. The first implementation pass passed its tests and still failed review. The pyATS/Unicon connection performed an extra, implicit read when it connected. One approved show version has to mean one command, so we treated that as a finding, not a footnote. The second pass removed it.

Evidence. 36 of 36 tasks, accepted November 2. Both the OpenClaw and Hermes runtimes ran one approved show version with zero setup commands. Each produced signed results and acknowledgements, and the IOS-XE 17.18.2 output was checked independently. 311 protected Python tests back it, plus the HUD and boundary suites.

Spec and PR. specs/171-execution-parity, merged to main together with R03 in PR #323.

Five-step pipeline: Model proposes, Owner approves the exact read, Broker validates, Member enforces, Signed result + ACK; a terminal showing show version; a changed request bouncing off the Member enforces step marked refused.
R02: the model proposes, the owner approves one exact read, the broker validates and the Member enforces. Change anything and the Member refuses.

Phases 1 to 3: one region

R03: a GCP Border with on-prem Members

Goal. Put a Border in GCP and keep the devices, credentials and models at home.

What we built. A fresh, control-only Border in GCP Montréal, with no model, no tools and no device credentials. A fresh Mac Member dialed out to it over iN2N. The Member ran OpenClaw with Claude Sonnet 4.6 and a pinned pyATS/Unicon adapter against four IOS-XE devices in a CML lab: R1, R2, SW1 and SW2. Each read was reviewed and approved separately in the cloud HUD. The Border never reached into the LAN.

What surprised us. Mostly that the boring things held. We injected failures on purpose: a lost submit reply after durable admission on R1, and a lost acknowledgement reply on R2. We also restarted the service and the Member and reloaded the browser. Afterwards there were still four device commands and eight model calls, the same as before. Recovery granted no new approval and executed nothing twice.

Evidence. 36 of 36 tasks complete. Each read returned parsed IOS 17.18.2 from exactly one SSH show version, with zero setup, enable or configuration commands. The cloud Border made zero protected model calls. Afterwards we removed all 13 cloud resources, the DNS records and the temporary SSH key, leaving no billable residue.

Spec and PR. specs/172-gcp-hybrid-border · PR #323.

Control-only Border in a cloud with crossed-out brain and wrench icons and an Owner approvals window; below the Internet line, an on-prem Member laptop sends iN2N outbound up and SSH show version to R1, R2, SW1 and SW2 in a CML lab.
R03: a control-only Border in GCP. The on-prem Member holds the model, the tools and the device credentials.

R04: a cloud Member, and the clients that caught up

Goal. Move a cloud-friendly capability into GCP as a Member, with its own secrets, behind a hosted model.

What we built. A GCP Member running the existing NetBox MCP and skills behind a guarded hosted model, with its integration credentials kept at the Member. The Border still holds no integration secrets. Three separately approved workloads ran through it: a NetBox device read, a prefix read and a subnet calculation.

What surprised us. Two earlier approved device attempts ended with unknown outcomes before any model or tool call. We kept both, with their original approvals, and did not replay them. Our first cost estimate used a zero-price IPv4 tier by mistake, and we caught it before anything was created.

Then came the client gap. The backend passed, but VS Code could not connect to the cloud Border, and that gap became R04a. We rejected a diagnostic landing page and an operations-only client as not good enough. Connecting by name and URL now opens the normal NetClaw interface (Overview, Chat, Canvas, RAG and the rest) backed by the cloud Border. R04b added Mobile QR enrollment from the HUD and from VS Code. Neither added user authentication. From R04a onward, every milestone has to assess VS Code and Mobile separately.

Evidence. R04 closed 33 of 33 tasks. Each workload made one attempt, one MCP call and two guarded model calls, and produced a verified signed result and a durable acknowledgement. Independent arithmetic confirmed that 192.0.2.0/26 has 62 usable hosts. A credential scan across 365 exported files found zero matches. R04a closed 22 of 22 tasks and R04b closed 30 of 30. Budget alerts at 50, 90 and 100 percent were in place; alerts are not hard caps.

Specs and PRs. specs/173-gcp-cloud-member · PR #324; specs/174-cloud-client-integration · PR #327; specs/175-cloud-mobile-enrollment · PR #328, PR #329.

Cloud containing a Border marked No integration secrets and a Cloud Member with NetBox MCP, Hosted model, Guard and a Secrets stay here vault; Device, Prefix and Subnet result cards flow back to the Border; an external NetBox database.
R04: the cloud Member holds its own secrets. The Border holds none.

R05: Ollama Members over IPv6 and IPv4

Goal. Run local-model Members, one in the cloud and one at home, under the same guard and approval rules.

What we built. Two guarded Ollama Members: a GCP AMD64 Member reached over IPv6 and a local ARM64 Member reached over IPv4, each running a native subnet-calculator MCP. We reused the patterns, tests and lessons of an earlier prototype, Thermidor, without importing its runtime or its Tailscale transport.

What surprised us. Mobile found a real case the browser had not: a review expiring while the provider outcome was uncertain. We implemented and verified a profile-rebind recovery step for it. We also relearned a cleanup lesson that comes back in every cloud stage: stopping a VM is not tearing it down, because disks and IP addresses keep billing.

Evidence. 33 of 33 tasks, accepted November 6. Each Member completed two model turns and one tool call with verified arithmetic, a signed result and a durable acknowledgement. A combined browser request with two separate approvals returned both results and survived a reload. Matching journeys passed in VS Code and on the iOS simulator.

Spec and PR. specs/176-ollama-members · PR #331.

A Border above two guarded Member tiles: an Ollama member in the cloud on AMD64 tagged IPv6 and an Ollama member on-prem on ARM64 tagged IPv4, each with a Subnet calculator.
R05: two guarded Ollama Members, one in GCP over IPv6 and one on-prem over IPv4.

eN2N: talking to other organizations

R06: external Borders over eN2N/NCFED

Goal. Prove that cloud Borders federate with independently run organizations the same way local Borders already did.

What we built. Three independent organizations, two in GCP and one local, in a full triangle. All six directions passed inventory exchange, contextual peer chat, native subnet-tool execution, asynchronous skill delegation and retrieval of the original result. Like R01, this ran over direct eN2N, not ngrok. The local Border sat behind NAT and simply opened outbound sessions to both clouds.

What surprised us. The limit we have to state honestly: the two cloud organizations shared one GCP project, with separate VMs, networks, service identities and NetClaw state. That proves separate NetClaw authority, not separate cloud tenants. Separately, the HUD’s four-request admission could return HTTP 429 during initial polling. Refreshing recovers the read without replaying any work.

Evidence. 35 of 35 tasks. We correlated 16 subnet MCP calls, 3 permitted skill-file reads and 38 provider responses. Refusal tests covered identity, consent, grants, budgets, internal-member access and transit. Browser, Mac VS Code and iPhone simulator journeys passed, and loss and restart recovery returned original outcomes without another execution.

Spec and PR. specs/177-external-en2n-federation · PR #333.

Three dashed organization boundaries, Org A (cloud), Org B (cloud) and Org C (local), each with a Border and two Members, connected Border to Border by six arrows with Consent handshakes and the label eN2N / NCFED.
R06: three independent organizations, two in the cloud and one local, in all six directions over eN2N/NCFED with bilateral consent.

The gate

R07: recover safely before adding HA

Goal. Before a second Border exists, prove that one Border and its Members survive restarts, moved endpoints, revoked authority, lost replies, duplicates and cancellation, without losing identity or repeating work.

What we built. Authority is now checked across awaited work and response delivery. Revocation is persisted before channels close. Clients retain their send markers, and endpoint changes are verified before client bindings are replaced. Work that was already dispatched when a Border process died becomes an unknown item to reconcile, and recovery never replays it. Despite the tunnel lifecycle in its title, R07 ran locally and created no cloud resources.

What surprised us. One VS Code run hit a shared-storage timestamp incident whose impact on content we could not determine. It stays in the record as unknown, which is the point of the whole milestone: an honest unknown beats a confident guess.

Evidence. 31 of 31 tasks, accepted November 7. 1,756 federation tests passed (with 5 existing skips), alongside the HUD, editor and Flutter suites. The accepted investigation targets are 120 seconds to make a failure visible, 180 seconds for automatic reattachment, 30 seconds for explicit recovery and 130 seconds to load cold history. These are targets, not worst-case SLAs.

Spec and PR. specs/178-recovery-gate · PR #334.

Four fault cards (Restart, Lost reply, Duplicate, Revoked) feeding an Original record ledger, which leads to Same identity and Unknown → reconcile, with a struck-through replay icon labeled No replay.
R07: whatever breaks, the original record survives. Uncertain work becomes reconciliation, never a replay.

HA

R08: one execution authority

Goal. Let several Borders front one Risk without ever letting two of them authorize the same work.

What we built. Exactly one current execution authority per Risk, backed by three private etcd 3.6.16 voters. The voters use separate peer and client CAs, mutual TLS and RBAC. Followers forward admissions to the leader, and Members enforce the fence, so a stale leader is refused. Losing the majority blocks new protected admission, and uncertainty never permits an automatic second execution.

What surprised us. An early fixture failed because the application certificate did not establish a TLS DNS identity. The cluster transport now uses a separate transport CA. Our own process also failed once. The original ten-operation budget for real runs was spent with three slots stuck in unknown, because the implementing agent made ordering mistakes. Nothing was reset. The cap was lifted afterwards, and every deliberate unknown after that was limited to one per run.

Evidence. All 18 rows of the fault matrix have real evidence, using an actual guard, a small local model (qwen3:0.6b) and the native subnet tool. The rows cover leader loss, partitions, stale leaders, quorum loss, clock uncertainty, unknown consensus commits, lost results and acknowledgements, full restart, retired nodes, fenced restore, capacity, trust negatives, and the HUD, VS Code and iOS journeys.

In fixtures, leader replacement took 15.007 seconds. R08 was accepted with eleven written limits on November 8. For example, Border-to-Border NCFED cuts were not claimed until R09 performed one on a real network.

Spec and PR. specs/179-border-ha-authority, which holds the three-voter etcd design · PR #339.

Three Border servers each with an etcd voter: the top one labeled Leader: execution authority, two Followers forwarding to it over mTLS; below, a Member behind a gate labeled Member enforces the fence; an inset with two grayed-out Borders and a stop sign labeled No quorum = no new protected work.
R08: one execution authority backed by three etcd voters over mTLS. Members enforce the fence. Without quorum, no new protected work.

Phase 4: going global

R09: three meshed Borders in three regions

Goal. One Risk across three GCP regions, with Members attached to every Border, where losing a region costs service, not correctness.

What we built. Three e2-medium Borders in northamerica-northeast1 (Montréal), europe-west1 (Belgium) and asia-southeast1 (Singapore). Each had its own DNS name, its own Let’s Encrypt certificate and its own etcd voter. Members on a Mac attached outbound to all three. VS Code accepted all three Borders by DNS name and showed each region and the current leader. The HUD showed which Border held execution authority and which were forwarding to it.

What broke, in order. We measured before we built. Median round trips between the Border VMs were about 106 ms from Montréal to Belgium and 239 ms from Montréal to Singapore. Belgium to Singapore measured about 330 ms, nearly double our estimate, because that path routes through the US. We stretched etcd’s timers to match: a 500 ms heartbeat and a 5-second election timeout.

World map with Borders in Montréal (leader), Belgium and Singapore; links labeled ~106 ms, ~239 ms and ~330 ms; a home Member with three attachments to all three Borders.
R09: measured median round trips between Border VMs (TCP connect, 60 samples per direction). Belgium–Singapore is slower than the map suggests.

Then operations started stranding. The Member’s model handoff accepts only a narrow band of elapsed admission time: more than 250 ms and at most 3,000 ms must remain in the signed admission window. With admission_ms at 6,000, fast handoffs were refused; at 3,000, slow ones were. Observed elapsed time was about 3 seconds. At 4,500, all three entry Borders completed real operations with a Montréal leader. The band is 2.75 seconds wide whatever the setting, so leader placement changes admission reliability. Decoupling the two budgets is a product change we still have to make.

Timing chart from 0 to 6000 elapsed ms: admission_ms 6000 accepts roughly 3000 to 5750 (fails), admission_ms 3000 accepts 0 to 2750 (fails), admission_ms 4500 accepts 1500 to 4250 (passes), with an amber line at about 3 seconds observed.
R09: the model handoff accepted a 2.75-second band of elapsed admission time. At 4,500 ms that band covered the roughly 3 seconds we observed with a Montréal leader.

Leader election does not prefer a nearby region. When we stopped the Montréal Border, leadership moved to Belgium in about 29 seconds in one run, and to Singapore in about 20 seconds in a later one. In that later run we stepped Singapore down so Belgium led, and an operation entering via Singapore then left a Member latched as uncertain. We did not reset it. We retired it and enrolled a replacement. Next we cut Montréal off the cluster plane. Belgium and Singapore elected Belgium in about 30 seconds, and Montréal rejoined as a follower without reclaiming authority. However, a two-of-three quorum spanning the two far regions could not admit work inside the window. That is a degraded mode, not availability.

We also cut the Border-to-Border NCFED link on a real network, closing one of R08’s limits. Authority stayed where it was, and an operation entering through the cut Border was refused with HTTP 503 before anything reached the Member. The NetBox hosted Member took four live attempts, each one peeling back a layer: a guard port mismatch; a 13 KB request limit against an 18.6 KB NetBox skill (raised to 128 KiB on the hosted path only); a spent inference fence; and a guard configuration derived from the Ollama profile, which DefenseClaw rightly blocked.

Every one of those failures came back as a signed, acknowledged refusal, and the other Members were unaffected. The far follower also timed out while fetching the Member catalog. We now ask Members concurrently and raised the forward timeout from 5 to 15 seconds.

Evidence. Three Member kinds completed real operations through Montréal, Belgium and Singapore, each with a durable acknowledgement: a subnet calculation on a local qwen3:0.6b model, a NetBox device read on hosted Sonnet 4.6, and a pyATS show version on hosted Sonnet 4.6. A Chat turn to Sonnet 4.6 through the Montréal HUD completed in 7.4 seconds. The full workspace on these Borders ran as an open lab, so anyone with a Border URL could use Chat and spend the provider key on that VM. Teardown deleted the VMs, disks, static IPs, firewall rules, network and DNS records by exact name, then verified the project was empty.

Spec and PRs. specs/182-multi-region-ha-borders (see live-evidence.md, measurements.md and cost-model.md) · PR #340, merged. Two contributor PRs from @byrn-baker landed alongside it. PR #341 added Member scoping and brought spec 183-member-mcp-mappings with it, which is why the R09A spec was renumbered from 183 to 185. PR #342 fixed the pyATS Member fixture that was failing the n2n contract suite on main.

R09A: Borders inside OpenShell and DefenseClaw

Goal. Run the Border role, the HUD ingress and the model guard inside NVIDIA OpenShell sandboxes with Cisco DefenseClaw guardrails, without weakening any identity, authority or admission guarantee from R08 and R09. R09 had no sandbox on any Border, and its chat path held the provider key with no inspection. Sandboxing does not add sign-in; the HUD is still open.

What we built. Two Border VMs and one etcd witness in Montréal. Two Borders in one region was an accepted validation topology. Each Border ran inside an OpenShell 0.1.7 sandbox with DefenseClaw 1.0.4 and OpenClaw 2026.7.1-2. OpenShell injects the Anthropic key at egress, so the sandbox environment only ever holds a placeholder, and the network policy lets only the DefenseClaw binary reach the provider. A HUD posture endpoint and a VS Code NetClaw: Show Node Posture command report the guard’s mode, its state and its last block.

A Border VM containing an OpenShell sandbox with the Border and an Agent; the Agent’s path to the Model API passes through a DefenseClaw guard, with the key injected at egress and only a placeholder inside; blocked arrows for Metadata server, Unapproved egress and Member-to-member; an inset showing 2 Borders + etcd witness, one region; a tag reading ~17–19 ms per sandbox hop.
R09A: the Border role, HUD ingress and DefenseClaw inside an OpenShell sandbox. The sandbox only ever sees a placeholder for the model key.

Evidence. Every check below was observed live. A benign chat turn passed through DefenseClaw on both nodes. A prompt-extraction attempt and a blocked exec tool call were refused and audited. A malicious skill dropped into the skills folder was quarantined before it could be enabled. With the guard stopped, the turn failed and OpenShell logged no new connection to the provider.

The isolation suite refused 9 of 9 cases on both nodes: out-of-policy file reads, writes to frozen source and policy, the metadata server, unapproved egress, provider-credential reads and member-to-member connections.

A stopped leader was replaced in about 13 seconds. Isolating a node moved authority in about 15 seconds, and the healed node did not reclaim it. Stopping the witness changed nothing.

Through the sandbox, a request took a median of 17.4 to 19.2 ms, against 0.7 ms on the plain host. That is under half a percent of the 4,500 ms admission window. In a third rebuild, we enrolled a hosted subnet-calculator Member and ran the same operation through the leader and then through the follower. It completed both times with a durable acknowledgement.

What surprised us. OpenShell has opinions. The protocol must be rest, not https. OpenShell refuses IP literals, so etcd endpoints need names. Our firewall rule also missed etcd’s client port, 12379. The runner hook we used to start pieces in the lab is a lab-only backdoor and must not ship. The node guard configuration carries three documented deviations from the qualified one, including forward_custom_headers: true, without which Anthropic returned a 400.

And one open item we cannot explain yet. During the leader upgrade, the surviving follower did not take leadership before the old leader returned. The epoch jumped from 15 to 17 with no leader observed on the follower. The old leader came back as leader with no split brain, but the timing is undiagnosed.

Cost and teardown. Each window ran at about $0.20 an hour (estimate). The first window ran about five hours and cost roughly $1 of compute plus a few cents of egress. We built the stage and tore it down three times. Each time we deleted by exact name and checked that a listing by label and name prefix came back empty.

Spec and PR. specs/185-openshell-defenseclaw-gcp (see verification.md and cost-model.md) · brief · PR #345, merge commit eb9cea8b.

R09B: the GCP completeness survey

Goal. Decide which other GCP capabilities would make the deployment complete, before Azure and AWS repeat the pattern. The brief lists the candidates: load balancing and health-checked endpoints; edge protection (Cloud Armor, rate limiting); identity-aware admin access and workload identity; Secret Manager and KMS; Cloud NAT and private egress with allow-lists; firewall policy and VPC controls; managed instance groups; logging, monitoring, alerting and budgets; certificate and DNS automation; and the artifact and image supply chain.

For each candidate, the survey records the problem it solves, its cost, how it interacts with the HA authority and the admission-timing findings, and an adopt, defer or reject decision. The survey builds nothing itself. It is a reference that R10 (Azure) and R12 (AWS) can mirror.

Status. Complete and kept as a reference in specs/192-gcp-completeness-survey/survey.md, which has not been pushed yet.

Brief. R09B-gcp-completeness-survey.md.

A three-column board labeled Adopt, Defer and Reject with blank cards, and a thoughtful lobster mascot holding cards with icons for a load balancer, shield, key, log scroll, ID badge and network graph.
R09B: deciding what else GCP should contribute before the pattern repeats on Azure and AWS.

R09C: making it something you can run

Goal. Close out GCP by turning what we did by hand into something anyone can repeat: the installer, a TUI, the VS Code flow and a NetClaw in GCP guide for deploying Borders and Members to GCP. Every one of them carries the no-authentication warning.

Status. This is spec 193 on branch 193-gcp-docs-setup-refresh. It has not been pushed, so treat this section as a preview. Until it lands, the GCP drivers on main are the ones we wrote for our own runs, scripts/ha-gcp-multiregion.py (R09) and scripts/ha-r09a-deploy.sh (R09A). Their defaults are our values, so read them before you run them.

Three panels, a terminal TUI titled NetClaw in GCP, an editor window with a Connect button, and an open guidebook, under an amber banner reading No sign-in yet · Keep URLs private.
R09C: the installer, TUI, VS Code flow and guide, with the same warning everywhere.

What you can do today

Here are the shapes we know work and where we proved each one.

Topology What it looks like Where we proved it
GCP only A Border and its Members all in GCP. R04 (cloud NetBox Member), R05 (cloud Ollama Member)
GCP Border + on-prem Members A cloud front door. Devices, credentials and models stay at home, and Members dial out. R03, R05, R09
On-prem Border + GCP Members The Border stays at home and Members run in GCP. Not qualified in R01–R09A
HA Risk Two or three Borders sharing one execution authority over etcd, in one region. R08 (fixtures and real runs), R09A (two Borders + witness)
Distributed Risk Three Borders in three regions, with Members attached to all of them. R09
Five panels: GCP only; GCP Border + on-prem members; On-prem Border + GCP members; HA Risk with three Borders in one region; Distributed Risk with three Borders on a globe.
Five topologies. The on-prem Border with GCP Members is not yet qualified in any spec.

Choose by what has to stay where. Device access, and anything holding SSH keys, tends to stay on-prem and dial out. NetBox, hosted models and calculators move to the cloud happily. HA buys you a front door that survives a lost Border, but read R09 before you spread Borders across continents: the farther apart the quorum, the less it can admit. Our measurements put the leader next to its nearest follower (Montréal or Belgium in our runs), and never in the region whose nearest peer is 239 ms away.

Rough cost (estimates)

Shape Estimate Source
Two Borders (e2-standard-2) + an etcd witness, one region About $0.20 per hour; about $1.80 for a six-hour window including egress; about $147 a month if forgotten Spec 185 cost-model.md
Three Borders (e2-medium) in three regions, plus a NetBox VM, egress and snapshots About $155 a month (±15%) Spec 182 cost-model.md

These are estimates in USD at on-demand list prices, before tax. Your model provider bills API usage separately, and it is not included above. Static IPv4 addresses bill whether or not a VM is running.

Set the budget alert first. Create a budget on the throwaway project before you create anything else. R09’s plan set alerts at 50, 80 and 100 percent of a $300 monthly ceiling; R04 used 50, 90 and 100 percent. Alerts notify you; they do not stop spending. Pair them with a teardown date.

How to tear down

Teardown is a step, not an afterthought, and every stage in this post ended with one. Delete by exact name, from a list you have read: the VMs and their disks, static addresses, firewall rules, subnets and the network, plus the DNS records you created with your DNS provider. Our R09 driver only deletes after you pass the SHA-256 of the list it printed. The R09A driver lists first and refuses to delete anything without its label or name prefix. Stopping a VM is not teardown, because disks and addresses keep billing. Afterwards, list again by label and prefix and confirm the result is empty. Rotate the model API key you placed on the Borders, and delete the project if it was a throwaway.

A loop of Deploy, Validate, Export evidence, Tear down and Verify empty around a lobster sweeping up server and disk icons, with an amber bell labeled Budget alert.
Every stage ended the same way: export the evidence, tear down, and check that the project is empty.

What is not here yet

The ground floor works. The floors that make it safe to share are still being built.

Users, SSO and RBAC (R15 to R17). R15 adds the multi-user foundation: server-side identity and isolated, persistent sessions for Chat and Canvas, with nothing trusted from browser storage. R16 adds bring-your-own SSO over OIDC, with Entra ID, Google, GitHub and generic providers. R17 adds RBAC from identity-provider groups, enforced server-side for MCP, SSH, API and workflows, together with delegated authority, approvals for high-impact operations and full audit. Multi-user stays feature-flagged off in releases until R17 lands, so no release ships users without authorization and audit.

Personal memory and shared knowledge (R18 to R20). R18 lets each person retrieve, inspect, correct and delete their own memory. R19 adds a shared network knowledge base: publishing with confirmation, review, provenance, freshness, deduplication and search, and checking the current environment before an old fix is reused. R20 closes issue #310 with an isolation suite and the UX for sign-in, the current user, personal versus shared, and role-aware visibility.

Until then, there is one principal (the owner), workloads stay read-only, and there is no sign-in.

Next: Azure, AWS and multi-cloud. Azure single region (R10) and multi-region (R11) come next, then AWS single region (R12, which may run beside R10) and multi-region (R13). R14 then stitches them into one Risk of nine Borders across three providers, with 36 Border pairs. Each provider is qualified first as its own three-Border cluster, because consensus clusters are rebuilt by joining nodes, not merged in place.

A building whose finished ground floor reads Borders · Members · HA, with scaffolded upper floors labeled Users, SSO, RBAC, Personal memory, Knowledge base and Isolation suite, and an open, unlocked front door beside a sign reading No sign-in yet.
The ground floor works. The floors with users, SSO and RBAC are still being built.

Try it, and tell us what breaks

If you run networks and wonder whether this belongs in your toolbox, the best way to find out is to break it yourself. Here is the loop we used. Create a throwaway GCP project and set the budget alert first. Deploy a Border and a Member or two. Give it HA if you’re curious, and spread it across regions if you’re very curious. Keep the URLs to yourself. Run read-only work against a lab and watch what the evidence says. Then tear it all down and check that the project is empty.

Then tell us what broke, on issue #310 or in a new issue. Every milestone above got better each time something failed in a way we had not predicted.

One more time: it works, and it is not production-ready.

You can deploy NetClaw Borders and Members in GCP, on-prem or as a mix, with HA or across regions. There is no authentication in front of any of it: anyone with a Border URL can use its HUD, its Chat and the model key behind it. Keep URLs private, experiment in a throwaway project with a budget alert, and tear it down when you are done. Users, SSO and RBAC arrive in R15 to R17.

A lobster in a lab coat and goggles at a workbench with a glass jar holding a tiny cloud with three glowing nodes, a kitchen timer and a sticky note reading Tear it down.
Experiment, measure, tear it down.

Sources

← More field notes