ZopNight's Kubernetes Fixes Outweigh the New Feature

ZopNight’s Kubernetes Fixes Outweigh the New Feature

ZopNight’s newest release notes open with a new feature: you can now migrate a database from one GCP Cloud SQL instance to another without leaving ZopDay, the Kubernetes platform ZopNight ships alongside. Kubernetes is the open-source system that schedules and runs containerized workloads across a cluster of machines, and it’s what ZopDay provisions, configures, and deploys onto across AWS, GKE, and AKS. The migration feature gets two screenshots at the top of the release notes.

Read past the fold and the shape of the release changes. The technical details section carries nine fixes, each with a reproduced failure, a root cause, and a before/after number. That’s the actual weight of this release: a feature ledger with one entry, and a reliability ledger with nine.

The distinction matters because the two ledgers get evaluated differently. A feature ledger tells you what a tool can now do. A reliability ledger tells you what was silently going wrong before, and for how long. Teams picking a governance platform should read both, not just the one with screenshots.

The Feature Ledger: One Migration Flow, GCP Only

The Cloud SQL migration flow has three steps. You pick a source database and a compatible target in a Migrate modal, which filters out incompatible instances and flags missing permissions with a fix link instead of failing silently later. You watch progress in a Migrations panel. When it finishes, the new database shows up on its own.

The safety property is specific, not implied: the source instance is never touched, and credentials are never stored by ZopNight during the copy. That’s the claim a migration feature has to earn, because the failure mode of a bad migration tool is a corrupted or exposed source, not just a slow copy.

The scope is also specific. This ships for GCP Cloud SQL only. AWS and Azure will plug into the same provider-neutral migration engine later, but they aren’t there yet. A live end-to-end test moved two real tables in 3 minutes 14 seconds, with source and target MD5-identical afterward and the source database untouched.

This works when the source and target versions are compatible and the calling account already holds the needed permissions. It breaks, by design, when the target would be a version downgrade from the source: that check runs as a preflight at the first migration step rather than at creation time, so an incompatible pair still creates a migration record before failing it, instead of silently succeeding on data it should have refused to copy. The same two-phase validation also blocks a second import from starting against a target that’s already mid-migration, since a concurrent import against the same destination is exactly the kind of race a migration tool has to close before it earns trust for anything bigger than one database.

The Namespace Pool That Wasn’t Supposed to Exist

The most severe fix in this release started as a 409 storm. Creating a project on the shared ZopCloud VM could fail 26 times in two minutes for a single user, because every tenant on that VM was competing for one shared pool of namespace names.

The root cause is a mismatch between what the check assumed and what the deploy path actually needed. Namespace uniqueness matters on a shared Kubernetes cluster, where namespaces are a real, shared resource two tenants could collide on. VMs don’t use namespaces at all. The uniqueness check ran anyway, so tenants were blocked by a resource they never touched.

The fix removes namespaces from the VM deploy path entirely rather than trying to scope the check more narrowly. That’s a smaller surface to get wrong than adding another conditional to an existing check. It was verified with five organizations creating a project with the same name on one shared VM, with zero conflicts.

Deploys Now Take a Number

Three deploy-reliability problems shipped together in this release: deploys overwhelming the server when a team ships at the same time, a second deploy trigger leaving an older build running instead of the newest one, and rollback that didn’t reliably work across every service.

The fix is a FIFO queue in the database rather than a GitHub Actions concurrency group. That choice is deliberate: a GHA concurrency group cancels queued runs when a newer one arrives, which is fine for CI but wrong for deploys, since a cancelled deploy can leave a service on a half-applied build. A DB-backed queue keeps every deploy in order instead of dropping the ones in between.

Canary testing ran three waves of 20 simultaneous real deploys, with zero cap violations and zero failures across every container and its proxy config each time. The queue’s SQL was measured at 0.90 milliseconds typical, 2.0 milliseconds worst case, and 4.1 milliseconds with 500 rows queued, because the query only scans the in-flight set instead of the whole table.

A Pricing Key Collision Undercharged Azure SQL by 42%

Azure cost figures for several resource types were wrong, and the biggest single error was Azure SQL Database being undercharged by roughly 42%. The cause was a pricing key collision: bare Azure SKU prefixes matched MySQL Single Server’s rate, around nine hundredths of a cent per vCore-hour, instead of SQL Database’s real rate, over fifteen hundredths of a cent per vCore-hour.

Three more mispricings shipped fixed in the same pass: VM and scale-set over-counts, disk tiers where a 4 GiB disk billed as if it were 32 GiB, and pricing gaps in zone-redundant disks, Event Hubs, and flexible Postgres and MySQL. None of these needed a schema change or a migration; the fix is entirely in how prices resolve.

The release notes flag something worth noting directly: deploy-day cost figures will move in both directions once this ships, and the SQL Database correction, a 74% increase, is the largest single upward mover. That’s an expected side effect of fixing an undercharge, not a new bug.

One Resource Shouldn’t Cost 101 RPCs

The beta recommender was getting OOM-killed 12 times a day. The cause: every discovery event, one per resource, triggered a metrics fetch for the entire org instead of just that resource. Against a 5,000-resource test org, that pattern cost 101 RPCs to evaluate a single resource.

The fix scopes the fetch to the resource that actually changed: a 99% drop, from 101 RPCs to 1. A second, related bug sat next to it: the org-level kill switch meant to freeze a problem org’s recommendations only covered the manual refresh button. Three separate background triggers kept mutating that org’s recommendations anyway, carrying roughly 1,100 changes in 24 hours for an org that was supposedly frozen. The fix gates every path that can write a recommendation, not just the one with a button, with one deliberate exception: a workflow the cloud already finished executing still closes its own recommendation, because leaving a completed remediation marked open would be a worse lie than the kill switch was trying to prevent. Every time the switch now skips a write, it logs which org and which path, so a frozen org that still looks frozen a month from now is provable, not assumed.

Parity testing on a snapshot before and after the fix produced 71 recommendations both times, byte-identical on every scoring field. That’s the check that matters most for a fix like this: proving the output didn’t change, only the cost of producing it.

Locks That Forget They’re Dead, and Deletions That Get Lost

Three more fixes share a pattern: something stopped happening, and nothing said so. A crashed pod’s discovery lock used to linger for up to 60 minutes, silently blocking every retry while the system still reported the run as completed. The fix makes the lock self-heal in about 2 minutes, using a token-fenced TTL that renews every 30 seconds instead of a lock with no expiry. A live test that killed the pod mid-hold confirmed the orphaned lock cleared at roughly 117 seconds.

Activity history had a matching gap: when a cloud resource was deleted outside ZopNight, the deletion event itself was almost never captured. Before the fix, 791 externally-terminated recommendations existed against only 2 stored delete events. Deleted resources were disappearing from history instead of being recorded as deleted.

The MCP token-scope bug follows the same shape from a different angle: four of the eight deploy tools exposed over MCP, including deploy_service, were invisible to every scoped token minted from the UI. The scope check was vetoing the tools’ read steps, and no token UI could grant a read permission that fine-grained in the first place. The fix doesn’t widen access; every call still passes the caller’s own role-based permissions.

What the Reliability Ledger Tells You About Evaluating This Release

Every fix in this release traces back to the same mistake at a different layer: a check, a fetch, or a lock that was scoped to one org, one resource, or one pod, but got applied as if it covered every tenant or every event. The shared VM namespace pool, the recommender’s org-wide metrics call, and the discovery lock’s unbounded TTL are the same bug shape, three times.

That pattern is worth watching for in any platform’s release notes, not just this one. A feature list tells you what’s new. A fix list with real before-and-after numbers, 26 conflicts down to zero, 101 RPCs down to 1, 791 lost events down to zero, tells you what was quietly wrong and whether the team measured it honestly enough to publish the number.

If you’re evaluating ZopNight or a comparable governance platform, don’t stop at the migration feature. Read the fix list, check whether the numbers are specific, and ask what the equivalent bug looks like in the tool you’re using today.