Repository navigation
CKS: reconcile node capacity after live resize - #14223
andrijapanicsb wants to merge 15 commits into
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #14223 +/- ##
============================================
+ Coverage 20.45% 20.87% +0.42%
- Complexity 21415 22137 +722
============================================
Files 6487 6503 +16
Lines 589303 592255 +2952
Branches 72245 72678 +433
============================================
+ Hits 120525 123638 +3113
+ Misses 455643 455178 -465
- Partials 13135 13439 +304
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
andrijapanicsb
changed the title
|
@ACSHomeBot package G |
Packaging results:
Test packages are available at: |
🔴 Test Coverage Grade:
|
| Metric | Value |
|---|---|
| Line coverage | 26.00% |
| Branch coverage | 20.40% |
Grade Scale
| Grade | Line Coverage | Meaning |
|---|---|---|
| 🟢 A | ≥ 80% | Excellent - this code sleeps well at night 😴 |
| 🟡 B | 60-79% | Good - almost there, don't stop now 😉 |
| 🟠 C | 40-59% | Acceptable - your code is wearing a seatbelt, but no airbags 😬 |
| 🔴 D | 20-39% | Marginal - boldly shipping where no test has gone before 🖖 |
| ⛔ F | < 20% | Failing - tests? what tests? 🔥 |
Branch coverage is shown as a secondary signal. Grade is determined by line coverage.
View full Actions run
40b50b0 to
7e5a95a
Compare
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
|
Package building started on worker G. |
Packaging results:
Test packages are available at: |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Queued for worker G. Jobs ahead: 0. |
|
Package building started on worker G. |
Packaging results:
Test packages are available at: |
andrijapanicsb
changed the title
|
Hi @Pearl1594, thanks a lot for the detailed review, I really appreciate the time you put into it. I've pushed the changes in 306cc2e and replied to each of your comments above. While working through them I also found two problems I had missed: a node cordoned by an operator would never have been retried after a failure, and the node SSH address was wrong on Netris VPC tiers. Both are fixed now. In two places I went a slightly different way from what you suggested (the etcd check and the skip on SSH failure), and I explained why in those threads. Let me know if anything still looks off. |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
- Name the repeated READ_GUEST, metadata and status strings once. - Use \d in the guest and Kubernetes quantity patterns. - Move the node type selection of the offering preflight into its own method, so the loop has no continue statements. - Drop throws Exception from two tests that cannot throw it.
|
@blueorangutan package |
|
@andrijapanicsb a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. |
|
@DaanHoogland, unrelated to all this work - share with @rajujith some of the PR fixes that I've raised against virt-v2v that are critical for VMware migrations, I'm sure Jithin will appreciate those - https://github.com/libguestfs/virt-v2v/pulls?q=is%3Apr+author%3Aandrijapanicsb - all merged, and he can just take the source DEB/RPM (whatever your worker uses) and backport those, together with the stupid FSTRIM that virt-v2v does - I had a 30TB migration over weekened, and FSTRIM added 2h of duration inside virt-v2v on top of 200 seconds that it really took to convert a WIndows VM with 12 disks/30TB of space. Perhaps he wants to also backport the --no-fstrim option from upstream. @rajujith ping me if you want to test openvddk.com - it's NBD/NBDSSL and HotAdd - that is a 1:1 binary compatible replacement for VMware's VDDK - completely free. I'm sure you might want to test it in your in-house product. |
@andrijapanicsb , looks like a compatitor of the one mentioned in https://www.networkworld.com/article/4241265/fluidcloud-replaces-vddk-with-its-own-disk-copy-engine-for-vmware-exits.html needs a comparison? (in a discussion item, probably not here) |
Packaging results:
Test packages are available at: |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
|
code looks very clean now @andrija, thnks. |
@DaanHoogland you know me well enough - don't compare apples and oranges... ;) "The FluidCloud engine exports the VM disk using Network File Copy (NFC) through standard application programming interfaces (API) on the VMware ESXi hypervisor. It then streams the virtual machine disk (VMDK) file over the network to the destination. Omar said the export captures everything, with no functionality loss. The engine does not do delta sync, so every export is a full export of the VM." Their stuff is a workaorund but uses NFC, My artefact is the libvixdisk.so.8.0.3 that you replace and off you go (HotAdd, NBD/NBDSSL (via NFC) - 100% compatible with VDDK = so all tools previously designed to work with VMware VDDK now works with OpenVDDK. Check the sites, numbers, compatibility, etc. I will share this publicly soon. Free and open source. |
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
Packaging results:
Test packages are available at: |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19802 |
Packaging results:
Test packages are available at: |
|
@blueorangutan test |
|
@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-17329)
|
Defer node preparation until a live CPU or memory change can be dispatched. On a retry of the committed offering, reconcile owned pending work without changing individual VM offerings or operator cordons.
|
|
@ACSHomeBot package G |
|
Accepted: Planned: RPM + DEB, KVM SystemVM. Status: Completed. Packaging results. |
Packaging results:
Test packages are available at: |
Context
#13226 added live compute-offering changes for CKS nodes, building on #12975. The VM could gain CPU/RAM while Kubernetes still reported the old capacity. Operators had to activate guest resources and restart
kubeletmanually. #14213 tracks that gap.Changes
CKS now activates and verifies the added guest CPU/RAM, restarts
kubeletwhen needed, and waits for the Node to report the new capacity and allocatable resources before completing the scale operation.Nodes are handled one at a time without draining resident Pods. While a node's resize is in progress, the node carries a
cloudstack.apache.org/cks-live-resize-pendingannotation and, unless an operator had already cordoned it, a CKS cordon. Both are set and removed with single conditional Node updates. If a scale fails, the annotation stays, and the CKS cordon stays too if the VM already reached the new offering. A retry of the same scale finishes that node before it changes any other. An existing operator cordon is preserved.QA also exposed a pre-existing cluster CPU/RAM accounting bug, not a regression from this PR. Totals are now calculated from the configured per-node-type offerings and counts, so the API and UI reflect mixed control/worker offerings correctly.
Functional and regression testing
Test environment: Cloud-managed KVM, isolated network, published packages from
f92e52d736b9, before the review changes. The tests will be repeated on the final commit.VPC/direct-access and multi-control HA were not tested. Dedicated etcd VMs are resized the same way as on main; they are not Kubernetes nodes, so nothing is reconciled on them (not live-tested). The separate existing
nodeids-only scale-in issue found during QA is tracked in #14458, with its fix in #14459.UI evidence
Successful scale task and correct aggregate capacity:
Scale request and result after Refresh
Control RAM: 2048 → 3072 MiB; worker unchanged at 3 CPU / 4096 MiB.
After Refresh: 6 CPU / 7168 MiB.