Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-03-08
15:26:40 artom I need to clear my email filters and restart them from scratch
15:27:00 artom It's been incorrectly flagging actual people reviews as third party CIs, and marking them read
15:27:03 artom Leading to me missing reviews
15:38:31 openstackgerrit Claudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination https://review.opendev.org/c/openstack/nova/+/779285
15:40:32 openstackgerrit Claudiu Belu proposed openstack/nova master: live migration: Pass limits onto scheduler-chosen destination https://review.opendev.org/c/openstack/nova/+/779285
15:42:37 claudiub ^ solved my NUMA node 0 overcommit due to live-migration issue. :)
15:44:05 sean-k-mooney im not sure how that would solve your issue
15:44:40 sean-k-mooney claudiub: its still not going to result in the vms decrementing resouces in the host numa toplogy blob
15:48:44 claudiub basically, just passing those limits we get from the scheduler to nova-compute. Those limits were None previously, so the CPU and memory allocation ratios were never checked, so NUMA node 0 was always a valid candidate for live-migration, even though it shouldn't if it's over the limit already. Now, when live-migrating, if the NUMA node 0 is full, they will be placed in NUMA node 1.
15:49:41 sean-k-mooney if you just use hw:numa_nodes=1 it will never be full
15:49:52 sean-k-mooney and you should have got that full behavior before
15:50:08 claudiub it wasn't only full, it was overscheduled.
15:50:29 sean-k-mooney no hw:numa_nodes does not claim/decrement that avaiable resouce on the host
15:50:43 sean-k-mooney without hw:cpu_policy=dedicated or hw:mem_page_size set
15:53:02 sean-k-mooney claudiub: at a minium you need to add func test for this here https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_numa_servers.py
15:53:21 claudiub sure, will do.
15:53:24 sean-k-mooney and proably in https://github.com/openstack/nova/blob/master/nova/tests/functional/libvirt/test_numa_live_migration.py
15:54:55 sean-k-mooney basicaly you will need to boot 2 vms with just hw:numa_nodes=1 and show that the first one fills numa0 and the second get soft pinned to numa 1
15:56:34 openstackgerrit Merged openstack/nova master: libvirt: Add firmware auto-configuration support https://review.opendev.org/c/openstack/nova/+/776678
15:56:36 claudiub Anyways, that's what we've seen in our Train environment. For example, we had a node with 400+ instances on NUMA node 0 (max should have been ~320), and if we would do a live-migration and that node was chosen as the destination, the instance would be placed in NUMA node 0. After this fix, the instances were being placed in NUMA node 1 instead.
15:57:32 sean-k-mooney right but as i said as far as i am aware you fix should not actully fix it. it may be i doubt its a full fix for all edgecases
15:59:46 claudiub it's a simple fix for a simple usecase that can also be easily be backported. Definitely, I haven't fixed everything about NUMA in nova, but I don't think it detracts anything if we add it.
16:00:37 sean-k-mooney claudiub: we will see ill try an play with it after code feeze, are you ok waiting until after that.
16:01:19 sean-k-mooney if this is backporaable it can be backported after FF anyway
16:02:22 claudiub sure. As I've said, we tried this in Train, and we seem to be more stable now when it comes to live-migrations.
16:14:30 melwitt deprecation cycle(s) to get rid of because of the changes in behavior if we were to make soft deletion no longer possible
16:14:30 melwitt sean-k-mooney: no, not planning to talk about cascading deletes. that would require removing soft-deletion which from past attempts would be very involved and I don't have the bandwidth to get into it. it would also change behaviors like being able to view the instance action list of a deleted instance, there's also a literal soft delete instance API that we'd have to decide what to do with, and so on. I think it would likely need a
16:15:41 sean-k-mooney melwitt: ack cool
16:17:23 lemko kashyap, nope, I'm not using nested virtualizaiton
16:34:36 melwitt sean-k-mooney, gibi: the bug downstream will happen even if we don't purge at the same time as archive. the issue is only with archive. archive moves a record to the shadow table, so it deletes it from the main table and inserts it in the shadow table. we use the word "purge" to refer to deleting things from the shadow tables i.e. 'nova-manage db purge'
16:35:12 gibi melwitt: thanks
16:42:45 openstackgerrit Balazs Gibizer proposed openstack/nova stable/train: Warn when starting services with older than N-1 computes https://review.opendev.org/c/openstack/nova/+/779293
16:42:46 openstackgerrit Balazs Gibizer proposed openstack/nova stable/train: Add upgrade check about old computes https://review.opendev.org/c/openstack/nova/+/779294
16:58:51 sean-k-mooney bauzas: https://review.opendev.org/c/openstack/nova/+/773792 is the pci port numa policy patch. it nit a volume detach issue thats unrelated so just recheked it there a while ago
17:06:47 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Add tests for insecure UEFI configuration https://review.opendev.org/c/openstack/nova/+/777367
17:06:48 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Add missing type hints https://review.opendev.org/c/openstack/nova/+/775688
17:06:48 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Stop passing around virt_type, caps https://review.opendev.org/c/openstack/nova/+/775689
17:06:49 openstackgerrit Stephen Finucane proposed openstack/nova master: hardware: Start parsing 'os_secure_boot' https://review.opendev.org/c/openstack/nova/+/682628
17:06:49 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Clarify purpose of 'Host.supports_*' properties https://review.opendev.org/c/openstack/nova/+/778739
17:06:50 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Translate secure boot requests to trait https://review.opendev.org/c/openstack/nova/+/776680
17:06:50 openstackgerrit Stephen Finucane proposed openstack/nova master: compute: Report COMPUTE_SECURITY_UEFI_SECURE_BOOT https://review.opendev.org/c/openstack/nova/+/776679
17:06:51 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Report secure boot support to scheduler https://review.opendev.org/c/openstack/nova/+/775690
17:06:51 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Wire up 'os_secure_boot' property https://review.opendev.org/c/openstack/nova/+/776681
17:06:52 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Remove duplicated 'start_compute' helper https://review.opendev.org/c/openstack/nova/+/776683
17:06:52 openstackgerrit Stephen Finucane proposed openstack/nova master: tests: Add functional tests for UEFI, secure boot https://review.opendev.org/c/openstack/nova/+/776682
17:06:53 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Add parsing of firmware metadata files https://review.opendev.org/c/openstack/nova/+/779302
17:06:53 openstackgerrit Stephen Finucane proposed openstack/nova master: docs: Document UEFI secure boot feature https://review.opendev.org/c/openstack/nova/+/776684
17:06:54 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Use firmware metadata files to configure instance https://review.opendev.org/c/openstack/nova/+/779304
17:06:54 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Move some host arch checks to guest checks https://review.opendev.org/c/openstack/nova/+/779303
17:07:33 stephenfin Phew
17:13:20 lyarwood stephenfin: I'll hit that again in the morning
17:13:36 lyarwood or maybe later tonight if I get time
18:30:22 openstackgerrit Stephen Finucane proposed openstack/nova master: libvirt: Enable 'vmcoreinfo' feature by default https://review.opendev.org/c/openstack/nova/+/752912
18:30:23 openstackgerrit Stephen Finucane proposed openstack/nova master: WIP: libvirt: Make vmcoreinfo dependent on feature flag https://review.opendev.org/c/openstack/nova/+/779342
18:35:48 stephenfin artom: ^
18:36:10 stephenfin tl;dr: I will take the feature flag approach but not now. Too many test failures, too little time
18:36:18 artom stephenfin, \o/
18:36:27 artom Fair
18:36:57 stephenfin (as noted in the patch itself, the secure boot series should clear up most of those failures since I've significantly improved the FakeLibvirtFixture)
18:38:09 artom It's always the tests...
18:38:42 artom And we'll have re-implemented libvirt in Python, only for qemu and kvm
19:36:54 artom gibi, still active?
19:40:27 artom I guess not, but I think I'm confident enough in my thinking that I'll go ahead
19:40:43 artom Replied in review, in any case
19:49:53 sean-k-mooney stephenfin: replied on the port policy spec. bauzas suggested i check for the extenion to be extra safe so dont really want to remove it unless he is ok with it
20:19:29 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci manager: replace node_id parameter with compute_node https://review.opendev.org/c/openstack/nova/+/778747
20:19:30 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: implement the 'socket' NUMA affinity policy https://review.opendev.org/c/openstack/nova/+/772779
20:19:30 openstackgerrit Artom Lifshitz proposed openstack/nova master: pci: track host NUMA topology in stats https://review.opendev.org/c/openstack/nova/+/774149
20:57:20 melwitt lyarwood: fyi https://review.opendev.org/c/openstack/nova/+/777218 and https://review.opendev.org/c/openstack/nova/+/777219 are ready for review on stable/ussuri. there are two more patches under them (sorry!) that I included bc (1) it's a bug fix I had forgotten to backport and (2) they eliminate merge conflicts
21:30:31 openstackgerrit sean mooney proposed openstack/nova master: add vdpa nodedev parsing and interface config gen https://review.opendev.org/c/openstack/nova/+/770532
21:30:31 openstackgerrit sean mooney proposed openstack/nova master: add constants for vnic type vdpa https://review.opendev.org/c/openstack/nova/+/770474
21:30:32 openstackgerrit sean mooney proposed openstack/nova master: extend pci dev_type for vdpa https://review.opendev.org/c/openstack/nova/+/777481
21:30:33 openstackgerrit sean mooney proposed openstack/nova master: add vdpa device detection https://review.opendev.org/c/openstack/nova/+/770533
21:30:34 openstackgerrit sean mooney proposed openstack/nova master: add hw:mlock extra spec https://review.opendev.org/c/openstack/nova/+/778347
21:30:35 openstackgerrit sean mooney proposed openstack/nova master: add vdpa vnic to pci request mapping and filtering. https://review.opendev.org/c/openstack/nova/+/778350
21:32:34 openstackgerrit sean mooney proposed openstack/nova master: add vdpa nodedev parsing and interface config gen https://review.opendev.org/c/openstack/nova/+/770532
21:32:34 openstackgerrit sean mooney proposed openstack/nova master: add hw:mlock extra spec https://review.opendev.org/c/openstack/nova/+/778347
21:32:35 openstackgerrit sean mooney proposed openstack/nova master: add vdpa device detection https://review.opendev.org/c/openstack/nova/+/770533
21:32:35 openstackgerrit sean mooney proposed openstack/nova master: extend pci dev_type for vdpa https://review.opendev.org/c/openstack/nova/+/777481
21:32:36 openstackgerrit sean mooney proposed openstack/nova master: add vdpa vnic to pci request mapping and filtering. https://review.opendev.org/c/openstack/nova/+/778350
21:49:17 openstackgerrit Ghanshyam proposed openstack/nova master: Change default policy for GET '/os-hypervisors' API https://review.opendev.org/c/openstack/nova/+/765798
21:51:38 openstackgerrit Ghanshyam proposed openstack/nova master: Change default policy for GET '/os-hypervisors' API https://review.opendev.org/c/openstack/nova/+/765798
22:19:39 openstackgerrit Merged openstack/nova master: libvirt: Add tests for insecure UEFI configuration https://review.opendev.org/c/openstack/nova/+/777367
23:05:15 openstackgerrit sean mooney proposed openstack/nova master: Support per port numa policies with SR-IOV https://review.opendev.org/c/openstack/nova/+/773792
23:05:51 openstackgerrit sean mooney proposed openstack/nova master: Support per port numa policies with SR-IOV https://review.opendev.org/c/openstack/nova/+/773792
#openstack-nova - 2021-03-09
01:17:59 yonglihe gibi: thanks raise the VIC TYPE concerns, I tested by wire that in local repo. I try to verify Rodolfo's patch today: https://review.opendev.org/c/openstack/neutron/+/779292
01:31:11 gmann stephenfin: gibi updated the policy patch for modernize-os-hypervisors-api BP https://review.opendev.org/c/openstack/nova/+/765798/5
08:52:40 yonglihe gibi: alex_xu:, F.Y.I I've talke with ralonsoh and slaweq, that neutron patch is likely merged today: https://review.opendev.org/c/openstack/neutron/+/779292
09:01:11 lyarwood melwitt: ack np looking
09:04:19 lyarwood elod: https://review.opendev.org/c/openstack/nova/+/777217/ - I think you forgot to vote on this?
09:20:34 elod lyarwood: wow. apparently. thx. :S
09:20:56 lyarwood elod: np I've done that lots with the new UI
09:22:15 elod yeah, let's blame that o:) I don't know how i missed to vote :S sorry :S
09:48:41 stephenfin oh, no Gerrit bot?
09:48:56 stephenfin trivial fix to address some issues coming down the pipeline with setuptools here https://review.opendev.org/c/openstack/nova/+/779449
09:52:22 jrosser could i get some help with this when debugging why i can't rescue boot-from-volume images https://opendev.org/openstack/nova/src/branch/master/nova/api/openstack/compute/rescue.py#L60
09:53:32 jrosser if i print req.api_version_request i get "API Version Request Major: 2, Minor: 1"
09:53:54 jrosser that seems always going to fail if passed to api_version_request.is_supported(req, '2.87')
09:55:08 stephenfin jrosser: How are you making the request/
09:57:12 jrosser stephenfin: either through horizon or `openstack server rescue --image 14600a9b-a240-4210-8a32-cea43d0499ac 4f2b0a96-7b6b-4d01-bb75-1248574e71d6`

Earlier   Later