| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-24 | |||
| 18:17:08 | sean-k-mooney | i dont think so in ussuri but you should not be use the disk filter in ussuri for ironic or otherwise if that is why you are asking | |
| 18:19:29 | NobodyCam | hey hey Sean, in queens that was a value I used to validate the node was ready for deployment. | |
| 18:21:15 | sean-k-mooney | why not use the node status in ironic for that | |
| 18:21:25 | sean-k-mooney | its ready for deployment when its in the avaiable state | |
| 18:22:35 | NobodyCam | with that we hit no valid hosts, I should add that when we provision we use force_node to deploy to specific nodes | |
| 18:24:46 | NobodyCam | here is a sample if what I was seeing: | |
| 18:24:57 | NobodyCam | https://www.irccloud.com/pastebin/fuSfO22M/ | |
| 18:25:39 | sean-k-mooney | NobodyCam: ironic stopped reporting resouces for cpu ram and disk several cycles ago | |
| 18:26:04 | NobodyCam | ack | |
| 18:26:24 | sean-k-mooney | i dont rember off the top of my head when we moved to trackign ironic nodes with custom resouce classes but im sure tha tlikely had an effect of os-hyperviors output eventually | |
| 18:26:39 | NobodyCam | yep | |
| 18:27:07 | sean-k-mooney | in wally we have removed disk_avaiable_lset https://specs.openstack.org/openstack/nova-specs/specs/wallaby/implemented/modernize-os-hypervisors-api.html | |
| 18:27:39 | NobodyCam | ++ | |
| 18:28:02 | sean-k-mooney | NobodyCam: what you could do is see if the node exist in os-hyperviors rather then check for that value specifically | |
| 18:28:29 | sean-k-mooney | but still the expected way to check this would be via ironic | |
| 18:28:52 | NobodyCam | yea, let me try a few thing... yea we do check ironic | |
| 18:29:09 | sean-k-mooney | NobodyCam: i would proably check the state filed | |
| 18:29:16 | sean-k-mooney | and mayby status | |
| 18:29:28 | sean-k-mooney | so check for sate:up status:enabled | |
| 18:29:57 | sean-k-mooney | that should mean its schduleable | |
| 18:30:33 | NobodyCam | ++ Awesome, let try some deploys with that... Thank you very much sean-k-mooney !!!! | |
| 18:30:41 | NobodyCam | I'll let you know what I find | |
| 18:30:47 | sean-k-mooney | cool | |
| 18:31:23 | sean-k-mooney | the other thing you could check woudl be placment to see that the node is there but i think os-hyperviors will only have the node after its in placment anyway | |
| 18:31:29 | opendevreview | Merged openstack/nova master: scheduler: Merge driver into manager https://review.opendev.org/c/openstack/nova/+/773644 | |
| 18:31:41 | opendevreview | Merged openstack/nova master: tests: Merge 'test_utils', 'test_scheduler_utils' https://review.opendev.org/c/openstack/nova/+/773646 | |
| 18:31:51 | opendevreview | Merged openstack/nova master: conf: Remove deprecated aliases https://review.opendev.org/c/openstack/nova/+/773647 | |
| 18:55:46 | lyarwood | melwitt: https://review.opendev.org/c/openstack/placement/+/679441/ - if you're happy to look at the nits from me and Sean in a fup I'm happy to +W this now | |
| 18:57:23 | melwitt | lyarwood: sure thing, sorry I had only skimmed them so far | |
| 18:58:57 | lyarwood | melwitt: no issues they are nits after all, I'll look at the AttributeCache change now | |
| 19:00:55 | melwitt | lyarwood: thanks, appreciate it. and I will look at the respin of the nova-manage volume patch today | |
| 19:01:49 | lyarwood | melwitt: thanks, https://review.opendev.org/c/openstack/nova/+/805667 and https://review.opendev.org/c/openstack/nova/+/802317 could also use another core if you get time | |
| 19:02:13 | lyarwood | the first fixing a gate-failure I introduced | |
| 19:02:41 | melwitt | will do, added to the queue. thanks | |
| 19:30:58 | lyarwood | sean-k-mooney: are you about? | |
| 19:31:45 | sean-k-mooney | yes | |
| 19:32:07 | lyarwood | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/screen-n-cpu.txt#13396 & https://e31f9be4742ab987838e-5d50e8ef24c23b6b3512cdd707b25e49.ssl.cf1.rackcdn.com/804286/4/gate/tempest-ipv6-only/ddb6cf5/testr_results.html - So I'm seeing a test failure dump out console logs suggesting that CirrOS is still trying and failing to load apic related stuff even with the feature disabled | |
| 19:32:08 | lyarwood | in the domain | |
| 19:35:13 | lyarwood | I wonder if we actually want <ioapic driver='qemu'/> | |
| 19:36:09 | sean-k-mooney | i dont think soe but we migtht want driver='none' | |
| 19:36:56 | sean-k-mooney | wait ioapic | |
| 19:37:04 | sean-k-mooney | that is different from the apic | |
| 19:37:15 | lyarwood | yeah | |
| 19:37:27 | lyarwood | ioapic | |
| 19:37:27 | lyarwood | Tune the I/O APIC. Possible values for the driver attribute are: kvm (default for KVM domains) and qemu which puts I/O APIC in userspace which is also known as a split I/O APIC mode. Since 3.4.0 (QEMU/KVM only) | |
| 19:37:54 | lyarwood | https://libvirt.org/formatdomain.html#hypervisor-features | |
| 19:38:33 | sean-k-mooney | let me look at the kernel message again | |
| 19:38:45 | sean-k-mooney | the one i was fixing initally was a hagn related to the apic not the ioapic | |
| 19:39:21 | sean-k-mooney | ok so that case ya is deffinetly the io apic | |
| 19:39:53 | sean-k-mooney | am i would still be tempted to say driver=none | |
| 19:39:57 | sean-k-mooney | but that is not documented | |
| 19:40:22 | sean-k-mooney | lyarwood: i dont think changing it to qemu will help | |
| 19:41:08 | lyarwood | Why given it's suggesting you should use that when using QEMU | |
| 19:41:55 | sean-k-mooney | well libvirt woudl do that automtically | |
| 19:42:03 | sean-k-mooney | kvm is only the default for kvm domains | |
| 19:42:10 | sean-k-mooney | it wont use kvm on a qemu domain | |
| 19:43:08 | lyarwood | Are you sure libvirt will add this automatically to the domain? | |
| 19:43:55 | sean-k-mooney | well we are not requesting it if it provdie an api it will set the correct drvier im pretty sure | |
| 19:44:06 | sean-k-mooney | if no i would consider that a libvirt bug | |
| 19:44:24 | sean-k-mooney | one that we likely should not fix in nova we could but that not how i would expect this to work | |
| 19:44:43 | sean-k-mooney | without requesting an ioapic by the way i would not expect to get one | |
| 19:44:47 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/screen-n-cpu.txt#7489 | |
| 19:44:55 | sean-k-mooney | we are not requesting it | |
| 19:45:11 | sean-k-mooney | perhaps its nto considerd a toggelable feature | |
| 19:45:19 | sean-k-mooney | "All features are listed within the features element, omitting a togglable feature tag turns it off." | |
| 19:46:45 | lyarwood | okay so we've never enabled the ioapic somehow? | |
| 19:47:34 | lyarwood | https://paste.centos.org/view/e7a2611c - we've only had three hits of this recently in our gate and I think two of these were before the latest workaround apic patch landed | |
| 19:47:46 | lyarwood | so maybe this can wait until kashyap is back | |
| 19:49:27 | sean-k-mooney | we have not enabeld it in the xml no | |
| 19:51:26 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/libvirt/libvirt/qemu/instance-00000001_log.txt | |
| 19:51:36 | sean-k-mooney | we also done seam to be enablit it on the qemu commandline | |
| 19:52:54 | sean-k-mooney | so either we need to explictly disable it which there seams to be no documented way to do or this is somethign qemu can disabel but libvirt cant | |
| 19:56:15 | opendevreview | Artom Lifshitz proposed openstack/nova master: Reproduce bug 1941005 https://review.opendev.org/c/openstack/nova/+/805882 | |
| 19:56:29 | artom | sean-k-mooney, ^^ confirmed, resize with extra specs is b0rked | |
| 19:56:33 | artom | We don't update the request spec | |
| 19:56:49 | sean-k-mooney | still on master? | |
| 19:56:54 | sean-k-mooney | it used to work in the past | |
| 19:56:54 | artom | Yep | |
| 19:57:04 | artom | stephenfin's patch fixed the numa topology case, not the general case | |
| 19:57:15 | sean-k-mooney | even before stephens patch | |
| 19:57:25 | sean-k-mooney | we regressed this at some point | |
| 19:57:37 | sean-k-mooney | im sure we used to update it at least in memory | |
| 19:57:46 | artom | It's kinda tricky to see - *adding* stuff normally fails as expected if no hosts can provide it, because the resource claim uses the flavor and image directly | |
| 19:57:49 | sean-k-mooney | we may not have commited it back to the db | |
| 19:58:03 | sean-k-mooney | but we did use them at some point for scudliign and creatign the xml | |
| 19:58:33 | artom | But *removing* stuff can go wrong, because despite having hosts that can fit the new thing, the scheduler tries to schedule according to the old request spec | |
| 19:59:54 | sean-k-mooney | ok well its broke now so we should fix an backport in anycase | |
| 20:00:21 | artom | Yar | |
| 20:00:39 | sean-k-mooney | but im pretty sure this used to work in icehouse as im pretty sure i tested resizeing to and from vms with pci deviecs back then | |
| 20:01:12 | artom | Perhaps - icehouse was a while ago - and it's pretty specific to hit | |
| 20:01:58 | lyarwood | cuts to titanic 80 years gif | |
| 20:02:17 | sean-k-mooney | hehe | |
| 20:02:57 | sean-k-mooney | yes it was i just remember testing this in the past althogu hit was specific realted to cpu pinnign and pci passtough so i dont know | |
| 20:03:36 | sean-k-mooney | no one complained and the xmls seams to eb right althgu we know that there are several placese where this si broken for move operations in the past so im not that surprised | |
| 20:04:01 | sean-k-mooney | artom: dont we have whitebox test fo this by the way | |
| 20:04:23 | artom | sean-k-mooney, we might... | |
| 20:04:42 | artom | But say you have two hosts with PCI devices - and you resize the instance from "PCI" to "no PCI" | |
| 20:05:03 | artom | It'll work fine - scheduling will pass with the bad old request spec because all hosts have PCI devices | |
| 20:05:17 | artom | It'll only bit you if you only have left hosts with *no* PCI devices | |
| 20:05:24 | artom | *bite you | |