| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-24 | |||
| 17:00:26 | sean-k-mooney | if not you could hardcode the hostname to ip mapping in /etc/hosts | |
| 17:02:47 | sean-k-mooney | so the two ways we have for copyting the block sotrage is scp or rsync https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/volume/remotefs.py#L172-L222 | |
| 17:03:55 | sean-k-mooney | i think this is how libvirt is calling it indirectly https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/utils.py#L214-L251 | |
| 17:04:43 | rmart04 | hmm, we did have hardcoded etc/hosts but we were trying to remove that dependency as we have a lot of hosts and updating this and restarting nova_compute on hundreds of boxes every time was getting tiresome (even in ansible) | |
| 17:05:14 | rmart04 | maybe we might have to look to re-implement that | |
| 17:05:57 | rmart04 | I guess I should mention we are still on stein too | |
| 17:06:15 | rmart04 | _sad face_ | |
| 17:08:49 | ganso | melwitt, lyarwood, bauzas: If you have a spare moment could you please take a look at this backport that already has a +2? Keep in mind that the merge conflict is also proposed and also has a +2: https://review.opendev.org/c/openstack/nova/+/805304 Thanks in advance! | |
| 17:13:31 | sean-k-mooney | im not direclty seeing it. to be honest i tought for live migration libvirt actully did the copy and we only used scp for cold migration | |
| 17:15:02 | sean-k-mooney | ah so e do this here https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/libvirt/driver.py#L9855-L9870 | |
| 17:15:33 | sean-k-mooney | rmart04: so we do the copy form the dest passing the source host name | |
| 17:15:57 | sean-k-mooney | although that is for the config drive | |
| 17:17:24 | rmart04 | the thing that flagged this up today was migrating amphora, unsure if octavia uses config drive by default... could be | |
| 17:17:29 | rmart04 | i'll check | |
| 17:18:33 | johnsom | Yes | |
| 17:18:54 | rmart04 | ahhhh.... yes, disk.config | |
| 17:20:14 | sean-k-mooney | you could work around this by using vfat config drives | |
| 17:20:39 | sean-k-mooney | also we dont need this anymore so we can remove it on master i think | |
| 17:20:53 | sean-k-mooney | the libvirt bug has been fixed | |
| 17:22:09 | sean-k-mooney | rmart04: so ya https://github.com/openstack/nova/blob/stable/stein/nova/conf/configdrive.py#L30 you can use vfat if that is supported by the guests instead if you need too | |
| 17:23:38 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Add migration to resolve shadow table discrepancies https://review.opendev.org/c/openstack/nova/+/805738 | |
| 17:27:35 | rmart04 | ok thanks for all the info, will see how i get on tomorrow | |
| 18:04:56 | opendevreview | Lee Yarwood proposed openstack/nova master: fup: Fix os-volume_attachments api-ref parameters https://review.opendev.org/c/openstack/nova/+/805876 | |
| 18:06:34 | lyarwood | gmann: ack fixed in a fup on top of the api change to avoid the respin, apologies again! | |
| 18:06:45 | lyarwood | ganso: ack I'll review this evening | |
| 18:11:49 | opendevreview | Lee Yarwood proposed openstack/python-novaclient master: Microversion 2.89 - os-volume_attachments https://review.opendev.org/c/openstack/python-novaclient/+/805022 | |
| 18:14:47 | NobodyCam | Good Morning Nova folks, was disk_available_least hypervisor stat removed in Ussuri? | |
| 18:14:58 | NobodyCam | for Ironic hypervisors? | |
| 18:17:08 | sean-k-mooney | i dont think so in ussuri but you should not be use the disk filter in ussuri for ironic or otherwise if that is why you are asking | |
| 18:19:29 | NobodyCam | hey hey Sean, in queens that was a value I used to validate the node was ready for deployment. | |
| 18:21:15 | sean-k-mooney | why not use the node status in ironic for that | |
| 18:21:25 | sean-k-mooney | its ready for deployment when its in the avaiable state | |
| 18:22:35 | NobodyCam | with that we hit no valid hosts, I should add that when we provision we use force_node to deploy to specific nodes | |
| 18:24:46 | NobodyCam | here is a sample if what I was seeing: | |
| 18:24:57 | NobodyCam | https://www.irccloud.com/pastebin/fuSfO22M/ | |
| 18:25:39 | sean-k-mooney | NobodyCam: ironic stopped reporting resouces for cpu ram and disk several cycles ago | |
| 18:26:04 | NobodyCam | ack | |
| 18:26:24 | sean-k-mooney | i dont rember off the top of my head when we moved to trackign ironic nodes with custom resouce classes but im sure tha tlikely had an effect of os-hyperviors output eventually | |
| 18:26:39 | NobodyCam | yep | |
| 18:27:07 | sean-k-mooney | in wally we have removed disk_avaiable_lset https://specs.openstack.org/openstack/nova-specs/specs/wallaby/implemented/modernize-os-hypervisors-api.html | |
| 18:27:39 | NobodyCam | ++ | |
| 18:28:02 | sean-k-mooney | NobodyCam: what you could do is see if the node exist in os-hyperviors rather then check for that value specifically | |
| 18:28:29 | sean-k-mooney | but still the expected way to check this would be via ironic | |
| 18:28:52 | NobodyCam | yea, let me try a few thing... yea we do check ironic | |
| 18:29:09 | sean-k-mooney | NobodyCam: i would proably check the state filed | |
| 18:29:16 | sean-k-mooney | and mayby status | |
| 18:29:28 | sean-k-mooney | so check for sate:up status:enabled | |
| 18:29:57 | sean-k-mooney | that should mean its schduleable | |
| 18:30:33 | NobodyCam | ++ Awesome, let try some deploys with that... Thank you very much sean-k-mooney !!!! | |
| 18:30:41 | NobodyCam | I'll let you know what I find | |
| 18:30:47 | sean-k-mooney | cool | |
| 18:31:23 | sean-k-mooney | the other thing you could check woudl be placment to see that the node is there but i think os-hyperviors will only have the node after its in placment anyway | |
| 18:31:29 | opendevreview | Merged openstack/nova master: scheduler: Merge driver into manager https://review.opendev.org/c/openstack/nova/+/773644 | |
| 18:31:41 | opendevreview | Merged openstack/nova master: tests: Merge 'test_utils', 'test_scheduler_utils' https://review.opendev.org/c/openstack/nova/+/773646 | |
| 18:31:51 | opendevreview | Merged openstack/nova master: conf: Remove deprecated aliases https://review.opendev.org/c/openstack/nova/+/773647 | |
| 18:55:46 | lyarwood | melwitt: https://review.opendev.org/c/openstack/placement/+/679441/ - if you're happy to look at the nits from me and Sean in a fup I'm happy to +W this now | |
| 18:57:23 | melwitt | lyarwood: sure thing, sorry I had only skimmed them so far | |
| 18:58:57 | lyarwood | melwitt: no issues they are nits after all, I'll look at the AttributeCache change now | |
| 19:00:55 | melwitt | lyarwood: thanks, appreciate it. and I will look at the respin of the nova-manage volume patch today | |
| 19:01:49 | lyarwood | melwitt: thanks, https://review.opendev.org/c/openstack/nova/+/805667 and https://review.opendev.org/c/openstack/nova/+/802317 could also use another core if you get time | |
| 19:02:13 | lyarwood | the first fixing a gate-failure I introduced | |
| 19:02:41 | melwitt | will do, added to the queue. thanks | |
| 19:30:58 | lyarwood | sean-k-mooney: are you about? | |
| 19:31:45 | sean-k-mooney | yes | |
| 19:32:07 | lyarwood | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/screen-n-cpu.txt#13396 & https://e31f9be4742ab987838e-5d50e8ef24c23b6b3512cdd707b25e49.ssl.cf1.rackcdn.com/804286/4/gate/tempest-ipv6-only/ddb6cf5/testr_results.html - So I'm seeing a test failure dump out console logs suggesting that CirrOS is still trying and failing to load apic related stuff even with the feature disabled | |
| 19:32:08 | lyarwood | in the domain | |
| 19:35:13 | lyarwood | I wonder if we actually want <ioapic driver='qemu'/> | |
| 19:36:09 | sean-k-mooney | i dont think soe but we migtht want driver='none' | |
| 19:36:56 | sean-k-mooney | wait ioapic | |
| 19:37:04 | sean-k-mooney | that is different from the apic | |
| 19:37:15 | lyarwood | yeah | |
| 19:37:27 | lyarwood | ioapic | |
| 19:37:27 | lyarwood | Tune the I/O APIC. Possible values for the driver attribute are: kvm (default for KVM domains) and qemu which puts I/O APIC in userspace which is also known as a split I/O APIC mode. Since 3.4.0 (QEMU/KVM only) | |
| 19:37:54 | lyarwood | https://libvirt.org/formatdomain.html#hypervisor-features | |
| 19:38:33 | sean-k-mooney | let me look at the kernel message again | |
| 19:38:45 | sean-k-mooney | the one i was fixing initally was a hagn related to the apic not the ioapic | |
| 19:39:21 | sean-k-mooney | ok so that case ya is deffinetly the io apic | |
| 19:39:53 | sean-k-mooney | am i would still be tempted to say driver=none | |
| 19:39:57 | sean-k-mooney | but that is not documented | |
| 19:40:22 | sean-k-mooney | lyarwood: i dont think changing it to qemu will help | |
| 19:41:08 | lyarwood | Why given it's suggesting you should use that when using QEMU | |
| 19:41:55 | sean-k-mooney | well libvirt woudl do that automtically | |
| 19:42:03 | sean-k-mooney | kvm is only the default for kvm domains | |
| 19:42:10 | sean-k-mooney | it wont use kvm on a qemu domain | |
| 19:43:08 | lyarwood | Are you sure libvirt will add this automatically to the domain? | |
| 19:43:55 | sean-k-mooney | well we are not requesting it if it provdie an api it will set the correct drvier im pretty sure | |
| 19:44:06 | sean-k-mooney | if no i would consider that a libvirt bug | |
| 19:44:24 | sean-k-mooney | one that we likely should not fix in nova we could but that not how i would expect this to work | |
| 19:44:43 | sean-k-mooney | without requesting an ioapic by the way i would not expect to get one | |
| 19:44:47 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/screen-n-cpu.txt#7489 | |
| 19:44:55 | sean-k-mooney | we are not requesting it | |
| 19:45:11 | sean-k-mooney | perhaps its nto considerd a toggelable feature | |
| 19:45:19 | sean-k-mooney | "All features are listed within the features element, omitting a togglable feature tag turns it off." | |
| 19:46:45 | lyarwood | okay so we've never enabled the ioapic somehow? | |
| 19:47:34 | lyarwood | https://paste.centos.org/view/e7a2611c - we've only had three hits of this recently in our gate and I think two of these were before the latest workaround apic patch landed | |
| 19:47:46 | lyarwood | so maybe this can wait until kashyap is back | |
| 19:49:27 | sean-k-mooney | we have not enabeld it in the xml no | |
| 19:51:26 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/ddb6cf53e65a404c89dac4dc4ca1cd51/log/controller/logs/libvirt/libvirt/qemu/instance-00000001_log.txt | |
| 19:51:36 | sean-k-mooney | we also done seam to be enablit it on the qemu commandline | |
| 19:52:54 | sean-k-mooney | so either we need to explictly disable it which there seams to be no documented way to do or this is somethign qemu can disabel but libvirt cant | |