| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-10-11 | |||
| 10:00:55 | sean-k-mooney | basicaly some large customes have said to use we cant live migrate every vm we have but we can stop them cause we run everything in duplex mode and can go simplex for a period of time without issue | |
| 10:01:45 | gibi | OK so there is two parallel direction to pursue. 1) add a workaround flag for something like wait_for_plug_event_at_reboot and 2) force_rebind_on_reboot to allow in place update of binding related data | |
| 10:02:06 | sean-k-mooney | so they are fine with stop(tearing down all networking) and start(setting up all networking) to "migrate" form one firewall driver to another (iptables->ovs) or between mech drvers(ml2/ovs->,l2/ovn) | |
| 10:02:31 | sean-k-mooney | gibi: am yes i guess so | |
| 10:02:51 | gibi | the problem (most probably) on my side is that they do VM reboots in big batches due to upgrade and nova sends a lot of unplug - plug event to ODL which fails to finish setting up OVS before the VM starts requesting IP via DHCP | |
| 10:04:08 | gibi | so if they can set a WA flag in nova that nova should wait for the network-vif-plugged before letting the VM run at reboot would help to control the situation | |
| 10:04:28 | sean-k-mooney | yep | |
| 10:04:38 | gibi | and they need it on Pike :/ | |
| 10:05:51 | gibi | I will go with a backportable WA flag | |
| 10:07:13 | sean-k-mooney | ya so on master i think we need to look at how and when we use events and for pike workaround flag is definetly the simplest thing as only the custoemr that is affect can opt in by settting it minimising the risk to others that dont need the change | |
| 10:09:44 | sean-k-mooney | gibi: i had planned to bring up ovn live migration at the ptg(which is affected by the vif-plugged-events) but i might just make tha ta general topic since i think the ovn issue will require ovn changes to fully resolve | |
| 10:19:47 | jpic | hi all, i don't understand what is supposed to keep the websocket between the spice html5 client and the spice server? browsers will kill inactive websockets after 60 secs of inactivity, so i suppose the server is supposed to send ping packets regularily | |
| 10:40:28 | gibi | sean-k-mooney: ack | |
| 10:47:40 | gibi | sean-k-mooney: one more thing, I don't see "bound_drivers" ever got implemented in neutron. Only neturon-lib has the field definition https://review.opendev.org/c/openstack/neutron-lib/+/645288/6/neutron_lib/api/definitions/portbindings.py#55 but neutron does not use that | |
| 10:47:50 | gibi | I only see the connectivity field being implemented | |
| 10:47:55 | gibi | in https://review.opendev.org/c/openstack/neutron/+/645645/ | |
| 11:05:52 | sean-k-mooney | i see... am i think it was ment to be implemented at the ml2 plugin level above the drivers but you might be right | |
| 11:06:36 | sean-k-mooney | https://review.opendev.org/c/openstack/neutron-lib/+/645288/6/neutron_lib/api/definitions/portbindings.py added bound_drivers to the extension | |
| 11:06:42 | sean-k-mooney | but im not sure if they have used it | |
| 11:08:32 | sean-k-mooney | looks like its only in neutron-lib https://codesearch.opendev.org/?q=VIF_DETAILS_BOUND_DRIVERS&i=nope&literal=nope&files=&excludeFiles=&repos= | |
| 11:08:48 | gibi | yeah that matches with my grep results | |
| 11:09:13 | sean-k-mooney | so ya looks like we would need to actully implement that on master before we can use it | |
| 11:09:20 | gibi | yepp, it looks like | |
| 11:09:21 | sean-k-mooney | i feel less bad for not using it yet | |
| 11:10:19 | gibi | :) | |
| 12:15:55 | sean-k-mooney | has anyone else had devstack time out on pip resolver issues | |
| 12:22:33 | sean-k-mooney | Collecting oslo.policy>=3.8.1 | |
| 12:22:35 | sean-k-mooney | Using cached oslo.policy-3.8.2-py3-none-any.whl (86 kB) | |
| 12:22:37 | sean-k-mooney | INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking | |
| 12:40:30 | gibi | sean-k-mooney: I haven't seen that pip issue yet | |
| 12:47:25 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419 | |
| 12:47:51 | gibi | sean-k-mooney: ^^ this is my first try on the hard reboot - vif plug issue | |
| 12:48:56 | sean-k-mooney | i think its cause by cinder's dep on oslo.vmware which is pulling in suds-jurko | |
| 12:49:06 | sean-k-mooney | which is an unmaintained soap lib fork | |
| 12:50:22 | sean-k-mooney | gibi: that is just the events right are you working on a second patch for rebinding | |
| 12:51:02 | sean-k-mooney | i guess you might not need that in your case | |
| 12:51:06 | gibi | sean-k-mooney: it is jut the event waiting | |
| 12:51:15 | sean-k-mooney | was pikes version of netwokring-odl new enough to send plug time events | |
| 12:51:29 | gibi | sean-k-mooney: yes, the plug time event patch was merged in pike | |
| 12:51:37 | sean-k-mooney | ah ok cool | |
| 12:52:04 | gibi | https://review.opendev.org/c/openstack/networking-odl/+/465463 | |
| 12:52:06 | gibi | this is the patch | |
| 12:52:26 | sean-k-mooney | do you have a DNM patch that sets that to true or set it in the job so we can see it work | |
| 12:52:42 | gibi | sean-k-mooney: good point I will add that | |
| 12:52:52 | gibi | sean-k-mooney: ml2/ovs works for me with this locally | |
| 12:53:06 | gibi | sean-k-mooney: so we can enable this WA in our normal ovs jobs | |
| 12:53:11 | sean-k-mooney | cool, ya currently its disabled so we wont see if this breaks anything although it will break ovn actully | |
| 12:53:29 | sean-k-mooney | ya we could enable it in nova-next | |
| 12:53:30 | gibi | I think I will enable this flaf in nova-next | |
| 12:53:34 | gibi | exactly :D | |
| 12:53:58 | sean-k-mooney | yep also we might want to renable a multinode jobs with ml2/ovs at some point | |
| 12:54:27 | sean-k-mooney | im not sure what coverage we have/have lost with the ovn default swap | |
| 12:55:08 | sean-k-mooney | i think nova-next might be the only job that is still using ml2/ovs (other then linuxbrige job) in our gate today | |
| 12:55:42 | sean-k-mooney | i dont think we have a multi node job with a backend that send plugtime events currently | |
| 12:56:59 | gibi | sean-k-mooney: nova-next is multinode afaik | |
| 12:57:11 | gibi | name: nova-next | |
| 12:57:11 | gibi | parent: tempest-multinode-full-py3 | |
| 12:57:16 | gibi | yepp it is multinode | |
| 12:58:00 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813419 | |
| 13:05:07 | sean-k-mooney | gibi: oh ok cool | |
| 13:20:09 | gibi | bauzas: fyi, I now remember that there will by TC + PTL discussion slot on the PTG. https://etherpad.opendev.org/p/tc-yoga-ptg L15 (Monday 15 UTC) | |
| 13:20:29 | gibi | bauzas: https://etherpad.opendev.org/p/tc-ptl-interaction | |
| 13:20:37 | gibi | I will join too | |
| 13:20:45 | gibi | just to listen in | |
| 13:20:53 | gibi | or fill in info from the past if needed | |
| 13:44:04 | bauzas | gibi: sorry was off | |
| 13:44:15 | bauzas | gibi: ack, I was wondering when the slot was there | |
| 14:02:07 | opendevreview | Balazs Gibizer proposed openstack/nova stable/pike: Add a WA flag waiting for vif-plugged event during reboot https://review.opendev.org/c/openstack/nova/+/813437 | |
| 14:26:17 | opendevreview | norman shen proposed openstack/nova master: Add a timeout for waiting volume snapshot ready https://review.opendev.org/c/openstack/nova/+/813443 | |
| 15:13:07 | dansmith | gibi: wow, pike huh? :) | |
| 15:13:53 | gibi | dansmith: last week I was pulled into a discussion upgrading Mitaka deployments to Victoria. Pike is easy compared to that | |
| 15:14:01 | dansmith | hah | |
| 15:14:47 | gibi | telcos are slooow | |
| 15:14:55 | dansmith | :) | |
| 15:24:46 | melwitt | bauzas: these backports are finally ready for review after the stable/train gate fixes https://review.opendev.org/c/openstack/nova/+/806629 and the patch under it, if you could take a look | |
| 15:25:19 | bauzas | melwitt: ack, opening a tab and looking at them once we discuss about other issues ;) | |
| 15:25:28 | melwitt | thanks :) | |
| 15:35:15 | opendevreview | Elod Illes proposed openstack/nova stable/stein: [stable-only] Pin virtualenv and setuptools https://review.opendev.org/c/openstack/nova/+/813451 | |
| 17:31:32 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | |
| 17:31:33 | opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports https://review.opendev.org/c/openstack/nova/+/812111 | |
| 19:39:54 | mdbooth | Not really an OpenStack issue, but as I'm hitting it running devstack on Ubuntu Focal I'm hoping somebody here has seen it before 🤞 c-vol fails to come up because lvcreate fails to create a thin volume due to: "modprobe: FATAL: Module dm-thin-pool not found in directory /lib/modules/5.4.0-1047-kvm". Anybody seen that? | |
| 19:41:02 | mdbooth | My google-fu is unfortunately not up to the task. | |
| 19:57:42 | clarkb | mdbooth: you might have to install something like linux-image-extra-virtual but for the kvm kernel | |
| 19:58:34 | mdbooth | clarkb: Thanks. I was just coming to the realisation that this image might have a stripped down kernel for kvm in it. I think I'm going to toss it and try a different cloud image. | |
| 19:59:23 | clarkb | linux-image-virtual doesn't have that module either fwiw, but they do ship linux-image-extra-virtual which I expect includes it | |
| 20:00:08 | mdbooth | Thanks! | |
| 20:00:11 | clarkb | I don't see a virtual package for extra kvm image kernel modules but there are package for specific kernel versions like linux-modules-5.4.0-1009-kvm | |
| 20:01:54 | clarkb | I did check the kernel config for the linux-image-virtual kernel I've got and it lists DM_THIN_PROVISIONING=m so there should be a module for it in a package somewhere | |
| 20:21:19 | mdbooth | Looking good: # find /usr/lib/modules/5.4.0-88-generic/ -name "*thin*" | |
| 20:21:32 | mdbooth | /usr/lib/modules/5.4.0-88-generic/kernel/drivers/md/dm-thin-pool.ko | |
| 20:21:38 | mdbooth | clarkb: Thanks again :) | |
| #openstack-nova - 2021-10-12 | |||
| 07:45:30 | bauzas | good morning Nova | |
| 07:57:36 | pslestang | Hey all, I'm facing an issue on a fresh installed devstack when running tox in nova directory which seems related to a conflict between oslo-vmware and suds-jurko | |
| 07:57:45 | pslestang | ERROR: Cannot install -r /opt/stack/nova/test-requirements.txt (line 28) because these package versions have conflicting dependencies. | |
| 07:57:47 | pslestang | The conflict is caused by: | |
| 07:57:49 | pslestang | oslo-vmware 3.9.1 depends on suds-jurko>=0.6 | |
| 07:57:51 | pslestang | The user requested (constraint) suds-jurko===0.6 | |
| 07:58:16 | pslestang | Is there a recommended fix or a solution to handle that? | |
| 07:58:34 | bauzas | pslestang: weirdo | |
| 07:58:59 | bauzas | which tox target ? | |