| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-08-19 | |||
| 15:57:17 | sean-k-mooney | ot me its tech debt and i didnte really like doign this as a one off | |
| 15:58:48 | stephenfin | it's still better than what we have though | |
| 15:59:22 | stephenfin | perfect is the enemy of good and all that | |
| 15:59:42 | sean-k-mooney | so my patch is also better then what we have | |
| 16:28:52 | dansmith | melwitt: yawna +W this or prefer to wait for bauzas? https://review.opendev.org/c/openstack/nova/+/926144 | |
| 16:51:11 | opendevreview | sean mooney proposed openstack/nova master: imporve nova object logging https://review.opendev.org/c/openstack/nova/+/898057 | |
| 16:52:14 | sean-k-mooney | stephenfin: melwitt ^ i updated the name and added a unit test as bauzas asked | |
| 17:01:58 | sean-k-mooney | dansmith: any idea if https://review.opendev.org/c/openstack/python-glanceclient/+/890634 will merge in time to allow https://review.opendev.org/c/openstack/nova/+/891036 to proceed this cycle | |
| 17:02:46 | sean-k-mooney | it looks like the sdk patch is also not ready https://review.opendev.org/c/openstack/openstacksdk/+/883271 | |
| 17:02:55 | dansmith | abhishekk: ^ | |
| 17:03:07 | dansmith | sean-k-mooney: I have been too busy to follow the progress lately, so I'm not sure | |
| 17:03:30 | sean-k-mooney | dansmith: no worreis i was just doing jira paperwork... hence checking | |
| 17:03:32 | abhishekk | ack | |
| 17:03:57 | abhishekk | I think glance client is good to go | |
| 17:04:19 | abhishekk | not sure about openstacksdk though | |
| 17:05:10 | sean-k-mooney | well we dont actully need the sdk change | |
| 17:05:17 | sean-k-mooney | dans currenct change uses cinder client | |
| 17:05:28 | sean-k-mooney | but we would need a cinder client release before feature freeze | |
| 17:05:49 | sean-k-mooney | what are the changes for merging the clinet change and doing a release this week | |
| 17:06:04 | sean-k-mooney | so that we can follow up with dans patch next week in advance of FF | |
| 17:06:05 | abhishekk | client just need one more +2 | |
| 17:06:26 | sean-k-mooney | well that and an actual release | |
| 17:06:35 | abhishekk | I think rosmaita can do that | |
| 17:06:54 | sean-k-mooney | ack well if this lines up it woudl be nice to get incldued in this release | |
| 17:06:57 | abhishekk | yes, once patch is merged I can post a release patch | |
| 17:07:03 | sean-k-mooney | if now i guess we will do the nova part next cycle | |
| 17:07:13 | sean-k-mooney | abhishekk++ | |
| 17:07:15 | abhishekk | right | |
| 17:07:45 | abhishekk | let me ping rosmaita on glance channel | |
| 17:10:35 | melwitt | dansmith: have we discussed with bauzas before? I was thinking if not then it might be best for him to review it | |
| 17:11:16 | sean-k-mooney | the deprecation is somethign we can merge post FF but i think they are back from PTO next week in either case | |
| 17:11:57 | dansmith | melwitt: yeah we did sort of informally, but ack, as long as we get it done in this release, that's fine | |
| 17:12:16 | melwitt | yeah he is back next monday iirc | |
| 17:15:06 | opendevreview | Merged openstack/nova master: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926407 | |
| 17:15:31 | sean-k-mooney | @artom ^ | |
| 17:15:58 | artom | sean-k-mooney, ack, backport time | |
| 17:16:06 | sean-k-mooney | yes please :) | |
| 17:17:49 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.2: Functional test test_boot_reschedule_with_proper_pci_device_count https://review.opendev.org/c/openstack/nova/+/926554 | |
| 17:17:50 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.2: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926555 | |
| 17:18:04 | artom | Wait, I skipped 2024.1, didn't I? | |
| 17:18:25 | artom | Actual names were better, fight me | |
| 17:18:44 | sean-k-mooney | yes you did | |
| 17:19:07 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2024.1: Functional test test_boot_reschedule_with_proper_pci_device_count https://review.opendev.org/c/openstack/nova/+/926556 | |
| 17:19:08 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2024.1: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926557 | |
| 17:20:33 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.2: Functional test test_boot_reschedule_with_proper_pci_device_count https://review.opendev.org/c/openstack/nova/+/926554 | |
| 17:20:34 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.2: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926555 | |
| 17:21:33 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Functional test test_boot_reschedule_with_proper_pci_device_count https://review.opendev.org/c/openstack/nova/+/926558 | |
| 17:21:34 | opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Fix PCI passthrough cleanup on reschedule https://review.opendev.org/c/openstack/nova/+/926559 | |
| #openstack-nova - 2024-08-20 | |||
| 00:37:39 | opendevreview | Takashi Kajinami proposed openstack/nova master: Report availability of stateless firmware support https://review.opendev.org/c/openstack/nova/+/908888 | |
| 00:37:39 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Launch instances with stateless firmware https://review.opendev.org/c/openstack/nova/+/908890 | |
| 00:37:40 | opendevreview | Takashi Kajinami proposed openstack/nova master: Add hw_firmware_stateless image property https://review.opendev.org/c/openstack/nova/+/926590 | |
| 03:37:36 | opendevreview | Merged openstack/nova master: [libvirt]log XML if nova fails to parse it https://review.opendev.org/c/openstack/nova/+/906364 | |
| 05:23:43 | opendevreview | Zhang Hua proposed openstack/nova master: Fix deepcopy usage for BlockDeviceMapping in get_root_info https://review.opendev.org/c/openstack/nova/+/920374 | |
| 08:39:34 | ralonsoh | sean-k-mooney, hello! I have an investigation in progress (well, very little progress so far) | |
| 08:39:57 | ralonsoh | I'm fixing an issue with eventlet, wsgi and tls with ML2/OVN | |
| 08:40:21 | ralonsoh | in a nutshell: I'm pushing https://review.opendev.org/c/openstack/neutron/+/925376/ | |
| 08:40:37 | ralonsoh | this patch works fine with ML2/OVN but not so great with ML2/OVS | |
| 08:41:21 | ralonsoh | the same test is randomly (but very frequently) failing in the OVS jobs: test_established_tcp_session_after_re_attachinging_sg | |
| 08:41:49 | ralonsoh | the issue, so far, is because n-compute is deleting the VM used to test the ping | |
| 08:42:00 | ralonsoh | for example: https://32db135de77b6f5b2e24-00cf68680e4576901ac5e284377f47f2.ssl.cf2.rackcdn.com/924317/7/experimental/neutron-tempest-plugin-openvswitch-distributed-dhcp/43f8b60/controller/logs/index.html | |
| 09:10:11 | ralonsoh | sean-k-mooney, if you have time to review these logs, can you explain why nova is trying to delete the VM? | |
| 09:10:20 | ralonsoh | Aug 19 12:41:14.998800 np0038215967 devstack@n-api.service[58447]: DEBUG nova.compute.api [None req-54123910-92d9-491b-b845-f8e7d37490dd tempest-StatefulNetworkSecGroupTest-1423213631 tempest-StatefulNetworkSecGroupTest-1423213631-project-member] [instance: 4623b509-a848-41a4-9fc2-1cf8227bd4ca] Going to try to terminate instance {{(pid=58447) delete /opt/stack/nova/nova/compute/api.py:2725}} | |
| 09:10:21 | ralonsoh | sorry, I don't find the reason | |
| 09:19:21 | mikal | sean-k-mooney: I'd do a change for the usb redirect devices as extra specs, but I'm starting to be a bit worried about any of these changes landing in time. What would you prefer? | |
| 09:19:45 | mikal | sean-k-mooney: naively I'd just cargo cult the sound device extra spec one for usb redirect devices, so I was kind of waiting for someone to tell me I did that one wrong first. | |
| 09:40:45 | sean-k-mooney[m] | mikal: ill review that shortly and let you know | |
| 09:43:47 | sean-k-mooney[m] | ralonsoh: ill try and take a look nova wont delete the vm unless asked | |
| 09:44:21 | sean-k-mooney[m] | even missing a vif plugged event willl just cause it to go to error althoguh we might stop it | |
| 09:49:43 | mikal | sean-k-mooney: yeah, I wont get to it tonight anyways as I'm oncall and that always makes me tired. But yeah, I'd appreciate the feedback, I've never done one of those before. | |
| 10:00:58 | ralonsoh | sean-k-mooney[m], I think I found something: the VM is created at 12:40:08 | |
| 10:01:25 | ralonsoh | but the OVS agent receives the tap port creation almost one minute later and doesn't have time to send the vif-plugged event | |
| 10:01:38 | ralonsoh | and the same time, n-compute decides to delete the VM | |
| 10:01:49 | ralonsoh | how can I increase this timeout? | |
| 10:02:13 | leavelet | Hi! I encountered a cinderclient.exceptions.Unauthorized error when creating an instance. I also received the message: "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible." I've confirmed that my nova service_user is configured correctly, and nova-status upgrade check shows all checks as successful. The error logs can be found at https://paste.aosc.io/paste/hMJM8omukSVYzgvLj | |
| 10:02:13 | leavelet | HEhqQ. Should I file an issue on Launchpad, or is this likely a configuration problem on my end? | |
| 10:02:49 | leavelet | I'm new to IRC and not very familiar with it, so if I'm in the wrong place, please let me know | |
| 10:03:28 | ralonsoh | sean-k-mooney[m], vif_plugging_timeout, that by default is 300 (now I don't understand...) | |
| 10:03:33 | sean-k-mooney | ralonsoh: you really should not need too but i think we have a time out that defaults to 300 seconds | |
| 10:03:39 | sean-k-mooney | ya | |
| 10:03:53 | ralonsoh | so I don't understand why the VM is deleted | |
| 10:04:05 | sean-k-mooney | its already longer then is reasonable and you are using kvm in this job | |
| 10:04:14 | ralonsoh | It is not commanded by the tempest test | |
| 10:04:17 | sean-k-mooney | so the vm should be quick to boot/start | |
| 10:05:26 | sean-k-mooney | i did notice that libvirt is a littel unhappy on that node and it failed to detach an interface in one case | |
| 10:05:54 | sean-k-mooney | you are also using very old guest images (20.04 and cirros 0.5.3) im not sure if that is related or not | |
| 10:05:57 | ralonsoh | btw, the VM id is 4623b509-a848-41a4-9fc2-1cf8227bd4ca and the port 9bc40eec-ae67-4b56-9e75-551609520b27 | |
| 10:06:29 | ralonsoh | that is working in the CI well, but not with this patch in particular | |
| 10:07:31 | sean-k-mooney | so the domain was defied at Aug 19 12:40:14.414263 | |
| 10:08:23 | sean-k-mooney | and we get the vif plugged event at ug 19 12:40:16.547804 | |
| 10:08:34 | sean-k-mooney | so 2 seconds later | |
| 10:09:14 | sean-k-mooney | ralonsoh: is this using the iptables firewall or openvswitch i assuem the latter | |
| 10:09:37 | ralonsoh | let me check (by default is the later) | |
| 10:09:46 | sean-k-mooney | "port_filter": true, "ovs_hybrid_plug": false, | |
| 10:09:53 | sean-k-mooney | its the ovs firewall | |
| 10:09:56 | ralonsoh | yes, ovs fw | |
| 10:10:28 | sean-k-mooney | that means that the ml2 agent only sees the tap when the vm is started vs seeign the veth pair | |
| 10:10:40 | sean-k-mooney | slightly earilier | |
| 10:11:13 | sean-k-mooney | so if we look at the ovs log we should see when it sees the tap | |
| 10:11:36 | ralonsoh | at Aug 19 12:41:15.653623 | |
| 10:11:48 | ralonsoh | just before it is deleted | |