| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2025-08-27 | |||
| 09:33:12 | tkajinam | s/loose/lose/ | |
| 09:33:13 | tkajinam | yeah | |
| 09:33:27 | tkajinam | but for sev we know the exact names, as you said | |
| 09:33:48 | bauzas | anyway, I'll quickly review your patch and then I'll see how much I can help gibi and sambork on their eventlet conductor patches | |
| 09:33:49 | tkajinam | (I updated the functional tests to prove it, too | |
| 09:33:56 | bauzas | tkajinam: perfect, thanks | |
| 09:35:31 | gibi | I confirmed it is the executor.sshutdow's behavior that is different https://paste.opendev.org/show/bmZZr707naiWYHMt7qim/ | |
| 09:36:38 | gibi | I feel this is a bug in futurist as python says all pending tasks should finish https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor.shutdown | |
| 09:39:42 | gibi | yepp python's own concurrent.future.ThreadPoolExecutor works as expected so this is a futuris bug. I will file it | |
| 09:40:52 | bauzas | gibi: excuse my ignorance but I'm getting hard to find how you found that only one host got the image when comparing paste.opendev.org/show/b1TPrh7ESmcIqWuTZx3D/ and paste.opendev.org/show/bGwpu8My66GcyR8T12sW/ | |
| 09:41:31 | bauzas | I see that compute1 emitted a notification saying it cached the image but I have no clue where the other compute said nooope | |
| 09:41:42 | sambork | ok, so I assume that we are block here (creating some class witch will overwrite shutdown behaviour sounds like to much) | |
| 09:41:53 | gibi | Aug 27 09:07:07 aio nova-conductor[117076]: INFO nova.conductor.manager [None req-8fc723e3-ecc0-4ee3-a4aa-fa33aa2eaf94 admin admin] Preparing to request pre-caching of image(s) 342349b7-351a-470d-92ee-c99f581e55e7 on 2 hosts across 1 cells. | |
| 09:41:58 | bauzas | I mean, I trust you but I want to be able to recognize that in logs | |
| 09:41:59 | gibi | Aug 27 09:07:11 aio nova-conductor[117076]: INFO nova.conductor.manager [None req-8fc723e3-ecc0-4ee3-a4aa-fa33aa2eaf94 admin admin] Image pre-cache operation for image(s) 342349b7-351a-470d-92ee-c99f581e55e7 completed in 4.54 seconds; 1 cached, 0 existing, 0 errors, 0 unsupported, 0 skipped (down) hosts | |
| 09:42:24 | bauzas | oh the "1 cached, gotcha" | |
| 09:42:34 | bauzas | thanks | |
| 09:42:35 | gibi | if you check the same log in the non threading case or in the executor=2 case the you will see the second log showing two cached hosts | |
| 09:42:50 | bauzas | yup, saw it thanks | |
| 09:42:56 | gibi | also the ls command from the two nodes shows that only aio got the image in _base compute1 does not | |
| 09:43:45 | gibi | sambork: yeah I feel the same, I don't think we should try to fix it in one and a half days and even if we have the fix it will be more surgery that I can expect folks will be able to review in short notice | |
| 09:44:17 | gibi | I have to drop for a lunch now sorry... | |
| 09:44:51 | sambork | gibi thanks for testing sorry that I didn't catch it | |
| 09:46:26 | gibi | no worries this is team work :) | |
| 10:04:29 | opendevreview | Merged openstack/nova stable/2024.1: Ignore metadata tags in pci/stats _find_pool logic https://review.opendev.org/c/openstack/nova/+/945101 | |
| 10:04:43 | opendevreview | Merged openstack/nova stable/2024.1: Update Nova bdm with updated swap info https://review.opendev.org/c/openstack/nova/+/937611 | |
| 11:02:25 | sean-k-mooney | gibi: so i belive if you call shutdwon on an executor pool that has pending task it just discards them | |
| 11:03:17 | sean-k-mooney | gibi: at least i belive that how the tread pools in the std lib concurrent.threadpool and process pool | |
| 11:04:49 | sean-k-mooney | gibi: whil i could not express point to this in doc this is part of the reason i orginallys asked that we do not rely on shutdown for the pool to finish all work. i wanted to keep a list of all the future for the exectuted tasks then wait for them to complete | |
| 11:06:42 | bauzas | sean-k-mooney: the SEV-ES is fully reviewed by me, I think we have a good opportunity to add that in Flamingo | |
| 11:07:07 | sean-k-mooney | bauzas: ok ill take a pass on it soon then | |
| 11:07:43 | sean-k-mooney | just sarting a littel late today, i got new ubiquit netowrking gear and may have bene up till 4 tweaking it.... | |
| 11:22:38 | gibi | sean-k-mooney: the python stdlib and the greenpool in futurist behaves the same and waits for all pending tasks, the futurist threadpool only waits for the running tasks not the queued ones. I think this is a bug in futurist as it should mimick the stdlib behavior | |
| 11:22:55 | gibi | sean-k-mooney: and yes we agreed that we should refactor the logic to be nicer than shutdown | |
| 11:23:02 | gibi | now we will have plenty of time for it | |
| 11:23:03 | gibi | :) | |
| 11:27:10 | gibi | still I will file a bug to futurist as this is a problem even for others switching from GreenThreadPoolExecutor to ThreadPoolExecutor within futurist | |
| 11:28:05 | sean-k-mooney | gibi: are you goign ot unparent the untit test form the conductor change if we are not proceedign with it | |
| 11:28:22 | gibi | yepp I will | |
| 11:28:40 | sean-k-mooney | ack did you seem my comment about the db transaction issue | |
| 11:28:44 | gibi | conductor will wait for Gazpatcho the unit test can go a bit if we want into F after FF | |
| 11:29:05 | sean-k-mooney | i think im just going to call that the GG release | |
| 11:29:10 | gibi | haven't seen the db comment yet, I will check, I assume it is abotu an unit test issue | |
| 11:29:20 | gibi | sean-k-mooney: hehe GG sounds good :) | |
| 11:29:38 | sean-k-mooney | so there was only one failing test case in the non voting zuul job patch | |
| 11:29:47 | sean-k-mooney | it passed on the next 2 | |
| 11:30:02 | sean-k-mooney | so either that test need to be after the fixture change | |
| 11:30:19 | sean-k-mooney | or its flaky under threading and need more investigation | |
| 11:30:30 | sean-k-mooney | so i was going to suggest adding it to the exclude list | |
| 11:30:45 | sean-k-mooney | i was tired last night so i didnt dign into the error much | |
| 11:31:50 | sean-k-mooney | gibi: i might do this as well but you may want to run it in a loop, im thinking of just letting the voting patch run in a loop executing unit test for an hour or so on my home server | |
| 11:32:27 | sean-k-mooney | actully i will kick that off now | |
| 11:42:51 | gibi | sean-k-mooney: I did run them in a loop locally. But I will investigate your comment | |
| 12:00:31 | sean-k-mooney | it was exactly 1 failng test so it could be just a one off cause by some test to test interaction | |
| 12:32:33 | opendevreview | Takashi Kajinami proposed openstack/nova master: Purge nested SEV RPs when SEV is disabled https://review.opendev.org/c/openstack/nova/+/958626 | |
| 12:39:27 | tkajinam | my bad. I though I run the whole functional tests but I didn't (and didn't caught some tests also needing update ) | |
| 12:39:32 | tkajinam | I thought * | |
| 12:53:55 | ralonsoh | sean-k-mooney, hello! maybe you missed my message about SR-IOV trusted VFs and MAC changing (https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/sriov-trusted-vfs.html) | |
| 12:56:07 | sean-k-mooney | ralonsoh: ya i didnt see it but i think we talked about it in the past | |
| 12:56:28 | sean-k-mooney | trusted vf shoudl not allow the mac to be changed form the guest | |
| 12:56:46 | sean-k-mooney | i think we talked about there beign a bug in one of the intel drivers? | |
| 12:57:24 | sean-k-mooney | the tursted vf spec allow the VF to be in promisous mode | |
| 12:57:38 | sean-k-mooney | so it can recive traffic that is not destented for its mack | |
| 12:57:53 | sean-k-mooney | but its mac shoudl be still enforced by nova/neutron to be the neutron port mac | |
| 12:59:03 | sean-k-mooney | the kernel dirver is free to rest the mack to all 0000 when the vm is stopped but if it is started again it shoudl be programed by nova/libvirt to the requsted mac again | |
| 12:59:20 | sean-k-mooney | changes to things like link state in the guest or phsysical on the host shoudl not impact the mac at all | |
| 13:00:24 | sean-k-mooney | chanign the mac in the guest shoudl not propagate to the vf in trusted mode and if it does i woudl consider that to be a security issue | |
| 13:00:37 | sean-k-mooney | specificly in the vendors PF and VF driver | |
| 13:01:40 | ralonsoh | sean-k-mooney, but port bonding (and MAC changing) is also considered in the nova spec | |
| 13:01:49 | sean-k-mooney | yes an no | |
| 13:01:54 | sean-k-mooney | what we were descibeing there | |
| 13:02:11 | sean-k-mooney | was the ablity to create a bond port in the guest with a diffent mac then what is on the VF | |
| 13:02:17 | sean-k-mooney | on the host | |
| 13:02:36 | sean-k-mooney | how this used to work is if you change the mac on the guest interface the host mac woudl remain the same | |
| 13:02:45 | sean-k-mooney | in ip tools | |
| 13:02:50 | sean-k-mooney | or ifconfig | |
| 13:03:05 | sean-k-mooney | btu the mac on the packet that was sent on the wire would be updated | |
| 13:03:25 | ralonsoh | sean-k-mooney, ahhhh I thought that if you add two interfaces to a bond (inside the VM) you also change the VF MAC | |
| 13:03:26 | sean-k-mooney | ralonsoh: i think that is what has changed | |
| 13:03:38 | sean-k-mooney | you can change the guest visable side | |
| 13:04:03 | ralonsoh | ahhhh so you can update the bond MAC but it will eventually use the VF MAC outside the VM | |
| 13:04:06 | ralonsoh | I didn't know that | |
| 13:04:12 | sean-k-mooney | btu on niantic for exampel that never actully changed the mac on the host side form the perspecitive of libvirt or ifconfig | |
| 13:04:38 | sean-k-mooney | it would change the mac on the wire which wi why it worked btu the comtople plane view was nto updated | |
| 13:04:58 | sean-k-mooney | i woudl assume at some point they hooked up the mailbox protocol to propagate the mac change | |
| 13:05:21 | ralonsoh | yeah, this is what is happening now | |
| 13:05:30 | sean-k-mooney | ralonsoh: ideaaly the sriov nic agent would use the pci adddress not the the mac/name | |
| 13:05:35 | ralonsoh | and this is of course an issue (at least from Neutron point of view) | |
| 13:05:58 | ralonsoh | sriov agent uses the MAC/PCI tuple to identify a port | |
| 13:06:04 | sean-k-mooney | is this the cause of the customer bug where tehy were seeing the state flap | |
| 13:06:19 | ralonsoh | yes | |
| 13:06:38 | sean-k-mooney | ah ok. im glad i updated that to the netwrokign team so :P | |
| 13:06:55 | sean-k-mooney | so ya i woudl consider just using the pci adress | |
| 13:07:07 | sean-k-mooney | the reason you check both is because of connectx-3 | |
| 13:07:15 | ralonsoh | so that would require a refactor of the sriov agent | |
| 13:07:19 | sean-k-mooney | in that generation of nic and only in that nic | |
| 13:07:39 | sean-k-mooney | you could have 2 pfs with the same pci adress | |
| 13:07:57 | sean-k-mooney | it was dumb and broke a bunch of other things and no one does that anymore | |
| 13:08:09 | ralonsoh | that's another story... | |
| 13:08:22 | ralonsoh | but of course, that won't work with ML2/SRIOV | |