| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-03-11 | |||
| 14:59:36 | sean-k-mooney | gibi and i spoke about this a while ago | |
| 14:59:58 | sean-k-mooney | kashyap: can proably verify but i think qemu now considers it an error to detach an already detaching vloume | |
| 15:00:00 | dansmith | is libvirt/qemu only requesting the guest drop it once? because subsequent ones seem to be a libvirt refusal to try again | |
| 15:00:16 | dansmith | yeah I see that error in the logs after the first attempt | |
| 15:00:50 | sean-k-mooney | ya so it used to be undefiend behavior that on some release aborted the detach then qemu started rejecting it and | |
| 15:00:55 | kashyap | sean-k-mooney: Yes, that's right - a device that is already being unplugged, QEMU will consider another attempt at it an error | |
| 15:01:03 | sean-k-mooney | im not really sure what the intended behavior is now | |
| 15:01:26 | dansmith | so .. what to do? if the guest isn't ready to handle the signal we're just screwed until reboot or something? | |
| 15:01:29 | dansmith | that seems pretty broken | |
| 15:02:21 | sean-k-mooney | i dont know honestly. i dont think we want to do what we did in the past for snapshot which was stop and start the vm | |
| 15:02:28 | sean-k-mooney | when live snapshots were not possible | |
| 15:02:36 | dansmith | yeah :/ | |
| 15:02:43 | sean-k-mooney | other then that i dont know of a way to force this form nova | |
| 15:02:52 | dansmith | but again, I'm still curious about why this seems to only happen on centos hosts | |
| 15:03:03 | dansmith | unless you think that behavior changed between 20.04's libvirt and now? | |
| 15:03:12 | sean-k-mooney | we were seeign it on ubuntu too with q35 | |
| 15:03:19 | kashyap | dansmith: So, this was the DNM patch of 120sec delay: https://review.opendev.org/c/openstack/devstack/+/828705/8/.zuul.yaml | |
| 15:03:39 | sean-k-mooney | i dont think we have seen this with pc and ubuntu | |
| 15:03:52 | dansmith | sean-k-mooney: oh, are the centos jobs all q35 by default? | |
| 15:04:03 | sean-k-mooney | no i dont think so | |
| 15:04:04 | kashyap | The thing is, CirrOS needs 10 sec to boot in our CI, but Nova returns "ACTIVE" when the guest spawns. | |
| 15:04:23 | dansmith | kashyap: ah that's the timeout not a delay, and just runs more futile retries I guess? | |
| 15:04:28 | sean-k-mooney | but it could be a combination of the differnet way attach is done with q35 and the new versions treatment of pc | |
| 15:04:57 | kashyap | Right, I read the timeout as "delay before the detach" | |
| 15:04:59 | sean-k-mooney | dansmith: we coudl try using a debian or maybe ubuntu latest job to verify | |
| 15:05:12 | sean-k-mooney | ill check nodepool quickly | |
| 15:05:22 | sean-k-mooney | but i think we have a debian 11 image avaiable | |
| 15:05:37 | sean-k-mooney | and i think it will have a similar libvirt as c9s | |
| 15:05:43 | kashyap | dansmith: Also, yes, Q35 does have some additional hidden special bugs with hot unplug. I was told <cough> CentOS/RHEL 8.6 has better fixes in that area | |
| 15:06:02 | kashyap | (Ouch, I should not use the c-word in these times) | |
| 15:06:18 | dansmith | kashyap: may I suggest <ahem> | |
| 15:07:04 | kashyap | Heh, sure | |
| 15:07:57 | sean-k-mooney | debian-bullseye is there so we could try recreating it on that | |
| 15:08:35 | kashyap | dansmith: Incidentally, I was supposed to work with Red Hat QE today/Monday to test those bits | |
| 15:08:53 | dansmith | sean-k-mooney: so what's the point of doing that? to see if it seems to be characteristic of new libvirt/qemu and not something else about stream9 itself? | |
| 15:09:00 | kashyap | ("those bits" == supposed fixes in QEMU from 8.6) | |
| 15:09:15 | dansmith | kashyap: does that make it to stream9 at some point I hope? | |
| 15:09:33 | kashyap | Yes, definitely. They should. | |
| 15:10:21 | kashyap | "There were number of improvements for both native PCI-E (albeit it still slow to react (due to how it's implemented in guest OS) and now q35 supports ACPI base hotplug, can you check with latest machine type (which supposedly should use ACPI hotplug) and see if it resolved the issue." | |
| 15:10:33 | sean-k-mooney | dansmith: yep basically | |
| 15:10:36 | sean-k-mooney | https://bugzilla.redhat.com/show_bug.cgi?id=2007129 | |
| 15:10:38 | kashyap | That's the comment from a PCI(e) dev from QEMU (from a RHT bug) | |
| 15:10:44 | sean-k-mooney | ^ that is the main bug right | |
| 15:10:46 | kashyap | (Where "latest machine type" == 8.6 / 9) | |
| 15:12:08 | dansmith | ack | |
| 15:12:10 | sean-k-mooney | from that bug the say "This bug is related to some change in qemu-6.2 so it should not be there in RHEL 8.4/8.2," | |
| 15:12:32 | sean-k-mooney | however we have had detach issue on 8.4 | |
| 15:12:48 | sean-k-mooney | anyway it might just be down to the use of 6.2+ on c9s | |
| 15:50:21 | opendevreview | Andre Aranha proposed openstack/nova master: Move FIPS jobs to experimental and periodic queue https://review.opendev.org/c/openstack/nova/+/833431 | |
| 15:55:14 | opendevreview | sean mooney proposed openstack/nova master: [WIP] enable block VDPA operations https://review.opendev.org/c/openstack/nova/+/832330 | |
| 15:55:45 | opendevreview | sean mooney proposed openstack/nova master: [WIP] enable blocked VDPA operations https://review.opendev.org/c/openstack/nova/+/832330 | |
| 15:56:31 | opendevreview | Andre Aranha proposed openstack/nova master: Test setting the nova job to centos-9-stream https://review.opendev.org/c/openstack/nova/+/831844 | |
| 15:56:34 | sean-k-mooney | i need to add a few unit tests and a release note but i think ^ that is basically done | |
| 15:57:03 | sean-k-mooney | i still want to test this with real hardware however before i do and i proably need to update the docs too | |
| 15:58:30 | bauzas | \o | |
| 16:01:24 | sean-k-mooney | o/ | |
| 16:14:50 | opendevreview | sean mooney proposed openstack/nova stable/xena: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833411 | |
| 16:21:28 | opendevreview | sean mooney proposed openstack/nova stable/wallaby: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833435 | |
| 16:22:33 | opendevreview | sean mooney proposed openstack/nova stable/victoria: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833436 | |
| 16:23:28 | opendevreview | sean mooney proposed openstack/nova stable/ussuri: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833437 | |
| 16:23:48 | opendevreview | sean mooney proposed openstack/nova stable/train: reenable greendns in nova. https://review.opendev.org/c/openstack/nova/+/833438 | |
| 16:49:35 | opendevreview | Takashi Natsume proposed openstack/nova master: Update min supported service version for Zed https://review.opendev.org/c/openstack/nova/+/833440 | |
| 17:05:35 | zigo | I'm packaging nova RC1. I've seen that os-win is removed. Is that library useless now? | |
| 17:06:18 | zigo | Oh... setup.cfg ... :P | |
| 17:06:51 | opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for Zed https://review.opendev.org/c/openstack/nova/+/833441 | |
| 17:21:40 | sean-k-mooney | zigo: its an optional dep (and always was) so its no in extras | |
| 17:22:46 | sean-k-mooney | zigo: https://github.com/openstack/nova/commit/86d87be8db588cc3125d53cd92e271fb45b1a3aa for context | |
| 17:23:27 | sean-k-mooney | zigo: this was partly propeted by unmaintained packages that were breakign the gate | |
| 17:24:17 | zigo | sean-k-mooney: Is zVMCloudConnector completely gone? | |
| 17:24:28 | zigo | Or will it stay ... | |
| 17:25:03 | zigo | In other words: should I ask for its removal from Debian and erase all traces of it? | |
| 17:25:10 | gmann | dansmith: kashyap sean-k-mooney for detach failure/SSHable things, this last one needs to be merged, rescue negative test which this patch making SSHable was failing in reported bug. https://review.opendev.org/c/openstack/tempest/+/831608 | |
| 17:25:26 | gmann | it is not ready, need to debug on change failure though | |
| 17:29:34 | dansmith | gmann: ah are you saying that the sshable patches that already merged are working and that there are just a few remaining ones needing to be converted (in that patch)? | |
| 17:30:05 | sean-k-mooney | zigo: we still have it in tree https://github.com/openstack/nova/tree/master/nova/virt/zvm im not sure what its state is | |
| 17:30:48 | gmann | dansmith: patches merged are few volume detach are made SSH-able but the failing test in centos9-stream was rescue negative which is in-progress in 831608 | |
| 17:31:46 | gmann | dansmith: those merged were not failing, may be due to the wait between server create and detach operation call. in rescue negative timing were playing key role | |
| 17:32:38 | dansmith | gmann: okay the latest failure I'm looking at includes test_rescued_vm_detach_volume but there's another in there, which may or may not be related | |
| 17:32:46 | dansmith | but yeah good to know | |
| 17:34:16 | sean-k-mooney | dansmith: how oftten are you seeing the failure by the way | |
| 17:34:24 | sean-k-mooney | is it blockign the gate consitently | |
| 17:34:25 | gmann | dansmith: yeah that test and in negative test just try the detach and assert expected failure as detach cannot be done on rescue server but later this test does un-rescue server and detach in cleanup there it stuck | |
| 17:34:40 | dansmith | sean-k-mooney: on centos, this was one very common one we were suffering in the glance job | |
| 17:34:42 | gmann | * in that negative test | |
| 17:35:04 | sean-k-mooney | does glance need to test this? | |
| 17:35:04 | dansmith | gmann: do those tests actually ssh for some reason, or are we just using ssh to determine readiness? | |
| 17:35:34 | gmann | dansmith: just to check readiness | |
| 17:35:36 | dansmith | sean-k-mooney: yeah, this was a glance-cinder-multistore job which needs to run volume-related tests of course | |
| 17:35:46 | sean-k-mooney | some volume operation are certenly glance related but detach proably isnt | |
| 17:36:01 | dansmith | gmann: we could use the login prompt via console instead to reduce the need for secgroups, if that's hard for some reason | |
| 17:36:25 | dansmith | sean-k-mooney: it's a job that tests cinder-glance multistore arrangements | |
| 17:36:28 | gmann | dansmith: that is one try if that fix it. but there might be some other issue. we will see if that patch (once pass gate) can pass cento9 job too | |
| 17:36:46 | sean-k-mooney | dansmith: we removed usign the console in an eairler patch | |
| 17:36:49 | dansmith | sean-k-mooney: but the only reason we were running on centos was because we were trying to get a fips job and used one of our existing wide-coverage jobs to do that | |
| 17:37:01 | dansmith | gmann: ack | |
| 17:37:11 | gmann | yeah SSH-able was preferred than console check | |
| 17:37:25 | dansmith | it's definitely good, it's just more than required for this but fair enough | |
| 17:37:58 | gmann | let me debug sec group thing today or monday. I compared and it was same as other test doing but i might have missing something | |
| 17:38:17 | sean-k-mooney | i think the console check failed in a specific edgecase but i dont recall what it was exactly. | |
| 17:38:32 | dansmith | it's just out-of-band, so a little less fragile, | |