| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-26 | |||
| 09:20:39 | kashyap | gibi: Oh dear ... that "full house" part sounds like a petri dish to let the virus "flourish" | |
| 09:22:02 | frickler | gibi: yes, except it doesn't seem to be related to either py36 nor libvirt, but eventlet, as the jobs are passing with the cap melwitt made | |
| 09:25:13 | gibi | frickler: do you mean https://review.opendev.org/#/c/759552 as the cap? this caps libvirt-python | |
| 09:29:37 | bauzas | frickler: I thought melwitt reported the problem | |
| 09:30:04 | lyarwood | FWIW I'm rasing the libvirt-python issue with danpb downstream now | |
| 09:30:05 | frickler | gibi: oh, indeed, I didn't notice there was a new version | |
| 09:30:24 | bauzas | frickler: gibi: https://bugs.launchpad.net/nova/+bug/1901383 | |
| 09:30:24 | openstack | Launchpad bug 1901383 in OpenStack Compute (nova) "nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0" [Critical,New] | |
| 09:30:33 | lyarwood | we might want to land the revert anyway while we get a bug filed against libvirt-python etc | |
| 09:30:38 | bauzas | oh man, I'm burned by gibi | |
| 09:32:25 | kashyap | Yeah, perhaps copy/pasting (with credit) what Mel wrote in here - https://gitlab.com/libvirt/libvirt-python/-/issues | |
| 09:57:48 | gibi | lyarwood: I agree | |
| 10:29:13 | lyarwood | gibi: any objections if I pushed an updated PS for the revert? | |
| 10:29:38 | bauzas | lyarwood: I'm also good with | |
| 10:29:56 | lyarwood | ack, lets do that then | |
| 10:32:10 | lyarwood | gibi / bauzas ; okay done - https://review.opendev.org/759552 | |
| 10:32:29 | lyarwood | melwitt: ^ btw once you're online, we are going to push ahead with the revert to unblock the gate | |
| 10:33:56 | bauzas | lyarwood: also, we should make sure that libvirt-python==6.8.0 shouldn't be used in nova | |
| 10:34:30 | lyarwood | bauzas: it's not | |
| 10:34:44 | lyarwood | bauzas: oh wait, you mean an actual startup check? | |
| 10:35:00 | bauzas | no | |
| 10:35:10 | lyarwood | bauzas: ah just a requirements check | |
| 10:35:12 | bauzas | actually https://github.com/openstack/nova/blob/master/requirements.txt | |
| 10:35:15 | lyarwood | yeah | |
| 10:35:26 | lyarwood | hmm well | |
| 10:35:29 | lyarwood | the issue there is | |
| 10:35:34 | bauzas | something like libvirt-python!=6.8.0 | |
| 10:35:37 | lyarwood | that not all drivers actually need it | |
| 10:35:47 | lyarwood | so I'm not sure that we can put it in our requirements | |
| 10:35:56 | bauzas | well | |
| 10:36:08 | bauzas | sure, but then all the drivers wouldn't need 6.8.0, right? | |
| 10:36:37 | lyarwood | right so that's why it's in upper-constraints.txt | |
| 10:36:51 | bauzas | mmmm, ok | |
| 10:37:05 | bauzas | but then once we get 6.8.1 ? | |
| 10:37:26 | lyarwood | yeah I'm not sure if we can add !=6.8.0 in uc | |
| 10:37:36 | lyarwood | but I'd assume that would live somewhere in the requirements repo | |
| 10:38:03 | lyarwood | or we just move lc to 6.8.1 | |
| 10:38:09 | lyarwood | but that smells wrong | |
| 10:49:31 | bauzas | lyarwood: I was thinking on the nova req.txt file for not accepting 6.8.0 | |
| 10:50:09 | bauzas | something like https://github.com/openstack/nova/blob/master/requirements.txt#L41 | |
| 11:00:11 | lyarwood | bauzas: yeah, I note pypowervm is actually listed there so I wonder if we can get away with virt driver specific things in there | |
| 11:10:59 | bauzas | ah I see your point, you'd be afraid of having libvirt releases to be in the reqs file as it's not needed for all | |
| 11:11:06 | bauzas | then it's a good point | |
| 11:11:15 | bauzas | I dunno what to say honestly | |
| 12:56:41 | tobias-urdin | sean-k-mooney: if you have some seconds, can you see my last comment on https://review.opendev.org/#/c/741529/ | |
| 14:05:06 | tobias-urdin | sean-k-mooney: and new comment in bug https://bugs.launchpad.net/neutron/+bug/1815989 | |
| 14:05:06 | openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 14:17:50 | sean-k-mooney | im replying now | |
| 14:18:18 | sean-k-mooney | neutron is shoudl have finished wiring up the port before sending network-vif-plugged | |
| 14:18:32 | sean-k-mooney | waiting for the port binding to be activiated is not correct. | |
| 14:18:45 | sean-k-mooney | well partly | |
| 14:18:56 | sean-k-mooney | tobias-urdin: are you usuing post-copy | |
| 14:19:27 | sean-k-mooney | there is some ambiguity in the spec if the prot should be active before the port binding is active or not | |
| 14:20:12 | sean-k-mooney | but the semantics of network-vif-plugged is when its sent the vm shoudl have full network connectivity | |
| 14:21:39 | sean-k-mooney | tobias-urdin: that race is a differnt race by the way so we should not use the same bug to track it | |
| 14:32:43 | tobias-urdin | sean-k-mooney: after looking through the nova notifier in neutron, the port updates is subscribing to the BEFORE_RESPONSE event, so i assume that the event is sent before neutron-server sends the action out to the agents | |
| 14:32:54 | tobias-urdin | so it's not really send when the agents are done binding the port | |
| 14:33:25 | tobias-urdin | but i assume that's only when the port goes to active state, but not really is up by the agents | |
| 14:34:16 | tobias-urdin | the reason it works better for instance spawns is because a spawn of a instance takes some seconds, checking there is not really any neutron events that can be sent out when an agents is done with a port, that's what we would like | |
| 14:34:44 | tobias-urdin | https://opendev.org/openstack/neutron/src/branch/master/neutron/notifiers/nova.py#L102 | |
| 14:34:52 | tobias-urdin | https://opendev.org/openstack/neutron/src/branch/master/neutron/pecan_wsgi/hooks/notifier.py#L101 | |
| 14:38:07 | sean-k-mooney | tobias-urdin: the port shoudl have alreay been active | |
| 14:38:16 | sean-k-mooney | the port is in the active state on the souce host | |
| 14:38:28 | sean-k-mooney | so it a little ambigiuos | |
| 14:38:44 | sean-k-mooney | prior to rocky neutron would have considered it active | |
| 14:39:13 | sean-k-mooney | with the multipe port bidnign workflow we prebinding the port on the dest but that binding is not active | |
| 14:39:54 | sean-k-mooney | until post live migrate | |
| 14:40:41 | sean-k-mooney | we are expecting that the link state on the interface should be up but you could argue that it shold not be up until the binding is active | |
| 14:40:48 | sean-k-mooney | we disucssed that in the spec | |
| 14:40:56 | sean-k-mooney | ill have to compare to what we agreed | |
| 14:41:16 | dansmith | gibi: you were +2 on this initially, can you +W now that I've addressed the feedback? https://review.opendev.org/#/c/756534/ | |
| 14:46:03 | tobias-urdin | sean-k-mooney: that means the race is there, so the better way would be that neutron notifiers nova when the port is done on the destination node (i.e the neutron agent has fixed the port and it's ready for use) | |
| 14:46:30 | tobias-urdin | i can provide more debug information from the neutron ovs agent details if you want, i.e more lines than i posted in the launchpad bug | |
| 14:46:35 | sean-k-mooney | tobias-urdin: to be clear this is a seperate issue form the bug your commeting on | |
| 14:47:05 | tobias-urdin | sean-k-mooney: yeah, i understand that now, hence we will be running the patch is linked as a workaround for now | |
| 14:47:06 | sean-k-mooney | tobias-urdin: there is an implcit race in this logic by desgin | |
| 14:47:10 | tobias-urdin | or do you think there is a better workaround? | |
| 14:47:29 | tobias-urdin | i got a little big scared by your comment on that patch, did u see the question there? | |
| 14:47:36 | sean-k-mooney | it seam like you are hitting that implict race but its not clear we can ever fix that | |
| 14:47:51 | gibi | dansmith: looking... | |
| 14:48:51 | sean-k-mooney | tobias-urdin: the conclict/question basicely comes down to should neutron set the link state to active for inactive port bindings | |
| 14:50:11 | gibi | dansmith: done +2 +A | |
| 14:50:58 | tobias-urdin | sean-k-mooney: im thinking in the terms of neutron sending a network-vif-up to nova, so that nova can wait for the port to come up, but maybe the nova driver hands of the resume of a migrated instance to libvirt so it never handles the last part of the live-migration | |
| 14:51:23 | dansmith | gibi: thanks | |
| 14:51:37 | sean-k-mooney | when there was only one port binding pre rocky neutorn would wire up the port and sicne there was only one binding it would consider it active at that point | |
| 14:52:12 | tobias-urdin | sean-k-mooney: it's a pretty bad race though, since it's a complete outage for a vm until some traffic is generated from inside (which could be pretty much never) | |
| 14:52:28 | tobias-urdin | sean-k-mooney: what do you see as the best workaround? | |
| 14:52:32 | tobias-urdin | for now | |
| 14:52:50 | tobias-urdin | also my last comment on https://review.opendev.org/#/c/741529/ | |
| 14:52:53 | sean-k-mooney | tobias-urdin: are you using postcopy | |
| 14:53:52 | tobias-urdin | live_migration_permit_post_copy is unset so default (false) | |
| 14:54:06 | sean-k-mooney | tobias-urdin: so our libvirt/qemu team say that once you use the qemu commands it marks the instacne as tainted and they nolonger will support it | |
| 14:54:38 | sean-k-mooney | e.g. if you ever use raw qemu command vi libvirt you nolonger have support for that vm from redhat virt team | |
| 14:55:43 | sean-k-mooney | tobias-urdin: some qemu monitor command can break libvirt or currpt the guest | |
| 14:56:10 | sean-k-mooney | tobias-urdin: specificly some of the lowlevel block deivce commands | |
| 14:56:40 | tobias-urdin | sean-k-mooney: ok, i'm more curious the reasoning because, if the command im running adds a risk | |
| 14:56:52 | tobias-urdin | but maybe that more of a question for somebody on that team | |
| 14:56:53 | sean-k-mooney | that command does not | |
| 14:57:12 | sean-k-mooney | but we dont want ot say yes and no on a per command basis | |
| 14:57:42 | sean-k-mooney | that said danpb said adding a libvirt api for this would be a resonable thing to do | |