| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-26 | |||
| 09:04:40 | bauzas | :p | |
| 09:05:30 | bauzas | at least the coffee is good :p | |
| 09:05:39 | gibi | it was raining the whole weekend here and the TZ change hit here too. Our governent now tries not to do anyithing about the virus, so even though the numbers are raising rapidly nobody talks about lockdown | |
| 09:07:00 | bauzas | gibi: nw, I was joking ;-) | |
| 09:07:27 | gibi | it could be worse :) | |
| 09:07:50 | bauzas | indeed ;) | |
| 09:08:13 | kashyap | gibi: Yeah, wholesale lockdown is complex :-( | |
| 09:09:29 | kashyap | gibi: One the one side, it is effective at controlling the spread of virus; on the other, complete sectors of economies become a train wreck that is Jupiter-sized | |
| 09:09:51 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Support vGPU management by Cyborg https://review.opendev.org/750116 | |
| 09:13:42 | gibi | kashyap: we had a controlled lockdown during spring, we have nothing right now (exepct some lukewarm rule about masks). I'm not an economist but I imagine there is a lot of possible actions between full lockdown and nothing | |
| 09:14:36 | kashyap | gibi: Yeah, I hear you; controlled lockdown makes sense. | |
| 09:14:41 | gibi | e.g. we had controlled timeslots for groceries during sping. | |
| 09:14:47 | gibi | spring | |
| 09:15:10 | frickler | bauzas: gibi: do you have a bug report already for the eventlet update issue (https://review.opendev.org/759506)? seeing this coming up in multiple channels now because it seems to break almost everyone | |
| 09:15:14 | kashyap | (Full lockdown is also a "policital suicide") | |
| 09:16:20 | gibi | frickler: I was on PTO on Friday I need to look back what happened there | |
| 09:16:37 | gibi | kashyap: I agree on the full lockdown thing | |
| 09:17:27 | kashyap | gibi: Right now, here in Belgium is a "controlled lockdown" - all cafes/bars/restaurants closed for a month; no going out from midnight to 05:00 AM (it's even stricter in Brussels); etc. | |
| 09:18:09 | gibi | frickler: we have https://review.opendev.org/#/q/topic:dnm-eventlet-revert+(status:open+OR+status:merged) but I don't see a bugreport | |
| 09:18:57 | openstackgerrit | Wenping Song proposed openstack/nova-specs master: Support vGPU management by Cyborg https://review.opendev.org/750116 | |
| 09:19:12 | gibi | kashyap: that sounsd something I can welcome here too. We have football games with full house here, crazy | |
| 09:19:58 | openstack | Launchpad bug 1901383 in OpenStack Compute (nova) "nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0" [Critical,New] | |
| 09:19:58 | gibi | frickler: is this the bug? https://bugs.launchpad.net/nova/+bug/1901383 | |
| 09:20:39 | kashyap | gibi: Oh dear ... that "full house" part sounds like a petri dish to let the virus "flourish" | |
| 09:22:02 | frickler | gibi: yes, except it doesn't seem to be related to either py36 nor libvirt, but eventlet, as the jobs are passing with the cap melwitt made | |
| 09:25:13 | gibi | frickler: do you mean https://review.opendev.org/#/c/759552 as the cap? this caps libvirt-python | |
| 09:29:37 | bauzas | frickler: I thought melwitt reported the problem | |
| 09:30:04 | lyarwood | FWIW I'm rasing the libvirt-python issue with danpb downstream now | |
| 09:30:05 | frickler | gibi: oh, indeed, I didn't notice there was a new version | |
| 09:30:24 | openstack | Launchpad bug 1901383 in OpenStack Compute (nova) "nova-compute fails with "TypeError: Parameterized generics cannot be used with class" with python 3.6 and libvirt-python 6.8.0" [Critical,New] | |
| 09:30:24 | bauzas | frickler: gibi: https://bugs.launchpad.net/nova/+bug/1901383 | |
| 09:30:33 | lyarwood | we might want to land the revert anyway while we get a bug filed against libvirt-python etc | |
| 09:30:38 | bauzas | oh man, I'm burned by gibi | |
| 09:32:25 | kashyap | Yeah, perhaps copy/pasting (with credit) what Mel wrote in here - https://gitlab.com/libvirt/libvirt-python/-/issues | |
| 09:57:48 | gibi | lyarwood: I agree | |
| 10:29:13 | lyarwood | gibi: any objections if I pushed an updated PS for the revert? | |
| 10:29:38 | bauzas | lyarwood: I'm also good with | |
| 10:29:56 | lyarwood | ack, lets do that then | |
| 10:32:10 | lyarwood | gibi / bauzas ; okay done - https://review.opendev.org/759552 | |
| 10:32:29 | lyarwood | melwitt: ^ btw once you're online, we are going to push ahead with the revert to unblock the gate | |
| 10:33:56 | bauzas | lyarwood: also, we should make sure that libvirt-python==6.8.0 shouldn't be used in nova | |
| 10:34:30 | lyarwood | bauzas: it's not | |
| 10:34:44 | lyarwood | bauzas: oh wait, you mean an actual startup check? | |
| 10:35:00 | bauzas | no | |
| 10:35:10 | lyarwood | bauzas: ah just a requirements check | |
| 10:35:12 | bauzas | actually https://github.com/openstack/nova/blob/master/requirements.txt | |
| 10:35:15 | lyarwood | yeah | |
| 10:35:26 | lyarwood | hmm well | |
| 10:35:29 | lyarwood | the issue there is | |
| 10:35:34 | bauzas | something like libvirt-python!=6.8.0 | |
| 10:35:37 | lyarwood | that not all drivers actually need it | |
| 10:35:47 | lyarwood | so I'm not sure that we can put it in our requirements | |
| 10:35:56 | bauzas | well | |
| 10:36:08 | bauzas | sure, but then all the drivers wouldn't need 6.8.0, right? | |
| 10:36:37 | lyarwood | right so that's why it's in upper-constraints.txt | |
| 10:36:51 | bauzas | mmmm, ok | |
| 10:37:05 | bauzas | but then once we get 6.8.1 ? | |
| 10:37:26 | lyarwood | yeah I'm not sure if we can add !=6.8.0 in uc | |
| 10:37:36 | lyarwood | but I'd assume that would live somewhere in the requirements repo | |
| 10:38:03 | lyarwood | or we just move lc to 6.8.1 | |
| 10:38:09 | lyarwood | but that smells wrong | |
| 10:49:31 | bauzas | lyarwood: I was thinking on the nova req.txt file for not accepting 6.8.0 | |
| 10:50:09 | bauzas | something like https://github.com/openstack/nova/blob/master/requirements.txt#L41 | |
| 11:00:11 | lyarwood | bauzas: yeah, I note pypowervm is actually listed there so I wonder if we can get away with virt driver specific things in there | |
| 11:10:59 | bauzas | ah I see your point, you'd be afraid of having libvirt releases to be in the reqs file as it's not needed for all | |
| 11:11:06 | bauzas | then it's a good point | |
| 11:11:15 | bauzas | I dunno what to say honestly | |
| 12:56:41 | tobias-urdin | sean-k-mooney: if you have some seconds, can you see my last comment on https://review.opendev.org/#/c/741529/ | |
| 14:05:06 | openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 14:05:06 | tobias-urdin | sean-k-mooney: and new comment in bug https://bugs.launchpad.net/neutron/+bug/1815989 | |
| 14:17:50 | sean-k-mooney | im replying now | |
| 14:18:18 | sean-k-mooney | neutron is shoudl have finished wiring up the port before sending network-vif-plugged | |
| 14:18:32 | sean-k-mooney | waiting for the port binding to be activiated is not correct. | |
| 14:18:45 | sean-k-mooney | well partly | |
| 14:18:56 | sean-k-mooney | tobias-urdin: are you usuing post-copy | |
| 14:19:27 | sean-k-mooney | there is some ambiguity in the spec if the prot should be active before the port binding is active or not | |
| 14:20:12 | sean-k-mooney | but the semantics of network-vif-plugged is when its sent the vm shoudl have full network connectivity | |
| 14:21:39 | sean-k-mooney | tobias-urdin: that race is a differnt race by the way so we should not use the same bug to track it | |
| 14:32:43 | tobias-urdin | sean-k-mooney: after looking through the nova notifier in neutron, the port updates is subscribing to the BEFORE_RESPONSE event, so i assume that the event is sent before neutron-server sends the action out to the agents | |
| 14:32:54 | tobias-urdin | so it's not really send when the agents are done binding the port | |
| 14:33:25 | tobias-urdin | but i assume that's only when the port goes to active state, but not really is up by the agents | |
| 14:34:16 | tobias-urdin | the reason it works better for instance spawns is because a spawn of a instance takes some seconds, checking there is not really any neutron events that can be sent out when an agents is done with a port, that's what we would like | |
| 14:34:44 | tobias-urdin | https://opendev.org/openstack/neutron/src/branch/master/neutron/notifiers/nova.py#L102 | |
| 14:34:52 | tobias-urdin | https://opendev.org/openstack/neutron/src/branch/master/neutron/pecan_wsgi/hooks/notifier.py#L101 | |
| 14:38:07 | sean-k-mooney | tobias-urdin: the port shoudl have alreay been active | |
| 14:38:16 | sean-k-mooney | the port is in the active state on the souce host | |
| 14:38:28 | sean-k-mooney | so it a little ambigiuos | |
| 14:38:44 | sean-k-mooney | prior to rocky neutron would have considered it active | |
| 14:39:13 | sean-k-mooney | with the multipe port bidnign workflow we prebinding the port on the dest but that binding is not active | |
| 14:39:54 | sean-k-mooney | until post live migrate | |
| 14:40:41 | sean-k-mooney | we are expecting that the link state on the interface should be up but you could argue that it shold not be up until the binding is active | |
| 14:40:48 | sean-k-mooney | we disucssed that in the spec | |
| 14:40:56 | sean-k-mooney | ill have to compare to what we agreed | |
| 14:41:16 | dansmith | gibi: you were +2 on this initially, can you +W now that I've addressed the feedback? https://review.opendev.org/#/c/756534/ | |
| 14:46:03 | tobias-urdin | sean-k-mooney: that means the race is there, so the better way would be that neutron notifiers nova when the port is done on the destination node (i.e the neutron agent has fixed the port and it's ready for use) | |
| 14:46:30 | tobias-urdin | i can provide more debug information from the neutron ovs agent details if you want, i.e more lines than i posted in the launchpad bug | |
| 14:46:35 | sean-k-mooney | tobias-urdin: to be clear this is a seperate issue form the bug your commeting on | |
| 14:47:05 | tobias-urdin | sean-k-mooney: yeah, i understand that now, hence we will be running the patch is linked as a workaround for now | |
| 14:47:06 | sean-k-mooney | tobias-urdin: there is an implcit race in this logic by desgin | |
| 14:47:10 | tobias-urdin | or do you think there is a better workaround? | |