| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-26 | |||
| 10:36:37 | lyarwood | right so that's why it's in upper-constraints.txt | |
| 10:36:51 | bauzas | mmmm, ok | |
| 10:37:05 | bauzas | but then once we get 6.8.1 ? | |
| 10:37:26 | lyarwood | yeah I'm not sure if we can add !=6.8.0 in uc | |
| 10:37:36 | lyarwood | but I'd assume that would live somewhere in the requirements repo | |
| 10:38:03 | lyarwood | or we just move lc to 6.8.1 | |
| 10:38:09 | lyarwood | but that smells wrong | |
| 10:49:31 | bauzas | lyarwood: I was thinking on the nova req.txt file for not accepting 6.8.0 | |
| 10:50:09 | bauzas | something like https://github.com/openstack/nova/blob/master/requirements.txt#L41 | |
| 11:00:11 | lyarwood | bauzas: yeah, I note pypowervm is actually listed there so I wonder if we can get away with virt driver specific things in there | |
| 11:10:59 | bauzas | ah I see your point, you'd be afraid of having libvirt releases to be in the reqs file as it's not needed for all | |
| 11:11:06 | bauzas | then it's a good point | |
| 11:11:15 | bauzas | I dunno what to say honestly | |
| 12:56:41 | tobias-urdin | sean-k-mooney: if you have some seconds, can you see my last comment on https://review.opendev.org/#/c/741529/ | |
| 14:05:06 | tobias-urdin | sean-k-mooney: and new comment in bug https://bugs.launchpad.net/neutron/+bug/1815989 | |
| 14:05:06 | openstack | Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | |
| 14:17:50 | sean-k-mooney | im replying now | |
| 14:18:18 | sean-k-mooney | neutron is shoudl have finished wiring up the port before sending network-vif-plugged | |
| 14:18:32 | sean-k-mooney | waiting for the port binding to be activiated is not correct. | |
| 14:18:45 | sean-k-mooney | well partly | |
| 14:18:56 | sean-k-mooney | tobias-urdin: are you usuing post-copy | |
| 14:19:27 | sean-k-mooney | there is some ambiguity in the spec if the prot should be active before the port binding is active or not | |
| 14:20:12 | sean-k-mooney | but the semantics of network-vif-plugged is when its sent the vm shoudl have full network connectivity | |
| 14:21:39 | sean-k-mooney | tobias-urdin: that race is a differnt race by the way so we should not use the same bug to track it | |
| 14:32:43 | tobias-urdin | sean-k-mooney: after looking through the nova notifier in neutron, the port updates is subscribing to the BEFORE_RESPONSE event, so i assume that the event is sent before neutron-server sends the action out to the agents | |
| 14:32:54 | tobias-urdin | so it's not really send when the agents are done binding the port | |
| 14:33:25 | tobias-urdin | but i assume that's only when the port goes to active state, but not really is up by the agents | |
| 14:34:16 | tobias-urdin | the reason it works better for instance spawns is because a spawn of a instance takes some seconds, checking there is not really any neutron events that can be sent out when an agents is done with a port, that's what we would like | |
| 14:34:44 | tobias-urdin | https://opendev.org/openstack/neutron/src/branch/master/neutron/notifiers/nova.py#L102 | |
| 14:34:52 | tobias-urdin | https://opendev.org/openstack/neutron/src/branch/master/neutron/pecan_wsgi/hooks/notifier.py#L101 | |
| 14:38:07 | sean-k-mooney | tobias-urdin: the port shoudl have alreay been active | |
| 14:38:16 | sean-k-mooney | the port is in the active state on the souce host | |
| 14:38:28 | sean-k-mooney | so it a little ambigiuos | |
| 14:38:44 | sean-k-mooney | prior to rocky neutron would have considered it active | |
| 14:39:13 | sean-k-mooney | with the multipe port bidnign workflow we prebinding the port on the dest but that binding is not active | |
| 14:39:54 | sean-k-mooney | until post live migrate | |
| 14:40:41 | sean-k-mooney | we are expecting that the link state on the interface should be up but you could argue that it shold not be up until the binding is active | |
| 14:40:48 | sean-k-mooney | we disucssed that in the spec | |
| 14:40:56 | sean-k-mooney | ill have to compare to what we agreed | |
| 14:41:16 | dansmith | gibi: you were +2 on this initially, can you +W now that I've addressed the feedback? https://review.opendev.org/#/c/756534/ | |
| 14:46:03 | tobias-urdin | sean-k-mooney: that means the race is there, so the better way would be that neutron notifiers nova when the port is done on the destination node (i.e the neutron agent has fixed the port and it's ready for use) | |
| 14:46:30 | tobias-urdin | i can provide more debug information from the neutron ovs agent details if you want, i.e more lines than i posted in the launchpad bug | |
| 14:46:35 | sean-k-mooney | tobias-urdin: to be clear this is a seperate issue form the bug your commeting on | |
| 14:47:05 | tobias-urdin | sean-k-mooney: yeah, i understand that now, hence we will be running the patch is linked as a workaround for now | |
| 14:47:06 | sean-k-mooney | tobias-urdin: there is an implcit race in this logic by desgin | |
| 14:47:10 | tobias-urdin | or do you think there is a better workaround? | |
| 14:47:29 | tobias-urdin | i got a little big scared by your comment on that patch, did u see the question there? | |
| 14:47:36 | sean-k-mooney | it seam like you are hitting that implict race but its not clear we can ever fix that | |
| 14:47:51 | gibi | dansmith: looking... | |
| 14:48:51 | sean-k-mooney | tobias-urdin: the conclict/question basicely comes down to should neutron set the link state to active for inactive port bindings | |
| 14:50:11 | gibi | dansmith: done +2 +A | |
| 14:50:58 | tobias-urdin | sean-k-mooney: im thinking in the terms of neutron sending a network-vif-up to nova, so that nova can wait for the port to come up, but maybe the nova driver hands of the resume of a migrated instance to libvirt so it never handles the last part of the live-migration | |
| 14:51:23 | dansmith | gibi: thanks | |
| 14:51:37 | sean-k-mooney | when there was only one port binding pre rocky neutorn would wire up the port and sicne there was only one binding it would consider it active at that point | |
| 14:52:12 | tobias-urdin | sean-k-mooney: it's a pretty bad race though, since it's a complete outage for a vm until some traffic is generated from inside (which could be pretty much never) | |
| 14:52:28 | tobias-urdin | sean-k-mooney: what do you see as the best workaround? | |
| 14:52:32 | tobias-urdin | for now | |
| 14:52:50 | tobias-urdin | also my last comment on https://review.opendev.org/#/c/741529/ | |
| 14:52:53 | sean-k-mooney | tobias-urdin: are you using postcopy | |
| 14:53:52 | tobias-urdin | live_migration_permit_post_copy is unset so default (false) | |
| 14:54:06 | sean-k-mooney | tobias-urdin: so our libvirt/qemu team say that once you use the qemu commands it marks the instacne as tainted and they nolonger will support it | |
| 14:54:38 | sean-k-mooney | e.g. if you ever use raw qemu command vi libvirt you nolonger have support for that vm from redhat virt team | |
| 14:55:43 | sean-k-mooney | tobias-urdin: some qemu monitor command can break libvirt or currpt the guest | |
| 14:56:10 | sean-k-mooney | tobias-urdin: specificly some of the lowlevel block deivce commands | |
| 14:56:40 | tobias-urdin | sean-k-mooney: ok, i'm more curious the reasoning because, if the command im running adds a risk | |
| 14:56:52 | tobias-urdin | but maybe that more of a question for somebody on that team | |
| 14:56:53 | sean-k-mooney | that command does not | |
| 14:57:12 | sean-k-mooney | but we dont want ot say yes and no on a per command basis | |
| 14:57:42 | sean-k-mooney | that said danpb said adding a libvirt api for this would be a resonable thing to do | |
| 14:58:39 | tobias-urdin | sean-k-mooney: i c, so the workaround is to stop using the new multiple port binding feature (which means patching code) or using my patch (which means patching code) | |
| 14:58:55 | sean-k-mooney | tobias-urdin: no there is a longer race in that case | |
| 14:59:08 | sean-k-mooney | tobias-urdin: i think neutron change teh behavior in the l2 agent | |
| 14:59:24 | sean-k-mooney | to require that it has an active portbining on the host | |
| 14:59:27 | tobias-urdin | so if no changes are merged upstream, i still have to fix it downstream in our running nova/neutron code by patching it | |
| 14:59:58 | sean-k-mooney | i think this is something we need to talk about with the neutron folks | |
| 15:00:21 | sean-k-mooney | and come up with a supportable way forward | |
| 15:00:29 | tobias-urdin | sean-k-mooney: roger that, thanks for your time! | |
| 15:40:12 | gibi | stephenfin: mypy is killing me. :) these tests mypy tests work until I add a simple member function to the class here https://github.com/gibizer/ovo-mypy-plugin/blob/main/ovo_mypy_plugin/tests/test.py#L54 | |
| 15:40:51 | gibi | stephenfin: somehow it makes the fields.IntegerField name undefined in the scope of the fields dict | |
| 15:41:00 | gibi | craaazyyy | |
| 15:59:18 | bauzas | gibi: one of my points => if you need more time for providing mypy types to all the methods than the time you need to review new changes, then why should we supporting it ? | |
| 15:59:38 | bauzas | we have gerrit for this :) | |
| 16:00:18 | lbragstad | gmann o/ is there a way for project admins to discover host information? | |
| 16:00:19 | lbragstad | https://opendev.org/openstack/nova/src/branch/master/nova/policies/servers.py#L178-L190 | |
| 16:01:30 | lbragstad | gmann i'm wondering if a deployment opted into using the new defaults for nova, would that expose an API to project-administrators that they shouldn't have? | |
| 16:01:47 | lbragstad | or if it's not really useful since they can't discover compute hosts anyway | |
| 16:05:09 | bauzas | anyway, /me needs to leave \o | |
| 16:05:14 | gibi | bauzas: I'm working on a mypy plugin that understands oslo versionedobjects and inject the dynamic field information to the static analyzer. If I can make it work then it potentially opens up the possibility to typecheck a lot of nova code tight to o.vos | |
| 16:05:51 | bauzas | well ok | |
| 16:06:12 | bauzas | but honestly, I can still review on any object modification and look at this | |
| 16:06:29 | gibi | bauzas: sure it is not to replace code review | |
| 16:06:35 | bauzas | it's simple for o.vo objects | |
| 16:06:35 | sean-k-mooney | lbragstad: we dont really have project admins | |
| 16:06:45 | bauzas | gibi: but fair enough | |
| 16:06:46 | sean-k-mooney | lbragstad: project admins are admins | |
| 16:06:58 | lbragstad | sean-k-mooney ok - i was just looking through the policy and noticed that comment | |
| 16:07:13 | lbragstad | it looks like only system administrators can list and view hypervisors and services | |
| 16:07:27 | sean-k-mooney | i dont think we make a distinciton | |
| 16:07:48 | sean-k-mooney | lbragstad: between a system admin and any other type of admin | |
| 16:08:02 | sean-k-mooney | unless that has changed with the scopes thing | |