Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-10-26
14:05:06 openstack Launchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney)
14:05:06 tobias-urdin sean-k-mooney: and new comment in bug https://bugs.launchpad.net/neutron/+bug/1815989
14:17:50 sean-k-mooney im replying now
14:18:18 sean-k-mooney neutron is shoudl have finished wiring up the port before sending network-vif-plugged
14:18:32 sean-k-mooney waiting for the port binding to be activiated is not correct.
14:18:45 sean-k-mooney well partly
14:18:56 sean-k-mooney tobias-urdin: are you usuing post-copy
14:19:27 sean-k-mooney there is some ambiguity in the spec if the prot should be active before the port binding is active or not
14:20:12 sean-k-mooney but the semantics of network-vif-plugged is when its sent the vm shoudl have full network connectivity
14:21:39 sean-k-mooney tobias-urdin: that race is a differnt race by the way so we should not use the same bug to track it
14:32:43 tobias-urdin sean-k-mooney: after looking through the nova notifier in neutron, the port updates is subscribing to the BEFORE_RESPONSE event, so i assume that the event is sent before neutron-server sends the action out to the agents
14:32:54 tobias-urdin so it's not really send when the agents are done binding the port
14:33:25 tobias-urdin but i assume that's only when the port goes to active state, but not really is up by the agents
14:34:16 tobias-urdin the reason it works better for instance spawns is because a spawn of a instance takes some seconds, checking there is not really any neutron events that can be sent out when an agents is done with a port, that's what we would like
14:34:44 tobias-urdin https://opendev.org/openstack/neutron/src/branch/master/neutron/notifiers/nova.py#L102
14:34:52 tobias-urdin https://opendev.org/openstack/neutron/src/branch/master/neutron/pecan_wsgi/hooks/notifier.py#L101
14:38:07 sean-k-mooney tobias-urdin: the port shoudl have alreay been active
14:38:16 sean-k-mooney the port is in the active state on the souce host
14:38:28 sean-k-mooney so it a little ambigiuos
14:38:44 sean-k-mooney prior to rocky neutron would have considered it active
14:39:13 sean-k-mooney with the multipe port bidnign workflow we prebinding the port on the dest but that binding is not active
14:39:54 sean-k-mooney until post live migrate
14:40:41 sean-k-mooney we are expecting that the link state on the interface should be up but you could argue that it shold not be up until the binding is active
14:40:48 sean-k-mooney we disucssed that in the spec
14:40:56 sean-k-mooney ill have to compare to what we agreed
14:41:16 dansmith gibi: you were +2 on this initially, can you +W now that I've addressed the feedback? https://review.opendev.org/#/c/756534/
14:46:03 tobias-urdin sean-k-mooney: that means the race is there, so the better way would be that neutron notifiers nova when the port is done on the destination node (i.e the neutron agent has fixed the port and it's ready for use)
14:46:30 tobias-urdin i can provide more debug information from the neutron ovs agent details if you want, i.e more lines than i posted in the launchpad bug
14:46:35 sean-k-mooney tobias-urdin: to be clear this is a seperate issue form the bug your commeting on
14:47:05 tobias-urdin sean-k-mooney: yeah, i understand that now, hence we will be running the patch is linked as a workaround for now
14:47:06 sean-k-mooney tobias-urdin: there is an implcit race in this logic by desgin
14:47:10 tobias-urdin or do you think there is a better workaround?
14:47:29 tobias-urdin i got a little big scared by your comment on that patch, did u see the question there?
14:47:36 sean-k-mooney it seam like you are hitting that implict race but its not clear we can ever fix that
14:47:51 gibi dansmith: looking...
14:48:51 sean-k-mooney tobias-urdin: the conclict/question basicely comes down to should neutron set the link state to active for inactive port bindings
14:50:11 gibi dansmith: done +2 +A
14:50:58 tobias-urdin sean-k-mooney: im thinking in the terms of neutron sending a network-vif-up to nova, so that nova can wait for the port to come up, but maybe the nova driver hands of the resume of a migrated instance to libvirt so it never handles the last part of the live-migration
14:51:23 dansmith gibi: thanks
14:51:37 sean-k-mooney when there was only one port binding pre rocky neutorn would wire up the port and sicne there was only one binding it would consider it active at that point
14:52:12 tobias-urdin sean-k-mooney: it's a pretty bad race though, since it's a complete outage for a vm until some traffic is generated from inside (which could be pretty much never)
14:52:28 tobias-urdin sean-k-mooney: what do you see as the best workaround?
14:52:32 tobias-urdin for now
14:52:50 tobias-urdin also my last comment on https://review.opendev.org/#/c/741529/
14:52:53 sean-k-mooney tobias-urdin: are you using postcopy
14:53:52 tobias-urdin live_migration_permit_post_copy is unset so default (false)
14:54:06 sean-k-mooney tobias-urdin: so our libvirt/qemu team say that once you use the qemu commands it marks the instacne as tainted and they nolonger will support it
14:54:38 sean-k-mooney e.g. if you ever use raw qemu command vi libvirt you nolonger have support for that vm from redhat virt team
14:55:43 sean-k-mooney tobias-urdin: some qemu monitor command can break libvirt or currpt the guest
14:56:10 sean-k-mooney tobias-urdin: specificly some of the lowlevel block deivce commands
14:56:40 tobias-urdin sean-k-mooney: ok, i'm more curious the reasoning because, if the command im running adds a risk
14:56:52 tobias-urdin but maybe that more of a question for somebody on that team
14:56:53 sean-k-mooney that command does not
14:57:12 sean-k-mooney but we dont want ot say yes and no on a per command basis
14:57:42 sean-k-mooney that said danpb said adding a libvirt api for this would be a resonable thing to do
14:58:39 tobias-urdin sean-k-mooney: i c, so the workaround is to stop using the new multiple port binding feature (which means patching code) or using my patch (which means patching code)
14:58:55 sean-k-mooney tobias-urdin: no there is a longer race in that case
14:59:08 sean-k-mooney tobias-urdin: i think neutron change teh behavior in the l2 agent
14:59:24 sean-k-mooney to require that it has an active portbining on the host
14:59:27 tobias-urdin so if no changes are merged upstream, i still have to fix it downstream in our running nova/neutron code by patching it
14:59:58 sean-k-mooney i think this is something we need to talk about with the neutron folks
15:00:21 sean-k-mooney and come up with a supportable way forward
15:00:29 tobias-urdin sean-k-mooney: roger that, thanks for your time!
15:40:12 gibi stephenfin: mypy is killing me. :) these tests mypy tests work until I add a simple member function to the class here https://github.com/gibizer/ovo-mypy-plugin/blob/main/ovo_mypy_plugin/tests/test.py#L54
15:40:51 gibi stephenfin: somehow it makes the fields.IntegerField name undefined in the scope of the fields dict
15:41:00 gibi craaazyyy
15:59:18 bauzas gibi: one of my points => if you need more time for providing mypy types to all the methods than the time you need to review new changes, then why should we supporting it ?
15:59:38 bauzas we have gerrit for this :)
16:00:18 lbragstad gmann o/ is there a way for project admins to discover host information?
16:00:19 lbragstad https://opendev.org/openstack/nova/src/branch/master/nova/policies/servers.py#L178-L190
16:01:30 lbragstad gmann i'm wondering if a deployment opted into using the new defaults for nova, would that expose an API to project-administrators that they shouldn't have?
16:01:47 lbragstad or if it's not really useful since they can't discover compute hosts anyway
16:05:09 bauzas anyway, /me needs to leave \o
16:05:14 gibi bauzas: I'm working on a mypy plugin that understands oslo versionedobjects and inject the dynamic field information to the static analyzer. If I can make it work then it potentially opens up the possibility to typecheck a lot of nova code tight to o.vos
16:05:51 bauzas well ok
16:06:12 bauzas but honestly, I can still review on any object modification and look at this
16:06:29 gibi bauzas: sure it is not to replace code review
16:06:35 sean-k-mooney lbragstad: we dont really have project admins
16:06:35 bauzas it's simple for o.vo objects
16:06:45 bauzas gibi: but fair enough
16:06:46 sean-k-mooney lbragstad: project admins are admins
16:06:58 lbragstad sean-k-mooney ok - i was just looking through the policy and noticed that comment
16:07:13 lbragstad it looks like only system administrators can list and view hypervisors and services
16:07:27 sean-k-mooney i dont think we make a distinciton
16:07:48 sean-k-mooney lbragstad: between a system admin and any other type of admin
16:08:02 sean-k-mooney unless that has changed with the scopes thing
16:08:27 lbragstad https://opendev.org/openstack/nova/src/branch/master/nova/policies/hypervisors.py#L37
16:08:27 lbragstad looks like it did
16:08:32 lbragstad https://opendev.org/openstack/nova/src/branch/master/nova/policies/base.py#L111
16:18:51 lbragstad sean-k-mooney i think i figured it out - thanks
16:22:40 openstackgerrit Kashyap Chamarthy proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" for Wallaby https://review.opendev.org/759731
17:53:35 gmann lbragstad: yeah that is one of the todo to make system admin to create server for projects so that host info etc can be passed in.
17:54:12 lbragstad gmann got it
17:54:13 gmann lbragstad: project admin can get host info of existing server via GET /servers
17:55:00 lbragstad any project user can do that, but only project administrators can exploit creating a server on a specific host, right?
17:55:02 gmann and that way they can use current default which is PROJECT_ADMIN. not perfect default for now but something to improve with microversion
17:55:12 gmann lbragstad: yes
17:55:27 lbragstad ok - cool
18:00:55 admin0 hi guys .. how to fix placement database .. a compute node with instances has now a different uuid than what it was originally added as
18:01:29 admin0 so when nova-compute runs, it says unable to add host as name already exists

Earlier   Later