| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-03 | |||
| 15:13:06 | mriedem | otherwise nova-compute on the source host is going to continue thinking it owns the instance | |
| 15:14:02 | s10 | mriedem: thank you | |
| 15:14:24 | nicolasbock | Ok, other than that this sounds mildly scary, could you give me a pointer where I find that value mriedem ? | |
| 15:14:53 | mriedem | do you know where the guest is actively running now? | |
| 15:14:57 | openstackgerrit | Merged openstack/nova stable/ocata: [Stable Only] Add amd-ssbd and amd-no-ssb CPU flags https://review.openstack.org/607296 | |
| 15:15:03 | nicolasbock | Yes | |
| 15:15:05 | mriedem | it should be in the last live-migration migration record for the instance | |
| 15:15:14 | nicolasbock | Ok | |
| 15:15:16 | mriedem | well then you just update the table record in the nova db | |
| 15:16:00 | mriedem | update instances set host=<host> where uuid=<instance uuid>; | |
| 15:16:01 | nicolasbock | Ok, sounds so straightforward when you put it like that ;) | |
| 15:16:02 | nicolasbock | I'll give it a try | |
| 15:16:08 | nicolasbock | Thanks! | |
| 15:19:04 | openstackgerrit | Vlad Gusev proposed openstack/nova master: Not set instance to ERROR if set_admin_password failed https://review.openstack.org/555160 | |
| 15:19:04 | sean-k-mooney | mriedem: out of interest what would happen if you tried to do a hard reboot or other lifecycle action on a vm in this state with the wrong host set | |
| 15:19:26 | sean-k-mooney | woudl it repare the instance or try to start it on the wrong host? | |
| 15:21:06 | mriedem | it would try to start it on the wrong host | |
| 15:21:44 | mriedem | i do'nt know what would then happen - would you get the same instance running on two hosts? or a domain not found from the wrong host when trying to reboot it? | |
| 15:21:47 | mriedem | i'd hope the latter | |
| 15:21:51 | sean-k-mooney | right which if it was using share storage could lead to data curoption correct | |
| 15:22:10 | mriedem | well i'd hope reboot would fail if the guest isn't actually on the hypervisor | |
| 15:22:14 | sean-k-mooney | i would hope the latter too | |
| 15:22:23 | openstackgerrit | Elod Illes proposed openstack/nova stable/ocata: Don't delete neutron port when attach failed https://review.openstack.org/607614 | |
| 15:23:41 | mriedem | doesn't look like it would fail though https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L849 | |
| 15:23:47 | mriedem | we just handle the not found and assume the guest is already gone | |
| 15:24:53 | mriedem | so maybe there is something to be said for putting hte instance into ERROR state on failed post live migration | |
| 15:24:57 | mriedem | to force the admin to fix it | |
| 15:25:02 | sean-k-mooney | mriedem: right we need that in case where we are usign hard reboot to "fix" things | |
| 15:25:05 | mriedem | so the user doesn't try to reboot the thing and screw it up | |
| 15:26:15 | sean-k-mooney | mriedem: perhaps when i discussed this wit mdbooth previorsly i was suggesting always updating the host to the correct location fo the vm | |
| 15:26:53 | sean-k-mooney | then you could decide if it shoudl stay in error or active state seperately without haveing more bugs if you left it in active | |
| 15:27:39 | openstackgerrit | Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211 | |
| 15:28:33 | sean-k-mooney | mriedem: or to put that another way i think there are two issues in the other case. 1 the host is not upstaed when the vm is moved in some cases and 2 what to do when cleanup fails post migration | |
| 15:28:55 | mriedem | if post live migration set the instance.host to the correct host on which it's running then yeah my concern about the user rebooting it and now having the same guest on different hosts is less of an issue | |
| 15:32:08 | sean-k-mooney | nicolasbock: if you are still around the clip notes version of that conversation is you might want to consider locking the instance untill you have reparied the db to prevent any lifecylce envents | |
| 15:33:07 | mdbooth | mriedem: IIRC my thought at the time was that we should completely update the instance record for the destination immediately after we switch it, then run source cleanup. | |
| 15:33:36 | mdbooth | So if we fail for whatever reason we've recorded that we're running on the dest. | |
| 15:37:46 | mriedem | s10: i think the unit test is going to fail in that patch, see comments for why | |
| 15:37:51 | mdbooth | Incidentally, to reiterate something I said earlier, when this patch landed late last week it made the test_parallel_evacuate_with_server_group about 20 times more likely to occur: https://review.openstack.org/#/c/604859/ | |
| 15:38:08 | mdbooth | That test is now failing around 50% for me. | |
| 15:38:14 | mriedem | we can skip the test for now | |
| 15:38:20 | mriedem | while the fix is being reviewed | |
| 15:38:44 | mdbooth | mriedem: ack. Given ^^^ it seems that the test has never been good. | |
| 15:40:52 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Add native implementation OVSDB API https://review.openstack.org/482226 | |
| 15:43:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip test_parallel_evacuate_with_server_group until fixed https://review.openstack.org/607620 | |
| 15:43:19 | mriedem | dansmith: efried: mdbooth: ^ gives time to be comfortable with the fix | |
| 15:44:21 | mdbooth | mriedem: ack. | |
| 15:44:22 | efried | mriedem: What will the criteria for comfort be? | |
| 15:45:05 | efried | mriedem: https://review.openstack.org/#/c/605436/ has three +1s and a +2 | |
| 15:45:33 | mriedem | i guess that's up to whoever +Ws it | |
| 15:46:12 | efried | mriedem: Well, I'm comfortable that it fixes the problem. But don't feel confident enough in the actual code change to +W. I would think someone like.... mriedem would be able to have that confidence. | |
| 15:47:47 | mriedem | i'm not very confident in anything atm | |
| 15:48:28 | dansmith | I think that change needs a lot of inspection | |
| 15:48:35 | dansmith | which I can't do right this moment | |
| 15:59:45 | mriedem | who enjoys a good UnboundLocalError? https://github.com/openstack/nova/blob/237ced4737a28728408eb30c3d20c6b2c13b4a8d/nova/network/neutronv2/api.py#L1429 | |
| 16:02:54 | mriedem | oh i guess it's not unbound, it's a module import... | |
| 16:09:43 | mriedem | so uh, if we hit ^ shouldn't we fail the build? | |
| 16:10:08 | mriedem | clearly the user isn't going to get the sriov port attached to the guest that they requested | |
| 16:11:12 | openstackgerrit | Jay Pipes proposed openstack/nova stable/ocata: Re-use existing ComputeNode on ironic rebalance https://review.openstack.org/607626 | |
| 16:12:10 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix logging parameter in _populate_pci_mac_address https://review.openstack.org/607628 | |
| 16:12:54 | openstack | Launchpad bug 1795064 in OpenStack Compute (nova) "SR-IOV error IndexError: pop from empty list" [Undecided,New] | |
| 16:12:54 | mriedem | sean-k-mooney: you might be interested in https://bugs.launchpad.net/nova/+bug/1795064 | |
| 16:13:06 | mriedem | something something sriov and kernel versions | |
| 16:13:26 | sean-k-mooney | looking | |
| 16:13:47 | jaypipes | mriedem: https://review.openstack.org/#/c/607626/ is that stable/ocata backport for the duplicate hypervisor_hostname thingie | |
| 16:13:56 | jaypipes | mriedem: thx for your help earlier. | |
| 16:15:11 | mriedem | jaypipes: so you cherry-picked that from master? | |
| 16:15:16 | mriedem | https://review.openstack.org/#/c/508555/ | |
| 16:15:39 | mriedem | or did you cherry pick from the pike backport but forget the -x option on the cherry-pick command? | |
| 16:15:44 | spatel_ | Hi folks | |
| 16:15:50 | jaypipes | mriedem: no, I cherry-picked the SHA1 from the stable/pike patch | |
| 16:16:06 | jaypipes | mriedem: oh, sorry, I don't know about -x :( | |
| 16:16:28 | spatel_ | I am having issue with SR-IOV with shared PCI device between numa and reading this blueprint : https://blueprints.launchpad.net/nova/+spec/share-pci-between-numa-nodes | |
| 16:17:25 | spatel_ | I have set hw:pci_numa_affinity_policy='preferred' in flavor but still its not allowing me to run instance on NUMA-1 | |
| 16:17:25 | openstack | Launchpad bug 1795064 in OpenStack Compute (nova) "SR-IOV error IndexError: pop from empty list" [Undecided,New] | |
| 16:17:25 | sean-k-mooney | spatel_: this bug https://bugs.launchpad.net/nova/+bug/1795064? or another? | |
| 16:17:58 | spatel_ | sean-k-mooney: that problem got resolved by downgrading kernel to 3.x | |
| 16:18:39 | sean-k-mooney | spatel_: i think you issue with 4.18 was that you did not have a netdev associated with the vf | |
| 16:19:04 | spatel_ | Is that configuration issue or BUG? | |
| 16:19:46 | sean-k-mooney | spatel_: i would say config issue. i would guess the default options for the gernel module cahnged and or you are using a different driver by default | |
| 16:19:47 | jaypipes | mriedem: apologies. how can I fix appropriately? do I need to re-do the git cherry-pick with -x? or can/should I just edit the commit message with seomthing? | |
| 16:20:27 | sean-k-mooney | spatel_: for example if the device was bound to vfio_pci instead fo the broadcom driver then it would existit in lspci but not have a netdev | |
| 16:20:59 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Service version check for NUMA live migration https://review.openstack.org/566723 | |
| 16:20:59 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Libvirt live migration: update NUMA XML for dest https://review.openstack.org/575179 | |
| 16:21:01 | spatel_ | hmmm! | |
| 16:21:02 | sean-k-mooney | spatel_: do you whitelist devices using the devname option? | |
| 16:21:04 | mriedem | jaypipes: see my other comment in the ocata backport about documenting the conflicts? | |
| 16:21:18 | spatel_ | sean-k-mooney: yes i am using devname option to specify my interface | |
| 16:21:29 | spatel_ | pci_passthrough_whitelist = "{ "physical_network":"vlan", "devname":"eno2" }" | |
| 16:21:36 | sean-k-mooney | spatel_: in general i advise against that for this exact reason | |
| 16:21:52 | sean-k-mooney | spatel_: if you use the pci adddress instead 4.18 would likely be fine | |
| 16:22:14 | spatel_ | pci address ? | |
| 16:22:27 | sean-k-mooney | spatel_: the whitelist can have 3 modes of whitelisting | |
| 16:22:35 | spatel_ | you mean vendor_id or product_id ? | |
| 16:23:09 | sean-k-mooney | spatel_: you can used devname, (vendor_id and product_id) or you can pass a pci address | |
| 16:23:24 | spatel_ | sean-k-mooney: i will give it a try and report back to BUG | |
| 16:23:41 | openstack | Launchpad bug 1795920 in OpenStack Compute (nova) "SR-IOV shared PCI numa not working " [Undecided,New] | |
| 16:23:41 | spatel_ | sean-k-mooney: currently i am dealing with this issue :( https://bugs.launchpad.net/nova/+bug/1795920 | |
| 16:23:58 | spatel_ | Do you know what wrong i am doing here | |
| 16:24:14 | spatel_ | I have 2 NUMA node and running SR-IOV with shared PCI | |