Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-11
21:24:08 openstackgerrit Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
21:26:50 mriedem +1 on a couple, and +2 on the others that i didn't touch in non-trivial ways
21:35:07 sean-k-mooney melwitt: i have address your momments on https://review.openstack.org/#/c/602384 by the way. ill be starting on the backport tomorrow.
21:35:25 sean-k-mooney /momments/comments/
21:35:56 melwitt k, will look
21:36:21 sean-k-mooney mriedem: mnaser a different edgecase that will likely need a more invovled fix was flagged to me for that bug earlier today
21:37:45 sean-k-mooney mriedem: mnaser i dont think the current patch or os-vif on its own can address it if i confirm it tommrow and i think will require nova and or neutron change to fix the second vector.
21:38:14 mnaser sounds like a boatload of fun
21:39:41 sean-k-mooney mnaser: yes its one metric "boatload of fun".
21:42:18 cfriesen mriedem: any chance you could take a look at this? It's been sitting around for a while without any real reviews and it's in code you know about. https://review.openstack.org/#/c/603844/
21:42:41 sean-k-mooney mnaser: what version of openstack are you running by the way.
21:43:16 mnaser sean-k-mooney: queens in process of upgrading to rocky in montreal and rocky in sjc
21:44:46 openstackgerrit Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
21:46:07 mriedem code i know about?
21:46:19 mriedem flattery will get you everywhere
21:49:10 cfriesen I figured I'd ping sfinucan tomorrow since he's mucked with the neutron api too
21:50:06 mriedem this is kind of sledgehammery
21:50:16 mriedem "neutron could be down at some point so let's always check"
21:51:39 cfriesen you mean the call to check if it's unbound/failed is unconditional?
21:53:28 dansmith mriedem: just checking that you don't think a local test is necessary here? https://review.openstack.org/#/c/606398/11/nova/tests/unit/api/openstack/compute/test_serversV21.py
21:53:33 dansmith I know the next file kinda has a little coverage
21:58:17 mriedem yeah that's in the next module
21:59:19 mriedem cfriesen: it seems sort of whack-a-mole to me to have to check if our networking is f'ed up always on rebuild and reboot
21:59:36 mriedem because the argument could be made that we need to do the same thing in lots of other operations right?
21:59:40 mriedem e.g. we don't trust the cache
21:59:46 dansmith aight
22:01:26 sean-k-mooney cfriesen: ill try and review that cahnge also.
22:01:31 cfriesen hard reboot and rebuild are a bit special in that they are used to recover from error scenarios (where the instance is in ERROR status)
22:01:47 cfriesen sean-k-mooney: thanks.
22:02:37 cfriesen so the idea is that we need to deal with scenarios like an evacuation that failed halfway through,
22:03:12 mriedem what about rescue
22:04:02 cfriesen can you rescue an instance in error state? (I don't use rescue much.)
22:04:08 mriedem yeah apparently
22:04:25 mriedem you can also stop/start an error instance but doesn't look like that messes with networking
22:05:01 sean-k-mooney mriedem: well stop/start would be the same as hard reboot right?
22:05:38 cfriesen looks like maybe rescue might make sense to handle as well
22:07:08 sean-k-mooney cfriesen: for evacuate you are refing to the rebuild we do and not nova host-evacuate-live
22:07:15 cfriesen sean-k-mooney: yes
22:08:12 sean-k-mooney ok because if the livemigation fails in the nova host-evacuate-live case it may have the wrong host set in the instance if it fails in the post migration cleanup code
22:09:53 openstackgerrit Merged openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947
22:10:00 openstackgerrit Merged openstack/nova master: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/607650
22:10:16 openstackgerrit Merged openstack/nova master: fix "you" typo https://review.openstack.org/609777
22:10:27 mriedem sean-k-mooney: no i don't think so
22:11:28 sean-k-mooney mriedem: ? you dont think you can have the wrong host set on the instace?
22:11:53 mriedem you said "same as" but i guess i don't know what you mean by same as
22:12:05 mriedem cfriesen: comments inline
22:12:14 mriedem it feels heavyweight to have to always check this for every reboot/rebuild
22:12:18 cfriesen is there a way to deal with all these scenarios generically? fundamentally I think we're talking about a mismatch between what nova thinks and what neutron thinks.
22:12:33 cfriesen I agree it's not ideal, just not sure how to handle it more cleanly.
22:12:36 mriedem refreshing the info cache won't help when the port bindings are wrong, we know that
22:12:37 cfriesen checking reveiw
22:12:54 mriedem the heal instance info cache periodic could be made to deal with failed port bindings and re-bind them
22:13:07 mriedem we could rollback the port bindings on a failed evacaute
22:13:09 mriedem *evacuate
22:13:15 mriedem that's what we do on a failed live migration
22:13:40 cfriesen there's nothing monitoring the evacuate though, is there?
22:14:03 sean-k-mooney mriedem: oh stop/start same as reboot --hard.
22:14:23 mriedem cfriesen: no, but we have try/except
22:14:32 mriedem and we'd know if we're past the point of updating port bindings to point at the dest
22:14:43 mriedem if we fail after that and haven't updated the instance.host, we could change the port bindings back to the source host
22:15:10 mriedem i started something like that here https://review.openstack.org/#/c/588087/1/nova/compute/manager.py
22:15:12 mriedem but gd it gets messy
22:15:24 sean-k-mooney mriedem: we are currently only doing the multiple port binding dance for live migrate right.
22:15:30 mriedem yes
22:17:10 sean-k-mooney we likely should try to convege all move opperation to that flow in the future. that said we have not needed to yet so we havent
22:17:59 openstackgerrit Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
22:18:58 cfriesen what about the case where neutron loses messages, leaving the vif_type as binding_failed or unbound?
22:19:53 sean-k-mooney cfriesen: we had talked about allowing a hard reboot to try and rebind the port in that case
22:20:23 mriedem sean-k-mooney: that's what this patch does
22:20:42 sean-k-mooney mriedem: oh i was still reading it.
22:20:46 mriedem on every reboot it checks if the vif type is binding_failed or unbound, or if the binding:host_id doesn't match the current host and re-bindgs
22:20:48 mriedem *binds
22:21:01 cfriesen is there a way to detect that we're in that state without needing to do the check on every reboot/rebuild/rescue?
22:21:04 mriedem cfriesen: if neutron shits the bed, that's hard to handle in any case
22:21:07 mriedem since it could be anywhere
22:21:21 mriedem cfriesen: well the _heal_instance_info_cache periodic could certainly do that right?
22:21:47 mriedem or some sort of heal_failed_port_bindings periodic
22:21:58 sean-k-mooney mriedem: i think that is what we orginially discussed as an option during the RC period
22:22:16 cfriesen okay...but then you end up in a scenario where a failed instance can't be recovered until that periodic task runs
22:22:43 sean-k-mooney on the patch you wrote to stop the network info cache gettin poisioned by binding faild and unbound vif types
22:22:50 mriedem cfriesen: yeah i know
22:22:54 mriedem all options are terrible
22:22:58 mriedem and it's 5:23
22:23:00 mriedem and i want to leave
22:23:08 cfriesen go. :)
23:00:14 openstackgerrit sean mooney proposed openstack/os-vif master: add support for generic tap device plug https://review.openstack.org/602384
23:09:30 openstackgerrit sean mooney proposed openstack/os-vif stable/rocky: add support for generic tap device plug https://review.openstack.org/609850
23:12:10 openstackgerrit sean mooney proposed openstack/os-vif stable/queens: add support for generic tap device plug https://review.openstack.org/609851
23:40:06 openstackgerrit Adam Spiers proposed openstack/nova-specs master: Add spec for libvirt driver launching AMD SEV-encrypted instances https://review.openstack.org/609779
#openstack-nova - 2018-10-12
00:48:47 alex_xu cfriesen: the qemu will take care the live migration for vTPM?
01:04:14 openstackgerrit fupingxie proposed openstack/nova master: Don't recreate inst_base on source when using rbd backend in resize https://review.openstack.org/605590
01:15:00 zhanglong alex_xu: what's the give scenario about your issue? to migrate [vtpm vm] to [another host], or to migrate [vtpm vm] to [another non-vtpm vm]?
01:16:15 openstackgerrit fupingxie proposed openstack/nova master: Don't recreate inst_base on source when using rbd backend in resize https://review.openstack.org/605590
01:32:40 openstackgerrit Naichuan Sun proposed openstack/nova master: os-xenapi(n-rp): add traits for vgpu n-rp https://review.openstack.org/604269
01:35:45 alex_xu zhanglong: migrate vtpm vm to another host
01:36:16 alex_xu zhanglong: live migrate
01:45:10 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P):Get vgpu info from `allocations` https://review.openstack.org/521717
01:45:33 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): support compute node resource provider update https://review.openstack.org/521041
01:45:52 openstackgerrit Naichuan Sun proposed openstack/nova master: os-xenapi(n-rp): add traits for vgpu n-rp https://review.openstack.org/604269
02:11:24 zhanglong alex_xu: OK. So to ensure [source Asymmetric key] only migrate to [trusted destination vTPM], it is requested that [destination vTPM] use AIK signed key.

Earlier   Later