Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-11
21:42:18 cfriesen mriedem: any chance you could take a look at this? It's been sitting around for a while without any real reviews and it's in code you know about. https://review.openstack.org/#/c/603844/
21:42:41 sean-k-mooney mnaser: what version of openstack are you running by the way.
21:43:16 mnaser sean-k-mooney: queens in process of upgrading to rocky in montreal and rocky in sjc
21:44:46 openstackgerrit Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
21:46:07 mriedem code i know about?
21:46:19 mriedem flattery will get you everywhere
21:49:10 cfriesen I figured I'd ping sfinucan tomorrow since he's mucked with the neutron api too
21:50:06 mriedem this is kind of sledgehammery
21:50:16 mriedem "neutron could be down at some point so let's always check"
21:51:39 cfriesen you mean the call to check if it's unbound/failed is unconditional?
21:53:28 dansmith mriedem: just checking that you don't think a local test is necessary here? https://review.openstack.org/#/c/606398/11/nova/tests/unit/api/openstack/compute/test_serversV21.py
21:53:33 dansmith I know the next file kinda has a little coverage
21:58:17 mriedem yeah that's in the next module
21:59:19 mriedem cfriesen: it seems sort of whack-a-mole to me to have to check if our networking is f'ed up always on rebuild and reboot
21:59:36 mriedem because the argument could be made that we need to do the same thing in lots of other operations right?
21:59:40 mriedem e.g. we don't trust the cache
21:59:46 dansmith aight
22:01:26 sean-k-mooney cfriesen: ill try and review that cahnge also.
22:01:31 cfriesen hard reboot and rebuild are a bit special in that they are used to recover from error scenarios (where the instance is in ERROR status)
22:01:47 cfriesen sean-k-mooney: thanks.
22:02:37 cfriesen so the idea is that we need to deal with scenarios like an evacuation that failed halfway through,
22:03:12 mriedem what about rescue
22:04:02 cfriesen can you rescue an instance in error state? (I don't use rescue much.)
22:04:08 mriedem yeah apparently
22:04:25 mriedem you can also stop/start an error instance but doesn't look like that messes with networking
22:05:01 sean-k-mooney mriedem: well stop/start would be the same as hard reboot right?
22:05:38 cfriesen looks like maybe rescue might make sense to handle as well
22:07:08 sean-k-mooney cfriesen: for evacuate you are refing to the rebuild we do and not nova host-evacuate-live
22:07:15 cfriesen sean-k-mooney: yes
22:08:12 sean-k-mooney ok because if the livemigation fails in the nova host-evacuate-live case it may have the wrong host set in the instance if it fails in the post migration cleanup code
22:09:53 openstackgerrit Merged openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947
22:10:00 openstackgerrit Merged openstack/nova master: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/607650
22:10:16 openstackgerrit Merged openstack/nova master: fix "you" typo https://review.openstack.org/609777
22:10:27 mriedem sean-k-mooney: no i don't think so
22:11:28 sean-k-mooney mriedem: ? you dont think you can have the wrong host set on the instace?
22:11:53 mriedem you said "same as" but i guess i don't know what you mean by same as
22:12:05 mriedem cfriesen: comments inline
22:12:14 mriedem it feels heavyweight to have to always check this for every reboot/rebuild
22:12:18 cfriesen is there a way to deal with all these scenarios generically? fundamentally I think we're talking about a mismatch between what nova thinks and what neutron thinks.
22:12:33 cfriesen I agree it's not ideal, just not sure how to handle it more cleanly.
22:12:36 mriedem refreshing the info cache won't help when the port bindings are wrong, we know that
22:12:37 cfriesen checking reveiw
22:12:54 mriedem the heal instance info cache periodic could be made to deal with failed port bindings and re-bind them
22:13:07 mriedem we could rollback the port bindings on a failed evacaute
22:13:09 mriedem *evacuate
22:13:15 mriedem that's what we do on a failed live migration
22:13:40 cfriesen there's nothing monitoring the evacuate though, is there?
22:14:03 sean-k-mooney mriedem: oh stop/start same as reboot --hard.
22:14:23 mriedem cfriesen: no, but we have try/except
22:14:32 mriedem and we'd know if we're past the point of updating port bindings to point at the dest
22:14:43 mriedem if we fail after that and haven't updated the instance.host, we could change the port bindings back to the source host
22:15:10 mriedem i started something like that here https://review.openstack.org/#/c/588087/1/nova/compute/manager.py
22:15:12 mriedem but gd it gets messy
22:15:24 sean-k-mooney mriedem: we are currently only doing the multiple port binding dance for live migrate right.
22:15:30 mriedem yes
22:17:10 sean-k-mooney we likely should try to convege all move opperation to that flow in the future. that said we have not needed to yet so we havent
22:17:59 openstackgerrit Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
22:18:58 cfriesen what about the case where neutron loses messages, leaving the vif_type as binding_failed or unbound?
22:19:53 sean-k-mooney cfriesen: we had talked about allowing a hard reboot to try and rebind the port in that case
22:20:23 mriedem sean-k-mooney: that's what this patch does
22:20:42 sean-k-mooney mriedem: oh i was still reading it.
22:20:46 mriedem on every reboot it checks if the vif type is binding_failed or unbound, or if the binding:host_id doesn't match the current host and re-bindgs
22:20:48 mriedem *binds
22:21:01 cfriesen is there a way to detect that we're in that state without needing to do the check on every reboot/rebuild/rescue?
22:21:04 mriedem cfriesen: if neutron shits the bed, that's hard to handle in any case
22:21:07 mriedem since it could be anywhere
22:21:21 mriedem cfriesen: well the _heal_instance_info_cache periodic could certainly do that right?
22:21:47 mriedem or some sort of heal_failed_port_bindings periodic
22:21:58 sean-k-mooney mriedem: i think that is what we orginially discussed as an option during the RC period
22:22:16 cfriesen okay...but then you end up in a scenario where a failed instance can't be recovered until that periodic task runs
22:22:43 sean-k-mooney on the patch you wrote to stop the network info cache gettin poisioned by binding faild and unbound vif types
22:22:50 mriedem cfriesen: yeah i know
22:22:54 mriedem all options are terrible
22:22:58 mriedem and it's 5:23
22:23:00 mriedem and i want to leave
22:23:08 cfriesen go. :)
23:00:14 openstackgerrit sean mooney proposed openstack/os-vif master: add support for generic tap device plug https://review.openstack.org/602384
23:09:30 openstackgerrit sean mooney proposed openstack/os-vif stable/rocky: add support for generic tap device plug https://review.openstack.org/609850
23:12:10 openstackgerrit sean mooney proposed openstack/os-vif stable/queens: add support for generic tap device plug https://review.openstack.org/609851
23:40:06 openstackgerrit Adam Spiers proposed openstack/nova-specs master: Add spec for libvirt driver launching AMD SEV-encrypted instances https://review.openstack.org/609779
#openstack-nova - 2018-10-12
00:48:47 alex_xu cfriesen: the qemu will take care the live migration for vTPM?
01:04:14 openstackgerrit fupingxie proposed openstack/nova master: Don't recreate inst_base on source when using rbd backend in resize https://review.openstack.org/605590
01:15:00 zhanglong alex_xu: what's the give scenario about your issue? to migrate [vtpm vm] to [another host], or to migrate [vtpm vm] to [another non-vtpm vm]?
01:16:15 openstackgerrit fupingxie proposed openstack/nova master: Don't recreate inst_base on source when using rbd backend in resize https://review.openstack.org/605590
01:32:40 openstackgerrit Naichuan Sun proposed openstack/nova master: os-xenapi(n-rp): add traits for vgpu n-rp https://review.openstack.org/604269
01:35:45 alex_xu zhanglong: migrate vtpm vm to another host
01:36:16 alex_xu zhanglong: live migrate
01:45:10 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P):Get vgpu info from `allocations` https://review.openstack.org/521717
01:45:33 openstackgerrit Naichuan Sun proposed openstack/nova master: xenapi(N-R-P): support compute node resource provider update https://review.openstack.org/521041
01:45:52 openstackgerrit Naichuan Sun proposed openstack/nova master: os-xenapi(n-rp): add traits for vgpu n-rp https://review.openstack.org/604269
02:11:24 zhanglong alex_xu: OK. So to ensure [source Asymmetric key] only migrate to [trusted destination vTPM], it is requested that [destination vTPM] use AIK signed key.
02:12:46 zhanglong alex_xu: And there are several solutions to solve this issue.see paper <vTPM: Virtualizing the Trusted Platform Module>.
02:19:36 alex_xu zhanglong: are you saying we don't have enough support in kvm to support live migrate with vTPM?
02:22:30 alex_xu zhanglong: btw, i'm revewing this spec https://review.openstack.org/571111, try to figure out the live migration whether is the scope of that proposal
02:24:35 zhanglong alex_xu: KVM? vm is related to QEMU, not KVM.
02:27:37 zhanglong alex_xu: As I know , the process of [live migrate with vTPM] is related to vm manager e.g. nova, and can be decoupling with host. :)
02:28:17 alex_xu zhanglong: yea, sorry, i'm just saying the generic word, i say kvm is mean to all those stuff qemu, kvm. i know qemu is about device emulation, kvm is kernel module...
02:29:21 zhanglong alex_xu: mm :)
05:25:45 openstackgerrit Dao Cong Tien proposed openstack/nova master: Add note for sending SIGHUP at rolling upgrade steps https://review.openstack.org/609910
05:45:33 cfriesen alex_xu: not sure if it's libvirt or qemu, but nova doesn't need to worry about it.

Earlier   Later