| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-11 | |||
| 20:42:23 | melwitt | that's a good one | |
| 20:48:00 | openstackgerrit | melanie witt proposed openstack/nova-specs master: Document Stein review priorities https://review.openstack.org/609807 | |
| 20:50:14 | melwitt | nova meeting in 10 minutes | |
| 20:58:33 | mriedem | well i figured out the 2.67 unit test failures | |
| 20:58:36 | mriedem | or lack thereof | |
| 21:12:59 | openstack | Launchpad bug 1797269 in OpenStack Compute (nova) "Nova fails creating multiple NVIDIA VGPU instances at the same time" [Undecided,New] | |
| 21:12:59 | cfriesen | bauzas: you might find this interesting: https://bugs.launchpad.net/nova/+bug/1797269 | |
| 21:17:34 | openstackgerrit | sean mooney proposed openstack/os-vif master: add support for generic tap device plug https://review.openstack.org/602384 | |
| 21:19:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add microversion 2.67 to support volume_type https://review.openstack.org/606398 | |
| 21:19:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add compute API validation for when a volume_type is requested https://review.openstack.org/605573 | |
| 21:19:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add compute version 36 to support ``volume_type`` https://review.openstack.org/579360 | |
| 21:21:37 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 21:23:11 | mriedem | dansmith: there ^ | |
| 21:23:16 | mriedem | not sure i can +2 those now | |
| 21:24:08 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 21:26:50 | mriedem | +1 on a couple, and +2 on the others that i didn't touch in non-trivial ways | |
| 21:35:07 | sean-k-mooney | melwitt: i have address your momments on https://review.openstack.org/#/c/602384 by the way. ill be starting on the backport tomorrow. | |
| 21:35:25 | sean-k-mooney | /momments/comments/ | |
| 21:35:56 | melwitt | k, will look | |
| 21:36:21 | sean-k-mooney | mriedem: mnaser a different edgecase that will likely need a more invovled fix was flagged to me for that bug earlier today | |
| 21:37:45 | sean-k-mooney | mriedem: mnaser i dont think the current patch or os-vif on its own can address it if i confirm it tommrow and i think will require nova and or neutron change to fix the second vector. | |
| 21:38:14 | mnaser | sounds like a boatload of fun | |
| 21:39:41 | sean-k-mooney | mnaser: yes its one metric "boatload of fun". | |
| 21:42:18 | cfriesen | mriedem: any chance you could take a look at this? It's been sitting around for a while without any real reviews and it's in code you know about. https://review.openstack.org/#/c/603844/ | |
| 21:42:41 | sean-k-mooney | mnaser: what version of openstack are you running by the way. | |
| 21:43:16 | mnaser | sean-k-mooney: queens in process of upgrading to rocky in montreal and rocky in sjc | |
| 21:44:46 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 21:46:07 | mriedem | code i know about? | |
| 21:46:19 | mriedem | flattery will get you everywhere | |
| 21:49:10 | cfriesen | I figured I'd ping sfinucan tomorrow since he's mucked with the neutron api too | |
| 21:50:06 | mriedem | this is kind of sledgehammery | |
| 21:50:16 | mriedem | "neutron could be down at some point so let's always check" | |
| 21:51:39 | cfriesen | you mean the call to check if it's unbound/failed is unconditional? | |
| 21:53:28 | dansmith | mriedem: just checking that you don't think a local test is necessary here? https://review.openstack.org/#/c/606398/11/nova/tests/unit/api/openstack/compute/test_serversV21.py | |
| 21:53:33 | dansmith | I know the next file kinda has a little coverage | |
| 21:58:17 | mriedem | yeah that's in the next module | |
| 21:59:19 | mriedem | cfriesen: it seems sort of whack-a-mole to me to have to check if our networking is f'ed up always on rebuild and reboot | |
| 21:59:36 | mriedem | because the argument could be made that we need to do the same thing in lots of other operations right? | |
| 21:59:40 | mriedem | e.g. we don't trust the cache | |
| 21:59:46 | dansmith | aight | |
| 22:01:26 | sean-k-mooney | cfriesen: ill try and review that cahnge also. | |
| 22:01:31 | cfriesen | hard reboot and rebuild are a bit special in that they are used to recover from error scenarios (where the instance is in ERROR status) | |
| 22:01:47 | cfriesen | sean-k-mooney: thanks. | |
| 22:02:37 | cfriesen | so the idea is that we need to deal with scenarios like an evacuation that failed halfway through, | |
| 22:03:12 | mriedem | what about rescue | |
| 22:04:02 | cfriesen | can you rescue an instance in error state? (I don't use rescue much.) | |
| 22:04:08 | mriedem | yeah apparently | |
| 22:04:25 | mriedem | you can also stop/start an error instance but doesn't look like that messes with networking | |
| 22:05:01 | sean-k-mooney | mriedem: well stop/start would be the same as hard reboot right? | |
| 22:05:38 | cfriesen | looks like maybe rescue might make sense to handle as well | |
| 22:07:08 | sean-k-mooney | cfriesen: for evacuate you are refing to the rebuild we do and not nova host-evacuate-live | |
| 22:07:15 | cfriesen | sean-k-mooney: yes | |
| 22:08:12 | sean-k-mooney | ok because if the livemigation fails in the nova host-evacuate-live case it may have the wrong host set in the instance if it fails in the post migration cleanup code | |
| 22:09:53 | openstackgerrit | Merged openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | |
| 22:10:00 | openstackgerrit | Merged openstack/nova master: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/607650 | |
| 22:10:16 | openstackgerrit | Merged openstack/nova master: fix "you" typo https://review.openstack.org/609777 | |
| 22:10:27 | mriedem | sean-k-mooney: no i don't think so | |
| 22:11:28 | sean-k-mooney | mriedem: ? you dont think you can have the wrong host set on the instace? | |
| 22:11:53 | mriedem | you said "same as" but i guess i don't know what you mean by same as | |
| 22:12:05 | mriedem | cfriesen: comments inline | |
| 22:12:14 | mriedem | it feels heavyweight to have to always check this for every reboot/rebuild | |
| 22:12:18 | cfriesen | is there a way to deal with all these scenarios generically? fundamentally I think we're talking about a mismatch between what nova thinks and what neutron thinks. | |
| 22:12:33 | cfriesen | I agree it's not ideal, just not sure how to handle it more cleanly. | |
| 22:12:36 | mriedem | refreshing the info cache won't help when the port bindings are wrong, we know that | |
| 22:12:37 | cfriesen | checking reveiw | |
| 22:12:54 | mriedem | the heal instance info cache periodic could be made to deal with failed port bindings and re-bind them | |
| 22:13:07 | mriedem | we could rollback the port bindings on a failed evacaute | |
| 22:13:09 | mriedem | *evacuate | |
| 22:13:15 | mriedem | that's what we do on a failed live migration | |
| 22:13:40 | cfriesen | there's nothing monitoring the evacuate though, is there? | |
| 22:14:03 | sean-k-mooney | mriedem: oh stop/start same as reboot --hard. | |
| 22:14:23 | mriedem | cfriesen: no, but we have try/except | |
| 22:14:32 | mriedem | and we'd know if we're past the point of updating port bindings to point at the dest | |
| 22:14:43 | mriedem | if we fail after that and haven't updated the instance.host, we could change the port bindings back to the source host | |
| 22:15:10 | mriedem | i started something like that here https://review.openstack.org/#/c/588087/1/nova/compute/manager.py | |
| 22:15:12 | mriedem | but gd it gets messy | |
| 22:15:24 | sean-k-mooney | mriedem: we are currently only doing the multiple port binding dance for live migrate right. | |
| 22:15:30 | mriedem | yes | |
| 22:17:10 | sean-k-mooney | we likely should try to convege all move opperation to that flow in the future. that said we have not needed to yet so we havent | |
| 22:17:59 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 22:18:58 | cfriesen | what about the case where neutron loses messages, leaving the vif_type as binding_failed or unbound? | |
| 22:19:53 | sean-k-mooney | cfriesen: we had talked about allowing a hard reboot to try and rebind the port in that case | |
| 22:20:23 | mriedem | sean-k-mooney: that's what this patch does | |
| 22:20:42 | sean-k-mooney | mriedem: oh i was still reading it. | |
| 22:20:46 | mriedem | on every reboot it checks if the vif type is binding_failed or unbound, or if the binding:host_id doesn't match the current host and re-bindgs | |
| 22:20:48 | mriedem | *binds | |
| 22:21:01 | cfriesen | is there a way to detect that we're in that state without needing to do the check on every reboot/rebuild/rescue? | |
| 22:21:04 | mriedem | cfriesen: if neutron shits the bed, that's hard to handle in any case | |
| 22:21:07 | mriedem | since it could be anywhere | |
| 22:21:21 | mriedem | cfriesen: well the _heal_instance_info_cache periodic could certainly do that right? | |
| 22:21:47 | mriedem | or some sort of heal_failed_port_bindings periodic | |
| 22:21:58 | sean-k-mooney | mriedem: i think that is what we orginially discussed as an option during the RC period | |
| 22:22:16 | cfriesen | okay...but then you end up in a scenario where a failed instance can't be recovered until that periodic task runs | |
| 22:22:43 | sean-k-mooney | on the patch you wrote to stop the network info cache gettin poisioned by binding faild and unbound vif types | |
| 22:22:50 | mriedem | cfriesen: yeah i know | |
| 22:22:54 | mriedem | all options are terrible | |
| 22:22:58 | mriedem | and it's 5:23 | |
| 22:23:00 | mriedem | and i want to leave | |
| 22:23:08 | cfriesen | go. :) | |
| 23:00:14 | openstackgerrit | sean mooney proposed openstack/os-vif master: add support for generic tap device plug https://review.openstack.org/602384 | |