| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-11 | |||
| 19:43:40 | artom | Hrmm, my old abandoned patches aren't telling me much. Anyways, it might be microversions, is what I'm saying. | |
| 20:11:39 | mriedem | well i think i know why the api samples are busted | |
| 20:12:23 | mriedem | https://github.com/openstack/nova/blob/7a5d7db7fc6ccfbd97508a1e8cd353ec25cf8ca2/nova/tests/functional/api_sample_tests/test_servers.py#L38 | |
| 20:12:32 | mriedem | it's not using the 2.67 request sample | |
| 20:15:23 | artom | Hey, I was right! Time to revive https://review.openstack.org/#/c/430352/ ? | |
| 20:17:34 | mriedem | i'm not sure i understand all that black magic | |
| 20:18:45 | mriedem | common_req_names is definitely f'ed though | |
| 20:19:22 | artom | It's basically a way to automagically load whatever the latest samples in the tree are, and adds a test case that runs with 2.latest | |
| 20:20:30 | openstackgerrit | Merged openstack/os-vif master: Reflow docs to 79 columns https://review.openstack.org/609726 | |
| 20:22:45 | artom | I still kinda want it to happen, but ideally there'd be a really awesome generous intelligent core who would commit to it, because it's loads of tedious work fixing all the samples that break with 2.latest | |
| 20:23:53 | mriedem | i don't see any cores like that around here | |
| 20:24:05 | openstackgerrit | melanie witt proposed openstack/nova-specs master: Document Stein review priorities https://review.openstack.org/609807 | |
| 20:24:23 | artom | Giant dicks indeed ;) | |
| 20:35:45 | mriedem | ima need this to fix these tests https://www.youtube.com/watch?v=3sClFZ21oSo | |
| 20:40:52 | melwitt | I could use the same music to triage a bunch of bugs | |
| 20:41:37 | mriedem | this was on the radio on the way home from lunch with laura today, https://www.youtube.com/watch?v=zNgcYGgtf8M - i threatened to roll the windows down and blast it | |
| 20:42:03 | mriedem | who me?! yes you! | |
| 20:42:23 | melwitt | that's a good one | |
| 20:48:00 | openstackgerrit | melanie witt proposed openstack/nova-specs master: Document Stein review priorities https://review.openstack.org/609807 | |
| 20:50:14 | melwitt | nova meeting in 10 minutes | |
| 20:58:33 | mriedem | well i figured out the 2.67 unit test failures | |
| 20:58:36 | mriedem | or lack thereof | |
| 21:12:59 | openstack | Launchpad bug 1797269 in OpenStack Compute (nova) "Nova fails creating multiple NVIDIA VGPU instances at the same time" [Undecided,New] | |
| 21:12:59 | cfriesen | bauzas: you might find this interesting: https://bugs.launchpad.net/nova/+bug/1797269 | |
| 21:17:34 | openstackgerrit | sean mooney proposed openstack/os-vif master: add support for generic tap device plug https://review.openstack.org/602384 | |
| 21:19:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add microversion 2.67 to support volume_type https://review.openstack.org/606398 | |
| 21:19:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add compute API validation for when a volume_type is requested https://review.openstack.org/605573 | |
| 21:19:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add compute version 36 to support ``volume_type`` https://review.openstack.org/579360 | |
| 21:21:37 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 21:23:11 | mriedem | dansmith: there ^ | |
| 21:23:16 | mriedem | not sure i can +2 those now | |
| 21:24:08 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 21:26:50 | mriedem | +1 on a couple, and +2 on the others that i didn't touch in non-trivial ways | |
| 21:35:07 | sean-k-mooney | melwitt: i have address your momments on https://review.openstack.org/#/c/602384 by the way. ill be starting on the backport tomorrow. | |
| 21:35:25 | sean-k-mooney | /momments/comments/ | |
| 21:35:56 | melwitt | k, will look | |
| 21:36:21 | sean-k-mooney | mriedem: mnaser a different edgecase that will likely need a more invovled fix was flagged to me for that bug earlier today | |
| 21:37:45 | sean-k-mooney | mriedem: mnaser i dont think the current patch or os-vif on its own can address it if i confirm it tommrow and i think will require nova and or neutron change to fix the second vector. | |
| 21:38:14 | mnaser | sounds like a boatload of fun | |
| 21:39:41 | sean-k-mooney | mnaser: yes its one metric "boatload of fun". | |
| 21:42:18 | cfriesen | mriedem: any chance you could take a look at this? It's been sitting around for a while without any real reviews and it's in code you know about. https://review.openstack.org/#/c/603844/ | |
| 21:42:41 | sean-k-mooney | mnaser: what version of openstack are you running by the way. | |
| 21:43:16 | mnaser | sean-k-mooney: queens in process of upgrading to rocky in montreal and rocky in sjc | |
| 21:44:46 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 21:46:07 | mriedem | code i know about? | |
| 21:46:19 | mriedem | flattery will get you everywhere | |
| 21:49:10 | cfriesen | I figured I'd ping sfinucan tomorrow since he's mucked with the neutron api too | |
| 21:50:06 | mriedem | this is kind of sledgehammery | |
| 21:50:16 | mriedem | "neutron could be down at some point so let's always check" | |
| 21:51:39 | cfriesen | you mean the call to check if it's unbound/failed is unconditional? | |
| 21:53:28 | dansmith | mriedem: just checking that you don't think a local test is necessary here? https://review.openstack.org/#/c/606398/11/nova/tests/unit/api/openstack/compute/test_serversV21.py | |
| 21:53:33 | dansmith | I know the next file kinda has a little coverage | |
| 21:58:17 | mriedem | yeah that's in the next module | |
| 21:59:19 | mriedem | cfriesen: it seems sort of whack-a-mole to me to have to check if our networking is f'ed up always on rebuild and reboot | |
| 21:59:36 | mriedem | because the argument could be made that we need to do the same thing in lots of other operations right? | |
| 21:59:40 | mriedem | e.g. we don't trust the cache | |
| 21:59:46 | dansmith | aight | |
| 22:01:26 | sean-k-mooney | cfriesen: ill try and review that cahnge also. | |
| 22:01:31 | cfriesen | hard reboot and rebuild are a bit special in that they are used to recover from error scenarios (where the instance is in ERROR status) | |
| 22:01:47 | cfriesen | sean-k-mooney: thanks. | |
| 22:02:37 | cfriesen | so the idea is that we need to deal with scenarios like an evacuation that failed halfway through, | |
| 22:03:12 | mriedem | what about rescue | |
| 22:04:02 | cfriesen | can you rescue an instance in error state? (I don't use rescue much.) | |
| 22:04:08 | mriedem | yeah apparently | |
| 22:04:25 | mriedem | you can also stop/start an error instance but doesn't look like that messes with networking | |
| 22:05:01 | sean-k-mooney | mriedem: well stop/start would be the same as hard reboot right? | |
| 22:05:38 | cfriesen | looks like maybe rescue might make sense to handle as well | |
| 22:07:08 | sean-k-mooney | cfriesen: for evacuate you are refing to the rebuild we do and not nova host-evacuate-live | |
| 22:07:15 | cfriesen | sean-k-mooney: yes | |
| 22:08:12 | sean-k-mooney | ok because if the livemigation fails in the nova host-evacuate-live case it may have the wrong host set in the instance if it fails in the post migration cleanup code | |
| 22:09:53 | openstackgerrit | Merged openstack/nova master: Add scatter-gather-single-cell utility https://review.openstack.org/594947 | |
| 22:10:00 | openstackgerrit | Merged openstack/nova master: Handle IndexError in _populate_neutron_binding_profile https://review.openstack.org/607650 | |
| 22:10:16 | openstackgerrit | Merged openstack/nova master: fix "you" typo https://review.openstack.org/609777 | |
| 22:10:27 | mriedem | sean-k-mooney: no i don't think so | |
| 22:11:28 | sean-k-mooney | mriedem: ? you dont think you can have the wrong host set on the instace? | |
| 22:11:53 | mriedem | you said "same as" but i guess i don't know what you mean by same as | |
| 22:12:05 | mriedem | cfriesen: comments inline | |
| 22:12:14 | mriedem | it feels heavyweight to have to always check this for every reboot/rebuild | |
| 22:12:18 | cfriesen | is there a way to deal with all these scenarios generically? fundamentally I think we're talking about a mismatch between what nova thinks and what neutron thinks. | |
| 22:12:33 | cfriesen | I agree it's not ideal, just not sure how to handle it more cleanly. | |
| 22:12:36 | mriedem | refreshing the info cache won't help when the port bindings are wrong, we know that | |
| 22:12:37 | cfriesen | checking reveiw | |
| 22:12:54 | mriedem | the heal instance info cache periodic could be made to deal with failed port bindings and re-bind them | |
| 22:13:07 | mriedem | we could rollback the port bindings on a failed evacaute | |
| 22:13:09 | mriedem | *evacuate | |
| 22:13:15 | mriedem | that's what we do on a failed live migration | |
| 22:13:40 | cfriesen | there's nothing monitoring the evacuate though, is there? | |
| 22:14:03 | sean-k-mooney | mriedem: oh stop/start same as reboot --hard. | |
| 22:14:23 | mriedem | cfriesen: no, but we have try/except | |
| 22:14:32 | mriedem | and we'd know if we're past the point of updating port bindings to point at the dest | |
| 22:14:43 | mriedem | if we fail after that and haven't updated the instance.host, we could change the port bindings back to the source host | |
| 22:15:10 | mriedem | i started something like that here https://review.openstack.org/#/c/588087/1/nova/compute/manager.py | |
| 22:15:12 | mriedem | but gd it gets messy | |
| 22:15:24 | sean-k-mooney | mriedem: we are currently only doing the multiple port binding dance for live migrate right. | |
| 22:15:30 | mriedem | yes | |
| 22:17:10 | sean-k-mooney | we likely should try to convege all move opperation to that flow in the future. that said we have not needed to yet so we havent | |
| 22:17:59 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: WIP: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 22:18:58 | cfriesen | what about the case where neutron loses messages, leaving the vif_type as binding_failed or unbound? | |
| 22:19:53 | sean-k-mooney | cfriesen: we had talked about allowing a hard reboot to try and rebind the port in that case | |
| 22:20:23 | mriedem | sean-k-mooney: that's what this patch does | |