| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-16 | |||
| 21:29:06 | artom | melwitt, ping me if you have questions / need review / whatever | |
| 21:29:34 | melwitt | thanks ++ | |
| 21:29:38 | artom | (/me needs pressure to "re-join" upstream) | |
| 21:29:45 | artom | I've been neglecting y'all | |
| 21:30:08 | mriedem | don't forget to loop dansmith into this when you want to talk about it, i'm sure he'd love to | |
| 21:30:38 | artom | You're such a good friend | |
| 21:30:38 | melwitt | artom: heh. I might end up running away from this screaming after I try to work on it, so if that doesn't happen maybe I'll ping you | |
| 21:31:01 | artom | melwitt, screaming would be a good sign, actually | |
| 21:31:02 | sean-k-mooney | ya we cant backport adopting multiple port bindings for evacuate | |
| 21:31:11 | artom | Means you're sane (inasmuch as that's still possible) | |
| 21:31:14 | sean-k-mooney | im not sure if we want to do that or not in general | |
| 21:31:18 | melwitt | lol | |
| 21:31:36 | sean-k-mooney | it might be useful but its alot of work to untangel things and make sure it works | |
| 21:32:14 | artom | Yep. We'd also need to run it with a couple of other Neutron backends | |
| 21:32:28 | artom | IIRC we created a DNM job to run against... OVS? OVN? | |
| 21:32:38 | artom | OVB? OG? | |
| 21:32:39 | sean-k-mooney | ovs ovn and lb | |
| 21:32:47 | artom | RunDMC? | |
| 21:33:20 | melwitt | 😂 | |
| 21:33:20 | melwitt | 😂 | |
| 21:33:30 | melwitt | had to use an emoji for that one | |
| 21:34:00 | sean-k-mooney | sure you "had too" :P | |
| 21:34:24 | melwitt | yeah, it wouldn't let me type anything else until I posted the emoji | |
| 21:35:21 | sean-k-mooney | did https://bugs.launchpad.net/nova/+bug/1813789 come up recently downstream or in relation to the nova-livemigation job | |
| 21:35:21 | openstack | Launchpad bug 1813789 in OpenStack Compute (nova) "Evacuate test intermittently fails with network-vif-plugged timeout exception" [Medium,In progress] - Assigned to Artom Lifshitz (notartom) | |
| 21:35:33 | sean-k-mooney | i rembere talking about it a few days ago | |
| 21:35:39 | sean-k-mooney | i just dont recall the context | |
| 21:35:53 | melwitt | lyarwood mentioned it in the nova meeting | |
| 21:36:15 | sean-k-mooney | ah ya it was in context of the zuul v3 migration | |
| 21:36:18 | melwitt | just saying he hit that bug and http://status.openstack.org/elastic-recheck/#1844929 a bunch of times while trying to get some work done | |
| 21:36:35 | sean-k-mooney | yep | |
| 21:37:08 | melwitt | so I started looking at http://status.openstack.org/elastic-recheck/#1844929 and got nowhere. and now I know http://status.openstack.org/elastic-recheck/#1813789 is also horrid | |
| 21:38:44 | sean-k-mooney | so given we dont use the multiple port bindings flow i think we can assume that as long as the port is still active it will recive a bind time even rather then plug time | |
| 21:38:50 | mriedem | those probably aren't even in the same ballpark of terrible | |
| 21:39:15 | sean-k-mooney | but i would need to think that true more carefully to make sure that is correct | |
| 21:39:46 | mriedem | in the bug i linked in logs where the things were happening so it's not really a question of where the race is | |
| 21:39:57 | mriedem | though those log links are going to be dead by now | |
| 21:40:11 | mriedem | unshelve has the same issue | |
| 21:40:39 | mriedem | 1. bind to new host triggers async network-vif-plugged event, 2. driver.spawn plugs vifs which sets up the callback handler | |
| 21:40:47 | mriedem | if you get the event before 2 you're stuck | |
| 21:40:55 | melwitt | right, ok | |
| 21:41:42 | sean-k-mooney | mriedem: in the unsevle case you could argue that when we go to shevle offloaded teh status of the port shoudl be down | |
| 21:42:08 | sean-k-mooney | which woudl prevent the event being sent in the ovs case atleast until its plugged in the driver | |
| 21:42:13 | mriedem | yup, i opened an old bug for that as well | |
| 21:42:32 | sean-k-mooney | but we would need to sitll use the bind_time vs plugtime thing to check | |
| 21:43:13 | sean-k-mooney | is the state of the port something we can contol from nova? | |
| 21:43:26 | sean-k-mooney | or rather are we allowed to set it | |
| 21:43:38 | sean-k-mooney | if so we could set it to down when we do an evaucate | |
| 21:44:34 | melwitt | well, I'll try looking at it, see how it goes | |
| 21:47:28 | mriedem | this is the thing i was thinking about for a shelve related bug https://github.com/openstack/nova/blob/stable/stein/nova/network/neutronv2/api.py#L3332 | |
| 21:47:55 | mriedem | looks like i updated that as part of the cross-cell series https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L3339 | |
| 21:48:43 | mriedem | https://review.opendev.org/#/c/697162/ | |
| 21:48:46 | mriedem | it's all coming back to me | |
| 21:48:53 | sean-k-mooney | so in that function we would jsut set the binding host to None | |
| 21:49:00 | sean-k-mooney | which would unbind it | |
| 21:49:18 | sean-k-mooney | and the status should go to down as a result | |
| 21:50:19 | mriedem | yeah like how _unbind_ports works | |
| 21:50:38 | mriedem | except you can't clear the device_owner on the port when shelve offloading | |
| 21:50:45 | mriedem | the nova instance needs to continue to "own" the port | |
| 21:51:09 | sean-k-mooney | we need to keep device_id which is the nova instance uuid too but ya | |
| 21:51:43 | mriedem | let us *shelve* this discussion for 6 months from now when it comes up again :) | |
| 21:51:56 | sean-k-mooney | :) | |
| 21:51:57 | mriedem | o/ | |
| 21:57:20 | sean-k-mooney | melwitt: so calling self.network_api.cleanup_instance_network_on_host on the source node durign an evacuate might allow us to use the bind vs plug time evnet code to determin when to wait in the evacuate case too | |
| 21:57:50 | sean-k-mooney | melwitt: it is currently only called for cross cell resize | |
| 21:58:37 | sean-k-mooney | but ya we should be doing that definetly during shelve offload to fix the shelve case | |
| 22:01:43 | melwitt | sean-k-mooney: so does that mean that you think evacuate is pretty "easy" but shelve will be more difficult? or are they a similar level of complexity | |
| 22:02:50 | sean-k-mooney | shelve shoudl be easy evaucate might be more difficutly but i think we just need to call cleanup_instance_network_on_host in the right place | |
| 22:03:03 | sean-k-mooney | so in shevel case we shoudl be calling it in shelve offload | |
| 22:03:24 | sean-k-mooney | and in evauate we need to do it before we call spawn on the dest host | |
| 22:03:31 | sean-k-mooney | so pretty early on | |
| 22:04:00 | sean-k-mooney | that will put the port into an unbound state which will set the port status to down | |
| 22:04:24 | melwitt | ok. just saying it sounds like the utilities are available, just have to leverage them | |
| 22:04:43 | melwitt | with the model bind_time stuff | |
| 22:04:47 | sean-k-mooney | they were not in place until recently | |
| 22:05:12 | melwitt | I'm not saying they were, just trying to understand what's the landscape today | |
| 22:05:26 | sean-k-mooney | yep | |
| 22:05:45 | sean-k-mooney | i think they were both added last cycle so they are there form train on | |
| 22:06:05 | sean-k-mooney | they should be backportable too i think | |
| 22:08:17 | melwitt | ok | |
| 22:31:26 | openstackgerrit | Merged openstack/nova stable/train: Unplug VIFs as part of cleanup of networks https://review.opendev.org/711251 | |
| 22:38:03 | openstackgerrit | sean mooney proposed openstack/nova master: [WIP] unbind port before evacuate and shelve offload https://review.opendev.org/713342 | |
| 22:39:20 | sean-k-mooney | melwitt: im not sure if ^ will work but i think it would be something like that, at least as a start | |
| 22:39:48 | melwitt | cool thanks sean-k-mooney | |
| #openstack-nova - 2020-03-17 | |||
| 00:55:27 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936 | |
| 00:55:28 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: Refine and introduce correct parameters for test_get_guest_config_numa_host_instance_topo_cpu_pinning https://review.opendev.org/713351 | |
| 00:55:28 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: scheduler: Introduce 'dedicated' and 'sharing' CPUs for InstanceNUMACell https://review.opendev.org/713352 | |
| 00:55:29 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: Derive 'cpu_pinning_requested' property from cpu_policy https://review.opendev.org/713353 | |
| 00:55:29 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: Introduce 'MIXED' CPU allocation policy for instance https://review.opendev.org/713354 | |
| 00:55:30 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: Introduce the interface of creating 'MIXED' policy instance through 'PCPU' and 'VCPU' https://review.opendev.org/713355 | |
| 01:23:08 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 01:23:08 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 01:27:53 | openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | |
| 01:27:53 | openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | |
| 02:25:35 | openstackgerrit | Merged openstack/nova master: Use fair locks in resource tracker https://review.opendev.org/711528 | |
| 02:25:41 | openstackgerrit | Merged openstack/nova master: Ensures that COMPUTE_RESOURCE_SEMAPHORE usage is fair https://review.opendev.org/712674 | |
| 02:57:40 | openstackgerrit | Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494 | |
| 04:20:24 | openstackgerrit | Abhishek Kekane proposed openstack/nova master: DNM: Add reproducer for subunit parser error https://review.opendev.org/700522 | |
| 05:50:36 | openstackgerrit | Kevin Zhao proposed openstack/nova master: Add default cpu model for aarch64 https://review.opendev.org/709494 | |
| 07:47:58 | openstackgerrit | Kevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/712607 | |