Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-18
14:24:45 sean-k-mooney nightmare_unreal: the openstack client uses storyborad
14:25:52 sean-k-mooney so here https://storyboard.openstack.org/#!/project/openstack/python-openstackclient but this is all the releated projects https://storyboard.openstack.org/#!/project_group/80
14:26:15 nightmare_unreal thanks :)
14:34:45 dansmith artom: your sigh about migration func testing is lack of review I assume?
14:34:55 dansmith I'm sure I'm at least partially to blame there
14:35:51 artom dansmith, no, mostly towards myself
14:35:57 dansmith oh okay
14:35:58 artom For not staying on top of it
14:36:33 artom I wanted to do a push, well, starting now
14:36:51 artom I still have stephenfin's feedback from the previous millenium to address (about the 2/3 NUMA nodes split)
14:37:16 artom But if we manage to land it before the VPMEM live migration patch, it would make things easier
14:37:25 dansmith ack
14:37:48 dansmith I'm pretty un-jazzed about that patch
14:38:24 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759
14:38:24 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475
14:38:37 artom dansmith, live migration with VPMEM?
14:38:44 dansmith yes
14:39:10 artom In terms of current implementation or the concept in general?
14:39:34 artom I'm just neutral-jazzed - I got involved because my pawprints are all over that code after NUMA live migration
14:39:56 dansmith well, I'm rather negative on the vpmem stuff in general, but that ship has sailed..But yeah, the current patch more acutely
14:40:09 openstackgerrit Balazs Gibizer proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766
14:40:17 dansmith it smacks of "let me just move around all these critical things so they work the way I need them"
14:40:44 artom Yeah...
14:40:49 lyarwood sean-k-mooney: https://bugs.launchpad.net/neutron/+bug/1867936 seen this before?
14:40:49 openstack Launchpad bug 1867936 in neutron "test_update_delete_extra_route failing due to timeout when creating subnets" [Undecided,New]
14:41:28 artom dansmith, you've essentially put words on a vague feeling I had
14:41:43 sean-k-mooney lyarwood: only in the failure of one of your patches
14:41:48 artom Which is why I would really be more comfortable if we at least had NUMA LM func tests in-tree before merging
14:42:07 gibi dansmith: hi! added the global service version check to https://review.opendev.org/#/c/705475
14:42:08 artom Those at least exercise those code paths
14:42:18 dansmith artom: well, I certainly don't think that's unreasonable, as long as we can get that moving
14:42:33 artom dansmith, ack, lemme get on that
14:42:56 lyarwood sean-k-mooney: yeah I've seen it a few times, was just wondering if someones already raised a more specific bug report for this
14:43:04 dansmith artom: using your patch to validate what we have in tree so we can more directly assess the changes this one makes would be helpful
14:43:07 dansmith gibi: ack
14:43:39 artom dansmith, the bottom two of https://review.opendev.org/#/c/705545/ would be ready, I'd say
14:43:41 sean-k-mooney lyarwood: there have been some issue with the dhcp agent by the way. e.g. it not seting up the dhcp config in a timely manner
14:43:52 dansmith gibi: I feel like we might be at the point with the cyborg set where someone else needs to take over it.. I'm not fully caught up yet this morning but I think feedback has been pending for a while now
14:43:58 sean-k-mooney this might just be a side effect of that althogh i though that neuton had fixed that
14:44:07 artom dansmith, if you have any -1s to get in on those, do it now ;)
14:44:28 lyarwood sean-k-mooney: kk
14:44:29 dansmith artom: at least the bottom one is -W, should it not be?
14:44:42 gibi dansmith: I agree. I can spend some review time on that series but I cannot promise to take over the impl
14:44:51 artom dansmith, it's to indicate that the top 2 aren't ready
14:45:06 artom It shouldn't stop you from -1'ing :)
14:45:13 dansmith gibi: no, not asking the latter, but it would be good if we could have the next few above your +2s reviewed and then make a big effort there
14:45:51 gibi dansmith: OK. I will look into those patches soon
14:46:26 dansmith gibi: thanks, I'll start in on your unshelve patch
14:46:36 gibi dansmith: thanks
14:47:52 gibi dansmith: I'm working on retrofitting the service version check to the other qos move operations implemented in Ussuri
14:48:06 gibi dansmith: but the pattern is there in the unshelve patch
14:48:46 dansmith gibi: okay, things that are already landed are kinda already out of the bag, unless you mean gating everything on this service version for people that haven't rolled yet
14:49:12 gibi dansmith: I'm planning to gate it on the service version I added at the unshelve patch
14:49:17 gibi dansmith: does it make sense to you?
14:49:37 gibi dansmith: I don't think many people rolled Ussuri yet
14:49:52 dansmith okay.. technically that is kinda making history go backwards, but for the practical case where people are just deploying releases, it makes sense
14:50:15 dansmith I don't really want to get too far into that line of thinking, because it leads us towards "break things during the release as long as we fix before GA" but... this isn't quite that
14:50:59 gibi dansmith: I consider it as a bug that was introduced in Ussuri and fixed in Ussuri
14:51:35 nightmare_unreal how can i report a bug , it's related to openstack client
14:51:51 nightmare_unreal use storyboard ?
14:53:54 stephenfin nightmare_unreal: yeah, I think so
15:03:55 lyarwood sean-k-mooney: https://zuul.opendev.org/t/openstack/build/b62f4ef603904e05bc34e2a32a4d2866/log/logs/screen-n-cpu.txt#6754 - so the nova-live-migration evacuation failure looks like another race between libvirtd and n-cpu btw
15:04:00 lyarwood at least in this example
15:08:24 lyarwood ah ignore me that's an earlier negative test
15:15:54 dansmith gibi: maybe you can save me some time checking on this, but... it looks like we do set reqspec.requested_resources in various places,
15:16:09 dansmith but this patch just blindly sets that on reqspec (as you do elsewhere)
15:16:42 dansmith is the _only_ time any of requested_resources gets used is for port qos right now and you're just taking the shortcut of being aggressive there?
15:17:01 dansmith I note the comment to that effect, but I just want to make sure you're positive that it's only used for that, and that it's always safe to blow those away
15:17:34 gibi dansmith: I'm positive it is use only for that (except the cyborg case where I'm not sure what is merged yet)
15:17:50 dansmith nothing for it is merged, so that's safe
15:18:05 gibi but I do remember that the cyborg change was proper and did not overwrite the requested_resources
15:18:37 gibi dansmith: so I think we are safe there
15:18:43 dansmith seems a little risky to just be blind about these early cases, and that it's going to be a source of bugs later where cyborg uses it for spawn, for example but unshelve overwrites those
15:19:13 openstackgerrit Lee Yarwood proposed openstack/nova master: gate: Ensure n-cpu is stopped on the subnode during evacuation testing https://review.opendev.org/713674
15:19:14 lyarwood sean-k-mooney: ^ so I wonder if n-cpu running on the subnode is borking things here even if it is marked as down in the API?
15:20:04 lyarwood I can see the subnode refreshing the instance network info cache for example
15:20:16 lyarwood while we are trying to evacuate
15:20:17 gibi dansmith: the field is not persisted so I don't think we will overwrite existing data from the past. every user of the requested_resources needs to re-fill it for each operation
15:20:31 dansmith ahh, okay that makes sense
15:25:50 sean-k-mooney lyarwood: well the mark host down api has a prequisite that the host is actully down and the agents are stoped
15:26:39 sean-k-mooney lyarwood: if we voilate that expectation then perhaps it coudl cause issues
15:27:41 sean-k-mooney lyarwood: in the evacuate case we proably should be storping n-cpu on the compute host as it is not properly fenced if it is running and able to connect to the message bus/db
15:28:10 lyarwood sean-k-mooney: yup, if it's pinging neutron behind our backs I'm assuming that's going to cause issues when it comes to evacuting the instance from the node
15:29:09 sean-k-mooney well i have not check how the info cache works but unless its per host then it could replace the binding info in the cache with the stale prot bindin info for the orginal host
15:29:33 sean-k-mooney although im not sure if that would cause an issue
15:29:50 sean-k-mooney its certenly safter to not have it messing with the info cache
15:30:54 sean-k-mooney the warning is pretty clear in the api docs too on this point "tting a service forced down without completely fencing it will likely result in the corruption of VMs on that host."
15:32:32 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679
15:32:33 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680
15:33:13 gibi dansmith: fyi I retrofitted the service version check to the rest of the Ussuri qos move code ^^
15:36:30 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
15:37:02 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682
15:37:25 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
15:37:35 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682
15:52:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: [Community goal] Update contributor documentation https://review.opendev.org/712420
16:01:59 dansmith gibi: can you look at my concern here real quick? https://review.opendev.org/#/c/713679/1/nova/api/openstack/compute/migrate_server.py
16:02:53 gibi dansmith: looking
16:03:27 openstackgerrit Merged openstack/os-resource-classes master: Switch to hacking 2.x https://review.opendev.org/710276
16:03:43 gibi dansmith: make sense. I can reword the API error, and LOG the detailed info
16:04:02 dansmith gibi: okay, thanks

Earlier   Later