Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-18
14:44:51 artom dansmith, it's to indicate that the top 2 aren't ready
14:45:06 artom It shouldn't stop you from -1'ing :)
14:45:13 dansmith gibi: no, not asking the latter, but it would be good if we could have the next few above your +2s reviewed and then make a big effort there
14:45:51 gibi dansmith: OK. I will look into those patches soon
14:46:26 dansmith gibi: thanks, I'll start in on your unshelve patch
14:46:36 gibi dansmith: thanks
14:47:52 gibi dansmith: I'm working on retrofitting the service version check to the other qos move operations implemented in Ussuri
14:48:06 gibi dansmith: but the pattern is there in the unshelve patch
14:48:46 dansmith gibi: okay, things that are already landed are kinda already out of the bag, unless you mean gating everything on this service version for people that haven't rolled yet
14:49:12 gibi dansmith: I'm planning to gate it on the service version I added at the unshelve patch
14:49:17 gibi dansmith: does it make sense to you?
14:49:37 gibi dansmith: I don't think many people rolled Ussuri yet
14:49:52 dansmith okay.. technically that is kinda making history go backwards, but for the practical case where people are just deploying releases, it makes sense
14:50:15 dansmith I don't really want to get too far into that line of thinking, because it leads us towards "break things during the release as long as we fix before GA" but... this isn't quite that
14:50:59 gibi dansmith: I consider it as a bug that was introduced in Ussuri and fixed in Ussuri
14:51:35 nightmare_unreal how can i report a bug , it's related to openstack client
14:51:51 nightmare_unreal use storyboard ?
14:53:54 stephenfin nightmare_unreal: yeah, I think so
15:03:55 lyarwood sean-k-mooney: https://zuul.opendev.org/t/openstack/build/b62f4ef603904e05bc34e2a32a4d2866/log/logs/screen-n-cpu.txt#6754 - so the nova-live-migration evacuation failure looks like another race between libvirtd and n-cpu btw
15:04:00 lyarwood at least in this example
15:08:24 lyarwood ah ignore me that's an earlier negative test
15:15:54 dansmith gibi: maybe you can save me some time checking on this, but... it looks like we do set reqspec.requested_resources in various places,
15:16:09 dansmith but this patch just blindly sets that on reqspec (as you do elsewhere)
15:16:42 dansmith is the _only_ time any of requested_resources gets used is for port qos right now and you're just taking the shortcut of being aggressive there?
15:17:01 dansmith I note the comment to that effect, but I just want to make sure you're positive that it's only used for that, and that it's always safe to blow those away
15:17:34 gibi dansmith: I'm positive it is use only for that (except the cyborg case where I'm not sure what is merged yet)
15:17:50 dansmith nothing for it is merged, so that's safe
15:18:05 gibi but I do remember that the cyborg change was proper and did not overwrite the requested_resources
15:18:37 gibi dansmith: so I think we are safe there
15:18:43 dansmith seems a little risky to just be blind about these early cases, and that it's going to be a source of bugs later where cyborg uses it for spawn, for example but unshelve overwrites those
15:19:13 openstackgerrit Lee Yarwood proposed openstack/nova master: gate: Ensure n-cpu is stopped on the subnode during evacuation testing https://review.opendev.org/713674
15:19:14 lyarwood sean-k-mooney: ^ so I wonder if n-cpu running on the subnode is borking things here even if it is marked as down in the API?
15:20:04 lyarwood I can see the subnode refreshing the instance network info cache for example
15:20:16 lyarwood while we are trying to evacuate
15:20:17 gibi dansmith: the field is not persisted so I don't think we will overwrite existing data from the past. every user of the requested_resources needs to re-fill it for each operation
15:20:31 dansmith ahh, okay that makes sense
15:25:50 sean-k-mooney lyarwood: well the mark host down api has a prequisite that the host is actully down and the agents are stoped
15:26:39 sean-k-mooney lyarwood: if we voilate that expectation then perhaps it coudl cause issues
15:27:41 sean-k-mooney lyarwood: in the evacuate case we proably should be storping n-cpu on the compute host as it is not properly fenced if it is running and able to connect to the message bus/db
15:28:10 lyarwood sean-k-mooney: yup, if it's pinging neutron behind our backs I'm assuming that's going to cause issues when it comes to evacuting the instance from the node
15:29:09 sean-k-mooney well i have not check how the info cache works but unless its per host then it could replace the binding info in the cache with the stale prot bindin info for the orginal host
15:29:33 sean-k-mooney although im not sure if that would cause an issue
15:29:50 sean-k-mooney its certenly safter to not have it messing with the info cache
15:30:54 sean-k-mooney the warning is pretty clear in the api docs too on this point "tting a service forced down without completely fencing it will likely result in the corruption of VMs on that host."
15:32:32 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679
15:32:33 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680
15:33:13 gibi dansmith: fyi I retrofitted the service version check to the rest of the Ussuri qos move code ^^
15:36:30 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
15:37:02 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682
15:37:25 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
15:37:35 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682
15:52:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: [Community goal] Update contributor documentation https://review.opendev.org/712420
16:01:59 dansmith gibi: can you look at my concern here real quick? https://review.opendev.org/#/c/713679/1/nova/api/openstack/compute/migrate_server.py
16:02:53 gibi dansmith: looking
16:03:27 openstackgerrit Merged openstack/os-resource-classes master: Switch to hacking 2.x https://review.opendev.org/710276
16:03:43 gibi dansmith: make sense. I can reword the API error, and LOG the detailed info
16:04:02 dansmith gibi: okay, thanks
16:04:20 dansmith also, vmware ci did report on the un-deprecate patch... but it failed and only ran two tempest tests (both of which failed)
16:04:58 gibi dansmith: I will respin th whole series and fix your nits on the bottom too.
16:05:08 dansmith gibi: same api concern here: https://review.opendev.org/#/c/705475/6/nova/api/openstack/compute/shelve.py
16:05:22 dansmith gibi: cool, I'll circle back quickly when you do
16:05:23 gibi dansmith: yepp, and in the evac patch too
16:05:57 gibi dansmith: regarding vmware. I think it is fair to say that we need those self selected tempest test to pass
16:06:33 dansmith I think we need to see it working on a wide scale for a substantial period of time,
16:06:54 dansmith because this pattern of making it work for two weeks until it breaks again doesn't really translate to quality
16:07:39 gibi dansmith: yes, you have a point
16:08:06 gibi dansmith: does it only triggers to vmware related patcheS? because then there will be not many changes that triggers them
16:08:25 dansmith AFAICT, it's manually triggered it seems
16:09:06 gibi that feels like cheating for a CI machinery
16:39:13 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
16:59:08 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697
17:05:12 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697
17:05:24 gmann johnthetubaguy: list flavor-access policy is little wired. old defaults are admin-or-owner which i think should be admin as multiple projects can have access to single flavor so owner is not suitable things for flavor. ^^ updating it with more accurate new defaults. you can see details in bug https://bugs.launchpad.net/nova/+bug/1867840
17:05:24 openstack Launchpad bug 1867840 in OpenStack Compute (nova) "os-flavor-access API policy should be admin only" [Undecided,New] - Assigned to Ghanshyam Mann (ghanshyammann)
17:06:44 gmann i did not fix the old default as bug because that can break existing deployment. so thinking to keep supporting them and cleanup during we move to system scope by default
17:53:32 openstackgerrit Merged openstack/nova master: db: Remove unused ec2 DB APIs https://review.opendev.org/698880
18:05:02 openstackgerrit Merged openstack/nova master: libvirt: Check the guest support UEFI https://review.opendev.org/685678
18:10:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759
18:10:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475
18:13:36 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679
18:13:52 openstackgerrit Balazs Gibizer proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766
18:15:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680
18:15:25 gibi dansmith: updated the whole qos series ^^
18:15:32 dansmith gibi: sweet
18:15:41 gibi dansmith: thanks for the review
18:15:52 dansmith np, will hit those soon
18:22:45 sean-k-mooney lyarwood: it looks like your evacuate change is hitting some where errors https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7789
18:23:37 sean-k-mooney it disconnect an iscsi volume then reports rbd issues and finally fails with a virtual interface createion faild error
18:23:44 sean-k-mooney although i dont know why that is being raided
19:03:23 lyarwood sean-k-mooney: which one was that sorry?
19:03:49 lyarwood sean-k-mooney: oh the evacuation fix
19:03:58 lyarwood sean-k-mooney: it looked like the same issue again in console.log
19:04:04 lyarwood sean-k-mooney: I didn't even think to look at n-cpu sorry
19:05:15 sean-k-mooney the n-cpu logs honestly confuse me
19:05:35 sean-k-mooney i did not look at teh code but i think the virutal interface creation excpetion is a redheiring
19:07:33 lyarwood sean-k-mooney: https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7760 it's the same issue
19:08:38 lyarwood sean-k-mooney: we hit the vif timeout first and call disconnect_volume
19:08:58 lyarwood sean-k-mooney: I think the RBD cleanup failures are due to the domain still being active on the subnode
19:09:50 lyarwood oh right because ceph.sh only sets up ephemeral storage on ceph
19:09:58 lyarwood not c-vol

Earlier   Later