Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-03-18
15:04:00 lyarwood at least in this example
15:08:24 lyarwood ah ignore me that's an earlier negative test
15:15:54 dansmith gibi: maybe you can save me some time checking on this, but... it looks like we do set reqspec.requested_resources in various places,
15:16:09 dansmith but this patch just blindly sets that on reqspec (as you do elsewhere)
15:16:42 dansmith is the _only_ time any of requested_resources gets used is for port qos right now and you're just taking the shortcut of being aggressive there?
15:17:01 dansmith I note the comment to that effect, but I just want to make sure you're positive that it's only used for that, and that it's always safe to blow those away
15:17:34 gibi dansmith: I'm positive it is use only for that (except the cyborg case where I'm not sure what is merged yet)
15:17:50 dansmith nothing for it is merged, so that's safe
15:18:05 gibi but I do remember that the cyborg change was proper and did not overwrite the requested_resources
15:18:37 gibi dansmith: so I think we are safe there
15:18:43 dansmith seems a little risky to just be blind about these early cases, and that it's going to be a source of bugs later where cyborg uses it for spawn, for example but unshelve overwrites those
15:19:13 openstackgerrit Lee Yarwood proposed openstack/nova master: gate: Ensure n-cpu is stopped on the subnode during evacuation testing https://review.opendev.org/713674
15:19:14 lyarwood sean-k-mooney: ^ so I wonder if n-cpu running on the subnode is borking things here even if it is marked as down in the API?
15:20:04 lyarwood I can see the subnode refreshing the instance network info cache for example
15:20:16 lyarwood while we are trying to evacuate
15:20:17 gibi dansmith: the field is not persisted so I don't think we will overwrite existing data from the past. every user of the requested_resources needs to re-fill it for each operation
15:20:31 dansmith ahh, okay that makes sense
15:25:50 sean-k-mooney lyarwood: well the mark host down api has a prequisite that the host is actully down and the agents are stoped
15:26:39 sean-k-mooney lyarwood: if we voilate that expectation then perhaps it coudl cause issues
15:27:41 sean-k-mooney lyarwood: in the evacuate case we proably should be storping n-cpu on the compute host as it is not properly fenced if it is running and able to connect to the message bus/db
15:28:10 lyarwood sean-k-mooney: yup, if it's pinging neutron behind our backs I'm assuming that's going to cause issues when it comes to evacuting the instance from the node
15:29:09 sean-k-mooney well i have not check how the info cache works but unless its per host then it could replace the binding info in the cache with the stale prot bindin info for the orginal host
15:29:33 sean-k-mooney although im not sure if that would cause an issue
15:29:50 sean-k-mooney its certenly safter to not have it messing with the info cache
15:30:54 sean-k-mooney the warning is pretty clear in the api docs too on this point "tting a service forced down without completely fencing it will likely result in the corruption of VMs on that host."
15:32:32 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679
15:32:33 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680
15:33:13 gibi dansmith: fyi I retrofitted the service version check to the rest of the Ussuri qos move code ^^
15:36:30 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
15:37:02 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682
15:37:25 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
15:37:35 openstackgerrit Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682
15:52:40 openstackgerrit Balazs Gibizer proposed openstack/nova master: [Community goal] Update contributor documentation https://review.opendev.org/712420
16:01:59 dansmith gibi: can you look at my concern here real quick? https://review.opendev.org/#/c/713679/1/nova/api/openstack/compute/migrate_server.py
16:02:53 gibi dansmith: looking
16:03:27 openstackgerrit Merged openstack/os-resource-classes master: Switch to hacking 2.x https://review.opendev.org/710276
16:03:43 gibi dansmith: make sense. I can reword the API error, and LOG the detailed info
16:04:02 dansmith gibi: okay, thanks
16:04:20 dansmith also, vmware ci did report on the un-deprecate patch... but it failed and only ran two tempest tests (both of which failed)
16:04:58 gibi dansmith: I will respin th whole series and fix your nits on the bottom too.
16:05:08 dansmith gibi: same api concern here: https://review.opendev.org/#/c/705475/6/nova/api/openstack/compute/shelve.py
16:05:22 dansmith gibi: cool, I'll circle back quickly when you do
16:05:23 gibi dansmith: yepp, and in the evac patch too
16:05:57 gibi dansmith: regarding vmware. I think it is fair to say that we need those self selected tempest test to pass
16:06:33 dansmith I think we need to see it working on a wide scale for a substantial period of time,
16:06:54 dansmith because this pattern of making it work for two weeks until it breaks again doesn't really translate to quality
16:07:39 gibi dansmith: yes, you have a point
16:08:06 gibi dansmith: does it only triggers to vmware related patcheS? because then there will be not many changes that triggers them
16:08:25 dansmith AFAICT, it's manually triggered it seems
16:09:06 gibi that feels like cheating for a CI machinery
16:39:13 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674
16:59:08 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697
17:05:12 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697
17:05:24 openstack Launchpad bug 1867840 in OpenStack Compute (nova) "os-flavor-access API policy should be admin only" [Undecided,New] - Assigned to Ghanshyam Mann (ghanshyammann)
17:05:24 gmann johnthetubaguy: list flavor-access policy is little wired. old defaults are admin-or-owner which i think should be admin as multiple projects can have access to single flavor so owner is not suitable things for flavor. ^^ updating it with more accurate new defaults. you can see details in bug https://bugs.launchpad.net/nova/+bug/1867840
17:06:44 gmann i did not fix the old default as bug because that can break existing deployment. so thinking to keep supporting them and cleanup during we move to system scope by default
17:53:32 openstackgerrit Merged openstack/nova master: db: Remove unused ec2 DB APIs https://review.opendev.org/698880
18:05:02 openstackgerrit Merged openstack/nova master: libvirt: Check the guest support UEFI https://review.opendev.org/685678
18:10:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475
18:10:51 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759
18:13:36 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679
18:13:52 openstackgerrit Balazs Gibizer proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766
18:15:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680
18:15:25 gibi dansmith: updated the whole qos series ^^
18:15:32 dansmith gibi: sweet
18:15:41 gibi dansmith: thanks for the review
18:15:52 dansmith np, will hit those soon
18:22:45 sean-k-mooney lyarwood: it looks like your evacuate change is hitting some where errors https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7789
18:23:37 sean-k-mooney it disconnect an iscsi volume then reports rbd issues and finally fails with a virtual interface createion faild error
18:23:44 sean-k-mooney although i dont know why that is being raided
19:03:23 lyarwood sean-k-mooney: which one was that sorry?
19:03:49 lyarwood sean-k-mooney: oh the evacuation fix
19:03:58 lyarwood sean-k-mooney: it looked like the same issue again in console.log
19:04:04 lyarwood sean-k-mooney: I didn't even think to look at n-cpu sorry
19:05:15 sean-k-mooney the n-cpu logs honestly confuse me
19:05:35 sean-k-mooney i did not look at teh code but i think the virutal interface creation excpetion is a redheiring
19:07:33 lyarwood sean-k-mooney: https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7760 it's the same issue
19:08:38 lyarwood sean-k-mooney: we hit the vif timeout first and call disconnect_volume
19:08:58 lyarwood sean-k-mooney: I think the RBD cleanup failures are due to the domain still being active on the subnode
19:09:50 lyarwood oh right because ceph.sh only sets up ephemeral storage on ceph
19:09:58 lyarwood not c-vol
19:10:00 prometheanfire looks like alembic is still failing for nova / neutron https://review.opendev.org/708706
19:10:05 sean-k-mooney ah ok
19:10:29 sean-k-mooney the ceph thing was confusing me
19:10:51 sean-k-mooney and ya that kind of makes sense
19:11:12 sean-k-mooney so we not only need to stop the compute agent but the vms on the sub node too
19:11:37 sean-k-mooney which i guess makes sense
19:12:00 sean-k-mooney so we should stop the compute agent. then use libvirt to stop all the vms on the subnode
19:12:06 sean-k-mooney then do an evacuate
19:16:14 lyarwood sean-k-mooney: kk, I'll give that a go.
19:16:58 lyarwood sean-k-mooney: here's a thought, shouldn't all devstack@* services be stopped on the subnode?
19:17:25 sean-k-mooney i was consider doing a power off on the subnode
19:17:31 sean-k-mooney but i think we need logs from it
19:18:02 sean-k-mooney but yes you could do "sudo systemctl stop devstack@*"
19:18:29 sean-k-mooney that would be totally valid to do provide we dont install cpeh there
19:19:00 sean-k-mooney if we are installing ceph in the devstack slice then that woudl be an issue
19:22:20 noonedeadpunk hi! is there any reason (like security or other) why flavor transition from public to private or vice versa is not supported?
19:22:47 sean-k-mooney noonedeadpunk: private flavors was a half impleneted feature that is not commonly used
19:23:18 sean-k-mooney noonedeadpunk: so support was not added.
19:23:25 noonedeadpunk oh really? Like I was using it pretty much in several clouds

Earlier   Later