| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-03-18 | |||
| 15:18:37 | gibi | dansmith: so I think we are safe there | |
| 15:18:43 | dansmith | seems a little risky to just be blind about these early cases, and that it's going to be a source of bugs later where cyborg uses it for spawn, for example but unshelve overwrites those | |
| 15:19:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: gate: Ensure n-cpu is stopped on the subnode during evacuation testing https://review.opendev.org/713674 | |
| 15:19:14 | lyarwood | sean-k-mooney: ^ so I wonder if n-cpu running on the subnode is borking things here even if it is marked as down in the API? | |
| 15:20:04 | lyarwood | I can see the subnode refreshing the instance network info cache for example | |
| 15:20:16 | lyarwood | while we are trying to evacuate | |
| 15:20:17 | gibi | dansmith: the field is not persisted so I don't think we will overwrite existing data from the past. every user of the requested_resources needs to re-fill it for each operation | |
| 15:20:31 | dansmith | ahh, okay that makes sense | |
| 15:25:50 | sean-k-mooney | lyarwood: well the mark host down api has a prequisite that the host is actully down and the agents are stoped | |
| 15:26:39 | sean-k-mooney | lyarwood: if we voilate that expectation then perhaps it coudl cause issues | |
| 15:27:41 | sean-k-mooney | lyarwood: in the evacuate case we proably should be storping n-cpu on the compute host as it is not properly fenced if it is running and able to connect to the message bus/db | |
| 15:28:10 | lyarwood | sean-k-mooney: yup, if it's pinging neutron behind our backs I'm assuming that's going to cause issues when it comes to evacuting the instance from the node | |
| 15:29:09 | sean-k-mooney | well i have not check how the info cache works but unless its per host then it could replace the binding info in the cache with the stale prot bindin info for the orginal host | |
| 15:29:33 | sean-k-mooney | although im not sure if that would cause an issue | |
| 15:29:50 | sean-k-mooney | its certenly safter to not have it messing with the info cache | |
| 15:30:54 | sean-k-mooney | the warning is pretty clear in the api docs too on this point "tting a service forced down without completely fencing it will likely result in the corruption of VMs on that host." | |
| 15:32:32 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679 | |
| 15:32:33 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680 | |
| 15:33:13 | gibi | dansmith: fyi I retrofitted the service version check to the rest of the Ussuri qos move code ^^ | |
| 15:36:30 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | |
| 15:37:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682 | |
| 15:37:25 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | |
| 15:37:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Fix intermittently failing regression case https://review.opendev.org/713682 | |
| 15:52:40 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: [Community goal] Update contributor documentation https://review.opendev.org/712420 | |
| 16:01:59 | dansmith | gibi: can you look at my concern here real quick? https://review.opendev.org/#/c/713679/1/nova/api/openstack/compute/migrate_server.py | |
| 16:02:53 | gibi | dansmith: looking | |
| 16:03:27 | openstackgerrit | Merged openstack/os-resource-classes master: Switch to hacking 2.x https://review.opendev.org/710276 | |
| 16:03:43 | gibi | dansmith: make sense. I can reword the API error, and LOG the detailed info | |
| 16:04:02 | dansmith | gibi: okay, thanks | |
| 16:04:20 | dansmith | also, vmware ci did report on the un-deprecate patch... but it failed and only ran two tempest tests (both of which failed) | |
| 16:04:58 | gibi | dansmith: I will respin th whole series and fix your nits on the bottom too. | |
| 16:05:08 | dansmith | gibi: same api concern here: https://review.opendev.org/#/c/705475/6/nova/api/openstack/compute/shelve.py | |
| 16:05:22 | dansmith | gibi: cool, I'll circle back quickly when you do | |
| 16:05:23 | gibi | dansmith: yepp, and in the evac patch too | |
| 16:05:57 | gibi | dansmith: regarding vmware. I think it is fair to say that we need those self selected tempest test to pass | |
| 16:06:33 | dansmith | I think we need to see it working on a wide scale for a substantial period of time, | |
| 16:06:54 | dansmith | because this pattern of making it work for two weeks until it breaks again doesn't really translate to quality | |
| 16:07:39 | gibi | dansmith: yes, you have a point | |
| 16:08:06 | gibi | dansmith: does it only triggers to vmware related patcheS? because then there will be not many changes that triggers them | |
| 16:08:25 | dansmith | AFAICT, it's manually triggered it seems | |
| 16:09:06 | gibi | that feels like cheating for a CI machinery | |
| 16:39:13 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | |
| 16:59:08 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697 | |
| 17:05:12 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697 | |
| 17:05:24 | openstack | Launchpad bug 1867840 in OpenStack Compute (nova) "os-flavor-access API policy should be admin only" [Undecided,New] - Assigned to Ghanshyam Mann (ghanshyammann) | |
| 17:05:24 | gmann | johnthetubaguy: list flavor-access policy is little wired. old defaults are admin-or-owner which i think should be admin as multiple projects can have access to single flavor so owner is not suitable things for flavor. ^^ updating it with more accurate new defaults. you can see details in bug https://bugs.launchpad.net/nova/+bug/1867840 | |
| 17:06:44 | gmann | i did not fix the old default as bug because that can break existing deployment. so thinking to keep supporting them and cleanup during we move to system scope by default | |
| 17:53:32 | openstackgerrit | Merged openstack/nova master: db: Remove unused ec2 DB APIs https://review.opendev.org/698880 | |
| 18:05:02 | openstackgerrit | Merged openstack/nova master: libvirt: Check the guest support UEFI https://review.opendev.org/685678 | |
| 18:10:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable unshelve with qos ports https://review.opendev.org/705475 | |
| 18:10:51 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support unshelve with qos ports https://review.opendev.org/704759 | |
| 18:13:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for live migrate with qos https://review.opendev.org/713679 | |
| 18:13:52 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Merge qos related renos for Ussuri https://review.opendev.org/706766 | |
| 18:15:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add service version check for evacuate with qos https://review.opendev.org/713680 | |
| 18:15:25 | gibi | dansmith: updated the whole qos series ^^ | |
| 18:15:32 | dansmith | gibi: sweet | |
| 18:15:41 | gibi | dansmith: thanks for the review | |
| 18:15:52 | dansmith | np, will hit those soon | |
| 18:22:45 | sean-k-mooney | lyarwood: it looks like your evacuate change is hitting some where errors https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7789 | |
| 18:23:37 | sean-k-mooney | it disconnect an iscsi volume then reports rbd issues and finally fails with a virtual interface createion faild error | |
| 18:23:44 | sean-k-mooney | although i dont know why that is being raided | |
| 19:03:23 | lyarwood | sean-k-mooney: which one was that sorry? | |
| 19:03:49 | lyarwood | sean-k-mooney: oh the evacuation fix | |
| 19:03:58 | lyarwood | sean-k-mooney: it looked like the same issue again in console.log | |
| 19:04:04 | lyarwood | sean-k-mooney: I didn't even think to look at n-cpu sorry | |
| 19:05:15 | sean-k-mooney | the n-cpu logs honestly confuse me | |
| 19:05:35 | sean-k-mooney | i did not look at teh code but i think the virutal interface creation excpetion is a redheiring | |
| 19:07:33 | lyarwood | sean-k-mooney: https://zuul.opendev.org/t/openstack/build/daabc75c8992406f8a089a808c2553b9/log/logs/screen-n-cpu.txt#7760 it's the same issue | |
| 19:08:38 | lyarwood | sean-k-mooney: we hit the vif timeout first and call disconnect_volume | |
| 19:08:58 | lyarwood | sean-k-mooney: I think the RBD cleanup failures are due to the domain still being active on the subnode | |
| 19:09:50 | lyarwood | oh right because ceph.sh only sets up ephemeral storage on ceph | |
| 19:09:58 | lyarwood | not c-vol | |
| 19:10:00 | prometheanfire | looks like alembic is still failing for nova / neutron https://review.opendev.org/708706 | |
| 19:10:05 | sean-k-mooney | ah ok | |
| 19:10:29 | sean-k-mooney | the ceph thing was confusing me | |
| 19:10:51 | sean-k-mooney | and ya that kind of makes sense | |
| 19:11:12 | sean-k-mooney | so we not only need to stop the compute agent but the vms on the sub node too | |
| 19:11:37 | sean-k-mooney | which i guess makes sense | |
| 19:12:00 | sean-k-mooney | so we should stop the compute agent. then use libvirt to stop all the vms on the subnode | |
| 19:12:06 | sean-k-mooney | then do an evacuate | |
| 19:16:14 | lyarwood | sean-k-mooney: kk, I'll give that a go. | |
| 19:16:58 | lyarwood | sean-k-mooney: here's a thought, shouldn't all devstack@* services be stopped on the subnode? | |
| 19:17:25 | sean-k-mooney | i was consider doing a power off on the subnode | |
| 19:17:31 | sean-k-mooney | but i think we need logs from it | |
| 19:18:02 | sean-k-mooney | but yes you could do "sudo systemctl stop devstack@*" | |
| 19:18:29 | sean-k-mooney | that would be totally valid to do provide we dont install cpeh there | |
| 19:19:00 | sean-k-mooney | if we are installing ceph in the devstack slice then that woudl be an issue | |
| 19:22:20 | noonedeadpunk | hi! is there any reason (like security or other) why flavor transition from public to private or vice versa is not supported? | |
| 19:22:47 | sean-k-mooney | noonedeadpunk: private flavors was a half impleneted feature that is not commonly used | |
| 19:23:18 | sean-k-mooney | noonedeadpunk: so support was not added. | |
| 19:23:25 | noonedeadpunk | oh really? Like I was using it pretty much in several clouds | |
| 19:23:26 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP gate: Ensure n-cpu is stopped on the subnode during evacuation https://review.opendev.org/713674 | |
| 19:23:59 | lyarwood | sean-k-mooney: ^ awful hacks, no idea if that would even work but there we go. This would actually be easier to do if this was a playbook... | |
| 19:24:16 | noonedeadpunk | Ok, I see. Just wondering if it makes sense to look into it to propose a patch | |
| 19:25:47 | noonedeadpunk | As there's no blockers which are aware for everyone, will try that out | |
| 19:26:27 | sean-k-mooney | noonedeadpunk: its an api change which would require a microverion bump and a spec | |
| 19:26:36 | sean-k-mooney | so it cant be done as a bug fix | |
| 19:26:40 | sean-k-mooney | https://docs.openstack.org/api-ref/compute/?expanded=remove-flavor-access-from-tenant-removetenantaccess-action-detail#flavors-access-flavors-os-flavor-access | |
| 19:26:56 | sean-k-mooney | the flavor acess api is what is normally used instead | |
| 19:28:44 | noonedeadpunk | sean-k-mooney: the thing I'm missing is transition from is_public: true to is_public: false.... | |