| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-16 | |||
| 19:29:09 | melwitt | this was the proposed fix https://review.openstack.org/#/c/318731/2/nova/compute/manager.py | |
| 19:29:14 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 19:29:15 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 19:29:35 | melwitt | orly | |
| 19:30:23 | mriedem | i'm thinking of this i think https://review.openstack.org/#/c/499237 | |
| 19:30:35 | melwitt | code looks the same AFAICT https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L640 | |
| 19:30:51 | melwitt | oh, okay. that should help then | |
| 19:31:02 | mriedem | right, if scheduling fails, set the migration status to 'error' | |
| 19:31:08 | mriedem | so when restarting the compute, the migration is not 'accepted' | |
| 19:31:11 | mriedem | or 'done' | |
| 19:31:16 | melwitt | maybe that bug could be considered closed with that fix, but we'd probably wanna write a test to make sure | |
| 19:31:25 | melwitt | yep, gotcha | |
| 19:32:10 | mriedem | i think you'd just have to update this test https://review.openstack.org/#/c/499237/4/nova/tests/functional/regressions/test_bug_1713783.py | |
| 19:32:12 | mriedem | to restart the compute | |
| 19:33:13 | melwitt | that would be sweet if that's all it'd take | |
| 19:33:28 | cfriesen | mriedem: I haven't reported an upstream bug yet, was trying to figure out if it was an upstream issue or not. My coworker reports that with the two fixes from stable/pike it doesn't fix the problem immediately but after some time the record has been updated in the DB. | |
| 19:34:03 | mriedem | after some time...like running the online data migrations? | |
| 19:34:11 | mriedem | or archiving deleted rows? | |
| 19:34:22 | cfriesen | melwitt: will the "read_deleted=yes" context change from https://git.openstack.org/cgit/openstack/nova/commit/?id=1407079d4008c6304799dd83f5bf4ba505d8e438 override the default model_query behaviour? | |
| 19:34:40 | mriedem | yes | |
| 19:34:47 | mriedem | context.read_deleted defaults to 'no' | |
| 19:35:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Report the allocated size of preallocated file based disks https://review.openstack.org/561704 | |
| 19:37:25 | cfriesen | mriedem: my coworker says he didn't do any online data migrations or archiving rows or anything "interesting". He suspects something querying the instances. | |
| 19:38:06 | cfriesen | running the online data migration doesn't normally affect deleted services, but applying "with utils.temporary_mutation(self._context, read_deleted='yes'):" to the online migration apparently fixed it all right up. | |
| 19:38:37 | melwitt | well, if the ordering of the returned dupe services (deleted one and the new one) is random, then eventually it would work fine if the non-deleted one was first, right? | |
| 19:38:44 | mriedem | cfriesen: ok so now you're saying that it's fixed? | |
| 19:39:11 | mriedem | i.e. https://review.openstack.org/#/c/545988/ | |
| 19:39:27 | mriedem | since everything is going to happen to that service with that mutated context, i expect it will update the deleted service with a uuid | |
| 19:39:56 | melwitt | yeah like, eventually maybe the non-deleted one is returned "first" in the list and then it goes through with the update and then there won't be a problem from then on | |
| 19:40:03 | mriedem | i just need to know if i should spend time trying to write a recreate test | |
| 19:40:11 | cfriesen | mriedem: that commit did not immediately fix things (he still hit some ServiceNotFound errors after applying it), but eventually the service did get a uuid | |
| 19:40:36 | melwitt | I think there's still a bug where for lazy-load of services it needs to not read deleted ones | |
| 19:41:15 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 19:41:16 | openstackgerrit | Jackie Truong proposed openstack/nova master: Plumb trusted_certs through libvirt driver image paths https://review.openstack.org/561262 | |
| 19:41:17 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 19:41:18 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 19:41:36 | melwitt | I hope services don't allow dupe by name if they are both non-deleted | |
| 19:41:38 | cfriesen | melwitt: arguably for the instance->service mapping the bug is that it maps by name | |
| 19:41:44 | melwitt | because then there would be a problem anyways | |
| 19:42:00 | mriedem | https://review.openstack.org/#/c/545988/ | |
| 19:42:01 | melwitt | cfriesen: yeah, true. | |
| 19:42:02 | mriedem | oops | |
| 19:42:11 | tmcm | sorry to repeat myself, but is it possible to run mitaka-based compute nodes with a queens nova-scheduler/placement? i'm looking at https://docs.openstack.org/nova/latest/user/placement.html but it's not clear to me from what i read there | |
| 19:42:11 | mriedem | name="uniq_services0host0binary0deleted"), | |
| 19:42:11 | mriedem | schema.UniqueConstraint("host", "binary", "deleted", | |
| 19:42:11 | mriedem | name="uniq_services0host0topic0deleted"), | |
| 19:42:11 | mriedem | schema.UniqueConstraint("host", "topic", "deleted", | |
| 19:42:35 | mriedem | tmcm: no | |
| 19:42:45 | tmcm | ok, thanks | |
| 19:42:46 | mriedem | it might work, but not by intention | |
| 19:42:49 | mriedem | we support n-1 computes | |
| 19:42:59 | mriedem | so controller can be queens and then computes can be pike+ | |
| 19:43:05 | tmcm | understood, thanks | |
| 19:43:17 | mriedem | at certain points we randomly delete compat code that's older than n-1 | |
| 19:44:10 | mriedem | tmcm: this should be a bit more clear https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | |
| 19:45:26 | cfriesen | mriedem: for a recreate test could we add a new DB entry with a deleted service that has the same name as the existing service? | |
| 19:46:16 | tmcm | mriedem: got it. i'd actually worked through that document already and my queens controller was happy from a db standpoint, i was just stuck at: | |
| 19:46:18 | tmcm | | Placement service and need to be upgraded and/or fixed. | | |
| 19:46:18 | tmcm | | This means no compute nodes are reporting into the | | |
| 19:46:18 | tmcm | | service but there are 80 compute nodes in the deployment. | | |
| 19:46:18 | tmcm | | Details: There are no compute resource providers in the Placement | | |
| 19:46:18 | tmcm | | Result: Warning | | |
| 19:46:18 | tmcm | | Check: Resource Providers | | |
| 19:46:18 | tmcm | +-------------------------------------------------------------------+ | |
| 19:46:19 | tmcm | | https://docs.openstack.org/nova/latest/user/placement.html | | |
| 19:46:19 | tmcm | | See | | |
| 19:46:20 | tmcm | +-------------------------------------------------------------------+ | |
| 19:46:20 | tmcm | | for more details. | | |
| 19:46:25 | tmcm | on the compute side | |
| 19:46:31 | openstackgerrit | Merged openstack/nova master: List instances performace optimization https://review.openstack.org/507854 | |
| 19:46:33 | tmcm | well, from the compute standpoint, i mean | |
| 19:47:27 | tmcm | so if there was any way to force the mitaka compute nodes to use the placement service, i might be OK | |
| 19:47:39 | artom | tmcm, there isn't, the code didn't exist in mitaka :) | |
| 19:47:46 | tmcm | got it | |
| 19:47:50 | tmcm | thanks to both of you | |
| 19:47:50 | artom | (Err, right?) | |
| 19:47:55 | tmcm | i didn't see it in there | |
| 19:48:12 | artom | This self-doubt thing is cripling | |
| 19:49:06 | mriedem | artom: yes, new in newton | |
| 19:49:13 | mriedem | optional in newton, required starting in ocata | |
| 19:49:16 | cfriesen | mriedem: apparently "nova show" on the instance will result in adding the uuid onto the service, but "nova list" hits the ServiceNotFound error | |
| 19:49:20 | artom | mriedem, thank you :) | |
| 19:49:33 | melwitt | you'd have to upgrade your compute nodes to at least newton to make them report in to placement. might even be ocata, not sure | |
| 19:49:44 | mriedem | cfriesen: those should both go down the same code path as far as i know, but maybe not | |
| 19:50:48 | melwitt | I think it would be random depending on the ordering of the returned dupe services | |
| 19:54:33 | melwitt | nevermind, it looks like it should try to generate a UUID for both services that match the instance via host, so it should fail reliably | |
| 19:54:35 | sean-k-mooney | dansmith: just finished looking at the patch. the broad strokes look fine. i left some nits inline. main concern is the name attribute of the driver field. i think we should just not set it for safty across livemirate in the future. anyway i left comments in the patch but over all it looks ok | |
| 19:56:10 | sean-k-mooney | dansmith: i might try an apply it locally tomorow and see if i have any issues. anyway im going to head for the evening. | |
| 19:57:12 | dansmith | sean-k-mooney: okay just to be sure, you know we're already setting it to vhost in almost all cases now right? | |
| 19:57:24 | dansmith | sean-k-mooney: it's done in _get_virtio_mq_settings() | |
| 19:57:43 | sean-k-mooney | dansmith: am i dont think we do that for vhost-user ports however | |
| 19:57:51 | dansmith | sean-k-mooney: we do | |
| 19:57:56 | dansmith | sean-k-mooney: even though it is wrong | |
| 19:57:59 | dansmith | that's the point :) | |
| 19:58:09 | sean-k-mooney | when we request vhost multiqueue only right | |
| 19:58:16 | dansmith | no | |
| 19:58:33 | dansmith | if we're in kvm, and MODEL_VIRTIO, we call that method and it sets it to vhost always I think | |
| 19:58:56 | TheJulia | efried: your python-ironicclient patch, Would you be okay if I updated the commit message? | |
| 19:59:08 | dansmith | sean-k-mooney: oh, actually, I guess you're right | |
| 19:59:16 | dansmith | sean-k-mooney: sahid said the opposite and I looked but didn't see | |