| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-16 | |||
| 19:07:11 | melwitt | so how did the service_update get the service id ... | |
| 19:07:55 | dansmith | it would be if we read_deleted when we looked it up | |
| 19:08:01 | dansmith | and got the first older deleted one by name | |
| 19:08:11 | dansmith | I haven't looked back into his latest stack trace, fyi | |
| 19:08:55 | melwitt | okay. yeah, the stacktrace shows ServiceNotFound being raised from service_update as service_update is trying to update via id and the id is not found because model_query | |
| 19:09:18 | dansmith | right | |
| 19:09:58 | mriedem | "the sequence of events was: boot instance on compute node, migrate instance, delete compute node and service, upgrade to pike, create new compute node with same name, migrate instance back to that compute node." | |
| 19:10:01 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | |
| 19:10:08 | melwitt | I haven't made the connection yet but somehow the service object associated with the instance is the deleted one. which I thought would be expected if you deleted the service out from under the instance. like, what would ever update the instance to point at a new service? I guess you say by name, which would find more than one matching service, and it's happening to find the deleted one first? | |
| 19:10:17 | dansmith | melwitt: it can't be | |
| 19:10:20 | dansmith | because it's only by name | |
| 19:10:50 | dansmith | I didn't see where they ened up starting down this path, I thought it was in the "upgrade to pike" section, | |
| 19:10:58 | melwitt | but two have the same name, the deleted one and the new one | |
| 19:11:02 | dansmith | and that they were implying we were handling deleted services in the migration | |
| 19:11:07 | dansmith | right | |
| 19:11:10 | mriedem | also waiting to hear if they have this https://review.openstack.org/#/c/545988/ | |
| 19:11:11 | melwitt | I haven't yet connected where the query is happening that would get the two returned | |
| 19:11:40 | melwitt | because like you all said earlier, that should be filtering out deleted rows, we would expect | |
| 19:11:46 | dansmith | yeah | |
| 19:12:07 | mriedem | if they have https://review.openstack.org/#/c/545988/ they will query deleted records too | |
| 19:12:33 | dansmith | mriedem: oh I didn't get the connection this morning you were making | |
| 19:12:44 | dansmith | mriedem: that we're using a read_deleted context to get the instance and thus the service too? | |
| 19:12:51 | mriedem | yes | |
| 19:12:55 | dansmith | gotcha that makes sense | |
| 19:13:42 | melwitt | mriedem: I think we should probably go with a specless blueprint on that one, so it gets bundled with the other new features | |
| 19:14:33 | mriedem | ack | |
| 19:14:50 | melwitt | thanks | |
| 19:19:01 | mriedem | trying to think of a test to recreate cfriesen's issue, could start two compute services, create the instance on one, migrate to the other, delete the old compute service/node, fake the upgrade to pike by deleting the uuid on the 2 services, start a new compute service with the same name as the old one, and then migrate the instance back to the other host | |
| 19:19:13 | mriedem | in his stacktrace, he's not doing a migration, simply listing instances | |
| 19:20:06 | mriedem | which unconditionally joins on services https://github.com/openstack/nova/blob/f0400313f30843fba114584f582b93f7cb5e5e48/nova/api/openstack/compute/servers.py#L295 | |
| 19:20:19 | melwitt | sounds like that should do it | |
| 19:21:28 | mriedem | cfriesen: have you reported an upstream bug for this issue? | |
| 19:23:54 | mriedem | bauzas: https://bugs.launchpad.net/nova/+bug/1764460 | |
| 19:23:54 | openstack | Launchpad bug 1764460 in OpenStack Compute (nova) queens "Cannot hard reboot an instance in error state" [High,Confirmed] | |
| 19:24:07 | mriedem | looks like a regression in the libvirt driver due to the mdev stuff in queens | |
| 19:25:49 | mriedem | this is fun https://bugs.launchpad.net/nova/+bug/1764385 | |
| 19:25:49 | openstack | Launchpad bug 1764385 in OpenStack Compute (nova) "no intimation to the admin that nova-api is stopped during execution of polling compute" [Undecided,Incomplete] | |
| 19:26:24 | melwitt | on the libvirt one, looks like a simple fix at least | |
| 19:27:54 | melwitt | I saw this old bug recently, evacuations that failed to schedule end up getting nuked during nova-compute restart https://bugs.launchpad.net/nova/+bug/1583504 | |
| 19:27:54 | openstack | Launchpad bug 1583504 in OpenStack Compute (nova) "The instances which didn't be evacuated will be destroyed when the nova-compute service is restarted." [Undecided,Confirmed] - Assigned to Margarita Shakhova (shakhova-margarita) | |
| 19:28:35 | melwitt | because the logic assumes that 'accepted' migration means it succeeded. I'm not sure what the original reasoning was for including 'accepted' in the logic (along with 'done') | |
| 19:28:36 | mriedem | hmm, is that from before the evac robustification to use migration records? | |
| 19:28:42 | mriedem | oh | |
| 19:29:03 | mriedem | that recently changed, the logic in the compute i mean, | |
| 19:29:08 | mriedem | it was discussed at the ptg in denver | |
| 19:29:09 | melwitt | this was the proposed fix https://review.openstack.org/#/c/318731/2/nova/compute/manager.py | |
| 19:29:14 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 19:29:15 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 19:29:35 | melwitt | orly | |
| 19:30:23 | mriedem | i'm thinking of this i think https://review.openstack.org/#/c/499237 | |
| 19:30:35 | melwitt | code looks the same AFAICT https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L640 | |
| 19:30:51 | melwitt | oh, okay. that should help then | |
| 19:31:02 | mriedem | right, if scheduling fails, set the migration status to 'error' | |
| 19:31:08 | mriedem | so when restarting the compute, the migration is not 'accepted' | |
| 19:31:11 | mriedem | or 'done' | |
| 19:31:16 | melwitt | maybe that bug could be considered closed with that fix, but we'd probably wanna write a test to make sure | |
| 19:31:25 | melwitt | yep, gotcha | |
| 19:32:10 | mriedem | i think you'd just have to update this test https://review.openstack.org/#/c/499237/4/nova/tests/functional/regressions/test_bug_1713783.py | |
| 19:32:12 | mriedem | to restart the compute | |
| 19:33:13 | melwitt | that would be sweet if that's all it'd take | |
| 19:33:28 | cfriesen | mriedem: I haven't reported an upstream bug yet, was trying to figure out if it was an upstream issue or not. My coworker reports that with the two fixes from stable/pike it doesn't fix the problem immediately but after some time the record has been updated in the DB. | |
| 19:34:03 | mriedem | after some time...like running the online data migrations? | |
| 19:34:11 | mriedem | or archiving deleted rows? | |
| 19:34:22 | cfriesen | melwitt: will the "read_deleted=yes" context change from https://git.openstack.org/cgit/openstack/nova/commit/?id=1407079d4008c6304799dd83f5bf4ba505d8e438 override the default model_query behaviour? | |
| 19:34:40 | mriedem | yes | |
| 19:34:47 | mriedem | context.read_deleted defaults to 'no' | |
| 19:35:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Report the allocated size of preallocated file based disks https://review.openstack.org/561704 | |
| 19:37:25 | cfriesen | mriedem: my coworker says he didn't do any online data migrations or archiving rows or anything "interesting". He suspects something querying the instances. | |
| 19:38:06 | cfriesen | running the online data migration doesn't normally affect deleted services, but applying "with utils.temporary_mutation(self._context, read_deleted='yes'):" to the online migration apparently fixed it all right up. | |
| 19:38:37 | melwitt | well, if the ordering of the returned dupe services (deleted one and the new one) is random, then eventually it would work fine if the non-deleted one was first, right? | |
| 19:38:44 | mriedem | cfriesen: ok so now you're saying that it's fixed? | |
| 19:39:11 | mriedem | i.e. https://review.openstack.org/#/c/545988/ | |
| 19:39:27 | mriedem | since everything is going to happen to that service with that mutated context, i expect it will update the deleted service with a uuid | |
| 19:39:56 | melwitt | yeah like, eventually maybe the non-deleted one is returned "first" in the list and then it goes through with the update and then there won't be a problem from then on | |
| 19:40:03 | mriedem | i just need to know if i should spend time trying to write a recreate test | |
| 19:40:11 | cfriesen | mriedem: that commit did not immediately fix things (he still hit some ServiceNotFound errors after applying it), but eventually the service did get a uuid | |
| 19:40:36 | melwitt | I think there's still a bug where for lazy-load of services it needs to not read deleted ones | |
| 19:41:15 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 19:41:16 | openstackgerrit | Jackie Truong proposed openstack/nova master: Plumb trusted_certs through libvirt driver image paths https://review.openstack.org/561262 | |
| 19:41:17 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 19:41:18 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 19:41:36 | melwitt | I hope services don't allow dupe by name if they are both non-deleted | |
| 19:41:38 | cfriesen | melwitt: arguably for the instance->service mapping the bug is that it maps by name | |
| 19:41:44 | melwitt | because then there would be a problem anyways | |
| 19:42:00 | mriedem | https://review.openstack.org/#/c/545988/ | |
| 19:42:01 | melwitt | cfriesen: yeah, true. | |
| 19:42:02 | mriedem | oops | |
| 19:42:11 | mriedem | schema.UniqueConstraint("host", "topic", "deleted", | |
| 19:42:11 | mriedem | name="uniq_services0host0topic0deleted"), | |
| 19:42:11 | mriedem | schema.UniqueConstraint("host", "binary", "deleted", | |
| 19:42:11 | mriedem | name="uniq_services0host0binary0deleted"), | |
| 19:42:11 | tmcm | sorry to repeat myself, but is it possible to run mitaka-based compute nodes with a queens nova-scheduler/placement? i'm looking at https://docs.openstack.org/nova/latest/user/placement.html but it's not clear to me from what i read there | |
| 19:42:35 | mriedem | tmcm: no | |
| 19:42:45 | tmcm | ok, thanks | |
| 19:42:46 | mriedem | it might work, but not by intention | |
| 19:42:49 | mriedem | we support n-1 computes | |
| 19:42:59 | mriedem | so controller can be queens and then computes can be pike+ | |
| 19:43:05 | tmcm | understood, thanks | |
| 19:43:17 | mriedem | at certain points we randomly delete compat code that's older than n-1 | |
| 19:44:10 | mriedem | tmcm: this should be a bit more clear https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | |