| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-16 | |||
| 19:02:45 | melwitt | cfriesen, mriedem, dansmith: looking at the code, it makes sense what's happening. the service was deleted while the instance still referred to it, then when you list the instance, it tries to generate a UUID for the service (that is deleted) and the service_update method tries to 'get' the service record by id and it gets ServiceNotFound because model_query will automatically filter out deleted things https://github.com/openstack/nova/ | |
| 19:02:50 | melwitt | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L554 | |
| 19:04:03 | tmcm | is it possible to run mitaka-based compute nodes with a queens nova-scheduler/placement? | |
| 19:04:23 | tmcm | i'm looking at https://docs.openstack.org/nova/latest/user/placement.html but it's not clear to me from what i read there | |
| 19:04:35 | melwitt | I would think if you're deleting services out from under live instances, after you create new services to replace them, wouldn't you have to manually associate the new service with existing instances? else it seems like other bad things could happen | |
| 19:05:37 | dansmith | they're only associated by name, right? | |
| 19:05:59 | melwitt | the db api stuff implied the backref is by service id | |
| 19:06:04 | melwitt | but lemme re-look | |
| 19:06:29 | dansmith | I don't think we have a service_id on instance, | |
| 19:06:31 | dansmith | I thought I looked | |
| 19:06:43 | dansmith | although I was looking at computenode specifically | |
| 19:06:45 | mriedem | i wonder why "from_db_object" is public in their stacktraces....totally unrelated | |
| 19:06:52 | melwitt | yeah looks like backref by host and binary | |
| 19:07:11 | melwitt | so how did the service_update get the service id ... | |
| 19:07:55 | dansmith | it would be if we read_deleted when we looked it up | |
| 19:08:01 | dansmith | and got the first older deleted one by name | |
| 19:08:11 | dansmith | I haven't looked back into his latest stack trace, fyi | |
| 19:08:55 | melwitt | okay. yeah, the stacktrace shows ServiceNotFound being raised from service_update as service_update is trying to update via id and the id is not found because model_query | |
| 19:09:18 | dansmith | right | |
| 19:09:58 | mriedem | "the sequence of events was: boot instance on compute node, migrate instance, delete compute node and service, upgrade to pike, create new compute node with same name, migrate instance back to that compute node." | |
| 19:10:01 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | |
| 19:10:08 | melwitt | I haven't made the connection yet but somehow the service object associated with the instance is the deleted one. which I thought would be expected if you deleted the service out from under the instance. like, what would ever update the instance to point at a new service? I guess you say by name, which would find more than one matching service, and it's happening to find the deleted one first? | |
| 19:10:17 | dansmith | melwitt: it can't be | |
| 19:10:20 | dansmith | because it's only by name | |
| 19:10:50 | dansmith | I didn't see where they ened up starting down this path, I thought it was in the "upgrade to pike" section, | |
| 19:10:58 | melwitt | but two have the same name, the deleted one and the new one | |
| 19:11:02 | dansmith | and that they were implying we were handling deleted services in the migration | |
| 19:11:07 | dansmith | right | |
| 19:11:10 | mriedem | also waiting to hear if they have this https://review.openstack.org/#/c/545988/ | |
| 19:11:11 | melwitt | I haven't yet connected where the query is happening that would get the two returned | |
| 19:11:40 | melwitt | because like you all said earlier, that should be filtering out deleted rows, we would expect | |
| 19:11:46 | dansmith | yeah | |
| 19:12:07 | mriedem | if they have https://review.openstack.org/#/c/545988/ they will query deleted records too | |
| 19:12:33 | dansmith | mriedem: oh I didn't get the connection this morning you were making | |
| 19:12:44 | dansmith | mriedem: that we're using a read_deleted context to get the instance and thus the service too? | |
| 19:12:51 | mriedem | yes | |
| 19:12:55 | dansmith | gotcha that makes sense | |
| 19:13:42 | melwitt | mriedem: I think we should probably go with a specless blueprint on that one, so it gets bundled with the other new features | |
| 19:14:33 | mriedem | ack | |
| 19:14:50 | melwitt | thanks | |
| 19:19:01 | mriedem | trying to think of a test to recreate cfriesen's issue, could start two compute services, create the instance on one, migrate to the other, delete the old compute service/node, fake the upgrade to pike by deleting the uuid on the 2 services, start a new compute service with the same name as the old one, and then migrate the instance back to the other host | |
| 19:19:13 | mriedem | in his stacktrace, he's not doing a migration, simply listing instances | |
| 19:20:06 | mriedem | which unconditionally joins on services https://github.com/openstack/nova/blob/f0400313f30843fba114584f582b93f7cb5e5e48/nova/api/openstack/compute/servers.py#L295 | |
| 19:20:19 | melwitt | sounds like that should do it | |
| 19:21:28 | mriedem | cfriesen: have you reported an upstream bug for this issue? | |
| 19:23:54 | openstack | Launchpad bug 1764460 in OpenStack Compute (nova) queens "Cannot hard reboot an instance in error state" [High,Confirmed] | |
| 19:23:54 | mriedem | bauzas: https://bugs.launchpad.net/nova/+bug/1764460 | |
| 19:24:07 | mriedem | looks like a regression in the libvirt driver due to the mdev stuff in queens | |
| 19:25:49 | openstack | Launchpad bug 1764385 in OpenStack Compute (nova) "no intimation to the admin that nova-api is stopped during execution of polling compute" [Undecided,Incomplete] | |
| 19:25:49 | mriedem | this is fun https://bugs.launchpad.net/nova/+bug/1764385 | |
| 19:26:24 | melwitt | on the libvirt one, looks like a simple fix at least | |
| 19:27:54 | openstack | Launchpad bug 1583504 in OpenStack Compute (nova) "The instances which didn't be evacuated will be destroyed when the nova-compute service is restarted." [Undecided,Confirmed] - Assigned to Margarita Shakhova (shakhova-margarita) | |
| 19:27:54 | melwitt | I saw this old bug recently, evacuations that failed to schedule end up getting nuked during nova-compute restart https://bugs.launchpad.net/nova/+bug/1583504 | |
| 19:28:35 | melwitt | because the logic assumes that 'accepted' migration means it succeeded. I'm not sure what the original reasoning was for including 'accepted' in the logic (along with 'done') | |
| 19:28:36 | mriedem | hmm, is that from before the evac robustification to use migration records? | |
| 19:28:42 | mriedem | oh | |
| 19:29:03 | mriedem | that recently changed, the logic in the compute i mean, | |
| 19:29:08 | mriedem | it was discussed at the ptg in denver | |
| 19:29:09 | melwitt | this was the proposed fix https://review.openstack.org/#/c/318731/2/nova/compute/manager.py | |
| 19:29:14 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 19:29:15 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 19:29:35 | melwitt | orly | |
| 19:30:23 | mriedem | i'm thinking of this i think https://review.openstack.org/#/c/499237 | |
| 19:30:35 | melwitt | code looks the same AFAICT https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L640 | |
| 19:30:51 | melwitt | oh, okay. that should help then | |
| 19:31:02 | mriedem | right, if scheduling fails, set the migration status to 'error' | |
| 19:31:08 | mriedem | so when restarting the compute, the migration is not 'accepted' | |
| 19:31:11 | mriedem | or 'done' | |
| 19:31:16 | melwitt | maybe that bug could be considered closed with that fix, but we'd probably wanna write a test to make sure | |
| 19:31:25 | melwitt | yep, gotcha | |
| 19:32:10 | mriedem | i think you'd just have to update this test https://review.openstack.org/#/c/499237/4/nova/tests/functional/regressions/test_bug_1713783.py | |
| 19:32:12 | mriedem | to restart the compute | |
| 19:33:13 | melwitt | that would be sweet if that's all it'd take | |
| 19:33:28 | cfriesen | mriedem: I haven't reported an upstream bug yet, was trying to figure out if it was an upstream issue or not. My coworker reports that with the two fixes from stable/pike it doesn't fix the problem immediately but after some time the record has been updated in the DB. | |
| 19:34:03 | mriedem | after some time...like running the online data migrations? | |
| 19:34:11 | mriedem | or archiving deleted rows? | |
| 19:34:22 | cfriesen | melwitt: will the "read_deleted=yes" context change from https://git.openstack.org/cgit/openstack/nova/commit/?id=1407079d4008c6304799dd83f5bf4ba505d8e438 override the default model_query behaviour? | |
| 19:34:40 | mriedem | yes | |
| 19:34:47 | mriedem | context.read_deleted defaults to 'no' | |
| 19:35:08 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Report the allocated size of preallocated file based disks https://review.openstack.org/561704 | |
| 19:37:25 | cfriesen | mriedem: my coworker says he didn't do any online data migrations or archiving rows or anything "interesting". He suspects something querying the instances. | |
| 19:38:06 | cfriesen | running the online data migration doesn't normally affect deleted services, but applying "with utils.temporary_mutation(self._context, read_deleted='yes'):" to the online migration apparently fixed it all right up. | |
| 19:38:37 | melwitt | well, if the ordering of the returned dupe services (deleted one and the new one) is random, then eventually it would work fine if the non-deleted one was first, right? | |
| 19:38:44 | mriedem | cfriesen: ok so now you're saying that it's fixed? | |
| 19:39:11 | mriedem | i.e. https://review.openstack.org/#/c/545988/ | |
| 19:39:27 | mriedem | since everything is going to happen to that service with that mutated context, i expect it will update the deleted service with a uuid | |
| 19:39:56 | melwitt | yeah like, eventually maybe the non-deleted one is returned "first" in the list and then it goes through with the update and then there won't be a problem from then on | |
| 19:40:03 | mriedem | i just need to know if i should spend time trying to write a recreate test | |
| 19:40:11 | cfriesen | mriedem: that commit did not immediately fix things (he still hit some ServiceNotFound errors after applying it), but eventually the service did get a uuid | |
| 19:40:36 | melwitt | I think there's still a bug where for lazy-load of services it needs to not read deleted ones | |
| 19:41:15 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 19:41:16 | openstackgerrit | Jackie Truong proposed openstack/nova master: Plumb trusted_certs through libvirt driver image paths https://review.openstack.org/561262 | |
| 19:41:17 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 19:41:18 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 19:41:36 | melwitt | I hope services don't allow dupe by name if they are both non-deleted | |
| 19:41:38 | cfriesen | melwitt: arguably for the instance->service mapping the bug is that it maps by name | |
| 19:41:44 | melwitt | because then there would be a problem anyways | |
| 19:42:00 | mriedem | https://review.openstack.org/#/c/545988/ | |
| 19:42:01 | melwitt | cfriesen: yeah, true. | |
| 19:42:02 | mriedem | oops | |