| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-16 | |||
| 18:23:32 | cdent | yeah, was worth finding anyway | |
| 18:29:24 | openstackgerrit | Merged openstack/nova master: Don't log a warning for InstanceNotFound with deleted VIFs https://review.openstack.org/554591 | |
| 18:33:28 | sean-k-mooney | dansmith: i was on a call but yes | |
| 18:33:53 | dansmith | sean-k-mooney: hey, so we had a question on friday and I think you're the right person to ask | |
| 18:34:10 | dansmith | sean-k-mooney: we still have non-os_vif handling of vhostuser interfaces in vif.pym | |
| 18:34:12 | sean-k-mooney | somthing about not implemented error for os-vif | |
| 18:34:19 | dansmith | yeah | |
| 18:34:19 | dansmith | but it's not clear to me/us if that is still needed or not | |
| 18:34:37 | sean-k-mooney | dansmith: that is used by both vpp and snabswitch | |
| 18:34:45 | dansmith | okay | |
| 18:34:48 | dansmith | why? | |
| 18:35:24 | sean-k-mooney | in both cases vhost-user is not being used with ovs so they dont need to have somting plug it into a vswitch | |
| 18:35:59 | sean-k-mooney | in the vpp case i belive they have an agent on the plathform that poll for the creation fo the socket and just connects when it shows up | |
| 18:36:10 | dansmith | so is it always going to be that way? | |
| 18:36:38 | sean-k-mooney | dansmith: we could add a new plugin to os-vif if you want to move it out of tree | |
| 18:36:52 | dansmith | seems like it would be less confusing | |
| 18:36:58 | dansmith | maybe a comment about who/what uses that path would be nice | |
| 18:37:08 | dansmith | regardless, can you have a look over https://review.openstack.org/#/c/484997 ? | |
| 18:37:42 | sean-k-mooney | we didnt to that initally as they are not reference plugings but its basically a noop so i dont really see the harm in a generic_vhost_user pluging in os-vif | |
| 18:38:18 | sean-k-mooney | oh the queue lenght stuff sure ill take a look now | |
| 18:38:53 | dansmith | thanks | |
| 18:39:39 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757 | |
| 18:47:30 | efried | bauzas: still around? | |
| 18:50:59 | openstackgerrit | sahid proposed openstack/nova master: libvirt: fix wrong driver name for vhostuser interface https://review.openstack.org/561562 | |
| 18:53:53 | melwitt | dansmith: yeah, saw your comment and -1ed it earlier this morning to add visibility to the fact that the changes were reverted | |
| 18:54:04 | dansmith | melwitt: ack | |
| 19:01:43 | mriedem | melwitt: i'm not sure if you want a simple specless blueprint for this or just let it slide as a wishlist bug fix https://review.openstack.org/#/c/555861/ - note we wouldn't backport that either way | |
| 19:02:45 | melwitt | blob/master/nova/db/sqlalchemy/api.py#L554 | |
| 19:02:45 | melwitt | cfriesen, mriedem, dansmith: looking at the code, it makes sense what's happening. the service was deleted while the instance still referred to it, then when you list the instance, it tries to generate a UUID for the service (that is deleted) and the service_update method tries to 'get' the service record by id and it gets ServiceNotFound because model_query will automatically filter out deleted things https://github.com/openstack/nova/ | |
| 19:02:50 | melwitt | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L554 | |
| 19:04:03 | tmcm | is it possible to run mitaka-based compute nodes with a queens nova-scheduler/placement? | |
| 19:04:23 | tmcm | i'm looking at https://docs.openstack.org/nova/latest/user/placement.html but it's not clear to me from what i read there | |
| 19:04:35 | melwitt | I would think if you're deleting services out from under live instances, after you create new services to replace them, wouldn't you have to manually associate the new service with existing instances? else it seems like other bad things could happen | |
| 19:05:37 | dansmith | they're only associated by name, right? | |
| 19:05:59 | melwitt | the db api stuff implied the backref is by service id | |
| 19:06:04 | melwitt | but lemme re-look | |
| 19:06:29 | dansmith | I don't think we have a service_id on instance, | |
| 19:06:31 | dansmith | I thought I looked | |
| 19:06:43 | dansmith | although I was looking at computenode specifically | |
| 19:06:45 | mriedem | i wonder why "from_db_object" is public in their stacktraces....totally unrelated | |
| 19:06:52 | melwitt | yeah looks like backref by host and binary | |
| 19:07:11 | melwitt | so how did the service_update get the service id ... | |
| 19:07:55 | dansmith | it would be if we read_deleted when we looked it up | |
| 19:08:01 | dansmith | and got the first older deleted one by name | |
| 19:08:11 | dansmith | I haven't looked back into his latest stack trace, fyi | |
| 19:08:55 | melwitt | okay. yeah, the stacktrace shows ServiceNotFound being raised from service_update as service_update is trying to update via id and the id is not found because model_query | |
| 19:09:18 | dansmith | right | |
| 19:09:58 | mriedem | "the sequence of events was: boot instance on compute node, migrate instance, delete compute node and service, upgrade to pike, create new compute node with same name, migrate instance back to that compute node." | |
| 19:10:01 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | |
| 19:10:08 | melwitt | I haven't made the connection yet but somehow the service object associated with the instance is the deleted one. which I thought would be expected if you deleted the service out from under the instance. like, what would ever update the instance to point at a new service? I guess you say by name, which would find more than one matching service, and it's happening to find the deleted one first? | |
| 19:10:17 | dansmith | melwitt: it can't be | |
| 19:10:20 | dansmith | because it's only by name | |
| 19:10:50 | dansmith | I didn't see where they ened up starting down this path, I thought it was in the "upgrade to pike" section, | |
| 19:10:58 | melwitt | but two have the same name, the deleted one and the new one | |
| 19:11:02 | dansmith | and that they were implying we were handling deleted services in the migration | |
| 19:11:07 | dansmith | right | |
| 19:11:10 | mriedem | also waiting to hear if they have this https://review.openstack.org/#/c/545988/ | |
| 19:11:11 | melwitt | I haven't yet connected where the query is happening that would get the two returned | |
| 19:11:40 | melwitt | because like you all said earlier, that should be filtering out deleted rows, we would expect | |
| 19:11:46 | dansmith | yeah | |
| 19:12:07 | mriedem | if they have https://review.openstack.org/#/c/545988/ they will query deleted records too | |
| 19:12:33 | dansmith | mriedem: oh I didn't get the connection this morning you were making | |
| 19:12:44 | dansmith | mriedem: that we're using a read_deleted context to get the instance and thus the service too? | |
| 19:12:51 | mriedem | yes | |
| 19:12:55 | dansmith | gotcha that makes sense | |
| 19:13:42 | melwitt | mriedem: I think we should probably go with a specless blueprint on that one, so it gets bundled with the other new features | |
| 19:14:33 | mriedem | ack | |
| 19:14:50 | melwitt | thanks | |
| 19:19:01 | mriedem | trying to think of a test to recreate cfriesen's issue, could start two compute services, create the instance on one, migrate to the other, delete the old compute service/node, fake the upgrade to pike by deleting the uuid on the 2 services, start a new compute service with the same name as the old one, and then migrate the instance back to the other host | |
| 19:19:13 | mriedem | in his stacktrace, he's not doing a migration, simply listing instances | |
| 19:20:06 | mriedem | which unconditionally joins on services https://github.com/openstack/nova/blob/f0400313f30843fba114584f582b93f7cb5e5e48/nova/api/openstack/compute/servers.py#L295 | |
| 19:20:19 | melwitt | sounds like that should do it | |
| 19:21:28 | mriedem | cfriesen: have you reported an upstream bug for this issue? | |
| 19:23:54 | openstack | Launchpad bug 1764460 in OpenStack Compute (nova) queens "Cannot hard reboot an instance in error state" [High,Confirmed] | |
| 19:23:54 | mriedem | bauzas: https://bugs.launchpad.net/nova/+bug/1764460 | |
| 19:24:07 | mriedem | looks like a regression in the libvirt driver due to the mdev stuff in queens | |
| 19:25:49 | openstack | Launchpad bug 1764385 in OpenStack Compute (nova) "no intimation to the admin that nova-api is stopped during execution of polling compute" [Undecided,Incomplete] | |
| 19:25:49 | mriedem | this is fun https://bugs.launchpad.net/nova/+bug/1764385 | |
| 19:26:24 | melwitt | on the libvirt one, looks like a simple fix at least | |
| 19:27:54 | openstack | Launchpad bug 1583504 in OpenStack Compute (nova) "The instances which didn't be evacuated will be destroyed when the nova-compute service is restarted." [Undecided,Confirmed] - Assigned to Margarita Shakhova (shakhova-margarita) | |
| 19:27:54 | melwitt | I saw this old bug recently, evacuations that failed to schedule end up getting nuked during nova-compute restart https://bugs.launchpad.net/nova/+bug/1583504 | |
| 19:28:35 | melwitt | because the logic assumes that 'accepted' migration means it succeeded. I'm not sure what the original reasoning was for including 'accepted' in the logic (along with 'done') | |
| 19:28:36 | mriedem | hmm, is that from before the evac robustification to use migration records? | |
| 19:28:42 | mriedem | oh | |
| 19:29:03 | mriedem | that recently changed, the logic in the compute i mean, | |
| 19:29:08 | mriedem | it was discussed at the ptg in denver | |
| 19:29:09 | melwitt | this was the proposed fix https://review.openstack.org/#/c/318731/2/nova/compute/manager.py | |
| 19:29:14 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 19:29:15 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 19:29:35 | melwitt | orly | |
| 19:30:23 | mriedem | i'm thinking of this i think https://review.openstack.org/#/c/499237 | |
| 19:30:35 | melwitt | code looks the same AFAICT https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L640 | |
| 19:30:51 | melwitt | oh, okay. that should help then | |
| 19:31:02 | mriedem | right, if scheduling fails, set the migration status to 'error' | |
| 19:31:08 | mriedem | so when restarting the compute, the migration is not 'accepted' | |
| 19:31:11 | mriedem | or 'done' | |
| 19:31:16 | melwitt | maybe that bug could be considered closed with that fix, but we'd probably wanna write a test to make sure | |
| 19:31:25 | melwitt | yep, gotcha | |
| 19:32:10 | mriedem | i think you'd just have to update this test https://review.openstack.org/#/c/499237/4/nova/tests/functional/regressions/test_bug_1713783.py | |
| 19:32:12 | mriedem | to restart the compute | |