| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-16 | |||
| 16:43:31 | cfriesen | stephenfin: pinning ignores the ratios, always 1:1 | |
| 16:43:36 | stephenfin | mriedem: I'd put money on it :) | |
| 16:44:03 | stephenfin | cfriesen: Not in the ComputeNode was reporting N * ratio free CPUs though, right? | |
| 16:44:07 | mriedem | (again) | |
| 16:44:22 | stephenfin | Although that's neither here nor there. I'll fix this up here and then fix up the log | |
| 16:44:23 | cfriesen | stephenfin: pinning doesn't use that count | |
| 16:44:27 | stephenfin | which seems broken | |
| 16:44:36 | stephenfin | cfriesen: Ah, potential clean up so | |
| 16:44:51 | cfriesen | stephenfin: at least, I'm about 90% sure it doesn't. :) | |
| 16:48:56 | cfriesen | jaypipes: do you know how we handle disk_allocation_ratio as a per-compute-node config option when we're using shared disk resource providers? | |
| 17:22:41 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757 | |
| 17:28:00 | openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Parse forbidden in extra_specs https://review.openstack.org/561677 | |
| 17:37:53 | jaypipes | cfriesen: nobody is yet using shared disk resource providers, so I don't know :) | |
| 18:04:03 | cfriesen | jaypipes: seems like we'd need to have the allocation ratio associated with the resource provider itself rather than the compute node | |
| 18:07:26 | cdent | cfriesen: yes, that would be the idea. efried has done some preliminary libvirt work, lemme find the link | |
| 18:07:31 | cdent | also you aware of #openstack-placement | |
| 18:07:51 | cdent | cfriesen: https://review.openstack.org/#/c/560459/ | |
| 18:08:26 | efried | cdent, cfriesen: In case you didn't catch the last nova meeting, bhagyashris agreed to take these patches over and fix up the tests. | |
| 18:09:51 | openstackgerrit | Jay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits https://review.openstack.org/531899 | |
| 18:09:52 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | |
| 18:10:43 | jaypipes | cfriesen: see https://review.openstack.org/#/c/544683/ | |
| 18:13:38 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Document how to disable notifications https://review.openstack.org/561684 | |
| 18:16:43 | dansmith | sean-k-mooney: around? | |
| 18:21:19 | mriedem | cdent: thanks for https://pypi.org/project/wsgi_intercept/ - i'll see about getting g-r updated | |
| 18:23:09 | mriedem | although i likely can't use that in this change until later since we might backport this patch | |
| 18:23:32 | cdent | yeah, was worth finding anyway | |
| 18:29:24 | openstackgerrit | Merged openstack/nova master: Don't log a warning for InstanceNotFound with deleted VIFs https://review.openstack.org/554591 | |
| 18:33:28 | sean-k-mooney | dansmith: i was on a call but yes | |
| 18:33:53 | dansmith | sean-k-mooney: hey, so we had a question on friday and I think you're the right person to ask | |
| 18:34:10 | dansmith | sean-k-mooney: we still have non-os_vif handling of vhostuser interfaces in vif.pym | |
| 18:34:12 | sean-k-mooney | somthing about not implemented error for os-vif | |
| 18:34:19 | dansmith | but it's not clear to me/us if that is still needed or not | |
| 18:34:19 | dansmith | yeah | |
| 18:34:37 | sean-k-mooney | dansmith: that is used by both vpp and snabswitch | |
| 18:34:45 | dansmith | okay | |
| 18:34:48 | dansmith | why? | |
| 18:35:24 | sean-k-mooney | in both cases vhost-user is not being used with ovs so they dont need to have somting plug it into a vswitch | |
| 18:35:59 | sean-k-mooney | in the vpp case i belive they have an agent on the plathform that poll for the creation fo the socket and just connects when it shows up | |
| 18:36:10 | dansmith | so is it always going to be that way? | |
| 18:36:38 | sean-k-mooney | dansmith: we could add a new plugin to os-vif if you want to move it out of tree | |
| 18:36:52 | dansmith | seems like it would be less confusing | |
| 18:36:58 | dansmith | maybe a comment about who/what uses that path would be nice | |
| 18:37:08 | dansmith | regardless, can you have a look over https://review.openstack.org/#/c/484997 ? | |
| 18:37:42 | sean-k-mooney | we didnt to that initally as they are not reference plugings but its basically a noop so i dont really see the harm in a generic_vhost_user pluging in os-vif | |
| 18:38:18 | sean-k-mooney | oh the queue lenght stuff sure ill take a look now | |
| 18:38:53 | dansmith | thanks | |
| 18:39:39 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757 | |
| 18:47:30 | efried | bauzas: still around? | |
| 18:50:59 | openstackgerrit | sahid proposed openstack/nova master: libvirt: fix wrong driver name for vhostuser interface https://review.openstack.org/561562 | |
| 18:53:53 | melwitt | dansmith: yeah, saw your comment and -1ed it earlier this morning to add visibility to the fact that the changes were reverted | |
| 18:54:04 | dansmith | melwitt: ack | |
| 19:01:43 | mriedem | melwitt: i'm not sure if you want a simple specless blueprint for this or just let it slide as a wishlist bug fix https://review.openstack.org/#/c/555861/ - note we wouldn't backport that either way | |
| 19:02:45 | melwitt | cfriesen, mriedem, dansmith: looking at the code, it makes sense what's happening. the service was deleted while the instance still referred to it, then when you list the instance, it tries to generate a UUID for the service (that is deleted) and the service_update method tries to 'get' the service record by id and it gets ServiceNotFound because model_query will automatically filter out deleted things https://github.com/openstack/nova/ | |
| 19:02:45 | melwitt | blob/master/nova/db/sqlalchemy/api.py#L554 | |
| 19:02:50 | melwitt | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L554 | |
| 19:04:03 | tmcm | is it possible to run mitaka-based compute nodes with a queens nova-scheduler/placement? | |
| 19:04:23 | tmcm | i'm looking at https://docs.openstack.org/nova/latest/user/placement.html but it's not clear to me from what i read there | |
| 19:04:35 | melwitt | I would think if you're deleting services out from under live instances, after you create new services to replace them, wouldn't you have to manually associate the new service with existing instances? else it seems like other bad things could happen | |
| 19:05:37 | dansmith | they're only associated by name, right? | |
| 19:05:59 | melwitt | the db api stuff implied the backref is by service id | |
| 19:06:04 | melwitt | but lemme re-look | |
| 19:06:29 | dansmith | I don't think we have a service_id on instance, | |
| 19:06:31 | dansmith | I thought I looked | |
| 19:06:43 | dansmith | although I was looking at computenode specifically | |
| 19:06:45 | mriedem | i wonder why "from_db_object" is public in their stacktraces....totally unrelated | |
| 19:06:52 | melwitt | yeah looks like backref by host and binary | |
| 19:07:11 | melwitt | so how did the service_update get the service id ... | |
| 19:07:55 | dansmith | it would be if we read_deleted when we looked it up | |
| 19:08:01 | dansmith | and got the first older deleted one by name | |
| 19:08:11 | dansmith | I haven't looked back into his latest stack trace, fyi | |
| 19:08:55 | melwitt | okay. yeah, the stacktrace shows ServiceNotFound being raised from service_update as service_update is trying to update via id and the id is not found because model_query | |
| 19:09:18 | dansmith | right | |
| 19:09:58 | mriedem | "the sequence of events was: boot instance on compute node, migrate instance, delete compute node and service, upgrade to pike, create new compute node with same name, migrate instance back to that compute node." | |
| 19:10:01 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | |
| 19:10:08 | melwitt | I haven't made the connection yet but somehow the service object associated with the instance is the deleted one. which I thought would be expected if you deleted the service out from under the instance. like, what would ever update the instance to point at a new service? I guess you say by name, which would find more than one matching service, and it's happening to find the deleted one first? | |
| 19:10:17 | dansmith | melwitt: it can't be | |
| 19:10:20 | dansmith | because it's only by name | |
| 19:10:50 | dansmith | I didn't see where they ened up starting down this path, I thought it was in the "upgrade to pike" section, | |
| 19:10:58 | melwitt | but two have the same name, the deleted one and the new one | |
| 19:11:02 | dansmith | and that they were implying we were handling deleted services in the migration | |
| 19:11:07 | dansmith | right | |
| 19:11:10 | mriedem | also waiting to hear if they have this https://review.openstack.org/#/c/545988/ | |
| 19:11:11 | melwitt | I haven't yet connected where the query is happening that would get the two returned | |
| 19:11:40 | melwitt | because like you all said earlier, that should be filtering out deleted rows, we would expect | |
| 19:11:46 | dansmith | yeah | |
| 19:12:07 | mriedem | if they have https://review.openstack.org/#/c/545988/ they will query deleted records too | |
| 19:12:33 | dansmith | mriedem: oh I didn't get the connection this morning you were making | |
| 19:12:44 | dansmith | mriedem: that we're using a read_deleted context to get the instance and thus the service too? | |
| 19:12:51 | mriedem | yes | |
| 19:12:55 | dansmith | gotcha that makes sense | |
| 19:13:42 | melwitt | mriedem: I think we should probably go with a specless blueprint on that one, so it gets bundled with the other new features | |
| 19:14:33 | mriedem | ack | |
| 19:14:50 | melwitt | thanks | |
| 19:19:01 | mriedem | trying to think of a test to recreate cfriesen's issue, could start two compute services, create the instance on one, migrate to the other, delete the old compute service/node, fake the upgrade to pike by deleting the uuid on the 2 services, start a new compute service with the same name as the old one, and then migrate the instance back to the other host | |
| 19:19:13 | mriedem | in his stacktrace, he's not doing a migration, simply listing instances | |
| 19:20:06 | mriedem | which unconditionally joins on services https://github.com/openstack/nova/blob/f0400313f30843fba114584f582b93f7cb5e5e48/nova/api/openstack/compute/servers.py#L295 | |
| 19:20:19 | melwitt | sounds like that should do it | |
| 19:21:28 | mriedem | cfriesen: have you reported an upstream bug for this issue? | |
| 19:23:54 | mriedem | bauzas: https://bugs.launchpad.net/nova/+bug/1764460 | |
| 19:23:54 | openstack | Launchpad bug 1764460 in OpenStack Compute (nova) queens "Cannot hard reboot an instance in error state" [High,Confirmed] | |