| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-17 | |||
| 16:50:29 | jaypipes | yeah | |
| 16:51:45 | mriedem | ok, so the first compute node never came back from GET /allocation_candidates probably | |
| 16:53:03 | mriedem | http://logs.openstack.org/64/483564/2/gate/gate-nova-tox-functional-ubuntu-xenial/0f4534d/console.html#_2017-07-17_15_20_35_765791 | |
| 16:53:08 | mriedem | that's when the first compute host starts up | |
| 16:53:46 | mriedem | and then "Compute_service record created for compute:fake-mini" | |
| 16:54:05 | mriedem | Another thread already created a resource provider with the UUID 3694a10a-e435-4400-83aa-7d6ade1fc866. Grabbing that record from the placement API. | |
| 16:54:35 | mriedem | i don't know what 3694a10a-e435-4400-83aa-7d6ade1fc866 is | |
| 16:54:46 | mriedem | i thought the first compute RP should be e9e74898-4332-4e50-a4c6-1ad75dd295e5 | |
| 16:55:18 | mriedem | hmm wtf GET /resource_providers/3694a10a-e435-4400-83aa-7d6ade1fc866" status: 404 | |
| 16:55:29 | mriedem | Unable to refresh my resource provider record | |
| 16:56:09 | mriedem | so whatever happened, it never posted inventory for the first compute host, | |
| 16:56:18 | mriedem | which is why it didn't come back in GET /allocation_candidates | |
| 16:56:30 | mriedem | seems like another case of the wsgi eventlet weirdness | |
| 16:56:33 | mriedem | crossing streams | |
| 16:58:18 | mriedem | on a side note, it's silly that we go through all the work of calling down to get_filtered_objects when the original list we got back from GET /allocation_candidates is empty to begin with | |
| 16:59:19 | mriedem | ah yes jay's todo: "Really, we should raise NoValidHosts here" | |
| 16:59:40 | jaypipes | tada. | |
| 16:59:53 | mriedem | so i think the failure is totally unrelated | |
| 17:03:44 | mriedem | i rechecked and left a big ass summary of the failure | |
| 17:10:20 | mriedem | aha 10 hours later i think i figured out the ValueError: Cannot convert NUMATopologyLimits(cpu_allocation_ratio=16.0,ram_allocation_ratio=1.5) to primitive | |
| 17:10:26 | mriedem | and is a real bug it looks like | |
| 17:13:18 | mriedem | oooo and one mr jay introduced :) | |
| 17:13:30 | mriedem | i guess our numa 3rd party ci isn't running? | |
| 17:13:42 | jaypipes | mriedem: in, like, how long? :) | |
| 17:13:47 | mriedem | https://review.openstack.org/#/c/478291/1/nova/scheduler/manager.py | |
| 17:13:50 | mriedem | L48 | |
| 17:13:52 | mriedem | that's not a primitive | |
| 17:14:04 | mriedem | so when passing to jsonutils.to_primitive | |
| 17:14:07 | mriedem | kablammo | |
| 17:14:11 | mriedem | in the latest oslo.serialization | |
| 17:14:18 | mriedem | http://logs.openstack.org/87/482287/6/check/gate-cross-nova-func-ubuntu-xenial/a645ce8/console.html#_2017-07-16_14_02_02_649384 | |
| 17:14:22 | mriedem | but not to worry, | |
| 17:14:31 | jaypipes | mmm | |
| 17:14:38 | mriedem | i've wrangled the oslo team to blacklist that version because it was breaking us, because of the bug :) | |
| 17:14:40 | mriedem | i'll push a fix | |
| 17:15:03 | jaypipes | k, thanks matt | |
| 17:17:51 | mriedem | sean-k-mooney: do you know if anyone is running the intel NUMA 3rd party ci anymore? | |
| 17:21:47 | mriedem | fwiw this could have always been busted https://review.openstack.org/#/c/478291/1/nova/scheduler/filter_scheduler.py | |
| 17:21:55 | mriedem | i don't see any fancy conversion before | |
| 17:28:42 | openstackgerrit | Merged openstack/nova master: Fix the releasenote and api-ref for quota-class API https://review.openstack.org/483170 | |
| 17:31:31 | jaypipes | alex_xu: https://review.openstack.org/#/c/480379/ reviewed. couple tiny nits. if you can fix those up and re-push, I'm +2. | |
| 17:39:00 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform servergroup.create notification https://review.openstack.org/454023 | |
| 17:39:18 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform servergroup.delete notification https://review.openstack.org/483902 | |
| 17:54:57 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform servergroup.delete notification https://review.openstack.org/483902 | |
| 17:59:37 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform servergroup.create notification https://review.openstack.org/454023 | |
| 17:59:55 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform servergroup.delete notification https://review.openstack.org/483902 | |
| 18:03:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert HostState.limits objcets to primitives https://review.openstack.org/484439 | |
| 18:03:43 | mriedem | dansmith: sfinucan: ^ i'm going to need someone more familiar with the numa limits stuff to look at that | |
| 18:03:47 | mriedem | like sahid or vladikr? | |
| 18:03:54 | mriedem | i don't know how this was actually working before | |
| 18:04:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Convert HostState.limits objects to primitives https://review.openstack.org/484439 | |
| 18:05:56 | mriedem | was conductor serializing that response before for us so the NUMATopologyLimits was getting serialized by conductor? | |
| 18:07:47 | oomichi | alex_xu: most patches are approved on https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/api-no-more-extensions-pike | |
| 18:08:09 | oomichi | alex_xu: need to update the remaining for conflicts | |
| 18:31:24 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log compute node uuid when the record is created https://review.openstack.org/484447 | |
| 18:52:46 | mriedem | dansmith: sfinucan: nvm i think i know how that worked before - jsonutils.to_primitive() wouldn't error out, but wouldn't convert the NUMATopologyLimits either, so it would go through NovaObjectSerializer to do the serialize/deserialize work on the object | |
| 18:53:08 | mriedem | the new change in to_primitive to raise ValueError is what blew it all up because it would try to serialize before hitting the NovaObjectSerializer | |
| 18:53:22 | dansmith | sorry I missed the above ping, | |
| 18:53:25 | dansmith | but I assume I can ignore now | |
| 18:53:53 | mriedem | validation of my understanding, and feelings, is always appreciated, but yeah you can ignore probably | |
| 18:54:26 | dansmith | jsonutils.to_primitive used to fail with "recursion error" or something like that when it hit a non-serializeable object | |
| 18:56:15 | dansmith | mriedem: what change introduced this? | |
| 18:56:30 | dansmith | oh I see, in oslo | |
| 18:58:29 | mriedem | yeah that's what the oslo.serialization change was trying to prevent, the infinite recursion | |
| 18:58:37 | mriedem | by just failing fast | |
| 18:59:29 | dansmith | mriedem: so the thing I'm concerned about is: are we returning something that is only using jsonutils for serialization? | |
| 18:59:45 | dansmith | our object serializer doesn't go hunting in an iterable for any instance of an object to serialize | |
| 19:00:55 | dansmith | doesn't go arbitrarily deep, I mean | |
| 19:01:14 | mriedem | hmm yeah because that's what i was looking at with https://github.com/openstack/nova/blob/master/nova/objects/base.py#L244 | |
| 19:01:24 | mriedem | so it goes 1 level? | |
| 19:01:32 | mriedem | well, actually, | |
| 19:01:55 | mriedem | so this would be a like {'limits': {'numa_topology': object}} | |
| 19:01:59 | mriedem | so that should hit it ^ | |
| 19:02:24 | dansmith | where is this exactly? | |
| 19:02:29 | dansmith | is this in a returned value? | |
| 19:02:32 | mriedem | yeah | |
| 19:02:54 | mriedem | https://review.openstack.org/#/c/484439/2/nova/scheduler/manager.py@140 | |
| 19:03:33 | mriedem | creates a list of these dicts, and each dict will have a 'limits' key that maps to a dict, and that thing has the numa_topology_limits=NUMATopologyLimits object | |
| 19:03:34 | mriedem | map | |
| 19:05:02 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Optimize creating servers with tags https://review.openstack.org/484170 | |
| 19:05:18 | dansmith | I guess it will go pretty deep | |
| 19:05:24 | dansmith | had to re-read that a couple times | |
| 19:05:43 | mriedem | yeah so i think that's how it wasn't blowing up before | |
| 19:05:52 | mriedem | well, and was actually serializing the thing | |
| 19:06:07 | dansmith | yeah | |
| 19:06:10 | mriedem | the fact jsonutils.to_primitive ignored it was to our advantage | |
| 19:06:25 | dansmith | yeah | |
| 19:06:58 | mriedem | that change is also being reverted https://review.openstack.org/#/c/484396/ | |
| 19:07:18 | mriedem | b/c i pointed out that releasing that under a patch release versoin w/o any prior warning <2 weeks to FF is likely to break other projects | |
| 19:07:19 | dansmith | heh | |
| 19:07:25 | dansmith | yeah I guess so | |
| 19:07:53 | mriedem | we only had prior warning b/c the requirements changes gate on nova unit/functional tests | |
| 19:25:51 | mriedem | oomichi: i'm +2 again on https://review.openstack.org/#/c/479774/ after your comments | |
| 19:39:33 | Ctesias | hi. I'm digging around in nova trying to figure out why my nova_override.conf file isn't overriding nova defaults and I'm curious where `import nova.conf` and `CONF = nova.conf.CONF` are coming from because I don't see those being defined anywhere | |
| 19:41:28 | mriedem | Ctesias: for nova-compute the options are loaded up here https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L42 | |
| 19:41:45 | mriedem | similar pattern for the other service entry points | |
| 19:41:59 | mriedem | https://github.com/openstack/nova/blob/master/nova/config.py#L49 | |
| 19:43:15 | Ctesias | Thanks for the pointer. | |
| 19:43:31 | Ctesias | Is putting an override_nova.conf file in the config directory the right way to get overrides into the nova config? | |
| 19:44:10 | mriedem | i've never used nova_override.conf | |