| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-27 | |||
| 17:46:36 | mriedem | hmm, didn't even get to my stuff | |
| 17:48:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | |
| 17:49:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | |
| 17:50:07 | dansmith | mriedem: check that out: https://imgur.com/a/2lmiw | |
| 17:50:11 | dansmith | sdague: you too ^ | |
| 17:50:29 | mriedem | jesus, graphs?! | |
| 17:51:04 | mriedem | well, looking at 2.46 and 2.47, i think it's 2.47 https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id41 | |
| 17:51:06 | dansmith | 2.26 was not free but 2.47 is killing us | |
| 17:51:07 | dansmith | yeah | |
| 17:51:41 | mriedem | this is 500 error/500 active right? | |
| 17:51:46 | mriedem | dansmith: want to report a bug with details? | |
| 17:51:51 | dansmith | mriedem: yes, 500/500 | |
| 17:53:04 | dansmith | so that was on my patch | |
| 17:53:19 | dansmith | I'm going to run it again on master but starting at maybe 2.24 to make sure the knees are in the same place | |
| 17:53:26 | mriedem | ok | |
| 17:53:28 | sdague | dansmith: interesting.... any idea why the display on the embedded data is causing things to go nuts | |
| 17:53:34 | mriedem | i did notice this w/o your change too though | |
| 17:53:50 | mriedem | i wonder if we're lazy-loading the flavor extra specs? | |
| 17:53:58 | dansmith | mriedem: yeah, I don't see differences in my numbers so I'm just doing it for completeness | |
| 17:54:02 | dansmith | sdague: I haven't looked yet | |
| 17:54:04 | sdague | mriedem: ah, right, that's probably it | |
| 17:54:20 | dansmith | we shouldn't be lazy-loading extra specs | |
| 17:54:28 | dansmith | they should be in the flavor in the instance | |
| 17:55:56 | sdague | I mean, it is a bunch more data. I guess it could just be serialization cost of more data, though it seems weird. | |
| 17:56:34 | mriedem | well, we were always getting the flavor | |
| 17:56:35 | dansmith | mriedem: https://bugs.launchpad.net/nova/+bug/1719966 | |
| 17:56:37 | openstack | Launchpad bug 1719966 in OpenStack Compute (nova) "Microversion 2.47 punches nova in its special place" [Undecided,New] | |
| 17:56:38 | mriedem | even before 2.27 | |
| 17:56:43 | mriedem | ha | |
| 17:56:57 | dansmith | sdague: right, no difference in what we're pulling from the db across that boundary, just what we do with it in the api | |
| 17:57:36 | dansmith | sdague: (I checked) | |
| 17:57:40 | mriedem | https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L269 | |
| 17:57:53 | mriedem | so we were always pulling it https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L263 | |
| 17:58:15 | mriedem | and we were always joining on it in the db https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L58 | |
| 17:58:26 | mriedem | so why is this so much slower? https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L248 | |
| 17:58:33 | mriedem | the policy check for each instance? | |
| 17:59:42 | dansmith | the only thing we can lazy-load from flavor isprojects, BTW | |
| 17:59:52 | dansmith | not extra_specs or anything else | |
| 18:00:07 | dansmith | https://github.com/openstack/nova/blob/master/nova/objects/flavor.py#L318-L319 | |
| 18:00:08 | mriedem | ok i thought that we always had extra_specs but didn't go back to check | |
| 18:00:13 | sdague | mriedem: yeh, the policy check is going to be per instance | |
| 18:00:29 | sdague | the policy check is a fs.stat as well | |
| 18:00:39 | dansmith | we should check it once and pass it to the per-instance flavor method right? | |
| 18:00:43 | mriedem | yes | |
| 18:00:50 | sdague | because policy file is dynamically reread | |
| 18:00:55 | mriedem | right | |
| 18:00:56 | mriedem | ... | |
| 18:00:58 | mriedem | jesus | |
| 18:00:58 | dansmith | that might explain why mriedem sees a bigger hit | |
| 18:01:15 | dansmith | you know what | |
| 18:01:19 | mriedem | where is cfriesen when it's time to talk about performance degradation? | |
| 18:01:19 | dansmith | I think we might want to backport this fix | |
| 18:01:27 | mriedem | we for sure do | |
| 18:01:33 | dansmith | I mean.. maybe | |
| 18:01:44 | mriedem | the policy thing is backportable | |
| 18:01:46 | mriedem | check once | |
| 18:01:54 | dansmith | we could leave it and just further relegate pike to the trashcan of releases | |
| 18:01:59 | mriedem | ha | |
| 18:02:02 | mriedem | but, | |
| 18:02:06 | mriedem | ocata is already in that trashcan | |
| 18:02:09 | dansmith | haha | |
| 18:02:36 | mriedem | i've literally been sending emails internally for weeks saying, "once you upgrade to pike, this should all be much better" | |
| 18:02:46 | mriedem | should* | |
| 18:02:57 | mriedem | *: not actual statement of fact backed up by any evidence | |
| 18:03:19 | melwitt | heh | |
| 18:05:23 | sdague | it would be nice if a context only evaluated a particular policy rule once | |
| 18:05:36 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Support traits in the Ironic driver https://review.openstack.org/507052 | |
| 18:06:29 | sdague | because it's going to be a little awkward to handle cases where you need to send down the preevaluted permission through a bunch of function calls | |
| 18:07:25 | mriedem | this one shoudn't be too terrible | |
| 18:07:28 | mriedem | dansmith: are you started on the fix? | |
| 18:07:29 | dansmith | right this one should be easy | |
| 18:07:37 | dansmith | mriedem: no but I can | |
| 18:07:57 | dansmith | I'd rather fix this and you finish reviewing my patch | |
| 18:08:08 | mriedem | which patch? the kahuna? | |
| 18:08:16 | mriedem | i can't finish what i haven't started | |
| 18:08:25 | dansmith | I'd rather fix this and you start reviewing my patch | |
| 18:08:49 | Tengu | hello! anyone can point me a valid doc for pike and host aggregation + flavor pinning? I'm stuck right now trying to get all working, and I find contradictory docs :/ | |
| 18:09:24 | mriedem | define "flavor pinning" | |
| 18:09:35 | mriedem | you can associate a host aggregate with a specific flavor via metadata / extra specs | |
| 18:09:46 | Tengu | mriedem: "m1.small must run on that aggregate, while m2.small must run on this aggregate" | |
| 18:09:48 | mriedem | however, any other aggregate which is not tied to that flavor can still use it | |
| 18:09:52 | mriedem | there is no exclusion | |
| 18:10:32 | mriedem | Tengu: i think you're looking for this then https://review.openstack.org/#/c/381912/ | |
| 18:10:32 | Tengu | what should I put in the metadata? | |
| 18:10:42 | Tengu | ah, will check that | |
| 18:11:13 | Tengu | 3 days ago? darn… pretty fresh | |
| 18:11:27 | mriedem | that spec has been around quite awhile | |
| 18:11:36 | mriedem | almost a year | |
| 18:11:45 | Tengu | we saw something like that for Icehouse | |
| 18:11:53 | mriedem | see L466 here https://etherpad.openstack.org/p/nova-ptg-queens | |
| 18:12:17 | mriedem | apparently the stakeholders were not yet synergized as promised | |
| 18:12:25 | Tengu | ah, that's for queens… we're running pike :/. don't tell me there isn't anything working right now? | |
| 18:12:46 | mriedem | well, read the spec first and confirm if that's what you're asking for | |
| 18:13:49 | Tengu | looks like what we want, yes. but that's strange, I found some doc, even at Redhat, saying "it works" but without proper example. | |
| 18:14:04 | mriedem | to summarize, we talked about this at the pike ptg in february, we needed to have the various use cases documented in the spec to make sure the solution would cover them, and there were at least 2 stakeholders in the room saying, "we have an out of tree filter that does something like this" and we said, ok read this and tell us if it will replace your out of tree filter, and those people never replied to ack that it does | |
| 18:14:35 | Tengu | erf | |
| 18:14:46 | Tengu | may I explain what I did? | |
| 18:15:13 | Tengu | and point to the doc I followed - maybe a solution might be found | |
| 18:16:25 | Tengu | mriedem: I followed https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/6/html/Administration_Guide/section-host-aggregates.html - I think that one has some equivalent in openstack "open" doc | |
| 18:17:16 | Tengu | mriedem: I activated AggregateInstanceExtraSpecsFilter filter in nova.conf, and created two aggregate - all hosts are in those aggregates (in fact, for now, only two hosts - hence once per group). | |