| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-27 | |||
| 17:26:03 | mriedem | i think ideally we don't want the ironic driver being a proxy to placement for this stuff | |
| 17:26:04 | johnthetubaguy | the problem is when an admin deletes a trait in ironic, how do we know to delete it in placement | |
| 17:26:13 | dansmith | johnthetubaguy: ironic could do it itself for sure | |
| 17:26:19 | dansmith | johnthetubaguy: for libvirt it'd be the virt driver | |
| 17:26:43 | johnthetubaguy | the problem is the nova creates the resource provider right now, using the compute node name, hashring details, etc | |
| 17:26:59 | dansmith | johnthetubaguy: no, the rp uuid is the ironic node uuid | |
| 17:27:15 | dansmith | johnthetubaguy: nova creates it if it's not there already, ironic could have done it | |
| 17:28:18 | johnthetubaguy | hmm, I thought it had both for some reason, I need to trace that all properly so its clear in my head | |
| 17:28:42 | johnthetubaguy | so I thought we said at the PTG the ironic virt driver would push this all up, but I am not totally against ironic doing that | |
| 17:29:08 | Tengu | hello! | |
| 17:29:41 | Tengu | I'm having some issues setting up host aggregation and flavor matching (i.e. "flavor m1.medium shall start only on that aggregate" | |
| 17:32:51 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/pike: Updated from global requirements https://review.openstack.org/493146 | |
| 17:34:23 | openstackgerrit | OpenStack Proposal Bot proposed openstack/python-novaclient stable/pike: Updated from global requirements https://review.openstack.org/493187 | |
| 17:34:55 | openstackgerrit | melanie witt proposed openstack/nova master: Set group_members when converting to legacy request spec https://review.openstack.org/507938 | |
| 17:38:18 | melwitt | mriedem: ^ I wrote that test by working from nova/tests/functional/regressions/test_bug_1671648.py and just now realized I guess I could have just added an instance group to the existing test to also test this. but maybe it's better to have the tests separated | |
| 17:39:55 | melwitt | food for thought | |
| 17:40:26 | mriedem | dansmith: L135 https://etherpad.openstack.org/p/nova-instance-list are my results for 1000 active instances with your change | |
| 17:40:40 | mriedem | i'm pretty surprised at the improvements there | |
| 17:42:18 | cdent | mriedem: which job results on https://review.openstack.org/#/c/507918/ are my best target for pokage? | |
| 17:42:40 | dansmith | mriedem: hmm | |
| 17:43:01 | dansmith | mriedem: if you roll back to the other patch does it go back to the perf you measured before? | |
| 17:43:37 | dansmith | mriedem: with my patch we iterate the list fewer times | |
| 17:44:01 | dansmith | I'd be surprised if it made that much difference, but it should make some | |
| 17:44:25 | mriedem | can try that in a bit | |
| 17:44:44 | dansmith | my microversion survey results are interesting | |
| 17:44:46 | dansmith | and not good | |
| 17:44:57 | dansmith | will be done in a few minutes | |
| 17:45:16 | mriedem | cdent: i'd think just the normal tempest dsvm job http://logs.openstack.org/18/507918/2/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d0a5723/ | |
| 17:45:27 | cdent | roger that | |
| 17:45:29 | mriedem | lots of copied bash in here so i likely screwed something up | |
| 17:46:36 | mriedem | hmm, didn't even get to my stuff | |
| 17:48:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | |
| 17:49:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | |
| 17:50:07 | dansmith | mriedem: check that out: https://imgur.com/a/2lmiw | |
| 17:50:11 | dansmith | sdague: you too ^ | |
| 17:50:29 | mriedem | jesus, graphs?! | |
| 17:51:04 | mriedem | well, looking at 2.46 and 2.47, i think it's 2.47 https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id41 | |
| 17:51:06 | dansmith | 2.26 was not free but 2.47 is killing us | |
| 17:51:07 | dansmith | yeah | |
| 17:51:41 | mriedem | this is 500 error/500 active right? | |
| 17:51:46 | mriedem | dansmith: want to report a bug with details? | |
| 17:51:51 | dansmith | mriedem: yes, 500/500 | |
| 17:53:04 | dansmith | so that was on my patch | |
| 17:53:19 | dansmith | I'm going to run it again on master but starting at maybe 2.24 to make sure the knees are in the same place | |
| 17:53:26 | mriedem | ok | |
| 17:53:28 | sdague | dansmith: interesting.... any idea why the display on the embedded data is causing things to go nuts | |
| 17:53:34 | mriedem | i did notice this w/o your change too though | |
| 17:53:50 | mriedem | i wonder if we're lazy-loading the flavor extra specs? | |
| 17:53:58 | dansmith | mriedem: yeah, I don't see differences in my numbers so I'm just doing it for completeness | |
| 17:54:02 | dansmith | sdague: I haven't looked yet | |
| 17:54:04 | sdague | mriedem: ah, right, that's probably it | |
| 17:54:20 | dansmith | we shouldn't be lazy-loading extra specs | |
| 17:54:28 | dansmith | they should be in the flavor in the instance | |
| 17:55:56 | sdague | I mean, it is a bunch more data. I guess it could just be serialization cost of more data, though it seems weird. | |
| 17:56:34 | mriedem | well, we were always getting the flavor | |
| 17:56:35 | dansmith | mriedem: https://bugs.launchpad.net/nova/+bug/1719966 | |
| 17:56:37 | openstack | Launchpad bug 1719966 in OpenStack Compute (nova) "Microversion 2.47 punches nova in its special place" [Undecided,New] | |
| 17:56:38 | mriedem | even before 2.27 | |
| 17:56:43 | mriedem | ha | |
| 17:56:57 | dansmith | sdague: right, no difference in what we're pulling from the db across that boundary, just what we do with it in the api | |
| 17:57:36 | dansmith | sdague: (I checked) | |
| 17:57:40 | mriedem | https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L269 | |
| 17:57:53 | mriedem | so we were always pulling it https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L263 | |
| 17:58:15 | mriedem | and we were always joining on it in the db https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L58 | |
| 17:58:26 | mriedem | so why is this so much slower? https://github.com/openstack/nova/blob/3174ee13a1541230a4b7b2a4737d679691fb14b3/nova/api/openstack/compute/views/servers.py#L248 | |
| 17:58:33 | mriedem | the policy check for each instance? | |
| 17:59:42 | dansmith | the only thing we can lazy-load from flavor isprojects, BTW | |
| 17:59:52 | dansmith | not extra_specs or anything else | |
| 18:00:07 | dansmith | https://github.com/openstack/nova/blob/master/nova/objects/flavor.py#L318-L319 | |
| 18:00:08 | mriedem | ok i thought that we always had extra_specs but didn't go back to check | |
| 18:00:13 | sdague | mriedem: yeh, the policy check is going to be per instance | |
| 18:00:29 | sdague | the policy check is a fs.stat as well | |
| 18:00:39 | dansmith | we should check it once and pass it to the per-instance flavor method right? | |
| 18:00:43 | mriedem | yes | |
| 18:00:50 | sdague | because policy file is dynamically reread | |
| 18:00:55 | mriedem | right | |
| 18:00:56 | mriedem | ... | |
| 18:00:58 | dansmith | that might explain why mriedem sees a bigger hit | |
| 18:00:58 | mriedem | jesus | |
| 18:01:15 | dansmith | you know what | |
| 18:01:19 | dansmith | I think we might want to backport this fix | |
| 18:01:19 | mriedem | where is cfriesen when it's time to talk about performance degradation? | |
| 18:01:27 | mriedem | we for sure do | |
| 18:01:33 | dansmith | I mean.. maybe | |
| 18:01:44 | mriedem | the policy thing is backportable | |
| 18:01:46 | mriedem | check once | |
| 18:01:54 | dansmith | we could leave it and just further relegate pike to the trashcan of releases | |
| 18:01:59 | mriedem | ha | |
| 18:02:02 | mriedem | but, | |
| 18:02:06 | mriedem | ocata is already in that trashcan | |
| 18:02:09 | dansmith | haha | |
| 18:02:36 | mriedem | i've literally been sending emails internally for weeks saying, "once you upgrade to pike, this should all be much better" | |
| 18:02:46 | mriedem | should* | |
| 18:02:57 | mriedem | *: not actual statement of fact backed up by any evidence | |
| 18:03:19 | melwitt | heh | |
| 18:05:23 | sdague | it would be nice if a context only evaluated a particular policy rule once | |
| 18:05:36 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Support traits in the Ironic driver https://review.openstack.org/507052 | |
| 18:06:29 | sdague | because it's going to be a little awkward to handle cases where you need to send down the preevaluted permission through a bunch of function calls | |
| 18:07:25 | mriedem | this one shoudn't be too terrible | |
| 18:07:28 | mriedem | dansmith: are you started on the fix? | |