| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 13:43:27 | jaypipes | mriedem: ok, so now I understand why CERN is setting max_placement_records so low... it's because a single HostMapping.get_by_host() is being a issued for each found host. | |
| 13:43:43 | jaypipes | instead of a batched approach. | |
| 13:45:09 | mriedem | sure, it's more than just the host mapping query | |
| 13:45:35 | jaypipes | https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L758 | |
| 13:45:40 | jaypipes | InstalceList too... | |
| 13:45:40 | mriedem | bingo | |
| 13:45:43 | jaypipes | ack. | |
| 13:45:47 | mriedem | yes, and i have already neutered that | |
| 13:45:56 | mriedem | that used to pull full Instance objects back | |
| 13:45:59 | mriedem | now it's just uuids | |
| 13:46:03 | mriedem | https://github.com/openstack/nova/commit/91f5af7ee7f7140eafb7237875f6cd6ea1abcd38 | |
| 13:46:07 | jaypipes | k | |
| 13:46:26 | frickler | mriedem: any chance to give these stable reviews progress? https://review.openstack.org/619220 https://review.openstack.org/619254 | |
| 13:46:45 | mriedem | frickler: there is a >0% chance yes | |
| 13:47:16 | openstackgerrit | Lee Yarwood proposed openstack/nova master: fixtures: Return a mocked class instead of method within fake_imagebackend https://review.openstack.org/619804 | |
| 13:48:13 | mriedem | frickler: why didn't you cherry pick the pike change from the queens change? looks like you had to redo some of the conflict resolution? | |
| 13:48:27 | mriedem | normally backports should just all go in a row, stein -> rocky -> queens -> pike | |
| 13:48:29 | mriedem | like a snowball | |
| 13:49:14 | frickler | mriedem: it was yet another conflict, yes, that function likes to do file hopping it seems | |
| 13:49:38 | mriedem | lyarwood: want to get this? https://review.openstack.org/#/c/619220/1 | |
| 13:49:43 | mriedem | ok | |
| 13:50:09 | lyarwood | mriedem: looking | |
| 13:50:57 | mriedem | ok scored both | |
| 13:51:52 | lyarwood | mriedem: okay done, would you mind taking a look at https://review.openstack.org/#/q/status:open+project:openstack/nova+topic:bug/1764883 when you get a chance? | |
| 13:56:00 | mriedem | i see how it is, i give you something easy and in return you give me this | |
| 13:56:22 | mriedem | nvm | |
| 13:56:22 | mriedem | oh these are backports.. | |
| 13:57:46 | lyarwood | mriedem: something just as easy, yeah, what a guy ;) | |
| 14:06:57 | openstackgerrit | sean mooney proposed openstack/nova-specs master: Add spec for sriov live migration https://review.openstack.org/605116 | |
| 14:07:25 | sean-k-mooney | bauzas: adrianc sorry for the delay on updating ^ | |
| 14:19:21 | adrianc | ack sean-k-mooney, will look into it | |
| 14:27:24 | openstackgerrit | Merged openstack/nova-specs master: Per aggregate scheduling weight (spec) https://review.openstack.org/599308 | |
| 14:28:38 | tssurya | hi back jaypipes :) | |
| 14:29:20 | tssurya | mriedem: thanks for the info on the config option, will see if enabling it makes it better | |
| 14:29:50 | tssurya | I guess the documentation says something like "f the configured filters and weighers do not need this information, disabling this option will improve performance" | |
| 14:30:14 | tssurya | but I didn't know what this exactly did until after looking into the code | |
| 14:31:23 | sean-k-mooney | adrianc: there are very minor changes as there seam to be little questions raised | |
| 14:31:37 | sean-k-mooney | adrianc: it should not affect you poc | |
| 14:35:16 | mriedem | tssurya: yeah it's mostly for the affinity type filters | |
| 14:36:04 | tssurya | mriedem: ah ok | |
| 14:36:22 | tssurya | yea | |
| 14:37:40 | mriedem | the performance thing there is probably about not needing to needlessly send instance information from all computes to the scheduler | |
| 14:37:53 | mriedem | but misses the part about how the scheduler then blindly pulls the information from the db anyway | |
| 14:38:06 | mriedem | hmm... | |
| 14:38:10 | mriedem | i wonder... | |
| 14:38:26 | mriedem | if we only got the HostState.instances in the scheduler if track_instance_changes was true | |
| 14:38:37 | mriedem | since they are kind of connected | |
| 14:38:52 | mriedem | we wouldn't want to make anything conditional on enabled filters b/c people can have out of tree filters | |
| 14:39:16 | mriedem | or we could just add a new option, CONF.filter_scheduler.do_your_filters_care_about_instances_on_hosts | |
| 14:39:24 | mriedem | with a better name | |
| 14:39:57 | mriedem | more config options is gross, but if you have split mq then track_instance_changes won't work and you don't need to enable it | |
| 14:40:32 | tssurya | right, I am pretty sure this option was enabled by default and three months ago we disabled it and saw a very slight performance improvement actually | |
| 14:40:54 | mriedem | because of less mq traffic/ | |
| 14:40:55 | mriedem | ? | |
| 14:41:04 | tssurya | but yea I am getting this from a tracking ticket on the issue, probably we will look into this next week | |
| 14:41:11 | tssurya | yea I think so | |
| 14:41:11 | mriedem | ok | |
| 14:41:41 | mriedem | as i said, i'd expect your mq traffic to go up by enabling it, but your overall scheduling time, at least for hosts in the cells reporting instance info, might go down | |
| 14:41:58 | mriedem | but ultimately if you aren't enabling filters that need that information, it's a waste of time all around | |
| 14:42:38 | tssurya | hmm also what do you think querying all the cell DBs to fetch the compute nodes could be a reason enough to add time ? | |
| 14:42:39 | sean-k-mooney | if people have time could they take a look at https://review.openstack.org/#/c/591607/11 | |
| 14:43:21 | tssurya | I honestly am not sure if the whole parallel works well enough even with scatter gather, | |
| 14:43:35 | tssurya | will have to dig through that too | |
| 14:44:50 | mriedem | tssurya: we're querying enabled cells right? but that could be 70 cells which don't contain the 10 compute nodes you care about? | |
| 14:45:12 | mriedem | also, how many scheduler workers are you running? | |
| 14:47:24 | dansmith | tssurya: why do you think the parallel thing doesn't work? | |
| 14:47:59 | dansmith | or you said "well enough"... | |
| 14:50:20 | sean-k-mooney | dansmith: stephenfin: can you add https://review.openstack.org/#/c/591607/11 to your review queue. | |
| 14:50:39 | tssurya | dansmith: well last time we tried to measure the parallel part in time over going to all 70 DBs we did not get any solid proof of fastness | |
| 14:50:41 | diliprenkila | Hi all, While creating instance snapshots , type conversion error occurs nova.api.openstack.wsgi [req-fd9b8a70-f455-42e1-b186-93fffaa6192e ff9650c86533492581513eca72b48409 2eea218eea984dd68f1378ea21c64b83 - 765703fcca634b149c7a012626847d2f 765703fcca634b149c7a012626847d2f] Unexpected exception in API method: TypeError: Unable to set 'os_hidden' to 'False'. Reason: u'False' is not of type u'boolean' | |
| 14:50:54 | stephenfin | sean-k-mooney: Sure | |
| 14:51:15 | openstack | Launchpad bug 1806239 in OpenStack Compute (nova) "nova-api should handle type conversion while creating server snapshots " [Undecided,New] | |
| 14:51:15 | mriedem | diliprenkila: you're talking about https://bugs.launchpad.net/nova/+bug/1806239 yes | |
| 14:51:20 | dansmith | tssurya: all of the processing of the database results happens in serial, which is generally the majority of the work | |
| 14:51:35 | tssurya | don't know was just wondering if someone had done the scatter-gather versus going sequential perf study | |
| 14:51:39 | dansmith | tssurya: that's because we're using eventlet instead of regular threads | |
| 14:51:46 | openstackgerrit | sean mooney proposed openstack/nova master: Add fill_virtual_interface_list online_data_migration script https://review.openstack.org/614167 | |
| 14:52:19 | tssurya | ah yea which is why the bottelneck is the time of the slowest DB still | |
| 14:52:46 | diliprenkila | mriedem>full log is at :https://etherpad.openstack.org/p/e20ERvBEkw | |
| 14:52:52 | dansmith | tssurya: no, not really, that's just because we have to wait for all the results in order to be able to produce them in order | |
| 14:53:02 | tssurya | mriedem: yea its still going to 70 cells out of which we care only about the 1 cells which might have those 10 nodes | |
| 14:53:19 | mriedem | hmm, wonder if we could front-filter the cells via host mappings, | |
| 14:53:24 | mriedem | that wouldn't be worth it in the case of 1-2 cells, | |
| 14:53:29 | mriedem | but for 70 it might | |
| 14:53:50 | dansmith | mriedem: like we do for projects on list.., and we have a knob to choose | |
| 14:53:55 | mriedem | sort of like how we get instance mappings by project_id and then filter the cells from those mappings | |
| 14:53:57 | mriedem | yeah | |
| 14:54:31 | tssurya | mriedem: yea we had a bug for that, and kind of have a patch to try that out, will first see if there is a major difference | |
| 14:55:03 | mriedem | diliprenkila: i don't see os_hidden here https://docs.openstack.org/glance/latest/admin/useful-image-properties.html | |
| 14:55:07 | mriedem | is it just not documented? | |
| 14:55:13 | dansmith | mriedem: and, it only helps for 70 cells when you regularly cut out 68 of those cells for any given scheduling request | |
| 14:55:31 | tssurya | dansmith: oh ok | |
| 14:56:02 | bauzas | mriedem: diliprenkila: I guess he means kvm_hidden | |
| 14:56:07 | openstack | Launchpad bug 1767303 in OpenStack Compute (nova) "Scheduler connects to all cells DBs to gather compute nodes info" [Undecided,Incomplete] - Assigned to Surya Seetharaman (tssurya) | |
| 14:56:07 | mriedem | tssurya: this bug? https://bugs.launchpad.net/nova/+bug/1767303 | |
| 14:56:16 | tssurya | yea | |
| 14:56:35 | mriedem | bauzas: no it's a property on the image? https://github.com/openstack/glance/blob/a308c444065307e99f18b521ed8d95714be24da7/glance/db/sqlalchemy/alembic_migrations/versions/rocky_expand01_add_os_hidden.py#L16 | |
| 14:57:12 | bauzas | ah my bad then | |
| 14:57:34 | bauzas | what does this ? | |
| 14:58:15 | bauzas | mmm https://github.com/openstack/nova/search?q=os_hidden&unscoped_q=os_hidden | |