| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-17 | |||
| 12:51:04 | gibi | mriedem: ^^ here it is | |
| 12:51:20 | mriedem | yup, watching zuul | |
| 12:54:42 | mriedem | tssurya: i guess belmiro didn't like that answer? | |
| 12:55:02 | tssurya | mriedem: I am sure that's not the case :D | |
| 12:55:09 | mriedem | he hates me now doesn't he | |
| 12:55:14 | tssurya | haha | |
| 12:55:19 | mriedem | I BLEW IT | |
| 12:55:48 | mriedem | tssurya: btw, got any cells v2 upgrade news? | |
| 12:56:32 | jroll | alex_xu: thanks | |
| 12:57:03 | tssurya | mriedem: its actually okay.. | |
| 12:57:24 | tssurya | we are just trying to understand what is taking time in the scheduler | |
| 12:57:39 | tssurya | I mean what takes most of the time in the path of the scheduling | |
| 12:58:13 | mriedem | my guess would be that it's pulling more compute node records from the db than it used to do with the per-cell scheduler | |
| 12:58:48 | tssurya | and what's more weird is, how do I test exactly this scatter-gather versus normal sequential querying ? | |
| 12:58:52 | gibi | mriedem: the lates ps of the osc_placement fix still failing the functional test locally :/ | |
| 12:58:56 | tssurya | like the time taken | |
| 12:59:02 | mriedem | so if you have 50 cells with 200 computes, before the top level scheduler would pick a cell and that would pull at most 200 compute nodes, right? but now it could be pulling at most 10K compute nodes | |
| 12:59:44 | tssurya | mriedem: we use dan's pre-placement filter to ensure we don't pull that much | |
| 12:59:54 | mriedem | tssurya: hmmm, we could add some debug logging to the scatter gather routine to log how long an operation took per cell... | |
| 13:00:26 | tssurya | mriedem: that is what I did, but it gives weird times, | |
| 13:00:32 | tssurya | not convincing ones | |
| 13:01:20 | mriedem | is the time being spent in pulling the compute nodes from the cells or from the filters? | |
| 13:02:21 | tssurya | mriedem: filters don't seem to take much time, pulling computes (the query directly done on the DB is very fast, however logging the scatter-gather routine gives weird longer times) | |
| 13:02:34 | tssurya | what is taking time is in the select destinations | |
| 13:02:44 | tssurya | I am working on it now to narrow it down further | |
| 13:02:47 | mriedem | also, do you know if you're hitting the same compute nodes for concurrent requests (packing)? just wondering if you're getting 409s from placement when trying to allocate resources against a target node | |
| 13:02:49 | gibi | mriedem: ohh, there is an upper() call on the message somewhere as {{{err_message:no such trait(s): custom_fake_hw_gpu_class_264c6d7df889415ea5d3b428d7a4a2fc. (http 404)}}} | |
| 13:03:32 | mriedem | tssurya: hmm, might be the alternate selection code? | |
| 13:03:51 | tssurya | mriedem: yea I am right at that point now | |
| 13:04:06 | tssurya | will see in prod if that's the one | |
| 13:04:52 | openstackgerrit | Balazs Gibizer proposed openstack/osc-placement master: Fix error message asserts in functional test https://review.openstack.org/569048 | |
| 13:05:11 | mriedem | that will add more time compared to before queens because we are trying to now find up to max_attempts alternate hosts in the same cell, and that means running those alternates through the filters again | |
| 13:05:38 | tssurya | mriedem: oh right! | |
| 13:05:42 | mriedem | and that's per instance | |
| 13:06:08 | mriedem | so if you do server multicreate in a single request with like 10 instances and max_attempts is 3, we are trying to find i think 30 hosts | |
| 13:07:37 | tssurya | mriedem: hmm yea, will know if this is the one taking time, will check the logs | |
| 13:07:45 | tssurya | meanwhile mriedem I have another issue : http://logs.openstack.org/71/568271/2/check/tempest-full-py3/7142c54/testr_results.html.gz , not sure on how to fix this | |
| 13:08:18 | gibi | mriedem: ps 3 passes the functional locally | |
| 13:08:19 | tssurya | it totally depends on if the host runs both cell and super conductors or runs one of them | |
| 13:09:40 | openstackgerrit | Nguyen Hai proposed openstack/nova-specs master: Follow the new PTI for document build https://review.openstack.org/551802 | |
| 13:11:53 | mriedem | tssurya: hmm, the cell conductor should be reporting into the cell1 database | |
| 13:12:22 | mriedem | this is n-super-cond: http://logs.openstack.org/71/568271/2/check/tempest-full-py3/7142c54/controller/logs/screen-n-super-cond.txt.gz#_May_17_08_20_09_133873 | |
| 13:12:35 | mriedem | http://logs.openstack.org/71/568271/2/check/tempest-full-py3/7142c54/controller/logs/etc/nova/nova_conf.txt.gz | |
| 13:12:43 | mriedem | which is in cell0 | |
| 13:12:44 | mriedem | [database] connection = mysql+pymysql://root:secretdatabase@127.0.0.1/nova_cell0?charset=utf8 | |
| 13:13:03 | mriedem | http://logs.openstack.org/71/568271/2/check/tempest-full-py3/7142c54/controller/logs/screen-n-cond-cell1.txt.gz#_May_17_08_20_10_145541 | |
| 13:13:03 | mriedem | the cell conductor is using the nova cell1 config | |
| 13:13:19 | mriedem | http://logs.openstack.org/71/568271/2/check/tempest-full-py3/7142c54/controller/logs/etc/nova/nova_cell1_conf.txt.gz | |
| 13:13:23 | mriedem | [database] connection = mysql+pymysql://root:secretdatabase@127.0.0.1/nova_cell1?charset=utf8 | |
| 13:14:43 | mriedem | tssurya: it's a buggy test http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/admin/test_services.py#n61 | |
| 13:14:59 | mriedem | it randomly picks the first service from the list and then looks that up by host and binary | |
| 13:15:03 | tssurya | mriedem: yea! | |
| 13:15:05 | mriedem | it's a single node devstack, so the host is going to be the same | |
| 13:15:18 | mriedem | so tempest needs to just not ever use conductor here | |
| 13:15:21 | mriedem | i'll report a tempest bug | |
| 13:15:35 | tssurya | mriedem: awesome thanks, | |
| 13:17:47 | mriedem | https://bugs.launchpad.net/tempest/+bug/1771817 | |
| 13:17:48 | openstack | Launchpad bug 1771817 in tempest "test_get_service_by_service_and_host_name intermittently fails if it tries to use nova-conductor on a single node devstack" [Undecided,New] | |
| 13:17:57 | mriedem | we should just delete that test from tempest, i'll ask the qa team | |
| 13:19:03 | tssurya | mriedem: ack | |
| 13:20:47 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add policy to InstanceGroup object and api models. https://review.openstack.org/563375 | |
| 13:21:22 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Microversion 2.63 - Use new format policy in server group https://review.openstack.org/567534 | |
| 13:31:51 | Kevin_Zheng | mriedem: I was thinking about count services matches the condition before assert | |
| 13:31:52 | belmoreira | mriedem sorry I needed to leave for few minutes | |
| 13:32:19 | openstackgerrit | sahid proposed openstack/nova master: libvirt: Improve 'qemu-img convert' performance https://review.openstack.org/522067 | |
| 13:33:27 | belmoreira | mriedem If "colour" is a property that we would like to consider during scheduling how this should be defined? | |
| 13:36:29 | dansmith | belmoreira: a property of what? | |
| 13:37:16 | mriedem | Kevin_Zheng: we do'nt need to do that | |
| 13:37:24 | mriedem | Kevin_Zheng: https://review.openstack.org/569112 | |
| 13:37:38 | mriedem | dansmith: of the compute node, | |
| 13:37:49 | mriedem | he wants to put a custom resource class or trait on the compute node resource provider | |
| 13:37:58 | mriedem | belmoreira: traits-based scheduling | |
| 13:38:09 | dansmith | mriedem: ah | |
| 13:38:14 | mriedem | belmoreira: see required traits here https://docs.openstack.org/nova/latest/user/flavors.html#extra-specs | |
| 13:38:29 | mriedem | belmoreira: you put a trait on the compute node resource provider (via osc-placement or REST API or whatever) and then put that required trait in a flavor | |
| 13:38:36 | mriedem | and then that flavor must land on a compute node with that trait | |
| 13:39:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova service-list use scatter-gather routine https://review.openstack.org/568271 | |
| 13:39:35 | belmoreira | mriedem: I understand that. What I was thinking is if the compute node can it self advertise the trait | |
| 13:39:52 | mriedem | belmoreira: as i said earlier, the compute node can report standard traits that it knows about, | |
| 13:40:04 | mriedem | for custom stuff, you can do that outside of nova-compute needing to report it for you by proxy | |
| 13:40:16 | Kevin_Zheng | mriedem: if that is the case, then most of the tests can be removed? | |
| 13:41:03 | efried | belmoreira: The virt driver can set any traits it wants. But one would have to make a case for a given trait being a sensible thing for the virt driver to set. | |
| 13:41:31 | mriedem | Kevin_Zheng: i'm starting small | |
| 13:41:38 | openstackgerrit | sahid proposed openstack/os-vif stable/pike: ovs-hybrid: should permanently keep MAC entries https://review.openstack.org/505731 | |
| 13:41:58 | belmoreira | the use case is when deploying a set of nodes with different custom traits all of this needs to be setup up using tha placement api. It would be easier to include them in a nova config option. This case is much easier to handle with a configuration management tool | |
| 13:42:21 | Kevin_Zheng | mriedem: :) | |
| 13:42:28 | efried | belmoreira: In Dublin we agreed that there would be a certain set of traits that the compute service "owns" - has the authority to set/unset on the providers it owns - and anything outside of that circle could be set externally via direct placement API access (via osc or whatever). | |
| 13:43:32 | belmoreira | efried mriedem thanks for the clarification | |
| 13:43:50 | efried | ? | |
| 13:43:50 | efried | compute_provider_additional_traits = CUSTOM_FOO, CUSTOM_BAR | |
| 13:43:50 | efried | belmoreira: You're suggesting a config option like | |
| 13:44:06 | mriedem | yes, and i don't think we should do that | |
| 13:44:07 | belmoreira | efried yes | |
| 13:44:10 | dansmith | you'd have to have a negative as well so you could ever remove traits you had previously set | |
| 13:44:13 | mriedem | it makes nova a proxy, | |
| 13:44:15 | dansmith | mriedem: agree | |
| 13:44:23 | mriedem | and blocks everyone on nova supporting this thing | |
| 13:44:28 | mriedem | when you can already do it external to nova | |
| 13:44:34 | efried | belmoreira: yes, I agree with mriedem and dansmith. | |
| 13:44:47 | mriedem | i already don't really care for the ironic trait proxy stuff we do | |