| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-06 | |||
| 07:41:25 | openstackgerrit | Merged openstack/nova master: Update mailinglist from dev to discuss https://review.openstack.org/621827 | |
| 07:41:36 | openstackgerrit | Merged openstack/nova master: modify the avaliable link https://review.openstack.org/616905 | |
| 09:16:53 | gibi | stephenfin: regarding shorted notification payload, would you like to see the samples stored in nova sorted only or also the json emitted on the message bus? | |
| 09:23:39 | gibi | mdbooth: in the failure in nova.tests.functional.regressions.test_bug_1550919.LibvirtFlatEvacuateTest in http://logs.openstack.org/22/606122/7/check/nova-tox-functional/1f3126b/testr_results.html.gz there are 9 seconds gap between '[nova.virt.libvirt.driver] Creating image' and the first polling of the server state. So someting is definitely slow there _before_ even the test try to wait for 5 seconds | |
| 09:23:45 | gibi | to see the server is ACTIVE state | |
| 09:24:06 | mdbooth | gibi: looking | |
| 09:24:33 | mdbooth | gibi: That's one of the ones which looked 'generally unhappy' I think. | |
| 09:25:08 | gibi | mdbooth: unfortunately we dont have such timing data from successful runs :/ | |
| 09:26:14 | mdbooth | gibi: Or DEBUG logs :( | |
| 09:26:19 | gibi | does 'Creating image' means that in these test we really generate the root fs for the instance? | |
| 09:27:28 | mdbooth | gibi: No. The disk creation is stubbed to just touch the file. | |
| 09:27:58 | mdbooth | However, it does execute _resolve_driver_format in imagebackend, which executes qemu-img. | |
| 09:28:16 | mdbooth | I have a patch locally to mock that out, but on my unloaded system it doesn't make a massive difference. | |
| 09:28:46 | mdbooth | I can see that it's about 3 seconds of wall clock time, although multithreaded. | |
| 09:32:59 | gibi | mdbooth: I see. Honestly I don't know what could be the real problem. In the run http://logs.openstack.org/22/606122/7/check/nova-tox-functional/1f3126b/testr_results.html.gz I see missing DB table exceptions as well that reminds me the race we fixed with cdent last week in the placement db fixture. Maybe that race had other side effects. However we only fixed that in the split out placement repo. | |
| 09:34:05 | mdbooth | gibi: Like I said yesterday, I suspect it's just a canary: the first thing to die when conditions get bad. | |
| 09:34:30 | mdbooth | I could mock another couple of things out, and increase the timeout. | |
| 09:35:00 | gibi | mdbooth: could very well be it. Still it woul be nice to know why 5 seconds in not enough in these tests. But I know that we don't have data to figure that out | |
| 09:35:18 | mdbooth | gibi: Why don't we have debug enabled, btw? | |
| 09:36:09 | gibi | mdbooth: I'm not sure, maybe we don't want to store that much of logs. | |
| 09:37:13 | gibi | mdbooth: but if you can propose a patch that turns the debug log on, I can support that. and we will see if there are other oppinions | |
| 09:42:47 | gibi | stephenfin: I can try to propose the sorting patch and see if people start puking | |
| 09:43:03 | gibi | stephenfin: anyhow, thanks for the reviews on those patches | |
| 09:43:27 | mdbooth | gibi: Thanks for spending time on this. | |
| 09:43:50 | gibi | mdbooth: I mysteries :) | |
| 09:44:14 | mdbooth | Sometimes, anyway. | |
| 10:10:40 | yan0s | I have a question about nova policies | |
| 10:11:27 | yan0s | are variables "compute:create", "compute:get" etc deprecated? | |
| 10:12:08 | yan0s | I see they are not mentioned in the latest documentation | |
| 10:12:09 | yan0s | https://docs.openstack.org/nova/rocky/configuration/policy.html | |
| 10:13:15 | yan0s | also testing some of them I think they affect access rights | |
| 10:13:42 | yan0s | only their "os_compute_api" equivalents work | |
| 10:19:22 | kashyap | gibi: Heya, Zuul was "-2" on this: https://review.openstack.org/#/c/620327/. Guess if I do a 'rechek', I'll "lose" all the ACKs? | |
| 10:20:00 | gibi | kashyap: I don't think so | |
| 10:20:05 | kashyap | Seems to be a 'neutron-grenade' failure | |
| 10:20:19 | gibi | kashyap: if you checked the fauilre and seems unrealated to your patch then feel free to recheck | |
| 10:20:53 | kashyap | Yeah, they're unrelated; I can't spot anything related to my patch: http://logs.openstack.org/27/620327/3/gate/neutron-grenade/8ffe31b/job-output.txt.gz | |
| 10:23:01 | gibi | kashyap: yeah the cells-v1 test failure seems unrelated | |
| 10:24:03 | gibi | kashyap: and I dont see anything suspicious in the grenade log either | |
| 10:24:33 | kashyap | Thanks! I hit a 'recheck' | |
| 10:24:34 | gibi | kashyap: btw, you only loose the +W if you need to rebase | |
| 10:24:49 | kashyap | gibi: Ah, noted. | |
| 10:48:43 | lyarwood | mdbooth: https://review.openstack.org/#/c/619804/ - I know you're busy with your own func test hell but can you add this to your queue today, happy to close it out if it isn't useful. | |
| 10:50:06 | mdbooth | lyarwood: Oh, nice. Does it work? | |
| 10:50:46 | mdbooth | lyarwood: I see you're using it in the patch above. | |
| 10:50:58 | lyarwood | mdbooth: I was, but I've dropped the patch above now | |
| 10:51:06 | lyarwood | mdbooth: so I either rebase this on your stuff or kill it | |
| 10:51:15 | lyarwood | mdbooth: and it works, I think. | |
| 11:28:54 | openstackgerrit | Merged openstack/nova master: Clean up cpu_shared_set config docs https://review.openstack.org/614864 | |
| 11:29:00 | openstackgerrit | Merged openstack/nova master: Delete NeutronLinuxBridgeInterfaceDriver https://review.openstack.org/616995 | |
| 11:53:02 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810 | |
| 11:53:03 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion 2.68: Handles Down Cells https://review.openstack.org/591657 | |
| 12:04:41 | openstackgerrit | Chris Dent proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | |
| 12:25:24 | openstackgerrit | Guo Jingyu proposed openstack/nova master: Add rfb.VNC support for novncproxy https://review.openstack.org/622336 | |
| 12:33:32 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Handle tags in _bury_in_cell0 https://review.openstack.org/621856 | |
| 12:58:49 | mriedem | tssurya: belmoreira: i'm not sure what i'm missing here https://bugs.launchpad.net/nova/+bug/1805989 | |
| 12:58:49 | openstack | Launchpad bug 1805989 in OpenStack Compute (nova) "Weight policy to stack/spread instances and "max_placement_results"" [Undecided,New] | |
| 13:03:17 | tssurya | mriedem: I guess from our prespective since we don't use any scheduler fitlers (except the Compute Filter), as opposed to a scenario where placement would return all results as in all possible available resources on which the weights would be applied versus what happens now when we do this only the max_placement_results number of available resources (hosts) | |
| 13:03:46 | tssurya | would effect the way packing/spreading is done | |
| 13:03:52 | mriedem | that's working as designed.... | |
| 13:04:04 | mriedem | max_placement_results is just a front-end filter | |
| 13:04:23 | sean-k-mooney | tssurya: you dont use any filetre so you dont use sriov/pci passthrough/cpu pinning/hugepages or numa in your cloud | |
| 13:04:26 | mriedem | like i said in the bug, even before placement, if you have 1k hosts, if the filters narrow that down to 10, then only 10 get weighed | |
| 13:04:33 | mriedem | sean-k-mooney: no they don't, | |
| 13:04:37 | mriedem | oh well, | |
| 13:04:39 | mriedem | nvm | |
| 13:04:55 | mriedem | b/c of cells v1 there were several things they couldn't support before, like affinity | |
| 13:05:04 | mriedem | idk about pci/numa | |
| 13:05:15 | sean-k-mooney | this seams kindof like a duplicate of https://bugs.launchpad.net/nova/+bug/1805984 | |
| 13:05:15 | openstack | Launchpad bug 1805984 in OpenStack Compute (nova) "Placement is not aware of disable compute nodes" [Wishlist,Triaged] | |
| 13:05:18 | tssurya | sean-k-mooney: we use pci passthrough, but not that much (just for one cell) | |
| 13:05:48 | mriedem | sean-k-mooney: that is definitely a different problem | |
| 13:06:02 | mriedem | that bug is that placement is returning only disabled computes | |
| 13:06:11 | mriedem | because cern sets max_placement_results very low | |
| 13:06:14 | sean-k-mooney | yes | |
| 13:06:21 | mriedem | this other one is different | |
| 13:06:35 | mriedem | placement has no relationship to weighers in nova-scheduler | |
| 13:06:47 | sean-k-mooney | it is a different problem but they are both because the limiting is done in placmeent not after teh filters | |
| 13:07:06 | mriedem | for the weigher bug it doesn't matter | |
| 13:07:13 | mriedem | filters are before weighers, | |
| 13:07:21 | mriedem | so if placement filtered out the hosts or the scheduler filters did, it doesn't matter | |
| 13:07:33 | mriedem | the weighers are only going to weigh what they get | |
| 13:07:54 | mriedem | so they will spread/pack across whatever comes through the filters | |
| 13:07:55 | sean-k-mooney | it maters if you have lost of disabled host in the limited set of result you got form plamcnet | |
| 13:08:40 | sean-k-mooney | it will skew the weighing by reducing the set of results | |
| 13:08:45 | tssurya | mriedem: exactly for us when not using "max_placement_results" filters would give us almost all the hosts from a cell | |
| 13:09:15 | mriedem | tssurya: yeah, so IMO bug 1805989 is not really a bug, it's working as designed | |
| 13:09:15 | openstack | bug 1805989 in OpenStack Compute (nova) "Weight policy to stack/spread instances and "max_placement_results"" [Undecided,New] https://launchpad.net/bugs/1805989 | |
| 13:09:20 | tssurya | but yea probably this iss not a bug its design thing | |
| 13:09:29 | mriedem | bug 1805984 is definitely a problem for which we have solutions | |
| 13:09:29 | openstack | bug 1805984 in OpenStack Compute (nova) "Placement is not aware of disable compute nodes" [Wishlist,Triaged] https://launchpad.net/bugs/1805984 | |
| 13:09:38 | sean-k-mooney | that said it is working as desinged as mriedem said. that desing my not be desireable and we might want to consider how to change it | |
| 13:09:58 | gibi | mriedem: I saw your comment on the notification deprecation patch. I have to organize my thoughts formulate an opinion | |
| 13:10:15 | mriedem | what we need to do is get CERN to the point that they don't have to workaround perf issues by setting max_placement_results to 10 | |
| 13:10:18 | mriedem | when they have 14K hosts | |
| 13:10:27 | mriedem | gibi: heh ok :) | |
| 13:10:29 | mriedem | no rush | |
| 13:10:35 | gibi | mriedem: yeah, I will take my time | |
| 13:10:38 | gibi | :) | |
| 13:10:46 | mriedem | gibi: it's a bit depressing huh? | |