| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 21:03:57 | mriedem | grep for CantStartEngineError in there | |
| 21:04:07 | mriedem | reschedules and then tries to get az from the cell conductor | |
| 21:04:22 | sean-k-mooney | mriedem: ya we were seeing CantStartEngineError | |
| 21:04:40 | mriedem | yup, known bug, see above | |
| 21:05:07 | mriedem | i don't think tempest does a ton of server group testing but there are a few tests for server affinity and anti-affinty group testing | |
| 21:05:07 | dansmith | mriedem: that fails from resouces unavailable on the compute | |
| 21:05:24 | ozzzo | I don't find "appa" in my libvirt config. Is it a default that I need to override? | |
| 21:05:26 | dansmith | "Requested instance NUMA topology cannot fit the given host NUMA topology" | |
| 21:05:38 | mriedem | dansmith: oh, maybe that's a race? | |
| 21:05:44 | mriedem | are the tests running in serial? | |
| 21:05:49 | dansmith | sean-k-mooney: ^ | |
| 21:05:56 | ozzzo | oic it looks like it is the default in qemu.conf | |
| 21:06:04 | ozzzo | # security_driver = [ "selinux", "apparmor" ] | |
| 21:06:06 | sean-k-mooney | dansmith: or the issue was the exception in update_resouce | |
| 21:06:18 | ozzzo | what would that look like, to disable both? | |
| 21:06:32 | ozzzo | security_driver = [] ? | |
| 21:06:35 | sean-k-mooney | ozzzo: i think by defualt it will use either | |
| 21:06:40 | sean-k-mooney | you have to expclity diable it | |
| 21:07:22 | sean-k-mooney | mriedem: and yes i have it running serially. | |
| 21:07:54 | mriedem | ozzzo: please see the channel topic, this is a mostly dev focused channel and today is feature freeze for the train release so as you can see there is quite a bit of dev activity trying to get this release closed out | |
| 21:08:09 | mriedem | and we're having a meeting at the same time over in #openstack-meeting | |
| 21:08:25 | mriedem | ozzzo: #openstack-operators or just #openstack might help, or the ask.o.o forum | |
| 21:08:37 | mriedem | otherwise try hitting up the openstack-discuss mailing list | |
| 21:09:08 | sean-k-mooney | ozzzo: i think you need to do [] yes but i dont know. | |
| 21:09:15 | ozzzo | ok i'll try it, ty | |
| 21:09:57 | mriedem | anyway, sean-k-mooney dansmith reschedule in gate jobs with devstack doesn't really work b/c of that up-call | |
| 21:10:19 | mriedem | we just don't normally see or care about it b/c if we hit it the job fails and we're likely toast anyway, but we don't normally see reschedules in tempest runs anyway, | |
| 21:10:27 | mriedem | unless like libvirtd dies on a compute and if that happens we fail anyway | |
| 21:10:55 | dansmith | that's why I'm wondering if we're failing and doing a reschedule because of the changes under test | |
| 21:12:17 | mriedem | could be latent races in the numa claims code but yeah, idk | |
| 21:12:34 | mriedem | if we're running tests in serial that should be less of a risk, unless some test is creating multiple servers in one run | |
| 21:12:47 | mriedem | and we're trying to pin all to the same cpus? | |
| 21:12:57 | dansmith | oh well, we'll just wait for mnaser to shake out all the bugs in here when he rolls it out | |
| 21:13:52 | mriedem | the job is setup to pin to the same cpus for the flavor we're using? | |
| 21:13:57 | mriedem | or how does that work? | |
| 21:14:14 | mriedem | iow, if a test creates more than one server on the same host, are we guaranteed to fail a claim for one and reschedule? | |
| 21:14:30 | sean-k-mooney | no | |
| 21:14:44 | sean-k-mooney | we have 8 cpus in the gate vms | |
| 21:14:50 | sean-k-mooney | the instace are each claiming 1 | |
| 21:15:00 | sean-k-mooney | and i have 7 of the 8 enabeld | |
| 21:15:10 | sean-k-mooney | but some tests do create multiple vms | |
| 21:17:17 | mriedem | yeah ok, but there aren't any tempest tests that create more than 3 servers i'm pretty sure | |
| 21:17:23 | mriedem | in a single test case i mean, | |
| 21:17:33 | mriedem | and i'm pretty sure the tempest tearDown waits for the servers to be gone, i.e. 404 from the API | |
| 21:18:10 | sean-k-mooney | the case weher we saw that errror the teardwon failed to delete before hiting the timeout | |
| 21:18:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Correctly handle non-CPU flag traits https://review.opendev.org/681932 | |
| 21:18:56 | stephenfin | mriedem, dansmith, efried: ^ | |
| 21:19:15 | mriedem | stephenfin: is that a master only regression? it should probably have a bug | |
| 21:19:23 | stephenfin | test is wrong - I rushed it | |
| 21:19:28 | stephenfin | I can do that | |
| 21:19:45 | mriedem | so only unit tests, no functional coverage | |
| 21:19:58 | sean-k-mooney | did that get intoduced with the sev stuff | |
| 21:20:02 | mriedem | this might be where i asked for a nova/tests/functional/regressions test case | |
| 21:20:18 | mriedem | code: do something wrong | |
| 21:20:25 | mriedem | unit test: assert the code did something wrong | |
| 21:23:20 | stephenfin | can someone give me an example of a non-CPU flag trait we support at the moment? | |
| 21:25:27 | mriedem | COMPUTE_SUPPORTS_MULTIATTACH | |
| 21:25:42 | sean-k-mooney | i was going to mention the once in my seriese that we punted... | |
| 21:25:47 | sean-k-mooney | ya that | |
| 21:26:07 | mriedem | so i bet we could recreate in devstack by just adding that to a tempest flavor yeah? | |
| 21:26:35 | sean-k-mooney | ya | |
| 21:27:01 | sean-k-mooney | you can do it the way im doing all the nfv/numa jobs | |
| 21:27:22 | mriedem | heh even just COMPUTE_ATTACH_INTERFACE | |
| 21:27:44 | stephenfin | mriedem: https://bugs.launchpad.net/nova/+bug/1843836 | |
| 21:27:45 | openstack | Launchpad bug 1843836 in OpenStack Compute (nova) "Failure to schedule if flavor contains non-CPU flag traits" [Undecided,New] | |
| 21:27:49 | sean-k-mooney | well all the compute capablity traits fall into that catagory right | |
| 21:28:16 | mriedem | stephenfin: yeah anything in here really https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L102 | |
| 21:28:31 | efried | artom et al, numalm bottom failed l-c on subunit parser https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_769/634827/60/check/openstack-tox-lower-constraints/7693dab/testr_results.html.gz | |
| 21:28:58 | artom | efried, ugh, recheck? | |
| 21:29:20 | artom | That's not a legit thing, right? | |
| 21:29:23 | efried | #2 failed functional on 6!=7 -- we fixed that mriedem right? Rebase? | |
| 21:29:33 | efried | artom: question whether to recheck or rebase | |
| 21:29:41 | efried | artom: you're way down the master branch | |
| 21:29:42 | mriedem | efried: gate should pick it up from master | |
| 21:29:54 | efried | you would think so mriedem, but it's possible this was queued from before that merged too. | |
| 21:29:59 | efried | https://ebfd4ec54dbc92b01f1e-dbc96e30a25abd79ad83d28f747c4e2b.ssl.cf2.rackcdn.com/635229/62/check/nova-tox-functional/a59c098/testr_results.html.gz | |
| 21:30:01 | artom | efried, yeah, that was intentional to make reviewing easier | |
| 21:30:03 | mriedem | the subunit to parser failure is when a test dumps too much shit to the logging output stream | |
| 21:30:04 | sean-k-mooney | efried: it will do a merge of the patch you see in gerrit and master | |
| 21:30:09 | mriedem | efried: yeah maybe | |
| 21:30:15 | mriedem | that claims one was in the gate b/c my fix was merged | |
| 21:30:25 | mriedem | *before | |
| 21:30:39 | efried | #3 hasn't failed yet, #4 is still queued | |
| 21:31:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Correctly handle non-CPU flag traits https://review.opendev.org/681932 | |
| 21:31:13 | efried | so the question is, do we recheck 1&2, hoping 3&4 pass, and save their place in the queue | |
| 21:31:13 | efried | or do we rebase, and lose 3&4's place in the queue | |
| 21:31:13 | sean-k-mooney | if we are going to rebase the LM series now is proably better then in a few hours | |
| 21:31:29 | efried | artom: your call | |
| 21:31:38 | stephenfin | mriedem: I'll whip up a functional test for that tomorrow. It's too late to face into that now | |
| 21:31:41 | mriedem | ha, got a node after 11 hours, then failed | |
| 21:31:51 | stephenfin | but unit test is fixed and I've linked the bug | |
| 21:31:55 | mriedem | stephenfin: sure, i was just going to push a simple devstack patch to show the failure | |
| 21:32:03 | artom | efried, does it really matter? I'm assuming if I rebase now, I'll get administrative +Ws | |
| 21:32:04 | efried | artom: first option we have to wait until gate posts actual failure, which will be a while. IMO second option would be best | |
| 21:32:14 | efried | artom: yes | |
| 21:32:15 | artom | It's not like we need this by tomorrow | |
| 21:32:40 | stephenfin | giant chain of patches | |
| 21:32:41 | stephenfin | wheee | |
| 21:32:44 | mriedem | artom: sorry but my +Ws on that series were only good through yesterday | |
| 21:32:44 | artom | It can wait the weekend or w/e, when the load on the gate isn't as high | |