| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 21:00:30 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 21:00:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 21:00:31 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | |
| 21:01:01 | dansmith | I was wondering earlier if we were failing to boot because of pcpu stuff and running that group recalc in a reschedule or something | |
| 21:01:07 | efried | stephenfin: https://review.opendev.org/#/c/678453/33..34/nova/tests/unit/objects/test_objects.py | |
| 21:01:10 | mriedem | efried: nova meeting? | |
| 21:01:29 | efried | yes | |
| 21:01:57 | mriedem | dansmith: yeah https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1457 | |
| 21:02:06 | ozzzo | I'm trying to get live migration working without shared storage. I got libvirtd to listen and set auth_tcp = "none" to fix the auth errrors; now the error is: | |
| 21:02:06 | ozzzo | 2019-09-12 13:05:10.154 60779 ERROR nova.virt.libvirt.driver [-] [instance: 01a9d185-f003-4f59-b87b-9256f5ea2eaa] Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor | |
| 21:02:06 | mriedem | though if it's a reschedule there are a couple of known up-call fails from that | |
| 21:02:13 | mriedem | dansmith: for resize and build reschedule | |
| 21:02:18 | ozzzo | Does anyone know what that means? I don't have apparmor running | |
| 21:02:21 | mriedem | to pull the AZ for the host which goes to aggregates | |
| 21:02:48 | mriedem | dansmith: https://bugs.launchpad.net/nova/+bug/1781286 | |
| 21:02:49 | openstack | Launchpad bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,Triaged] | |
| 21:03:03 | sean-k-mooney | ozzzo: that is proably what its complaining about | |
| 21:03:06 | dansmith | this one looked like setup_groups | |
| 21:03:09 | mriedem | Error trying to reschedule: oslo_messaging.rpc.client.RemoteError: Remote error: CantStartEngineError No sql_connection parameter is established | |
| 21:03:09 | dansmith | or whatever it's called | |
| 21:03:21 | ozzzo | I need apparmor for live migration? | |
| 21:03:29 | sean-k-mooney | no | |
| 21:03:50 | mriedem | dansmith: nope it's that aggs bug | |
| 21:03:52 | mriedem | https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/controller/logs/screen-n-cond-cell1.txt.gz | |
| 21:03:55 | sean-k-mooney | ozzzo: but if you dont have apparmor running you need to disable apparmor as the security driver in the libvirt config | |
| 21:03:57 | mriedem | grep for CantStartEngineError in there | |
| 21:04:07 | mriedem | reschedules and then tries to get az from the cell conductor | |
| 21:04:22 | sean-k-mooney | mriedem: ya we were seeing CantStartEngineError | |
| 21:04:40 | mriedem | yup, known bug, see above | |
| 21:05:07 | mriedem | i don't think tempest does a ton of server group testing but there are a few tests for server affinity and anti-affinty group testing | |
| 21:05:07 | dansmith | mriedem: that fails from resouces unavailable on the compute | |
| 21:05:24 | ozzzo | I don't find "appa" in my libvirt config. Is it a default that I need to override? | |
| 21:05:26 | dansmith | "Requested instance NUMA topology cannot fit the given host NUMA topology" | |
| 21:05:38 | mriedem | dansmith: oh, maybe that's a race? | |
| 21:05:44 | mriedem | are the tests running in serial? | |
| 21:05:49 | dansmith | sean-k-mooney: ^ | |
| 21:05:56 | ozzzo | oic it looks like it is the default in qemu.conf | |
| 21:06:04 | ozzzo | # security_driver = [ "selinux", "apparmor" ] | |
| 21:06:06 | sean-k-mooney | dansmith: or the issue was the exception in update_resouce | |
| 21:06:18 | ozzzo | what would that look like, to disable both? | |
| 21:06:32 | ozzzo | security_driver = [] ? | |
| 21:06:35 | sean-k-mooney | ozzzo: i think by defualt it will use either | |
| 21:06:40 | sean-k-mooney | you have to expclity diable it | |
| 21:07:22 | sean-k-mooney | mriedem: and yes i have it running serially. | |
| 21:07:54 | mriedem | ozzzo: please see the channel topic, this is a mostly dev focused channel and today is feature freeze for the train release so as you can see there is quite a bit of dev activity trying to get this release closed out | |
| 21:08:09 | mriedem | and we're having a meeting at the same time over in #openstack-meeting | |
| 21:08:25 | mriedem | ozzzo: #openstack-operators or just #openstack might help, or the ask.o.o forum | |
| 21:08:37 | mriedem | otherwise try hitting up the openstack-discuss mailing list | |
| 21:09:08 | sean-k-mooney | ozzzo: i think you need to do [] yes but i dont know. | |
| 21:09:15 | ozzzo | ok i'll try it, ty | |
| 21:09:57 | mriedem | anyway, sean-k-mooney dansmith reschedule in gate jobs with devstack doesn't really work b/c of that up-call | |
| 21:10:19 | mriedem | we just don't normally see or care about it b/c if we hit it the job fails and we're likely toast anyway, but we don't normally see reschedules in tempest runs anyway, | |
| 21:10:27 | mriedem | unless like libvirtd dies on a compute and if that happens we fail anyway | |
| 21:10:55 | dansmith | that's why I'm wondering if we're failing and doing a reschedule because of the changes under test | |
| 21:12:17 | mriedem | could be latent races in the numa claims code but yeah, idk | |
| 21:12:34 | mriedem | if we're running tests in serial that should be less of a risk, unless some test is creating multiple servers in one run | |
| 21:12:47 | mriedem | and we're trying to pin all to the same cpus? | |
| 21:12:57 | dansmith | oh well, we'll just wait for mnaser to shake out all the bugs in here when he rolls it out | |
| 21:13:52 | mriedem | the job is setup to pin to the same cpus for the flavor we're using? | |
| 21:13:57 | mriedem | or how does that work? | |
| 21:14:14 | mriedem | iow, if a test creates more than one server on the same host, are we guaranteed to fail a claim for one and reschedule? | |
| 21:14:30 | sean-k-mooney | no | |
| 21:14:44 | sean-k-mooney | we have 8 cpus in the gate vms | |
| 21:14:50 | sean-k-mooney | the instace are each claiming 1 | |
| 21:15:00 | sean-k-mooney | and i have 7 of the 8 enabeld | |
| 21:15:10 | sean-k-mooney | but some tests do create multiple vms | |
| 21:17:17 | mriedem | yeah ok, but there aren't any tempest tests that create more than 3 servers i'm pretty sure | |
| 21:17:23 | mriedem | in a single test case i mean, | |
| 21:17:33 | mriedem | and i'm pretty sure the tempest tearDown waits for the servers to be gone, i.e. 404 from the API | |
| 21:18:10 | sean-k-mooney | the case weher we saw that errror the teardwon failed to delete before hiting the timeout | |
| 21:18:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Correctly handle non-CPU flag traits https://review.opendev.org/681932 | |
| 21:18:56 | stephenfin | mriedem, dansmith, efried: ^ | |
| 21:19:15 | mriedem | stephenfin: is that a master only regression? it should probably have a bug | |
| 21:19:23 | stephenfin | test is wrong - I rushed it | |
| 21:19:28 | stephenfin | I can do that | |
| 21:19:45 | mriedem | so only unit tests, no functional coverage | |
| 21:19:58 | sean-k-mooney | did that get intoduced with the sev stuff | |
| 21:20:02 | mriedem | this might be where i asked for a nova/tests/functional/regressions test case | |
| 21:20:18 | mriedem | code: do something wrong | |
| 21:20:25 | mriedem | unit test: assert the code did something wrong | |
| 21:23:20 | stephenfin | can someone give me an example of a non-CPU flag trait we support at the moment? | |
| 21:25:27 | mriedem | COMPUTE_SUPPORTS_MULTIATTACH | |
| 21:25:42 | sean-k-mooney | i was going to mention the once in my seriese that we punted... | |
| 21:25:47 | sean-k-mooney | ya that | |
| 21:26:07 | mriedem | so i bet we could recreate in devstack by just adding that to a tempest flavor yeah? | |
| 21:26:35 | sean-k-mooney | ya | |
| 21:27:01 | sean-k-mooney | you can do it the way im doing all the nfv/numa jobs | |
| 21:27:22 | mriedem | heh even just COMPUTE_ATTACH_INTERFACE | |
| 21:27:44 | stephenfin | mriedem: https://bugs.launchpad.net/nova/+bug/1843836 | |
| 21:27:45 | openstack | Launchpad bug 1843836 in OpenStack Compute (nova) "Failure to schedule if flavor contains non-CPU flag traits" [Undecided,New] | |
| 21:27:49 | sean-k-mooney | well all the compute capablity traits fall into that catagory right | |
| 21:28:16 | mriedem | stephenfin: yeah anything in here really https://github.com/openstack/nova/blob/master/nova/virt/driver.py#L102 | |
| 21:28:31 | efried | artom et al, numalm bottom failed l-c on subunit parser https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_769/634827/60/check/openstack-tox-lower-constraints/7693dab/testr_results.html.gz | |
| 21:28:58 | artom | efried, ugh, recheck? | |
| 21:29:20 | artom | That's not a legit thing, right? | |
| 21:29:23 | efried | #2 failed functional on 6!=7 -- we fixed that mriedem right? Rebase? | |
| 21:29:33 | efried | artom: question whether to recheck or rebase | |
| 21:29:41 | efried | artom: you're way down the master branch | |