| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 20:53:48 | sean-k-mooney | i can add it expclitly to the job i guess | |
| 20:54:20 | sean-k-mooney | i also dont have that set locally so that is prably why i sometimes saw this an other times did not | |
| 20:54:30 | mriedem | sean-k-mooney: different patch sets for those jobs, | |
| 20:54:32 | mriedem | not sure if that matters, | |
| 20:54:34 | mriedem | i was looking at PS1 | |
| 20:54:48 | mriedem | https://review.opendev.org/#/c/681771/1..2/.zuul.yaml | |
| 20:55:01 | mriedem | do'nt know why that would change things | |
| 20:55:09 | sean-k-mooney | nor do i | |
| 20:55:18 | sean-k-mooney | that was needed to activate stephens code | |
| 20:55:24 | mriedem | unless there is a devstack or zuul bug or something | |
| 20:55:34 | sean-k-mooney | sice cpu_dedicated_set is what does the reshape | |
| 20:56:21 | mriedem | maybe https://github.com/openstack/devstack/commit/2468ceaa724aa5c8c44fb87ae223eb6687ff85f2 regressed something in devstack | |
| 20:56:26 | sean-k-mooney | anwyway the important thing is the last run pased with both seriese merged togoether so at least that good | |
| 20:56:58 | sean-k-mooney | maybe but both of thoes runs were done today | |
| 20:57:01 | mriedem | yeah, i guess we just keep an eye out for random failures like that | |
| 20:57:30 | sean-k-mooney | ya maybe check logstash to see if its happening | |
| 20:58:25 | sean-k-mooney | right so i was going to try and create a non legacy grenade job with all the onter non merged stuff. | |
| 20:59:02 | sean-k-mooney | what of the chaces 4 un merged unrelated seriese will all work togeher perficectly | |
| 20:59:02 | dansmith | mriedem: we only do that if we have an instance group right? | |
| 20:59:12 | dansmith | does the network advanced server ops thing use groups? | |
| 20:59:27 | sean-k-mooney | i dont think so | |
| 20:59:38 | sean-k-mooney | you mean server groups right | |
| 20:59:39 | dansmith | so I'm wondering why we'd be hitting this for that | |
| 20:59:42 | dansmith | yes, server grops | |
| 20:59:48 | dansmith | groups even | |
| 21:00:18 | sean-k-mooney | do we do check for resize/migrate | |
| 21:00:27 | sean-k-mooney | its for the anti afinity stuff right | |
| 21:00:27 | dansmith | only if they're in a group, IIRC | |
| 21:00:28 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 21:00:28 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 21:00:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 21:00:30 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 21:00:31 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | |
| 21:01:01 | dansmith | I was wondering earlier if we were failing to boot because of pcpu stuff and running that group recalc in a reschedule or something | |
| 21:01:07 | efried | stephenfin: https://review.opendev.org/#/c/678453/33..34/nova/tests/unit/objects/test_objects.py | |
| 21:01:10 | mriedem | efried: nova meeting? | |
| 21:01:29 | efried | yes | |
| 21:01:57 | mriedem | dansmith: yeah https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1457 | |
| 21:02:06 | mriedem | though if it's a reschedule there are a couple of known up-call fails from that | |
| 21:02:06 | ozzzo | 2019-09-12 13:05:10.154 60779 ERROR nova.virt.libvirt.driver [-] [instance: 01a9d185-f003-4f59-b87b-9256f5ea2eaa] Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor | |
| 21:02:06 | ozzzo | I'm trying to get live migration working without shared storage. I got libvirtd to listen and set auth_tcp = "none" to fix the auth errrors; now the error is: | |
| 21:02:13 | mriedem | dansmith: for resize and build reschedule | |
| 21:02:18 | ozzzo | Does anyone know what that means? I don't have apparmor running | |
| 21:02:21 | mriedem | to pull the AZ for the host which goes to aggregates | |
| 21:02:48 | mriedem | dansmith: https://bugs.launchpad.net/nova/+bug/1781286 | |
| 21:02:49 | openstack | Launchpad bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,Triaged] | |
| 21:03:03 | sean-k-mooney | ozzzo: that is proably what its complaining about | |
| 21:03:06 | dansmith | this one looked like setup_groups | |
| 21:03:09 | dansmith | or whatever it's called | |
| 21:03:09 | mriedem | Error trying to reschedule: oslo_messaging.rpc.client.RemoteError: Remote error: CantStartEngineError No sql_connection parameter is established | |
| 21:03:21 | ozzzo | I need apparmor for live migration? | |
| 21:03:29 | sean-k-mooney | no | |
| 21:03:50 | mriedem | dansmith: nope it's that aggs bug | |
| 21:03:52 | mriedem | https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/controller/logs/screen-n-cond-cell1.txt.gz | |
| 21:03:55 | sean-k-mooney | ozzzo: but if you dont have apparmor running you need to disable apparmor as the security driver in the libvirt config | |
| 21:03:57 | mriedem | grep for CantStartEngineError in there | |
| 21:04:07 | mriedem | reschedules and then tries to get az from the cell conductor | |
| 21:04:22 | sean-k-mooney | mriedem: ya we were seeing CantStartEngineError | |
| 21:04:40 | mriedem | yup, known bug, see above | |
| 21:05:07 | dansmith | mriedem: that fails from resouces unavailable on the compute | |
| 21:05:07 | mriedem | i don't think tempest does a ton of server group testing but there are a few tests for server affinity and anti-affinty group testing | |
| 21:05:24 | ozzzo | I don't find "appa" in my libvirt config. Is it a default that I need to override? | |
| 21:05:26 | dansmith | "Requested instance NUMA topology cannot fit the given host NUMA topology" | |
| 21:05:38 | mriedem | dansmith: oh, maybe that's a race? | |
| 21:05:44 | mriedem | are the tests running in serial? | |
| 21:05:49 | dansmith | sean-k-mooney: ^ | |
| 21:05:56 | ozzzo | oic it looks like it is the default in qemu.conf | |
| 21:06:04 | ozzzo | # security_driver = [ "selinux", "apparmor" ] | |
| 21:06:06 | sean-k-mooney | dansmith: or the issue was the exception in update_resouce | |
| 21:06:18 | ozzzo | what would that look like, to disable both? | |
| 21:06:32 | ozzzo | security_driver = [] ? | |
| 21:06:35 | sean-k-mooney | ozzzo: i think by defualt it will use either | |
| 21:06:40 | sean-k-mooney | you have to expclity diable it | |
| 21:07:22 | sean-k-mooney | mriedem: and yes i have it running serially. | |
| 21:07:54 | mriedem | ozzzo: please see the channel topic, this is a mostly dev focused channel and today is feature freeze for the train release so as you can see there is quite a bit of dev activity trying to get this release closed out | |
| 21:08:09 | mriedem | and we're having a meeting at the same time over in #openstack-meeting | |
| 21:08:25 | mriedem | ozzzo: #openstack-operators or just #openstack might help, or the ask.o.o forum | |
| 21:08:37 | mriedem | otherwise try hitting up the openstack-discuss mailing list | |
| 21:09:08 | sean-k-mooney | ozzzo: i think you need to do [] yes but i dont know. | |
| 21:09:15 | ozzzo | ok i'll try it, ty | |
| 21:09:57 | mriedem | anyway, sean-k-mooney dansmith reschedule in gate jobs with devstack doesn't really work b/c of that up-call | |
| 21:10:19 | mriedem | we just don't normally see or care about it b/c if we hit it the job fails and we're likely toast anyway, but we don't normally see reschedules in tempest runs anyway, | |
| 21:10:27 | mriedem | unless like libvirtd dies on a compute and if that happens we fail anyway | |
| 21:10:55 | dansmith | that's why I'm wondering if we're failing and doing a reschedule because of the changes under test | |
| 21:12:17 | mriedem | could be latent races in the numa claims code but yeah, idk | |
| 21:12:34 | mriedem | if we're running tests in serial that should be less of a risk, unless some test is creating multiple servers in one run | |
| 21:12:47 | mriedem | and we're trying to pin all to the same cpus? | |
| 21:12:57 | dansmith | oh well, we'll just wait for mnaser to shake out all the bugs in here when he rolls it out | |
| 21:13:52 | mriedem | the job is setup to pin to the same cpus for the flavor we're using? | |
| 21:13:57 | mriedem | or how does that work? | |
| 21:14:14 | mriedem | iow, if a test creates more than one server on the same host, are we guaranteed to fail a claim for one and reschedule? | |
| 21:14:30 | sean-k-mooney | no | |
| 21:14:44 | sean-k-mooney | we have 8 cpus in the gate vms | |
| 21:14:50 | sean-k-mooney | the instace are each claiming 1 | |
| 21:15:00 | sean-k-mooney | and i have 7 of the 8 enabeld | |
| 21:15:10 | sean-k-mooney | but some tests do create multiple vms | |
| 21:17:17 | mriedem | yeah ok, but there aren't any tempest tests that create more than 3 servers i'm pretty sure | |
| 21:17:23 | mriedem | in a single test case i mean, | |