| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 20:46:50 | sean-k-mooney | yeah the db is not configured on either node | |
| 20:46:51 | mriedem | talking about this job? https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_909/681840/2/experimental/nova-nfv-multinode/9090152/testr_results.html.gz | |
| 20:47:07 | stephenfin | sean-k-mooney: I'm looking but I don't see why it would. | |
| 20:47:08 | dansmith | no | |
| 20:47:15 | dansmith | mriedem: https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/screen-n-cpu.txt.gz | |
| 20:47:17 | efried | stephenfin: that rebase isn't going to work, because numalm is based waaay back on master, so that way conflicts with (at least) cpu models. I could rebase numalm, but it would lose its place in the queue (~10.5h) so I don't want to do that. Just going to ride it out, mkay? | |
| 20:47:38 | dansmith | mriedem: but I think this is just the late affinity check that must not be disabled on sean-k-mooney's | |
| 20:47:42 | stephenfin | (y) | |
| 20:48:10 | sean-k-mooney | this one https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/compute/logs/etc/nova/nova-cpu_conf.txt.gz#92 | |
| 20:48:25 | mriedem | disable_group_policy_check_upcall | |
| 20:48:32 | dansmith | sean-k-mooney: that's set on your job? | |
| 20:48:39 | mriedem | it's not in https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/etc/nova/nova_conf.txt.gz | |
| 20:48:41 | mriedem | in the subnode | |
| 20:48:44 | sean-k-mooney | yes | |
| 20:48:55 | dansmith | okay not being on the subnode would explain why we hit it sometimes and not others | |
| 20:48:56 | mriedem | devstack should configure that by default if you're using superconductor mode (which is the default) | |
| 20:49:22 | sean-k-mooney | its set on the contoler and compute | |
| 20:49:33 | mriedem | in that link above, | |
| 20:49:38 | mriedem | it's set on the compute on the controller host https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/controller/logs/etc/nova/nova-cpu_conf.txt.gz | |
| 20:49:39 | mriedem | the primary | |
| 20:49:41 | mriedem | but not the subnode | |
| 20:49:45 | mriedem | so that's why it's flaky | |
| 20:51:30 | mriedem | i'm not sure why it wouldn't be set, unless the zuul yaml post-config is overwriting the workarounds section | |
| 20:51:39 | mriedem | but the controller devstack run fixes it but the subnode one doesn't | |
| 20:51:56 | sean-k-mooney | i dont think that is what is happening | |
| 20:52:10 | sean-k-mooney | that is the comptue/subnode https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/compute/logs/etc/nova/nova-cpu_conf.txt.gz#92 | |
| 20:52:26 | sean-k-mooney | that is the contoler/primary https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/controller/logs/etc/nova/nova-cpu_conf.txt.gz#117 | |
| 20:52:37 | sean-k-mooney | on the run that pased its set on both | |
| 20:52:46 | mriedem | yeah i was looking at the one that failed | |
| 20:52:53 | mriedem | note that i'm coming into this 2 hours late :) | |
| 20:52:53 | sean-k-mooney | oh | |
| 20:53:08 | sean-k-mooney | so the one that failed didnt have it set on both | |
| 20:53:13 | dansmith | right | |
| 20:53:21 | sean-k-mooney | ok | |
| 20:53:36 | sean-k-mooney | i have no idea why that is then | |
| 20:53:48 | sean-k-mooney | i can add it expclitly to the job i guess | |
| 20:54:20 | sean-k-mooney | i also dont have that set locally so that is prably why i sometimes saw this an other times did not | |
| 20:54:30 | mriedem | sean-k-mooney: different patch sets for those jobs, | |
| 20:54:32 | mriedem | not sure if that matters, | |
| 20:54:34 | mriedem | i was looking at PS1 | |
| 20:54:48 | mriedem | https://review.opendev.org/#/c/681771/1..2/.zuul.yaml | |
| 20:55:01 | mriedem | do'nt know why that would change things | |
| 20:55:09 | sean-k-mooney | nor do i | |
| 20:55:18 | sean-k-mooney | that was needed to activate stephens code | |
| 20:55:24 | mriedem | unless there is a devstack or zuul bug or something | |
| 20:55:34 | sean-k-mooney | sice cpu_dedicated_set is what does the reshape | |
| 20:56:21 | mriedem | maybe https://github.com/openstack/devstack/commit/2468ceaa724aa5c8c44fb87ae223eb6687ff85f2 regressed something in devstack | |
| 20:56:26 | sean-k-mooney | anwyway the important thing is the last run pased with both seriese merged togoether so at least that good | |
| 20:56:58 | sean-k-mooney | maybe but both of thoes runs were done today | |
| 20:57:01 | mriedem | yeah, i guess we just keep an eye out for random failures like that | |
| 20:57:30 | sean-k-mooney | ya maybe check logstash to see if its happening | |
| 20:58:25 | sean-k-mooney | right so i was going to try and create a non legacy grenade job with all the onter non merged stuff. | |
| 20:59:02 | sean-k-mooney | what of the chaces 4 un merged unrelated seriese will all work togeher perficectly | |
| 20:59:02 | dansmith | mriedem: we only do that if we have an instance group right? | |
| 20:59:12 | dansmith | does the network advanced server ops thing use groups? | |
| 20:59:27 | sean-k-mooney | i dont think so | |
| 20:59:38 | sean-k-mooney | you mean server groups right | |
| 20:59:39 | dansmith | so I'm wondering why we'd be hitting this for that | |
| 20:59:42 | dansmith | yes, server grops | |
| 20:59:48 | dansmith | groups even | |
| 21:00:18 | sean-k-mooney | do we do check for resize/migrate | |
| 21:00:27 | sean-k-mooney | its for the anti afinity stuff right | |
| 21:00:27 | dansmith | only if they're in a group, IIRC | |
| 21:00:28 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 21:00:28 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 21:00:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470 | |
| 21:00:30 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |
| 21:00:31 | openstackgerrit | Eric Fried proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | |
| 21:01:01 | dansmith | I was wondering earlier if we were failing to boot because of pcpu stuff and running that group recalc in a reschedule or something | |
| 21:01:07 | efried | stephenfin: https://review.opendev.org/#/c/678453/33..34/nova/tests/unit/objects/test_objects.py | |
| 21:01:10 | mriedem | efried: nova meeting? | |
| 21:01:29 | efried | yes | |
| 21:01:57 | mriedem | dansmith: yeah https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1457 | |
| 21:02:06 | mriedem | though if it's a reschedule there are a couple of known up-call fails from that | |
| 21:02:06 | ozzzo | 2019-09-12 13:05:10.154 60779 ERROR nova.virt.libvirt.driver [-] [instance: 01a9d185-f003-4f59-b87b-9256f5ea2eaa] Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor | |
| 21:02:06 | ozzzo | I'm trying to get live migration working without shared storage. I got libvirtd to listen and set auth_tcp = "none" to fix the auth errrors; now the error is: | |
| 21:02:13 | mriedem | dansmith: for resize and build reschedule | |
| 21:02:18 | ozzzo | Does anyone know what that means? I don't have apparmor running | |
| 21:02:21 | mriedem | to pull the AZ for the host which goes to aggregates | |
| 21:02:48 | mriedem | dansmith: https://bugs.launchpad.net/nova/+bug/1781286 | |
| 21:02:49 | openstack | Launchpad bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,Triaged] | |
| 21:03:03 | sean-k-mooney | ozzzo: that is proably what its complaining about | |
| 21:03:06 | dansmith | this one looked like setup_groups | |
| 21:03:09 | dansmith | or whatever it's called | |
| 21:03:09 | mriedem | Error trying to reschedule: oslo_messaging.rpc.client.RemoteError: Remote error: CantStartEngineError No sql_connection parameter is established | |
| 21:03:21 | ozzzo | I need apparmor for live migration? | |
| 21:03:29 | sean-k-mooney | no | |
| 21:03:50 | mriedem | dansmith: nope it's that aggs bug | |
| 21:03:52 | mriedem | https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/controller/logs/screen-n-cond-cell1.txt.gz | |
| 21:03:55 | sean-k-mooney | ozzzo: but if you dont have apparmor running you need to disable apparmor as the security driver in the libvirt config | |
| 21:03:57 | mriedem | grep for CantStartEngineError in there | |
| 21:04:07 | mriedem | reschedules and then tries to get az from the cell conductor | |
| 21:04:22 | sean-k-mooney | mriedem: ya we were seeing CantStartEngineError | |
| 21:04:40 | mriedem | yup, known bug, see above | |
| 21:05:07 | dansmith | mriedem: that fails from resouces unavailable on the compute | |
| 21:05:07 | mriedem | i don't think tempest does a ton of server group testing but there are a few tests for server affinity and anti-affinty group testing | |
| 21:05:24 | ozzzo | I don't find "appa" in my libvirt config. Is it a default that I need to override? | |
| 21:05:26 | dansmith | "Requested instance NUMA topology cannot fit the given host NUMA topology" | |