Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-12
20:40:22 sean-k-mooney devstack-gate is a pain to get kvm working
20:40:38 sean-k-mooney yes it looks like its passing so ill try and modify it
20:40:45 dansmith cool
20:40:45 sean-k-mooney well depend on it
20:41:08 mriedem so...things are where they were 2 hours ago?
20:41:18 mriedem ooo down to 179 changes in the check queue
20:41:43 dansmith still nothing in the gate tho
20:42:03 mriedem yup, i don't expect nova stuff to probably be fully merged until maybe this weekend
20:42:08 mriedem if you account for rechecks
20:42:25 sean-k-mooney dansmith: by the way the latest test run for the combiniend numa + pcpus job does not appear to have the db error
20:42:48 dansmith sean-k-mooney: that db error wouldn't be flaky, so that makes me kinda wonder
20:43:16 sean-k-mooney well the difference is i am using cpu_dedicated_set instead of vcpu_pin_set
20:43:21 dansmith sean-k-mooney: unless one node is configured to have api db access and the other is not, and you don't get the fail if you're lucky and land on the one
20:45:11 sean-k-mooney stephenfin: could we be trying to lazy load the pinned cpus or something when we use vcpu_pin_set instead of cpu_dedicated_Set
20:45:44 mriedem devstack shouldn't configure nova-cpu.conf nor either of the cell conductor configs for api db access
20:45:46 mriedem by default anyway
20:46:22 dansmith mriedem: yeah I dunno why this would be legit failing here and not on master tho
20:46:50 sean-k-mooney yeah the db is not configured on either node
20:46:51 mriedem talking about this job? https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_909/681840/2/experimental/nova-nfv-multinode/9090152/testr_results.html.gz
20:47:07 stephenfin sean-k-mooney: I'm looking but I don't see why it would.
20:47:08 dansmith no
20:47:15 dansmith mriedem: https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/screen-n-cpu.txt.gz
20:47:17 efried stephenfin: that rebase isn't going to work, because numalm is based waaay back on master, so that way conflicts with (at least) cpu models. I could rebase numalm, but it would lose its place in the queue (~10.5h) so I don't want to do that. Just going to ride it out, mkay?
20:47:38 dansmith mriedem: but I think this is just the late affinity check that must not be disabled on sean-k-mooney's
20:47:42 stephenfin (y)
20:48:10 sean-k-mooney this one https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/compute/logs/etc/nova/nova-cpu_conf.txt.gz#92
20:48:25 mriedem disable_group_policy_check_upcall
20:48:32 dansmith sean-k-mooney: that's set on your job?
20:48:39 mriedem it's not in https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/etc/nova/nova_conf.txt.gz
20:48:41 mriedem in the subnode
20:48:44 sean-k-mooney yes
20:48:55 dansmith okay not being on the subnode would explain why we hit it sometimes and not others
20:48:56 mriedem devstack should configure that by default if you're using superconductor mode (which is the default)
20:49:22 sean-k-mooney its set on the contoler and compute
20:49:33 mriedem in that link above,
20:49:38 mriedem it's set on the compute on the controller host https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/controller/logs/etc/nova/nova-cpu_conf.txt.gz
20:49:39 mriedem the primary
20:49:41 mriedem but not the subnode
20:49:45 mriedem so that's why it's flaky
20:51:30 mriedem i'm not sure why it wouldn't be set, unless the zuul yaml post-config is overwriting the workarounds section
20:51:39 mriedem but the controller devstack run fixes it but the subnode one doesn't
20:51:56 sean-k-mooney i dont think that is what is happening
20:52:10 sean-k-mooney that is the comptue/subnode https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/compute/logs/etc/nova/nova-cpu_conf.txt.gz#92
20:52:26 sean-k-mooney that is the contoler/primary https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/controller/logs/etc/nova/nova-cpu_conf.txt.gz#117
20:52:37 sean-k-mooney on the run that pased its set on both
20:52:46 mriedem yeah i was looking at the one that failed
20:52:53 sean-k-mooney oh
20:52:53 mriedem note that i'm coming into this 2 hours late :)
20:53:08 sean-k-mooney so the one that failed didnt have it set on both
20:53:13 dansmith right
20:53:21 sean-k-mooney ok
20:53:36 sean-k-mooney i have no idea why that is then
20:53:48 sean-k-mooney i can add it expclitly to the job i guess
20:54:20 sean-k-mooney i also dont have that set locally so that is prably why i sometimes saw this an other times did not
20:54:30 mriedem sean-k-mooney: different patch sets for those jobs,
20:54:32 mriedem not sure if that matters,
20:54:34 mriedem i was looking at PS1
20:54:48 mriedem https://review.opendev.org/#/c/681771/1..2/.zuul.yaml
20:55:01 mriedem do'nt know why that would change things
20:55:09 sean-k-mooney nor do i
20:55:18 sean-k-mooney that was needed to activate stephens code
20:55:24 mriedem unless there is a devstack or zuul bug or something
20:55:34 sean-k-mooney sice cpu_dedicated_set is what does the reshape
20:56:21 mriedem maybe https://github.com/openstack/devstack/commit/2468ceaa724aa5c8c44fb87ae223eb6687ff85f2 regressed something in devstack
20:56:26 sean-k-mooney anwyway the important thing is the last run pased with both seriese merged togoether so at least that good
20:56:58 sean-k-mooney maybe but both of thoes runs were done today
20:57:01 mriedem yeah, i guess we just keep an eye out for random failures like that
20:57:30 sean-k-mooney ya maybe check logstash to see if its happening
20:58:25 sean-k-mooney right so i was going to try and create a non legacy grenade job with all the onter non merged stuff.
20:59:02 dansmith mriedem: we only do that if we have an instance group right?
20:59:02 sean-k-mooney what of the chaces 4 un merged unrelated seriese will all work togeher perficectly
20:59:12 dansmith does the network advanced server ops thing use groups?
20:59:27 sean-k-mooney i dont think so
20:59:38 sean-k-mooney you mean server groups right
20:59:39 dansmith so I'm wondering why we'd be hitting this for that
20:59:42 dansmith yes, server grops
20:59:48 dansmith groups even
21:00:18 sean-k-mooney do we do check for resize/migrate
21:00:27 dansmith only if they're in a group, IIRC
21:00:27 sean-k-mooney its for the anti afinity stuff right
21:00:28 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453
21:00:28 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454
21:00:29 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455
21:00:29 openstackgerrit Eric Fried proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456
21:00:30 openstackgerrit Eric Fried proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640
21:00:30 openstackgerrit Eric Fried proposed openstack/nova master: Add functional tests for virtual persistent memory https://review.opendev.org/678470
21:00:31 openstackgerrit Eric Fried proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397
21:01:01 dansmith I was wondering earlier if we were failing to boot because of pcpu stuff and running that group recalc in a reschedule or something
21:01:07 efried stephenfin: https://review.opendev.org/#/c/678453/33..34/nova/tests/unit/objects/test_objects.py
21:01:10 mriedem efried: nova meeting?
21:01:29 efried yes
21:01:57 mriedem dansmith: yeah https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1457
21:02:06 ozzzo I'm trying to get live migration working without shared storage. I got libvirtd to listen and set auth_tcp = "none" to fix the auth errrors; now the error is:
21:02:06 ozzzo 2019-09-12 13:05:10.154 60779 ERROR nova.virt.libvirt.driver [-] [instance: 01a9d185-f003-4f59-b87b-9256f5ea2eaa] Live Migration failure: unsupported configuration: Unable to find security driver for model apparmor: libvirtError: unsupported configuration: Unable to find security driver for model apparmor
21:02:06 mriedem though if it's a reschedule there are a couple of known up-call fails from that
21:02:13 mriedem dansmith: for resize and build reschedule
21:02:18 ozzzo Does anyone know what that means? I don't have apparmor running
21:02:21 mriedem to pull the AZ for the host which goes to aggregates
21:02:48 mriedem dansmith: https://bugs.launchpad.net/nova/+bug/1781286
21:02:49 openstack Launchpad bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,Triaged]

Earlier   Later