| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 20:36:48 | dansmith | I dunno where it is specifically but nova-cpu.conf yeah | |
| 20:36:48 | sean-k-mooney | yes we disabel the upcall | |
| 20:36:50 | sean-k-mooney | also https://8ab1fb0a384d9cbfa221-969de3c017bb40c2acaf4bf21edd2ff6.ssl.cf1.rackcdn.com/681771/2/experimental/nova-nfv-multi-numa-multinode/1faf214/testr_results.html.gz | |
| 20:36:56 | sean-k-mooney | https://review.opendev.org/#/c/681771/ | |
| 20:37:28 | dansmith | do we have any grenade results yet? | |
| 20:37:59 | sean-k-mooney | no i dont have a grenade job set up. but i can try to set one up quickly | |
| 20:38:30 | dansmith | I don't think quickly matters anymore, | |
| 20:38:36 | dansmith | but it'd be good to get a run on it while we have context, IMHO | |
| 20:39:21 | sean-k-mooney | so there is a patch to convert grenade to non legacy | |
| 20:39:37 | sean-k-mooney | if i depend on that i should be able to do the same config changes | |
| 20:39:58 | dansmith | oh, is that why it's hard because it's still a legacy job? | |
| 20:40:02 | sean-k-mooney | dansmith: this https://review.opendev.org/#/c/548936/ | |
| 20:40:07 | sean-k-mooney | dansmith: yes | |
| 20:40:10 | dansmith | gotcha | |
| 20:40:18 | dansmith | well, if that works currently, then yeah, try that I'd say | |
| 20:40:22 | sean-k-mooney | devstack-gate is a pain to get kvm working | |
| 20:40:38 | sean-k-mooney | yes it looks like its passing so ill try and modify it | |
| 20:40:45 | dansmith | cool | |
| 20:40:45 | sean-k-mooney | well depend on it | |
| 20:41:08 | mriedem | so...things are where they were 2 hours ago? | |
| 20:41:18 | mriedem | ooo down to 179 changes in the check queue | |
| 20:41:43 | dansmith | still nothing in the gate tho | |
| 20:42:03 | mriedem | yup, i don't expect nova stuff to probably be fully merged until maybe this weekend | |
| 20:42:08 | mriedem | if you account for rechecks | |
| 20:42:25 | sean-k-mooney | dansmith: by the way the latest test run for the combiniend numa + pcpus job does not appear to have the db error | |
| 20:42:48 | dansmith | sean-k-mooney: that db error wouldn't be flaky, so that makes me kinda wonder | |
| 20:43:16 | sean-k-mooney | well the difference is i am using cpu_dedicated_set instead of vcpu_pin_set | |
| 20:43:21 | dansmith | sean-k-mooney: unless one node is configured to have api db access and the other is not, and you don't get the fail if you're lucky and land on the one | |
| 20:45:11 | sean-k-mooney | stephenfin: could we be trying to lazy load the pinned cpus or something when we use vcpu_pin_set instead of cpu_dedicated_Set | |
| 20:45:44 | mriedem | devstack shouldn't configure nova-cpu.conf nor either of the cell conductor configs for api db access | |
| 20:45:46 | mriedem | by default anyway | |
| 20:46:22 | dansmith | mriedem: yeah I dunno why this would be legit failing here and not on master tho | |
| 20:46:50 | sean-k-mooney | yeah the db is not configured on either node | |
| 20:46:51 | mriedem | talking about this job? https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_909/681840/2/experimental/nova-nfv-multinode/9090152/testr_results.html.gz | |
| 20:47:07 | stephenfin | sean-k-mooney: I'm looking but I don't see why it would. | |
| 20:47:08 | dansmith | no | |
| 20:47:15 | dansmith | mriedem: https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/screen-n-cpu.txt.gz | |
| 20:47:17 | efried | stephenfin: that rebase isn't going to work, because numalm is based waaay back on master, so that way conflicts with (at least) cpu models. I could rebase numalm, but it would lose its place in the queue (~10.5h) so I don't want to do that. Just going to ride it out, mkay? | |
| 20:47:38 | dansmith | mriedem: but I think this is just the late affinity check that must not be disabled on sean-k-mooney's | |
| 20:47:42 | stephenfin | (y) | |
| 20:48:10 | sean-k-mooney | this one https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/compute/logs/etc/nova/nova-cpu_conf.txt.gz#92 | |
| 20:48:25 | mriedem | disable_group_policy_check_upcall | |
| 20:48:32 | dansmith | sean-k-mooney: that's set on your job? | |
| 20:48:39 | mriedem | it's not in https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/etc/nova/nova_conf.txt.gz | |
| 20:48:41 | mriedem | in the subnode | |
| 20:48:44 | sean-k-mooney | yes | |
| 20:48:55 | dansmith | okay not being on the subnode would explain why we hit it sometimes and not others | |
| 20:48:56 | mriedem | devstack should configure that by default if you're using superconductor mode (which is the default) | |
| 20:49:22 | sean-k-mooney | its set on the contoler and compute | |
| 20:49:33 | mriedem | in that link above, | |
| 20:49:38 | mriedem | it's set on the compute on the controller host https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/controller/logs/etc/nova/nova-cpu_conf.txt.gz | |
| 20:49:39 | mriedem | the primary | |
| 20:49:41 | mriedem | but not the subnode | |
| 20:49:45 | mriedem | so that's why it's flaky | |
| 20:51:30 | mriedem | i'm not sure why it wouldn't be set, unless the zuul yaml post-config is overwriting the workarounds section | |
| 20:51:39 | mriedem | but the controller devstack run fixes it but the subnode one doesn't | |
| 20:51:56 | sean-k-mooney | i dont think that is what is happening | |
| 20:52:10 | sean-k-mooney | that is the comptue/subnode https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/compute/logs/etc/nova/nova-cpu_conf.txt.gz#92 | |
| 20:52:26 | sean-k-mooney | that is the contoler/primary https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/controller/logs/etc/nova/nova-cpu_conf.txt.gz#117 | |
| 20:52:37 | sean-k-mooney | on the run that pased its set on both | |
| 20:52:46 | mriedem | yeah i was looking at the one that failed | |
| 20:52:53 | sean-k-mooney | oh | |
| 20:52:53 | mriedem | note that i'm coming into this 2 hours late :) | |
| 20:53:08 | sean-k-mooney | so the one that failed didnt have it set on both | |
| 20:53:13 | dansmith | right | |
| 20:53:21 | sean-k-mooney | ok | |
| 20:53:36 | sean-k-mooney | i have no idea why that is then | |
| 20:53:48 | sean-k-mooney | i can add it expclitly to the job i guess | |
| 20:54:20 | sean-k-mooney | i also dont have that set locally so that is prably why i sometimes saw this an other times did not | |
| 20:54:30 | mriedem | sean-k-mooney: different patch sets for those jobs, | |
| 20:54:32 | mriedem | not sure if that matters, | |
| 20:54:34 | mriedem | i was looking at PS1 | |
| 20:54:48 | mriedem | https://review.opendev.org/#/c/681771/1..2/.zuul.yaml | |
| 20:55:01 | mriedem | do'nt know why that would change things | |
| 20:55:09 | sean-k-mooney | nor do i | |
| 20:55:18 | sean-k-mooney | that was needed to activate stephens code | |
| 20:55:24 | mriedem | unless there is a devstack or zuul bug or something | |
| 20:55:34 | sean-k-mooney | sice cpu_dedicated_set is what does the reshape | |
| 20:56:21 | mriedem | maybe https://github.com/openstack/devstack/commit/2468ceaa724aa5c8c44fb87ae223eb6687ff85f2 regressed something in devstack | |
| 20:56:26 | sean-k-mooney | anwyway the important thing is the last run pased with both seriese merged togoether so at least that good | |
| 20:56:58 | sean-k-mooney | maybe but both of thoes runs were done today | |
| 20:57:01 | mriedem | yeah, i guess we just keep an eye out for random failures like that | |
| 20:57:30 | sean-k-mooney | ya maybe check logstash to see if its happening | |
| 20:58:25 | sean-k-mooney | right so i was going to try and create a non legacy grenade job with all the onter non merged stuff. | |
| 20:59:02 | dansmith | mriedem: we only do that if we have an instance group right? | |
| 20:59:02 | sean-k-mooney | what of the chaces 4 un merged unrelated seriese will all work togeher perficectly | |
| 20:59:12 | dansmith | does the network advanced server ops thing use groups? | |
| 20:59:27 | sean-k-mooney | i dont think so | |
| 20:59:38 | sean-k-mooney | you mean server groups right | |
| 20:59:39 | dansmith | so I'm wondering why we'd be hitting this for that | |
| 20:59:42 | dansmith | yes, server grops | |
| 20:59:48 | dansmith | groups even | |
| 21:00:18 | sean-k-mooney | do we do check for resize/migrate | |
| 21:00:27 | dansmith | only if they're in a group, IIRC | |
| 21:00:27 | sean-k-mooney | its for the anti afinity stuff right | |
| 21:00:28 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver discovering PMEM namespaces https://review.opendev.org/678453 | |
| 21:00:28 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: report VPMEM resources by provider tree https://review.opendev.org/678454 | |
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Support VM creation with vpmems and vpmems cleanup https://review.opendev.org/678455 | |
| 21:00:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | |
| 21:00:30 | openstackgerrit | Eric Fried proposed openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640 | |