| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 20:01:45 | dansmith | resize fails are not expected | |
| 20:01:59 | dansmith | Details: {'code': 400, 'created': '2019-09-12T18:52:42Z', 'message': 'CPU set to unpin [1] must be a subset of pinned CPU set []'} | |
| 20:02:30 | sean-k-mooney | you get set[] if the update resouce provider task fails | |
| 20:02:53 | sean-k-mooney | which happen if you live migrate and cause a pinning conflict | |
| 20:03:13 | sean-k-mooney | but i just got back form getting food so now im going to go eat in | |
| 20:03:15 | sean-k-mooney | brb | |
| 20:03:34 | dansmith | ah, so you think this had already caused a pinning conflict and then a resize failed or something? | |
| 20:03:41 | dansmith | I thought you were running these with concurrency=1 | |
| 20:03:42 | sean-k-mooney | ya | |
| 20:03:58 | sean-k-mooney | yes but it takes a while for it to get fixed | |
| 20:04:09 | sean-k-mooney | e.g. the perodic task time | |
| 20:04:09 | dansmith | even after the delete? | |
| 20:04:48 | sean-k-mooney | it wont be fixed until the update_resouces perodic runs i think | |
| 20:05:21 | dansmith | that seems odd | |
| 20:11:39 | dansmith | I see that both in the periodic as well as during a boot | |
| 20:18:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Note about Destination.forbidden_aggregates https://review.opendev.org/680945 | |
| 20:18:33 | dansmith | stephenfin: sorry I missed the ping above | |
| 20:19:00 | dansmith | stephenfin: I want to see tempest passing against it.. I'm glad you're doing lots of hand testing of it too, and I believe that whatever you're doing is working if you say it is | |
| 20:19:30 | dansmith | not that it matters, several other people are apparently fine +Wing this without seeing our test suite run on it, so this is just for funsies at this point | |
| 20:20:14 | stephenfin | I guess having a bajillion functional tests does help thing though | |
| 20:20:33 | stephenfin | Yes, I know they're not the same thing, but this is all touching the management'y parts of nova | |
| 20:20:40 | stephenfin | which the functional tests are excellent for | |
| 20:20:58 | dansmith | this is all touching plenty of the stuff in nova that breaks all the time | |
| 20:27:57 | sean-k-mooney | ok im back | |
| 20:29:00 | dansmith | sean-k-mooney: so what's the deal with the test run that was showing the object/db errors? | |
| 20:29:34 | sean-k-mooney | i dont know but thats what im going to look into now | |
| 20:29:46 | sean-k-mooney | did we see that in the ones for just stephens code | |
| 20:29:56 | sean-k-mooney | or is it only in the combined one | |
| 20:30:06 | dansmith | only combined, afaik | |
| 20:30:23 | sean-k-mooney | ok ill redeploy just artoms code and see if its there | |
| 20:30:59 | dansmith | sean-k-mooney: it would have caused the tests to fail when we originally run on artom's code if it was right? | |
| 20:31:04 | sean-k-mooney | i wonder if it could be releated to the instance.refresh() | |
| 20:31:13 | dansmith | no, I think it was on reqspec | |
| 20:31:23 | sean-k-mooney | am im not sure it would | |
| 20:31:38 | sean-k-mooney | when i saw that previoulsy the migrtaions worked | |
| 20:31:39 | dansmith | sean-k-mooney: I asked earlier why not and what is different | |
| 20:32:04 | stephenfin | Do we know if that instance was moved before attempting to rebuild? I'm assuming not | |
| 20:32:08 | dansmith | these were failing in the pcpu tests as a result no? | |
| 20:32:10 | efried | stephenfin: thanks for those reviews. There's going to be a merge conflict with numa lm on test_objects, so I'm going to rebase now to resolve preemptively, cool? | |
| 20:33:02 | stephenfin | efried: yeah, go for it. I've been meaning to do that myself but got sidetracked | |
| 20:33:17 | stephenfin | Lemme know what ones I need to rehit if the merge conflicts are non-trivial | |
| 20:33:43 | stephenfin | efried: Aaaactually, do you think you could rebase the cpu-resources series onto that too? | |
| 20:34:10 | efried | stephenfin: Yeah, I can do that, but in two stages if that's okay with you. | |
| 20:34:17 | stephenfin | I pointed out earlier that I think that's going to have the same issue | |
| 20:34:19 | stephenfin | cool by me | |
| 20:34:20 | dansmith | sean-k-mooney: File "/opt/stack/nova/nova/objects/instance_group.py" | |
| 20:34:48 | dansmith | sean-k-mooney: so maybe it is the late affinity check, but.. I don't understand why we'd hit that here but not in regular/other tests.. can you explain? | |
| 20:35:14 | sean-k-mooney | dansmith: we turn it off in the upstream gate but i might not be truning it off in my jobs | |
| 20:35:33 | dansmith | ah okay | |
| 20:35:39 | sean-k-mooney | it shoudl be in the nova-cpu.conf right | |
| 20:35:43 | sean-k-mooney | the workarounds section | |
| 20:36:33 | dansmith | oh you mean we turn off the check, not disable whatever test this is? | |
| 20:36:48 | dansmith | I dunno where it is specifically but nova-cpu.conf yeah | |
| 20:36:48 | sean-k-mooney | yes we disabel the upcall | |
| 20:36:50 | sean-k-mooney | also https://8ab1fb0a384d9cbfa221-969de3c017bb40c2acaf4bf21edd2ff6.ssl.cf1.rackcdn.com/681771/2/experimental/nova-nfv-multi-numa-multinode/1faf214/testr_results.html.gz | |
| 20:36:56 | sean-k-mooney | https://review.opendev.org/#/c/681771/ | |
| 20:37:28 | dansmith | do we have any grenade results yet? | |
| 20:37:59 | sean-k-mooney | no i dont have a grenade job set up. but i can try to set one up quickly | |
| 20:38:30 | dansmith | I don't think quickly matters anymore, | |
| 20:38:36 | dansmith | but it'd be good to get a run on it while we have context, IMHO | |
| 20:39:21 | sean-k-mooney | so there is a patch to convert grenade to non legacy | |
| 20:39:37 | sean-k-mooney | if i depend on that i should be able to do the same config changes | |
| 20:39:58 | dansmith | oh, is that why it's hard because it's still a legacy job? | |
| 20:40:02 | sean-k-mooney | dansmith: this https://review.opendev.org/#/c/548936/ | |
| 20:40:07 | sean-k-mooney | dansmith: yes | |
| 20:40:10 | dansmith | gotcha | |
| 20:40:18 | dansmith | well, if that works currently, then yeah, try that I'd say | |
| 20:40:22 | sean-k-mooney | devstack-gate is a pain to get kvm working | |
| 20:40:38 | sean-k-mooney | yes it looks like its passing so ill try and modify it | |
| 20:40:45 | dansmith | cool | |
| 20:40:45 | sean-k-mooney | well depend on it | |
| 20:41:08 | mriedem | so...things are where they were 2 hours ago? | |
| 20:41:18 | mriedem | ooo down to 179 changes in the check queue | |
| 20:41:43 | dansmith | still nothing in the gate tho | |
| 20:42:03 | mriedem | yup, i don't expect nova stuff to probably be fully merged until maybe this weekend | |
| 20:42:08 | mriedem | if you account for rechecks | |
| 20:42:25 | sean-k-mooney | dansmith: by the way the latest test run for the combiniend numa + pcpus job does not appear to have the db error | |
| 20:42:48 | dansmith | sean-k-mooney: that db error wouldn't be flaky, so that makes me kinda wonder | |
| 20:43:16 | sean-k-mooney | well the difference is i am using cpu_dedicated_set instead of vcpu_pin_set | |
| 20:43:21 | dansmith | sean-k-mooney: unless one node is configured to have api db access and the other is not, and you don't get the fail if you're lucky and land on the one | |
| 20:45:11 | sean-k-mooney | stephenfin: could we be trying to lazy load the pinned cpus or something when we use vcpu_pin_set instead of cpu_dedicated_Set | |
| 20:45:44 | mriedem | devstack shouldn't configure nova-cpu.conf nor either of the cell conductor configs for api db access | |
| 20:45:46 | mriedem | by default anyway | |
| 20:46:22 | dansmith | mriedem: yeah I dunno why this would be legit failing here and not on master tho | |
| 20:46:50 | sean-k-mooney | yeah the db is not configured on either node | |
| 20:46:51 | mriedem | talking about this job? https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_909/681840/2/experimental/nova-nfv-multinode/9090152/testr_results.html.gz | |
| 20:47:07 | stephenfin | sean-k-mooney: I'm looking but I don't see why it would. | |
| 20:47:08 | dansmith | no | |
| 20:47:15 | dansmith | mriedem: https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/screen-n-cpu.txt.gz | |
| 20:47:17 | efried | stephenfin: that rebase isn't going to work, because numalm is based waaay back on master, so that way conflicts with (at least) cpu models. I could rebase numalm, but it would lose its place in the queue (~10.5h) so I don't want to do that. Just going to ride it out, mkay? | |
| 20:47:38 | dansmith | mriedem: but I think this is just the late affinity check that must not be disabled on sean-k-mooney's | |
| 20:47:42 | stephenfin | (y) | |
| 20:48:10 | sean-k-mooney | this one https://zuul.opendev.org/t/openstack/build/1faf2148524e41638672ff627cb9a011/log/compute/logs/etc/nova/nova-cpu_conf.txt.gz#92 | |
| 20:48:25 | mriedem | disable_group_policy_check_upcall | |
| 20:48:32 | dansmith | sean-k-mooney: that's set on your job? | |
| 20:48:39 | mriedem | it's not in https://20aa241df59e72844ae9-597ff148d0ea9164d11e7cb764cf9b04.ssl.cf5.rackcdn.com/681771/1/experimental/nova-nfv-multi-numa-multinode/c1f6ccc/compute/logs/etc/nova/nova_conf.txt.gz | |
| 20:48:41 | mriedem | in the subnode | |
| 20:48:44 | sean-k-mooney | yes | |
| 20:48:55 | dansmith | okay not being on the subnode would explain why we hit it sometimes and not others | |
| 20:48:56 | mriedem | devstack should configure that by default if you're using superconductor mode (which is the default) | |