| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-12 | |||
| 19:46:18 | stephenfin | (that one was tricky because I (boldly) had pinned and unpinned instances on the same host, so I have to pick a value for 'cpu_dedicated_set' that match the pinset of the pinned instances and still leave some aside for 'cpu_shared_set') | |
| 19:47:13 | stephenfin | that I can continue to boot pinned instances after having set that, and they will go to both the host reporting PCPUs and the host without PCPUs | |
| 19:48:05 | stephenfin | and throughout all of that, placement's view of WTF is happening remains consistent | |
| 19:48:13 | stephenfin | *that placement's | |
| 19:49:07 | stephenfin | dansmith: How's that sounding so far? Anything in particular you want me to double back on or try? | |
| 19:50:07 | stephenfin | Happy to tar up the logs from both compute nodes for your perusal too. They're a bit messy because I've made a few (user-driven) mistakes along the way but they should do to quickly grep for ERRORs | |
| 19:50:40 | stephenfin | The main ones of those I've seen are errors from RabbitMQ about trying again in 1 sec, which I seem to always get in DevStack | |
| 19:51:40 | stephenfin | and libvirt b****** when I tried to set 'cpu_dedicated_set' to a range that excluded pinned cores from existing instances | |
| 19:52:42 | stephenfin | That's an issue already with vcpu_pin_set but I thought I'd fixed that with this bugger https://review.opendev.org/#/c/680107/ I've clearly missed something though and will keep investigating | |
| 19:53:15 | stephenfin | Also, there's a bug in the vCPU model selection code | |
| 19:54:47 | stephenfin | Namely, if I have a required trait and it's not a CPU flag, this will contain at least some None entries https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L600 | |
| 19:55:14 | stephenfin | and we have a later check to see if the result from the function is false'y | |
| 19:55:17 | stephenfin | set([None]) is not false'y | |
| 19:55:28 | stephenfin | patch incoming for that | |
| 20:00:02 | sean-k-mooney | stephenfin: here is the results for https://review.opendev.org/#/c/681827/2 https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_5bb/681827/2/experimental/nova-nfv-multinode/5bb9889/testr_results.html.gz | |
| 20:00:25 | sean-k-mooney | that is your code without artoms | |
| 20:00:39 | sean-k-mooney | the only thing that failed were 2 live migration tests | |
| 20:01:05 | stephenfin | that's exactly what we expected, right? | |
| 20:01:20 | sean-k-mooney | there are a coupld of extra failure in https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_909/681840/2/experimental/nova-nfv-multinode/9090152/testr_results.html.gz | |
| 20:01:35 | sean-k-mooney | whichi is using vcpu_pin_set again | |
| 20:01:45 | dansmith | resize fails are not expected | |
| 20:01:59 | dansmith | Details: {'code': 400, 'created': '2019-09-12T18:52:42Z', 'message': 'CPU set to unpin [1] must be a subset of pinned CPU set []'} | |
| 20:02:30 | sean-k-mooney | you get set[] if the update resouce provider task fails | |
| 20:02:53 | sean-k-mooney | which happen if you live migrate and cause a pinning conflict | |
| 20:03:13 | sean-k-mooney | but i just got back form getting food so now im going to go eat in | |
| 20:03:15 | sean-k-mooney | brb | |
| 20:03:34 | dansmith | ah, so you think this had already caused a pinning conflict and then a resize failed or something? | |
| 20:03:41 | dansmith | I thought you were running these with concurrency=1 | |
| 20:03:42 | sean-k-mooney | ya | |
| 20:03:58 | sean-k-mooney | yes but it takes a while for it to get fixed | |
| 20:04:09 | sean-k-mooney | e.g. the perodic task time | |
| 20:04:09 | dansmith | even after the delete? | |
| 20:04:48 | sean-k-mooney | it wont be fixed until the update_resouces perodic runs i think | |
| 20:05:21 | dansmith | that seems odd | |
| 20:11:39 | dansmith | I see that both in the periodic as well as during a boot | |
| 20:18:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Note about Destination.forbidden_aggregates https://review.opendev.org/680945 | |
| 20:18:33 | dansmith | stephenfin: sorry I missed the ping above | |
| 20:19:00 | dansmith | stephenfin: I want to see tempest passing against it.. I'm glad you're doing lots of hand testing of it too, and I believe that whatever you're doing is working if you say it is | |
| 20:19:30 | dansmith | not that it matters, several other people are apparently fine +Wing this without seeing our test suite run on it, so this is just for funsies at this point | |
| 20:20:14 | stephenfin | I guess having a bajillion functional tests does help thing though | |
| 20:20:33 | stephenfin | Yes, I know they're not the same thing, but this is all touching the management'y parts of nova | |
| 20:20:40 | stephenfin | which the functional tests are excellent for | |
| 20:20:58 | dansmith | this is all touching plenty of the stuff in nova that breaks all the time | |
| 20:27:57 | sean-k-mooney | ok im back | |
| 20:29:00 | dansmith | sean-k-mooney: so what's the deal with the test run that was showing the object/db errors? | |
| 20:29:34 | sean-k-mooney | i dont know but thats what im going to look into now | |
| 20:29:46 | sean-k-mooney | did we see that in the ones for just stephens code | |
| 20:29:56 | sean-k-mooney | or is it only in the combined one | |
| 20:30:06 | dansmith | only combined, afaik | |
| 20:30:23 | sean-k-mooney | ok ill redeploy just artoms code and see if its there | |
| 20:30:59 | dansmith | sean-k-mooney: it would have caused the tests to fail when we originally run on artom's code if it was right? | |
| 20:31:04 | sean-k-mooney | i wonder if it could be releated to the instance.refresh() | |
| 20:31:13 | dansmith | no, I think it was on reqspec | |
| 20:31:23 | sean-k-mooney | am im not sure it would | |
| 20:31:38 | sean-k-mooney | when i saw that previoulsy the migrtaions worked | |
| 20:31:39 | dansmith | sean-k-mooney: I asked earlier why not and what is different | |
| 20:32:04 | stephenfin | Do we know if that instance was moved before attempting to rebuild? I'm assuming not | |
| 20:32:08 | dansmith | these were failing in the pcpu tests as a result no? | |
| 20:32:10 | efried | stephenfin: thanks for those reviews. There's going to be a merge conflict with numa lm on test_objects, so I'm going to rebase now to resolve preemptively, cool? | |
| 20:33:02 | stephenfin | efried: yeah, go for it. I've been meaning to do that myself but got sidetracked | |
| 20:33:17 | stephenfin | Lemme know what ones I need to rehit if the merge conflicts are non-trivial | |
| 20:33:43 | stephenfin | efried: Aaaactually, do you think you could rebase the cpu-resources series onto that too? | |
| 20:34:10 | efried | stephenfin: Yeah, I can do that, but in two stages if that's okay with you. | |
| 20:34:17 | stephenfin | I pointed out earlier that I think that's going to have the same issue | |
| 20:34:19 | stephenfin | cool by me | |
| 20:34:20 | dansmith | sean-k-mooney: File "/opt/stack/nova/nova/objects/instance_group.py" | |
| 20:34:48 | dansmith | sean-k-mooney: so maybe it is the late affinity check, but.. I don't understand why we'd hit that here but not in regular/other tests.. can you explain? | |
| 20:35:14 | sean-k-mooney | dansmith: we turn it off in the upstream gate but i might not be truning it off in my jobs | |
| 20:35:33 | dansmith | ah okay | |
| 20:35:39 | sean-k-mooney | it shoudl be in the nova-cpu.conf right | |
| 20:35:43 | sean-k-mooney | the workarounds section | |
| 20:36:33 | dansmith | oh you mean we turn off the check, not disable whatever test this is? | |
| 20:36:48 | dansmith | I dunno where it is specifically but nova-cpu.conf yeah | |
| 20:36:48 | sean-k-mooney | yes we disabel the upcall | |
| 20:36:50 | sean-k-mooney | also https://8ab1fb0a384d9cbfa221-969de3c017bb40c2acaf4bf21edd2ff6.ssl.cf1.rackcdn.com/681771/2/experimental/nova-nfv-multi-numa-multinode/1faf214/testr_results.html.gz | |
| 20:36:56 | sean-k-mooney | https://review.opendev.org/#/c/681771/ | |
| 20:37:28 | dansmith | do we have any grenade results yet? | |
| 20:37:59 | sean-k-mooney | no i dont have a grenade job set up. but i can try to set one up quickly | |
| 20:38:30 | dansmith | I don't think quickly matters anymore, | |
| 20:38:36 | dansmith | but it'd be good to get a run on it while we have context, IMHO | |
| 20:39:21 | sean-k-mooney | so there is a patch to convert grenade to non legacy | |
| 20:39:37 | sean-k-mooney | if i depend on that i should be able to do the same config changes | |
| 20:39:58 | dansmith | oh, is that why it's hard because it's still a legacy job? | |
| 20:40:02 | sean-k-mooney | dansmith: this https://review.opendev.org/#/c/548936/ | |
| 20:40:07 | sean-k-mooney | dansmith: yes | |
| 20:40:10 | dansmith | gotcha | |
| 20:40:18 | dansmith | well, if that works currently, then yeah, try that I'd say | |
| 20:40:22 | sean-k-mooney | devstack-gate is a pain to get kvm working | |
| 20:40:38 | sean-k-mooney | yes it looks like its passing so ill try and modify it | |
| 20:40:45 | dansmith | cool | |
| 20:40:45 | sean-k-mooney | well depend on it | |
| 20:41:08 | mriedem | so...things are where they were 2 hours ago? | |
| 20:41:18 | mriedem | ooo down to 179 changes in the check queue | |
| 20:41:43 | dansmith | still nothing in the gate tho | |
| 20:42:03 | mriedem | yup, i don't expect nova stuff to probably be fully merged until maybe this weekend | |
| 20:42:08 | mriedem | if you account for rechecks | |
| 20:42:25 | sean-k-mooney | dansmith: by the way the latest test run for the combiniend numa + pcpus job does not appear to have the db error | |
| 20:42:48 | dansmith | sean-k-mooney: that db error wouldn't be flaky, so that makes me kinda wonder | |
| 20:43:16 | sean-k-mooney | well the difference is i am using cpu_dedicated_set instead of vcpu_pin_set | |
| 20:43:21 | dansmith | sean-k-mooney: unless one node is configured to have api db access and the other is not, and you don't get the fail if you're lucky and land on the one | |