Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-12
19:20:30 stephenfin sean-k-mooney, dansmith: WIP manual testing notes here too, btw https://etherpad.openstack.org/p/nova-cpu-resources
19:21:19 stephenfin I'm at controller+compute updated to use PCPU code with other compute using plain master (without PCPU code). Onto setting 'cpu_dedicated_set' on the former now
19:21:56 stephenfin I went with master because DevStack wouldn't deploy from stable/stein and I figured this would be easier simulate "upgrades" from nova's perspective
19:22:08 sean-k-mooney i ment to get food earilar and still havent so im gong to run for an 30 mins to an hour
19:22:38 stephenfin I just want to test the reshaper and create/move operations with nodes with and without the PCPU stuff, after all
19:22:56 dansmith sean-k-mooney: I dunno how it could be unrelated if it works on master (the lazy load)
19:26:20 sean-k-mooney its likely an issue in artoms code then if stephen has not seen it in his testing?
19:27:02 dansmith no, I would think it's more likely that stephen added something that looks at an object field, but hasn't been testing with computes and conductors that don't have access to the api database
19:27:39 dansmith artom's code eventually passed this right?
19:29:21 mnaser melwitt: thanks (re os-vif)
19:31:36 dansmith ...still not a damn thing in the gate
19:33:31 melwitt the Nova Penalty™
19:40:59 artom dansmith, stuff that's been +W'ed pre-FF can be rechecked until it merges, right?
19:41:06 artom It's not like "whelp, FF, killall the things"
19:42:44 dansmith ...mkay
19:43:56 stephenfin dansmith: okay, so, here's what I've tested
19:44:09 stephenfin that I can boot pinned and unpinned instances on master
19:44:33 stephenfin that if I "upgrade" a compute node to use the PCPU stuff, the existing instances keep working
19:44:48 stephenfin that I can boot new instances on that upgraded compute node
19:45:20 stephenfin that if I set 'cpu_dedicated_set', all my pinned instance magically transition to having PCPU inventory
19:46:18 stephenfin (that one was tricky because I (boldly) had pinned and unpinned instances on the same host, so I have to pick a value for 'cpu_dedicated_set' that match the pinset of the pinned instances and still leave some aside for 'cpu_shared_set')
19:47:13 stephenfin that I can continue to boot pinned instances after having set that, and they will go to both the host reporting PCPUs and the host without PCPUs
19:48:05 stephenfin and throughout all of that, placement's view of WTF is happening remains consistent
19:48:13 stephenfin *that placement's
19:49:07 stephenfin dansmith: How's that sounding so far? Anything in particular you want me to double back on or try?
19:50:07 stephenfin Happy to tar up the logs from both compute nodes for your perusal too. They're a bit messy because I've made a few (user-driven) mistakes along the way but they should do to quickly grep for ERRORs
19:50:40 stephenfin The main ones of those I've seen are errors from RabbitMQ about trying again in 1 sec, which I seem to always get in DevStack
19:51:40 stephenfin and libvirt b****** when I tried to set 'cpu_dedicated_set' to a range that excluded pinned cores from existing instances
19:52:42 stephenfin That's an issue already with vcpu_pin_set but I thought I'd fixed that with this bugger https://review.opendev.org/#/c/680107/ I've clearly missed something though and will keep investigating
19:53:15 stephenfin Also, there's a bug in the vCPU model selection code
19:54:47 stephenfin Namely, if I have a required trait and it's not a CPU flag, this will contain at least some None entries https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L600
19:55:14 stephenfin and we have a later check to see if the result from the function is false'y
19:55:17 stephenfin set([None]) is not false'y
19:55:28 stephenfin patch incoming for that
20:00:02 sean-k-mooney stephenfin: here is the results for https://review.opendev.org/#/c/681827/2 https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_5bb/681827/2/experimental/nova-nfv-multinode/5bb9889/testr_results.html.gz
20:00:25 sean-k-mooney that is your code without artoms
20:00:39 sean-k-mooney the only thing that failed were 2 live migration tests
20:01:05 stephenfin that's exactly what we expected, right?
20:01:20 sean-k-mooney there are a coupld of extra failure in https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_909/681840/2/experimental/nova-nfv-multinode/9090152/testr_results.html.gz
20:01:35 sean-k-mooney whichi is using vcpu_pin_set again
20:01:45 dansmith resize fails are not expected
20:01:59 dansmith Details: {'code': 400, 'created': '2019-09-12T18:52:42Z', 'message': 'CPU set to unpin [1] must be a subset of pinned CPU set []'}
20:02:30 sean-k-mooney you get set[] if the update resouce provider task fails
20:02:53 sean-k-mooney which happen if you live migrate and cause a pinning conflict
20:03:13 sean-k-mooney but i just got back form getting food so now im going to go eat in
20:03:15 sean-k-mooney brb
20:03:34 dansmith ah, so you think this had already caused a pinning conflict and then a resize failed or something?
20:03:41 dansmith I thought you were running these with concurrency=1
20:03:42 sean-k-mooney ya
20:03:58 sean-k-mooney yes but it takes a while for it to get fixed
20:04:09 sean-k-mooney e.g. the perodic task time
20:04:09 dansmith even after the delete?
20:04:48 sean-k-mooney it wont be fixed until the update_resouces perodic runs i think
20:05:21 dansmith that seems odd
20:11:39 dansmith I see that both in the periodic as well as during a boot
20:18:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Note about Destination.forbidden_aggregates https://review.opendev.org/680945
20:18:33 dansmith stephenfin: sorry I missed the ping above
20:19:00 dansmith stephenfin: I want to see tempest passing against it.. I'm glad you're doing lots of hand testing of it too, and I believe that whatever you're doing is working if you say it is
20:19:30 dansmith not that it matters, several other people are apparently fine +Wing this without seeing our test suite run on it, so this is just for funsies at this point
20:20:14 stephenfin I guess having a bajillion functional tests does help thing though
20:20:33 stephenfin Yes, I know they're not the same thing, but this is all touching the management'y parts of nova
20:20:40 stephenfin which the functional tests are excellent for
20:20:58 dansmith this is all touching plenty of the stuff in nova that breaks all the time
20:27:57 sean-k-mooney ok im back
20:29:00 dansmith sean-k-mooney: so what's the deal with the test run that was showing the object/db errors?
20:29:34 sean-k-mooney i dont know but thats what im going to look into now
20:29:46 sean-k-mooney did we see that in the ones for just stephens code
20:29:56 sean-k-mooney or is it only in the combined one
20:30:06 dansmith only combined, afaik
20:30:23 sean-k-mooney ok ill redeploy just artoms code and see if its there
20:30:59 dansmith sean-k-mooney: it would have caused the tests to fail when we originally run on artom's code if it was right?
20:31:04 sean-k-mooney i wonder if it could be releated to the instance.refresh()
20:31:13 dansmith no, I think it was on reqspec
20:31:23 sean-k-mooney am im not sure it would
20:31:38 sean-k-mooney when i saw that previoulsy the migrtaions worked
20:31:39 dansmith sean-k-mooney: I asked earlier why not and what is different
20:32:04 stephenfin Do we know if that instance was moved before attempting to rebuild? I'm assuming not
20:32:08 dansmith these were failing in the pcpu tests as a result no?
20:32:10 efried stephenfin: thanks for those reviews. There's going to be a merge conflict with numa lm on test_objects, so I'm going to rebase now to resolve preemptively, cool?
20:33:02 stephenfin efried: yeah, go for it. I've been meaning to do that myself but got sidetracked
20:33:17 stephenfin Lemme know what ones I need to rehit if the merge conflicts are non-trivial
20:33:43 stephenfin efried: Aaaactually, do you think you could rebase the cpu-resources series onto that too?
20:34:10 efried stephenfin: Yeah, I can do that, but in two stages if that's okay with you.
20:34:17 stephenfin I pointed out earlier that I think that's going to have the same issue
20:34:19 stephenfin cool by me
20:34:20 dansmith sean-k-mooney: File "/opt/stack/nova/nova/objects/instance_group.py"
20:34:48 dansmith sean-k-mooney: so maybe it is the late affinity check, but.. I don't understand why we'd hit that here but not in regular/other tests.. can you explain?
20:35:14 sean-k-mooney dansmith: we turn it off in the upstream gate but i might not be truning it off in my jobs
20:35:33 dansmith ah okay
20:35:39 sean-k-mooney it shoudl be in the nova-cpu.conf right
20:35:43 sean-k-mooney the workarounds section
20:36:33 dansmith oh you mean we turn off the check, not disable whatever test this is?
20:36:48 dansmith I dunno where it is specifically but nova-cpu.conf yeah
20:36:48 sean-k-mooney yes we disabel the upcall
20:36:50 sean-k-mooney also https://8ab1fb0a384d9cbfa221-969de3c017bb40c2acaf4bf21edd2ff6.ssl.cf1.rackcdn.com/681771/2/experimental/nova-nfv-multi-numa-multinode/1faf214/testr_results.html.gz
20:36:56 sean-k-mooney https://review.opendev.org/#/c/681771/
20:37:28 dansmith do we have any grenade results yet?
20:37:59 sean-k-mooney no i dont have a grenade job set up. but i can try to set one up quickly
20:38:30 dansmith I don't think quickly matters anymore,
20:38:36 dansmith but it'd be good to get a run on it while we have context, IMHO

Earlier   Later