Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-12
19:07:20 stephenfin oh Translating request for VCPU=2 to VCPU=0,PCPU=2
19:07:48 stephenfin yeah, sec
19:07:48 dansmith yeah
19:08:14 dansmith I saw that, forgot it was in the compute log and was like "cool, cool, we're ... wait a sec." :)
19:09:16 stephenfin so this is the code that does the translation https://review.opendev.org/#/c/671801/48/nova/scheduler/utils.py
19:09:34 stephenfin and the question is where do we build ResourceRequest object on the compute ndoe
19:09:36 stephenfin *node
19:09:43 stephenfin because I didn't add that :)
19:09:54 dansmith if we're legit logging that we should probably change that wording a little
19:10:32 stephenfin got it
19:10:45 stephenfin https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L595-L602
19:10:50 sean-k-mooney "cpu_dedicated_set: 0-6" is the correct spelling yes
19:11:02 stephenfin yup, and it has to be in the compute group
19:11:12 sean-k-mooney ya
19:12:10 stephenfin dansmith: That's from the vCPU model selection code and it was added as a quick way to extract the traits
19:12:54 openstackgerrit sean mooney proposed openstack/nova master: [DNM] numa + pcpus in placment live migration tests https://review.opendev.org/681771
19:13:04 stephenfin So agreed, it's misleading, and I didn't see it before because the vCPU model selection change only merged last night
19:13:43 dansmith s'cool, just sayin'
19:13:56 sean-k-mooney ok that re running
19:14:35 sean-k-mooney the 3 jobs that are just running the PCPU code shoudl report back in the next 20 mins or so the last one with numa will take about an hour and a half
19:14:45 stephenfin cool :)
19:15:04 stephenfin The solution, fwiw (and IMO), is what efried has asked for elsewhere: a util function to pull traits and resources from a flavor in a standard way, so we don't have a load of 'resourceNN:<type>' regexes about the place
19:15:14 stephenfin regexes? regexi?
19:15:33 dansmith sean-k-mooney: so what's the deal on the failed db queries?
19:15:50 dansmith sean-k-mooney: worried that maybe we're lazy-loading some api-only field or something
19:16:10 sean-k-mooney if they dont show up the ones that is stephens only then its from artoms series
19:16:16 sean-k-mooney if its in both its from master
19:17:24 sean-k-mooney this https://review.opendev.org/#/c/681771/1 is the combiend job and https://review.opendev.org/#/c/681807/2 is the 3 pcpu only jobs
19:17:50 sean-k-mooney i think i did see that with artoms series but not in all versions
19:18:22 sean-k-mooney could it be the late upcall for anti affintiy
19:19:02 sean-k-mooney if we are lazy loadign somethign im not sure what it would be off the top of my head
19:20:30 stephenfin sean-k-mooney, dansmith: WIP manual testing notes here too, btw https://etherpad.openstack.org/p/nova-cpu-resources
19:21:19 stephenfin I'm at controller+compute updated to use PCPU code with other compute using plain master (without PCPU code). Onto setting 'cpu_dedicated_set' on the former now
19:21:56 stephenfin I went with master because DevStack wouldn't deploy from stable/stein and I figured this would be easier simulate "upgrades" from nova's perspective
19:22:08 sean-k-mooney i ment to get food earilar and still havent so im gong to run for an 30 mins to an hour
19:22:38 stephenfin I just want to test the reshaper and create/move operations with nodes with and without the PCPU stuff, after all
19:22:56 dansmith sean-k-mooney: I dunno how it could be unrelated if it works on master (the lazy load)
19:26:20 sean-k-mooney its likely an issue in artoms code then if stephen has not seen it in his testing?
19:27:02 dansmith no, I would think it's more likely that stephen added something that looks at an object field, but hasn't been testing with computes and conductors that don't have access to the api database
19:27:39 dansmith artom's code eventually passed this right?
19:29:21 mnaser melwitt: thanks (re os-vif)
19:31:36 dansmith ...still not a damn thing in the gate
19:33:31 melwitt the Nova Penalty™
19:40:59 artom dansmith, stuff that's been +W'ed pre-FF can be rechecked until it merges, right?
19:41:06 artom It's not like "whelp, FF, killall the things"
19:42:44 dansmith ...mkay
19:43:56 stephenfin dansmith: okay, so, here's what I've tested
19:44:09 stephenfin that I can boot pinned and unpinned instances on master
19:44:33 stephenfin that if I "upgrade" a compute node to use the PCPU stuff, the existing instances keep working
19:44:48 stephenfin that I can boot new instances on that upgraded compute node
19:45:20 stephenfin that if I set 'cpu_dedicated_set', all my pinned instance magically transition to having PCPU inventory
19:46:18 stephenfin (that one was tricky because I (boldly) had pinned and unpinned instances on the same host, so I have to pick a value for 'cpu_dedicated_set' that match the pinset of the pinned instances and still leave some aside for 'cpu_shared_set')
19:47:13 stephenfin that I can continue to boot pinned instances after having set that, and they will go to both the host reporting PCPUs and the host without PCPUs
19:48:05 stephenfin and throughout all of that, placement's view of WTF is happening remains consistent
19:48:13 stephenfin *that placement's
19:49:07 stephenfin dansmith: How's that sounding so far? Anything in particular you want me to double back on or try?
19:50:07 stephenfin Happy to tar up the logs from both compute nodes for your perusal too. They're a bit messy because I've made a few (user-driven) mistakes along the way but they should do to quickly grep for ERRORs
19:50:40 stephenfin The main ones of those I've seen are errors from RabbitMQ about trying again in 1 sec, which I seem to always get in DevStack
19:51:40 stephenfin and libvirt b****** when I tried to set 'cpu_dedicated_set' to a range that excluded pinned cores from existing instances
19:52:42 stephenfin That's an issue already with vcpu_pin_set but I thought I'd fixed that with this bugger https://review.opendev.org/#/c/680107/ I've clearly missed something though and will keep investigating
19:53:15 stephenfin Also, there's a bug in the vCPU model selection code
19:54:47 stephenfin Namely, if I have a required trait and it's not a CPU flag, this will contain at least some None entries https://github.com/openstack/nova/blob/master/nova/virt/libvirt/utils.py#L600
19:55:14 stephenfin and we have a later check to see if the result from the function is false'y
19:55:17 stephenfin set([None]) is not false'y
19:55:28 stephenfin patch incoming for that
20:00:02 sean-k-mooney stephenfin: here is the results for https://review.opendev.org/#/c/681827/2 https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_5bb/681827/2/experimental/nova-nfv-multinode/5bb9889/testr_results.html.gz
20:00:25 sean-k-mooney that is your code without artoms
20:00:39 sean-k-mooney the only thing that failed were 2 live migration tests
20:01:05 stephenfin that's exactly what we expected, right?
20:01:20 sean-k-mooney there are a coupld of extra failure in https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_909/681840/2/experimental/nova-nfv-multinode/9090152/testr_results.html.gz
20:01:35 sean-k-mooney whichi is using vcpu_pin_set again
20:01:45 dansmith resize fails are not expected
20:01:59 dansmith Details: {'code': 400, 'created': '2019-09-12T18:52:42Z', 'message': 'CPU set to unpin [1] must be a subset of pinned CPU set []'}
20:02:30 sean-k-mooney you get set[] if the update resouce provider task fails
20:02:53 sean-k-mooney which happen if you live migrate and cause a pinning conflict
20:03:13 sean-k-mooney but i just got back form getting food so now im going to go eat in
20:03:15 sean-k-mooney brb
20:03:34 dansmith ah, so you think this had already caused a pinning conflict and then a resize failed or something?
20:03:41 dansmith I thought you were running these with concurrency=1
20:03:42 sean-k-mooney ya
20:03:58 sean-k-mooney yes but it takes a while for it to get fixed
20:04:09 dansmith even after the delete?
20:04:09 sean-k-mooney e.g. the perodic task time
20:04:48 sean-k-mooney it wont be fixed until the update_resouces perodic runs i think
20:05:21 dansmith that seems odd
20:11:39 dansmith I see that both in the periodic as well as during a boot
20:18:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Note about Destination.forbidden_aggregates https://review.opendev.org/680945
20:18:33 dansmith stephenfin: sorry I missed the ping above
20:19:00 dansmith stephenfin: I want to see tempest passing against it.. I'm glad you're doing lots of hand testing of it too, and I believe that whatever you're doing is working if you say it is
20:19:30 dansmith not that it matters, several other people are apparently fine +Wing this without seeing our test suite run on it, so this is just for funsies at this point
20:20:14 stephenfin I guess having a bajillion functional tests does help thing though
20:20:33 stephenfin Yes, I know they're not the same thing, but this is all touching the management'y parts of nova
20:20:40 stephenfin which the functional tests are excellent for
20:20:58 dansmith this is all touching plenty of the stuff in nova that breaks all the time
20:27:57 sean-k-mooney ok im back
20:29:00 dansmith sean-k-mooney: so what's the deal with the test run that was showing the object/db errors?
20:29:34 sean-k-mooney i dont know but thats what im going to look into now
20:29:46 sean-k-mooney did we see that in the ones for just stephens code
20:29:56 sean-k-mooney or is it only in the combined one
20:30:06 dansmith only combined, afaik

Earlier   Later