| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 18:55:48 | melwitt | thanks for chiming in, btw :) | |
| 18:55:53 | kmalloc | sean-k-mooney: yeah i'm not sure how a microversion affects policy enforcement -- it becomes weird. | |
| 18:55:58 | jaypipes | kmalloc: I struggle to have confidence in the firmness of the ideas after seeing keystone v3 domains, federation, PKI and other things go back and forth. | |
| 18:56:29 | kmalloc | jaypipes: well, please chime in on the scope types stuff. | |
| 18:56:37 | mlavalle | melwitt, rybridges: Neutron adds the hosts to the aggregate: https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L223 | |
| 18:57:04 | sean-k-mooney | mlavalle: that is for routed networks correct | |
| 18:57:21 | mlavalle | sean-k-mooney: yes it is | |
| 18:57:56 | kmalloc | jaypipes: there are a number of things we have not changed once other services lean on it. [also not sure what you mean firmness of federation, that's stayed the same and been built on, it hasn't been taken out] | |
| 18:58:16 | melwitt | mlavalle: a-ha, okay. so rybridges is seeing behavior where no hosts get added to the aggregate | |
| 18:58:18 | kmalloc | jaypipes: but i'd rather have more eyes/comments than less on the scope-types work. | |
| 18:58:28 | rybridges | Hey mlavalle! Thanks so much for the reply. We just checked the segmenthostmappings table in the DB and we see that the host mappings are in the table and our compute hosts are listed. But they are not added in the outpute of aggregate show | |
| 18:58:56 | kmalloc | sean-k-mooney: i'll have to see how the policy stuff shakes out with microversions, i'm sure it'll be 100% ok, just not clicking right this moment. | |
| 18:59:14 | kmalloc | sean-k-mooney: which means, i need to poke at things more. hehe :) | |
| 18:59:58 | kmalloc | sean-k-mooney: thanks for your time! | |
| 19:00:25 | cfriesen | jaypipes: cpu thread isolation is actually a valid issue for resource accounting, because the amount of resources consumed is variable depending on whether the host has enabled HT or not. | |
| 19:00:53 | sean-k-mooney | kmalloc: ya no worres i just didnot notice any microversion bumps in https://review.openstack.org/#/c/525772/4 and sice it has 1+2 already i taught i would ask since it proably should not merge until that is checked | |
| 19:01:09 | jaypipes | cfriesen: I don't doubt that. I'm saying that the cpu_threads_policy has nothing to do with resource accounting. | |
| 19:01:23 | kmalloc | sean-k-mooney: yeah, it might also not require a microversion, because it doesn't actually change anything yet, the change might need a microversion. | |
| 19:01:41 | cfriesen | jaypipes: but it does, since it determines whether we need to allocate additional resources on some compute nodes | |
| 19:01:58 | kmalloc | sean-k-mooney: i'll make sure to get some time to ensure we aren't introducing any breaking changes there (i don't think we are) that would require microversion bumps and if we are... make sure it's added | |
| 19:02:01 | sean-k-mooney | kmalloc: oh adding the scope_types=['project'] does not change the behavior | |
| 19:02:11 | kmalloc | sean-k-mooney: yeah it shouldn't. | |
| 19:02:18 | jaypipes | cfriesen: whether the host has HT enable impacts the reporting of CPU inventory for the host (or its NUMA nodes), but cpu_threads_policy only affects which host processors the guest is willing to be pinned to. | |
| 19:02:48 | jaypipes | cfriesen: are you referring to cpu_allocation_policy? | |
| 19:02:56 | jaypipes | or cpu_threads_policy? | |
| 19:02:57 | kmalloc | sean-k-mooney: yeah it's meant to just allow for richer policy allowing for defining the scope-types in policy as well | |
| 19:03:04 | cfriesen | jaypipes: no. If I ask for 1 PCPU and set cpu_thread_policy=ISOLATE then it will actually consume two host CPUs (the main one and the HT sibling) | |
| 19:03:25 | kmalloc | sean-k-mooney: but that should just be allowing for more specific policy enforcement, not actually making a change to behavior. | |
| 19:04:20 | kmalloc | sean-k-mooney: that is why i was feeling confused, but you know, i am 100% willing to assume I was mis-understanding something since it's in nova's code base and i don't spend as much time there. | |
| 19:04:22 | sean-k-mooney | kmalloc: ok cool sound like you have it well in hand in anycase. just taught i would ask for my own knollage i avoid api changes unless i have no choice | |
| 19:04:35 | kmalloc | sean-k-mooney: cool. and again ty very much! | |
| 19:04:57 | cfriesen | jaypipes: basically by enabling cpu_thread_policy=ISOLATE I'm asking for the entire host core, which maps to two ht siblings. But each of those siblings was exposed to placement as a PCPU. | |
| 19:05:24 | jaypipes | cfriesen: it doesn't always map to 2 HTs. | |
| 19:05:37 | cfriesen | jaypipes: true, it could be more | |
| 19:05:38 | jaypipes | cfriesen: yet more hardware vendor undefined randomness. | |
| 19:05:40 | sean-k-mooney | jaypipes: unfrotuently yes | |
| 19:05:49 | cfriesen | jaypipes: or it could be 1, if HT is disabled on the host | |
| 19:06:01 | jaypipes | cfriesen: no, that is the host inventory of things. | |
| 19:06:03 | sean-k-mooney | or 8 HT if its powerpc | |
| 19:06:13 | jaypipes | cfriesen: again, I understand the host inventory part of this. | |
| 19:06:35 | jaypipes | cfriesen: my problem is with flavors that consume different amounts of resources on different hosts. | |
| 19:06:59 | cfriesen | yep, that's exactly what this does, since it depends on the host config | |
| 19:07:06 | sean-k-mooney | jaypipes: ya thats what happens for cpu_thread_policy=ISOLATE today | |
| 19:07:27 | jaypipes | cfriesen: which, due to the design of cpu_threads_policy, being a string of "isolate|prefer|share" is entirely impossible to predict an integer amaount of some CPU resources that will *actually* be consumed by the guest. | |
| 19:08:07 | sean-k-mooney | jaypipes: yes. this was not an issue in icehouse but its biting us now | |
| 19:08:23 | mlavalle | rybridges, melwitt: Look at slide 28 in https://www.slideshare.net/MiguelLavalle/routed-networks-sydney. That shows you what you should see in Placement. For each segment in a routed network, that is the Placement structure that you should see | |
| 19:08:28 | cfriesen | jaypipes: the original goal was to improve flexibility by enabling hyperthreads, while allowing instances to ask for whole cores if they need it for performance. | |
| 19:08:31 | tblakes | mriedem: For bug https://bugs.launchpad.net/nova/+bug/1756360, it looks like we're going to need to implement __repr__ for NovaExceptions. Do you have any input on the format we want to return? | |
| 19:08:33 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:09:30 | sean-k-mooney | jaypipes: cfriesen ya the original intel proposal made pinning a host config option with no flavor extra specs. then you would just use aggregate to make teh desision | |
| 19:09:31 | cfriesen | jaypipes: actually, it *is* possible to predict what will be consumed by the guest given the host information | |
| 19:09:58 | cfriesen | jaypipes: it's just that it could be different from one compute node to another | |
| 19:09:59 | mlavalle | rybridges, melwitt: a routed network is a network where segments are associated to its subnets. In other words, if you do a GET of those subnets, all of them should have a valua in their 'sehment_id' attribute | |
| 19:10:10 | sean-k-mooney | cfriesen: not before the placement allocate_candidates request | |
| 19:10:26 | mlavalle | rybridges, melwitt: 'segment_id'^^^^ | |
| 19:10:44 | cfriesen | sean-k-mooney: correct, unless we wanted to model siblings in placement. :) | |
| 19:11:10 | mriedem | tblakes: can you reply to gibi's question about reproducing this in comment 1 | |
| 19:11:20 | sean-k-mooney | cfriesen: lets not unless its a trait | |
| 19:11:29 | melwitt | rybridges: did you verify whether the nova_api.aggregate_hosts table has anything in it? in case it's some kind of scoping issue with doing the 'openstack aggregate show' not showing it? | |
| 19:11:45 | sean-k-mooney | event then its a state ful trait which is kind of a bad thing | |
| 19:11:46 | cfriesen | sean-k-mooney: actually, a trait of "number of HT siblings" might make sense | |
| 19:11:49 | openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | |
| 19:12:09 | sean-k-mooney | cfriesen: it could but its missueing tratits | |
| 19:12:17 | sean-k-mooney | and it changes the request | |
| 19:13:29 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 19:13:55 | sean-k-mooney | i guess you could tag the PCPU RP with HW_1_HT and HW_2_HT if HT was on | |
| 19:14:34 | sean-k-mooney | then maybe combine that with forbiden tratis to avoid HT hosts | |
| 19:14:47 | sdeath | Q: upgrade from Ocata->Pike; now services are showing up as duplicates (same ID, UUID, etc). Any ideas as to what might be causing it? | |
| 19:15:05 | sdeath | nova service-list, openstack compute servicec list | |
| 19:15:20 | sdeath | (asked last couple days running on #openstack) | |
| 19:15:22 | sean-k-mooney | its a strech jay im assuming you would not like tratis for HT amount | |
| 19:15:46 | sdeath | interesting development today: evidently I can't delete them, bombs out, "Service id X refers to multiple services" (although the UUID is the same) | |
| 19:16:11 | sean-k-mooney | melwitt: ^ is the db race you were debugging | |
| 19:16:26 | cfriesen | how bad would it be if the initial pre-check didn't account for siblings properly but the accounting after we picked a compute node did? | |
| 19:16:40 | sean-k-mooney | melwitt: the reader writer lock upgrde thing | |
| 19:16:49 | cfriesen | the NUMATopology filter would still check for siblings properly | |
| 19:17:47 | cfriesen | sean-k-mooney: jaypipes: ^ | |
| 19:18:02 | melwitt | sean-k-mooney: hm, I thought that bug was preventing services without uuids from receiving new uuids. not resulting in duplicates of them? | |
| 19:18:23 | sdeath | there are no duplicate entries in nova.services; I suspect a join against that table is returning duplicate rows, maybe related to the upgrade? two versions of the API present at once? | |
| 19:18:31 | sean-k-mooney | melwitt: oh ok i taught i saw duplicate uuid in the title maybe not | |
| 19:18:39 | dansmith | sdeath: do you have two cells defined pointing at the same db? | |
| 19:18:40 | sean-k-mooney | sdeath: is this an ironic deployment? | |
| 19:18:52 | sdeath | dansmith: I do, it turns out… that my problem? | |
| 19:18:59 | dansmith | sdeath: yup | |
| 19:19:07 | sdeath | ah - very well then; cure for this being, then…? | |
| 19:19:12 | dansmith | sdeath: delete one | |
| 19:19:35 | sean-k-mooney | dansmith: wait sdeath do you also have duplicate host names across cells | |
| 19:19:57 | sean-k-mooney | dansmith: how would you get teh same uuid otherwise? | |
| 19:20:13 | dansmith | sean-k-mooney: two cell mappings pointing at the same db will cause us to list from it twice | |
| 19:20:13 | sdeath | sean-k: it returns the same rows both times… I can paste what I pasted to #openstack (and got kickbanned, thank you Freenode, I AM NOT A SPAMMER grumble mumble bah) | |
| 19:20:31 | jaypipes | cfriesen: "it *is* possible to predict what will be consumed by the guest given the host information" <-- but it's not possible to do scheduling that way. | |
| 19:20:32 | dansmith | sdeath: pastebin, yo | |
| 19:20:34 | melwitt | use paste.openstack.org | |
| 19:20:46 | melwitt | or pastebin | |
| 19:20:51 | sean-k-mooney | dansmith: ah ok so its not two compute services with the same uuid its two cell mappings | |
| 19:21:20 | melwitt | all of that said, you might hit the bug sean-k-mooney mentioned after that, and if so, the fix is up for review currently and will be backported https://bugs.launchpad.net/nova/+bug/1746509 | |
| 19:21:21 | openstack | Launchpad bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 19:21:22 | sdeath | so OK… I can't kill the cell because it's got hosts in it... | |
| 19:21:28 | sdeath | evidently… | |
| 19:21:36 | cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | |