| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:02:01 | sean-k-mooney | kmalloc: oh adding the scope_types=['project'] does not change the behavior | |
| 19:02:11 | kmalloc | sean-k-mooney: yeah it shouldn't. | |
| 19:02:18 | jaypipes | cfriesen: whether the host has HT enable impacts the reporting of CPU inventory for the host (or its NUMA nodes), but cpu_threads_policy only affects which host processors the guest is willing to be pinned to. | |
| 19:02:48 | jaypipes | cfriesen: are you referring to cpu_allocation_policy? | |
| 19:02:56 | jaypipes | or cpu_threads_policy? | |
| 19:02:57 | kmalloc | sean-k-mooney: yeah it's meant to just allow for richer policy allowing for defining the scope-types in policy as well | |
| 19:03:04 | cfriesen | jaypipes: no. If I ask for 1 PCPU and set cpu_thread_policy=ISOLATE then it will actually consume two host CPUs (the main one and the HT sibling) | |
| 19:03:25 | kmalloc | sean-k-mooney: but that should just be allowing for more specific policy enforcement, not actually making a change to behavior. | |
| 19:04:20 | kmalloc | sean-k-mooney: that is why i was feeling confused, but you know, i am 100% willing to assume I was mis-understanding something since it's in nova's code base and i don't spend as much time there. | |
| 19:04:22 | sean-k-mooney | kmalloc: ok cool sound like you have it well in hand in anycase. just taught i would ask for my own knollage i avoid api changes unless i have no choice | |
| 19:04:35 | kmalloc | sean-k-mooney: cool. and again ty very much! | |
| 19:04:57 | cfriesen | jaypipes: basically by enabling cpu_thread_policy=ISOLATE I'm asking for the entire host core, which maps to two ht siblings. But each of those siblings was exposed to placement as a PCPU. | |
| 19:05:24 | jaypipes | cfriesen: it doesn't always map to 2 HTs. | |
| 19:05:37 | cfriesen | jaypipes: true, it could be more | |
| 19:05:38 | jaypipes | cfriesen: yet more hardware vendor undefined randomness. | |
| 19:05:40 | sean-k-mooney | jaypipes: unfrotuently yes | |
| 19:05:49 | cfriesen | jaypipes: or it could be 1, if HT is disabled on the host | |
| 19:06:01 | jaypipes | cfriesen: no, that is the host inventory of things. | |
| 19:06:03 | sean-k-mooney | or 8 HT if its powerpc | |
| 19:06:13 | jaypipes | cfriesen: again, I understand the host inventory part of this. | |
| 19:06:35 | jaypipes | cfriesen: my problem is with flavors that consume different amounts of resources on different hosts. | |
| 19:06:59 | cfriesen | yep, that's exactly what this does, since it depends on the host config | |
| 19:07:06 | sean-k-mooney | jaypipes: ya thats what happens for cpu_thread_policy=ISOLATE today | |
| 19:07:27 | jaypipes | cfriesen: which, due to the design of cpu_threads_policy, being a string of "isolate|prefer|share" is entirely impossible to predict an integer amaount of some CPU resources that will *actually* be consumed by the guest. | |
| 19:08:07 | sean-k-mooney | jaypipes: yes. this was not an issue in icehouse but its biting us now | |
| 19:08:23 | mlavalle | rybridges, melwitt: Look at slide 28 in https://www.slideshare.net/MiguelLavalle/routed-networks-sydney. That shows you what you should see in Placement. For each segment in a routed network, that is the Placement structure that you should see | |
| 19:08:28 | cfriesen | jaypipes: the original goal was to improve flexibility by enabling hyperthreads, while allowing instances to ask for whole cores if they need it for performance. | |
| 19:08:31 | tblakes | mriedem: For bug https://bugs.launchpad.net/nova/+bug/1756360, it looks like we're going to need to implement __repr__ for NovaExceptions. Do you have any input on the format we want to return? | |
| 19:08:33 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:09:30 | sean-k-mooney | jaypipes: cfriesen ya the original intel proposal made pinning a host config option with no flavor extra specs. then you would just use aggregate to make teh desision | |
| 19:09:31 | cfriesen | jaypipes: actually, it *is* possible to predict what will be consumed by the guest given the host information | |
| 19:09:58 | cfriesen | jaypipes: it's just that it could be different from one compute node to another | |
| 19:09:59 | mlavalle | rybridges, melwitt: a routed network is a network where segments are associated to its subnets. In other words, if you do a GET of those subnets, all of them should have a valua in their 'sehment_id' attribute | |
| 19:10:10 | sean-k-mooney | cfriesen: not before the placement allocate_candidates request | |
| 19:10:26 | mlavalle | rybridges, melwitt: 'segment_id'^^^^ | |
| 19:10:44 | cfriesen | sean-k-mooney: correct, unless we wanted to model siblings in placement. :) | |
| 19:11:10 | mriedem | tblakes: can you reply to gibi's question about reproducing this in comment 1 | |
| 19:11:20 | sean-k-mooney | cfriesen: lets not unless its a trait | |
| 19:11:29 | melwitt | rybridges: did you verify whether the nova_api.aggregate_hosts table has anything in it? in case it's some kind of scoping issue with doing the 'openstack aggregate show' not showing it? | |
| 19:11:45 | sean-k-mooney | event then its a state ful trait which is kind of a bad thing | |
| 19:11:46 | cfriesen | sean-k-mooney: actually, a trait of "number of HT siblings" might make sense | |
| 19:11:49 | openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | |
| 19:12:09 | sean-k-mooney | cfriesen: it could but its missueing tratits | |
| 19:12:17 | sean-k-mooney | and it changes the request | |
| 19:13:29 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 19:13:55 | sean-k-mooney | i guess you could tag the PCPU RP with HW_1_HT and HW_2_HT if HT was on | |
| 19:14:34 | sean-k-mooney | then maybe combine that with forbiden tratis to avoid HT hosts | |
| 19:14:47 | sdeath | Q: upgrade from Ocata->Pike; now services are showing up as duplicates (same ID, UUID, etc). Any ideas as to what might be causing it? | |
| 19:15:05 | sdeath | nova service-list, openstack compute servicec list | |
| 19:15:20 | sdeath | (asked last couple days running on #openstack) | |
| 19:15:22 | sean-k-mooney | its a strech jay im assuming you would not like tratis for HT amount | |
| 19:15:46 | sdeath | interesting development today: evidently I can't delete them, bombs out, "Service id X refers to multiple services" (although the UUID is the same) | |
| 19:16:11 | sean-k-mooney | melwitt: ^ is the db race you were debugging | |
| 19:16:26 | cfriesen | how bad would it be if the initial pre-check didn't account for siblings properly but the accounting after we picked a compute node did? | |
| 19:16:40 | sean-k-mooney | melwitt: the reader writer lock upgrde thing | |
| 19:16:49 | cfriesen | the NUMATopology filter would still check for siblings properly | |
| 19:17:47 | cfriesen | sean-k-mooney: jaypipes: ^ | |
| 19:18:02 | melwitt | sean-k-mooney: hm, I thought that bug was preventing services without uuids from receiving new uuids. not resulting in duplicates of them? | |
| 19:18:23 | sdeath | there are no duplicate entries in nova.services; I suspect a join against that table is returning duplicate rows, maybe related to the upgrade? two versions of the API present at once? | |
| 19:18:31 | sean-k-mooney | melwitt: oh ok i taught i saw duplicate uuid in the title maybe not | |
| 19:18:39 | dansmith | sdeath: do you have two cells defined pointing at the same db? | |
| 19:18:40 | sean-k-mooney | sdeath: is this an ironic deployment? | |
| 19:18:52 | sdeath | dansmith: I do, it turns out… that my problem? | |
| 19:18:59 | dansmith | sdeath: yup | |
| 19:19:07 | sdeath | ah - very well then; cure for this being, then…? | |
| 19:19:12 | dansmith | sdeath: delete one | |
| 19:19:35 | sean-k-mooney | dansmith: wait sdeath do you also have duplicate host names across cells | |
| 19:19:57 | sean-k-mooney | dansmith: how would you get teh same uuid otherwise? | |
| 19:20:13 | sdeath | sean-k: it returns the same rows both times… I can paste what I pasted to #openstack (and got kickbanned, thank you Freenode, I AM NOT A SPAMMER grumble mumble bah) | |
| 19:20:13 | dansmith | sean-k-mooney: two cell mappings pointing at the same db will cause us to list from it twice | |
| 19:20:31 | jaypipes | cfriesen: "it *is* possible to predict what will be consumed by the guest given the host information" <-- but it's not possible to do scheduling that way. | |
| 19:20:32 | dansmith | sdeath: pastebin, yo | |
| 19:20:34 | melwitt | use paste.openstack.org | |
| 19:20:46 | melwitt | or pastebin | |
| 19:20:51 | sean-k-mooney | dansmith: ah ok so its not two compute services with the same uuid its two cell mappings | |
| 19:21:20 | melwitt | all of that said, you might hit the bug sean-k-mooney mentioned after that, and if so, the fix is up for review currently and will be backported https://bugs.launchpad.net/nova/+bug/1746509 | |
| 19:21:21 | openstack | Launchpad bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 19:21:22 | sdeath | so OK… I can't kill the cell because it's got hosts in it... | |
| 19:21:28 | sdeath | evidently… | |
| 19:21:36 | cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | |
| 19:21:41 | dansmith | sdeath: you'll have to delete it from sql I guess | |
| 19:21:48 | sdeath | DS: is that safe, then? | |
| 19:21:54 | sdeath | if I remove from the nova.cells table? | |
| 19:22:11 | dansmith | sdeath: from nova_api.cell_mappings | |
| 19:22:22 | sean-k-mooney | cfriesen: well that is what we were going to do anywya in jays current spec right? | |
| 19:22:22 | jaypipes | cfriesen: gross. | |
| 19:22:34 | mriedem | dansmith: sdeath: we have --force flag on delete-cell i thought? | |
| 19:22:39 | cfriesen | sean-k-mooney: except for the final allocations part at the end | |
| 19:22:44 | rybridges | mlavalle: melwitt: nova_api.aggregate_hosts is empty in the db. that is likely our problem. But as I said earlier the segmenthostmappings table in the neutron db has the right hosts in it.. | |
| 19:22:48 | dansmith | mriedem: that will delete all the hosts | |
| 19:22:52 | dansmith | mriedem: which he doesn't want | |
| 19:22:53 | jaypipes | sean-k-mooney: we weren't going to dynamically adjust the number of resources requested in the allocation requests depending on which host was picked! | |
| 19:22:59 | sean-k-mooney | cfriesen: the allcoation stil happen in the scheduler right | |
| 19:23:00 | mriedem | the mappings, right | |
| 19:23:01 | mriedem | nvm | |
| 19:23:30 | sean-k-mooney | jaypipes: ah ya but it could be done in the schduler and ask placement to validate it | |
| 19:24:07 | sean-k-mooney | jaypipes: we would have to have a down call to the compute host however to figure out if we need to adjust the claim amount | |
| 19:24:22 | dansmith | sdeath: figure out which one is right right one and delete the other | |
| 19:24:23 | cfriesen | sean-k-mooney: isnt' that already in the host infomration? | |
| 19:24:28 | mlavalle | rybridges: sehgmenthostmappings has a lot of rows that will naver make it to Placement. Only those segments that are part of a routed network will have a RP in Placement | |