| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:06:01 | jaypipes | cfriesen: no, that is the host inventory of things. | |
| 19:06:03 | sean-k-mooney | or 8 HT if its powerpc | |
| 19:06:13 | jaypipes | cfriesen: again, I understand the host inventory part of this. | |
| 19:06:35 | jaypipes | cfriesen: my problem is with flavors that consume different amounts of resources on different hosts. | |
| 19:06:59 | cfriesen | yep, that's exactly what this does, since it depends on the host config | |
| 19:07:06 | sean-k-mooney | jaypipes: ya thats what happens for cpu_thread_policy=ISOLATE today | |
| 19:07:27 | jaypipes | cfriesen: which, due to the design of cpu_threads_policy, being a string of "isolate|prefer|share" is entirely impossible to predict an integer amaount of some CPU resources that will *actually* be consumed by the guest. | |
| 19:08:07 | sean-k-mooney | jaypipes: yes. this was not an issue in icehouse but its biting us now | |
| 19:08:23 | mlavalle | rybridges, melwitt: Look at slide 28 in https://www.slideshare.net/MiguelLavalle/routed-networks-sydney. That shows you what you should see in Placement. For each segment in a routed network, that is the Placement structure that you should see | |
| 19:08:28 | cfriesen | jaypipes: the original goal was to improve flexibility by enabling hyperthreads, while allowing instances to ask for whole cores if they need it for performance. | |
| 19:08:31 | tblakes | mriedem: For bug https://bugs.launchpad.net/nova/+bug/1756360, it looks like we're going to need to implement __repr__ for NovaExceptions. Do you have any input on the format we want to return? | |
| 19:08:33 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:09:30 | sean-k-mooney | jaypipes: cfriesen ya the original intel proposal made pinning a host config option with no flavor extra specs. then you would just use aggregate to make teh desision | |
| 19:09:31 | cfriesen | jaypipes: actually, it *is* possible to predict what will be consumed by the guest given the host information | |
| 19:09:58 | cfriesen | jaypipes: it's just that it could be different from one compute node to another | |
| 19:09:59 | mlavalle | rybridges, melwitt: a routed network is a network where segments are associated to its subnets. In other words, if you do a GET of those subnets, all of them should have a valua in their 'sehment_id' attribute | |
| 19:10:10 | sean-k-mooney | cfriesen: not before the placement allocate_candidates request | |
| 19:10:26 | mlavalle | rybridges, melwitt: 'segment_id'^^^^ | |
| 19:10:44 | cfriesen | sean-k-mooney: correct, unless we wanted to model siblings in placement. :) | |
| 19:11:10 | mriedem | tblakes: can you reply to gibi's question about reproducing this in comment 1 | |
| 19:11:20 | sean-k-mooney | cfriesen: lets not unless its a trait | |
| 19:11:29 | melwitt | rybridges: did you verify whether the nova_api.aggregate_hosts table has anything in it? in case it's some kind of scoping issue with doing the 'openstack aggregate show' not showing it? | |
| 19:11:45 | sean-k-mooney | event then its a state ful trait which is kind of a bad thing | |
| 19:11:46 | cfriesen | sean-k-mooney: actually, a trait of "number of HT siblings" might make sense | |
| 19:11:49 | openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | |
| 19:12:09 | sean-k-mooney | cfriesen: it could but its missueing tratits | |
| 19:12:17 | sean-k-mooney | and it changes the request | |
| 19:13:29 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 19:13:55 | sean-k-mooney | i guess you could tag the PCPU RP with HW_1_HT and HW_2_HT if HT was on | |
| 19:14:34 | sean-k-mooney | then maybe combine that with forbiden tratis to avoid HT hosts | |
| 19:14:47 | sdeath | Q: upgrade from Ocata->Pike; now services are showing up as duplicates (same ID, UUID, etc). Any ideas as to what might be causing it? | |
| 19:15:05 | sdeath | nova service-list, openstack compute servicec list | |
| 19:15:20 | sdeath | (asked last couple days running on #openstack) | |
| 19:15:22 | sean-k-mooney | its a strech jay im assuming you would not like tratis for HT amount | |
| 19:15:46 | sdeath | interesting development today: evidently I can't delete them, bombs out, "Service id X refers to multiple services" (although the UUID is the same) | |
| 19:16:11 | sean-k-mooney | melwitt: ^ is the db race you were debugging | |
| 19:16:26 | cfriesen | how bad would it be if the initial pre-check didn't account for siblings properly but the accounting after we picked a compute node did? | |
| 19:16:40 | sean-k-mooney | melwitt: the reader writer lock upgrde thing | |
| 19:16:49 | cfriesen | the NUMATopology filter would still check for siblings properly | |
| 19:17:47 | cfriesen | sean-k-mooney: jaypipes: ^ | |
| 19:18:02 | melwitt | sean-k-mooney: hm, I thought that bug was preventing services without uuids from receiving new uuids. not resulting in duplicates of them? | |
| 19:18:23 | sdeath | there are no duplicate entries in nova.services; I suspect a join against that table is returning duplicate rows, maybe related to the upgrade? two versions of the API present at once? | |
| 19:18:31 | sean-k-mooney | melwitt: oh ok i taught i saw duplicate uuid in the title maybe not | |
| 19:18:39 | dansmith | sdeath: do you have two cells defined pointing at the same db? | |
| 19:18:40 | sean-k-mooney | sdeath: is this an ironic deployment? | |
| 19:18:52 | sdeath | dansmith: I do, it turns out… that my problem? | |
| 19:18:59 | dansmith | sdeath: yup | |
| 19:19:07 | sdeath | ah - very well then; cure for this being, then…? | |
| 19:19:12 | dansmith | sdeath: delete one | |
| 19:19:35 | sean-k-mooney | dansmith: wait sdeath do you also have duplicate host names across cells | |
| 19:19:57 | sean-k-mooney | dansmith: how would you get teh same uuid otherwise? | |
| 19:20:13 | dansmith | sean-k-mooney: two cell mappings pointing at the same db will cause us to list from it twice | |
| 19:20:13 | sdeath | sean-k: it returns the same rows both times… I can paste what I pasted to #openstack (and got kickbanned, thank you Freenode, I AM NOT A SPAMMER grumble mumble bah) | |
| 19:20:31 | jaypipes | cfriesen: "it *is* possible to predict what will be consumed by the guest given the host information" <-- but it's not possible to do scheduling that way. | |
| 19:20:32 | dansmith | sdeath: pastebin, yo | |
| 19:20:34 | melwitt | use paste.openstack.org | |
| 19:20:46 | melwitt | or pastebin | |
| 19:20:51 | sean-k-mooney | dansmith: ah ok so its not two compute services with the same uuid its two cell mappings | |
| 19:21:20 | melwitt | all of that said, you might hit the bug sean-k-mooney mentioned after that, and if so, the fix is up for review currently and will be backported https://bugs.launchpad.net/nova/+bug/1746509 | |
| 19:21:21 | openstack | Launchpad bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 19:21:22 | sdeath | so OK… I can't kill the cell because it's got hosts in it... | |
| 19:21:28 | sdeath | evidently… | |
| 19:21:36 | cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | |
| 19:21:41 | dansmith | sdeath: you'll have to delete it from sql I guess | |
| 19:21:48 | sdeath | DS: is that safe, then? | |
| 19:21:54 | sdeath | if I remove from the nova.cells table? | |
| 19:22:11 | dansmith | sdeath: from nova_api.cell_mappings | |
| 19:22:22 | jaypipes | cfriesen: gross. | |
| 19:22:22 | sean-k-mooney | cfriesen: well that is what we were going to do anywya in jays current spec right? | |
| 19:22:34 | mriedem | dansmith: sdeath: we have --force flag on delete-cell i thought? | |
| 19:22:39 | cfriesen | sean-k-mooney: except for the final allocations part at the end | |
| 19:22:44 | rybridges | mlavalle: melwitt: nova_api.aggregate_hosts is empty in the db. that is likely our problem. But as I said earlier the segmenthostmappings table in the neutron db has the right hosts in it.. | |
| 19:22:48 | dansmith | mriedem: that will delete all the hosts | |
| 19:22:52 | dansmith | mriedem: which he doesn't want | |
| 19:22:53 | jaypipes | sean-k-mooney: we weren't going to dynamically adjust the number of resources requested in the allocation requests depending on which host was picked! | |
| 19:22:59 | sean-k-mooney | cfriesen: the allcoation stil happen in the scheduler right | |
| 19:23:00 | mriedem | the mappings, right | |
| 19:23:01 | mriedem | nvm | |
| 19:23:30 | sean-k-mooney | jaypipes: ah ya but it could be done in the schduler and ask placement to validate it | |
| 19:24:07 | sean-k-mooney | jaypipes: we would have to have a down call to the compute host however to figure out if we need to adjust the claim amount | |
| 19:24:22 | dansmith | sdeath: figure out which one is right right one and delete the other | |
| 19:24:23 | cfriesen | sean-k-mooney: isnt' that already in the host infomration? | |
| 19:24:28 | mlavalle | rybridges: sehgmenthostmappings has a lot of rows that will naver make it to Placement. Only those segments that are part of a routed network will have a RP in Placement | |
| 19:24:30 | jaypipes | sean-k-mooney: no. I'm not willing to change the design of the placement service (and the allocation request atomicity/guarantees) just to meet these wack-o requirements. | |
| 19:24:34 | dansmith | sdeath: hopefully all your instance and host mappings refer to one of the two cell mappings | |
| 19:24:36 | melwitt | rybridges: yeah, so that implies the nova API call to add the host must be failing https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L224 else segment_host_mappings is empty there | |
| 19:24:42 | sean-k-mooney | cfriesen: what how many ht the host has | |
| 19:24:48 | sdeath | m: let's see if —force exists... | |
| 19:25:01 | sean-k-mooney | cfriesen: its in the numa toplogy blob kindof but not really | |
| 19:25:03 | dansmith | sdeath: you do not want --force | |
| 19:25:18 | mlavalle | rybridges: in other words. Have you created a routed network yet? | |
| 19:25:18 | dansmith | sdeath: --force on that is actually --recursive | |
| 19:25:25 | cfriesen | jaypipes: once we select a compute node in the scheduler we need to make a call to placement to actually consume the resources, right? | |
| 19:25:29 | sdeath | ds: so maybe we won't do that then | |
| 19:25:48 | sdeath | [backs slowly away from angry bomb with ominous red light glowing on it] | |
| 19:25:50 | dansmith | sdeath: PSA: I dunno what irc client you're using but I bet it supports tab nick completion | |
| 19:25:52 | melwitt | heh | |
| 19:26:05 | cfriesen | jaypipes: oh, but we tell it that we want to actually consume a specific allocation from earlier, | |
| 19:26:08 | sdeath | adium, evidently it does | |
| 19:26:35 | sdeath | so, backtrcing: nova_api.cell_mappings | |