| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:07:27 | jaypipes | cfriesen: which, due to the design of cpu_threads_policy, being a string of "isolate|prefer|share" is entirely impossible to predict an integer amaount of some CPU resources that will *actually* be consumed by the guest. | |
| 19:08:07 | sean-k-mooney | jaypipes: yes. this was not an issue in icehouse but its biting us now | |
| 19:08:23 | mlavalle | rybridges, melwitt: Look at slide 28 in https://www.slideshare.net/MiguelLavalle/routed-networks-sydney. That shows you what you should see in Placement. For each segment in a routed network, that is the Placement structure that you should see | |
| 19:08:28 | cfriesen | jaypipes: the original goal was to improve flexibility by enabling hyperthreads, while allowing instances to ask for whole cores if they need it for performance. | |
| 19:08:31 | tblakes | mriedem: For bug https://bugs.launchpad.net/nova/+bug/1756360, it looks like we're going to need to implement __repr__ for NovaExceptions. Do you have any input on the format we want to return? | |
| 19:08:33 | openstack | Launchpad bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] - Assigned to Tyler Blakeslee (tblakes) | |
| 19:09:30 | sean-k-mooney | jaypipes: cfriesen ya the original intel proposal made pinning a host config option with no flavor extra specs. then you would just use aggregate to make teh desision | |
| 19:09:31 | cfriesen | jaypipes: actually, it *is* possible to predict what will be consumed by the guest given the host information | |
| 19:09:58 | cfriesen | jaypipes: it's just that it could be different from one compute node to another | |
| 19:09:59 | mlavalle | rybridges, melwitt: a routed network is a network where segments are associated to its subnets. In other words, if you do a GET of those subnets, all of them should have a valua in their 'sehment_id' attribute | |
| 19:10:10 | sean-k-mooney | cfriesen: not before the placement allocate_candidates request | |
| 19:10:26 | mlavalle | rybridges, melwitt: 'segment_id'^^^^ | |
| 19:10:44 | cfriesen | sean-k-mooney: correct, unless we wanted to model siblings in placement. :) | |
| 19:11:10 | mriedem | tblakes: can you reply to gibi's question about reproducing this in comment 1 | |
| 19:11:20 | sean-k-mooney | cfriesen: lets not unless its a trait | |
| 19:11:29 | melwitt | rybridges: did you verify whether the nova_api.aggregate_hosts table has anything in it? in case it's some kind of scoping issue with doing the 'openstack aggregate show' not showing it? | |
| 19:11:45 | sean-k-mooney | event then its a state ful trait which is kind of a bad thing | |
| 19:11:46 | cfriesen | sean-k-mooney: actually, a trait of "number of HT siblings" might make sense | |
| 19:11:49 | openstackgerrit | Julia Kreger proposed openstack/nova master: WIP: Add microversion to ironic client wrapper call https://review.openstack.org/554762 | |
| 19:12:09 | sean-k-mooney | cfriesen: it could but its missueing tratits | |
| 19:12:17 | sean-k-mooney | and it changes the request | |
| 19:13:29 | openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | |
| 19:13:55 | sean-k-mooney | i guess you could tag the PCPU RP with HW_1_HT and HW_2_HT if HT was on | |
| 19:14:34 | sean-k-mooney | then maybe combine that with forbiden tratis to avoid HT hosts | |
| 19:14:47 | sdeath | Q: upgrade from Ocata->Pike; now services are showing up as duplicates (same ID, UUID, etc). Any ideas as to what might be causing it? | |
| 19:15:05 | sdeath | nova service-list, openstack compute servicec list | |
| 19:15:20 | sdeath | (asked last couple days running on #openstack) | |
| 19:15:22 | sean-k-mooney | its a strech jay im assuming you would not like tratis for HT amount | |
| 19:15:46 | sdeath | interesting development today: evidently I can't delete them, bombs out, "Service id X refers to multiple services" (although the UUID is the same) | |
| 19:16:11 | sean-k-mooney | melwitt: ^ is the db race you were debugging | |
| 19:16:26 | cfriesen | how bad would it be if the initial pre-check didn't account for siblings properly but the accounting after we picked a compute node did? | |
| 19:16:40 | sean-k-mooney | melwitt: the reader writer lock upgrde thing | |
| 19:16:49 | cfriesen | the NUMATopology filter would still check for siblings properly | |
| 19:17:47 | cfriesen | sean-k-mooney: jaypipes: ^ | |
| 19:18:02 | melwitt | sean-k-mooney: hm, I thought that bug was preventing services without uuids from receiving new uuids. not resulting in duplicates of them? | |
| 19:18:23 | sdeath | there are no duplicate entries in nova.services; I suspect a join against that table is returning duplicate rows, maybe related to the upgrade? two versions of the API present at once? | |
| 19:18:31 | sean-k-mooney | melwitt: oh ok i taught i saw duplicate uuid in the title maybe not | |
| 19:18:39 | dansmith | sdeath: do you have two cells defined pointing at the same db? | |
| 19:18:40 | sean-k-mooney | sdeath: is this an ironic deployment? | |
| 19:18:52 | sdeath | dansmith: I do, it turns out… that my problem? | |
| 19:18:59 | dansmith | sdeath: yup | |
| 19:19:07 | sdeath | ah - very well then; cure for this being, then…? | |
| 19:19:12 | dansmith | sdeath: delete one | |
| 19:19:35 | sean-k-mooney | dansmith: wait sdeath do you also have duplicate host names across cells | |
| 19:19:57 | sean-k-mooney | dansmith: how would you get teh same uuid otherwise? | |
| 19:20:13 | dansmith | sean-k-mooney: two cell mappings pointing at the same db will cause us to list from it twice | |
| 19:20:13 | sdeath | sean-k: it returns the same rows both times… I can paste what I pasted to #openstack (and got kickbanned, thank you Freenode, I AM NOT A SPAMMER grumble mumble bah) | |
| 19:20:31 | jaypipes | cfriesen: "it *is* possible to predict what will be consumed by the guest given the host information" <-- but it's not possible to do scheduling that way. | |
| 19:20:32 | dansmith | sdeath: pastebin, yo | |
| 19:20:34 | melwitt | use paste.openstack.org | |
| 19:20:46 | melwitt | or pastebin | |
| 19:20:51 | sean-k-mooney | dansmith: ah ok so its not two compute services with the same uuid its two cell mappings | |
| 19:21:20 | melwitt | all of that said, you might hit the bug sean-k-mooney mentioned after that, and if so, the fix is up for review currently and will be backported https://bugs.launchpad.net/nova/+bug/1746509 | |
| 19:21:21 | openstack | Launchpad bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 19:21:22 | sdeath | so OK… I can't kill the cell because it's got hosts in it... | |
| 19:21:28 | sdeath | evidently… | |
| 19:21:36 | cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | |
| 19:21:41 | dansmith | sdeath: you'll have to delete it from sql I guess | |
| 19:21:48 | sdeath | DS: is that safe, then? | |
| 19:21:54 | sdeath | if I remove from the nova.cells table? | |
| 19:22:11 | dansmith | sdeath: from nova_api.cell_mappings | |
| 19:22:22 | jaypipes | cfriesen: gross. | |
| 19:22:22 | sean-k-mooney | cfriesen: well that is what we were going to do anywya in jays current spec right? | |
| 19:22:34 | mriedem | dansmith: sdeath: we have --force flag on delete-cell i thought? | |
| 19:22:39 | cfriesen | sean-k-mooney: except for the final allocations part at the end | |
| 19:22:44 | rybridges | mlavalle: melwitt: nova_api.aggregate_hosts is empty in the db. that is likely our problem. But as I said earlier the segmenthostmappings table in the neutron db has the right hosts in it.. | |
| 19:22:48 | dansmith | mriedem: that will delete all the hosts | |
| 19:22:52 | dansmith | mriedem: which he doesn't want | |
| 19:22:53 | jaypipes | sean-k-mooney: we weren't going to dynamically adjust the number of resources requested in the allocation requests depending on which host was picked! | |
| 19:22:59 | sean-k-mooney | cfriesen: the allcoation stil happen in the scheduler right | |
| 19:23:00 | mriedem | the mappings, right | |
| 19:23:01 | mriedem | nvm | |
| 19:23:30 | sean-k-mooney | jaypipes: ah ya but it could be done in the schduler and ask placement to validate it | |
| 19:24:07 | sean-k-mooney | jaypipes: we would have to have a down call to the compute host however to figure out if we need to adjust the claim amount | |
| 19:24:22 | dansmith | sdeath: figure out which one is right right one and delete the other | |
| 19:24:23 | cfriesen | sean-k-mooney: isnt' that already in the host infomration? | |
| 19:24:28 | mlavalle | rybridges: sehgmenthostmappings has a lot of rows that will naver make it to Placement. Only those segments that are part of a routed network will have a RP in Placement | |
| 19:24:30 | jaypipes | sean-k-mooney: no. I'm not willing to change the design of the placement service (and the allocation request atomicity/guarantees) just to meet these wack-o requirements. | |
| 19:24:34 | dansmith | sdeath: hopefully all your instance and host mappings refer to one of the two cell mappings | |
| 19:24:36 | melwitt | rybridges: yeah, so that implies the nova API call to add the host must be failing https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L224 else segment_host_mappings is empty there | |
| 19:24:42 | sean-k-mooney | cfriesen: what how many ht the host has | |
| 19:24:48 | sdeath | m: let's see if —force exists... | |
| 19:25:01 | sean-k-mooney | cfriesen: its in the numa toplogy blob kindof but not really | |
| 19:25:03 | dansmith | sdeath: you do not want --force | |
| 19:25:18 | mlavalle | rybridges: in other words. Have you created a routed network yet? | |
| 19:25:18 | dansmith | sdeath: --force on that is actually --recursive | |
| 19:25:25 | cfriesen | jaypipes: once we select a compute node in the scheduler we need to make a call to placement to actually consume the resources, right? | |
| 19:25:29 | sdeath | ds: so maybe we won't do that then | |
| 19:25:48 | sdeath | [backs slowly away from angry bomb with ominous red light glowing on it] | |
| 19:25:50 | dansmith | sdeath: PSA: I dunno what irc client you're using but I bet it supports tab nick completion | |
| 19:25:52 | melwitt | heh | |
| 19:26:05 | cfriesen | jaypipes: oh, but we tell it that we want to actually consume a specific allocation from earlier, | |
| 19:26:08 | sdeath | adium, evidently it does | |
| 19:26:35 | sdeath | so, backtrcing: nova_api.cell_mappings | |
| 19:26:44 | cfriesen | jaypipes: I think I see what you mean, we can't retroactively change the size of an earlier allocation request | |
| 19:27:00 | sean-k-mooney | cfriesen: ya so in this case we would have to make another allocation_candiates request and then find the host n that again and consume that | |
| 19:27:49 | cfriesen | sean-k-mooney: right. Or else we throw out the cpu_thread_policy option in the flavor, or else we model siblings in placement. | |
| 19:28:02 | jaypipes | cfriesen: right. the whole point of the allocation candidates list was "these are the nodes that met your request for resources". if we then just modify that resource amount, we invalidate the result of allocation candidates. :( | |
| 19:28:27 | sdeath | dansmith: OK, rows delete (from host_mappings and from cell_mappings; host_mappings due to foreign_key constraint) | |
| 19:28:30 | sdeath | restarting nova… let's see if this work. | |