| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-22 | |||
| 19:14:34 | sean-k-mooney | then maybe combine that with forbiden tratis to avoid HT hosts | |
| 19:14:47 | sdeath | Q: upgrade from Ocata->Pike; now services are showing up as duplicates (same ID, UUID, etc). Any ideas as to what might be causing it? | |
| 19:15:05 | sdeath | nova service-list, openstack compute servicec list | |
| 19:15:20 | sdeath | (asked last couple days running on #openstack) | |
| 19:15:22 | sean-k-mooney | its a strech jay im assuming you would not like tratis for HT amount | |
| 19:15:46 | sdeath | interesting development today: evidently I can't delete them, bombs out, "Service id X refers to multiple services" (although the UUID is the same) | |
| 19:16:11 | sean-k-mooney | melwitt: ^ is the db race you were debugging | |
| 19:16:26 | cfriesen | how bad would it be if the initial pre-check didn't account for siblings properly but the accounting after we picked a compute node did? | |
| 19:16:40 | sean-k-mooney | melwitt: the reader writer lock upgrde thing | |
| 19:16:49 | cfriesen | the NUMATopology filter would still check for siblings properly | |
| 19:17:47 | cfriesen | sean-k-mooney: jaypipes: ^ | |
| 19:18:02 | melwitt | sean-k-mooney: hm, I thought that bug was preventing services without uuids from receiving new uuids. not resulting in duplicates of them? | |
| 19:18:23 | sdeath | there are no duplicate entries in nova.services; I suspect a join against that table is returning duplicate rows, maybe related to the upgrade? two versions of the API present at once? | |
| 19:18:31 | sean-k-mooney | melwitt: oh ok i taught i saw duplicate uuid in the title maybe not | |
| 19:18:39 | dansmith | sdeath: do you have two cells defined pointing at the same db? | |
| 19:18:40 | sean-k-mooney | sdeath: is this an ironic deployment? | |
| 19:18:52 | sdeath | dansmith: I do, it turns out… that my problem? | |
| 19:18:59 | dansmith | sdeath: yup | |
| 19:19:07 | sdeath | ah - very well then; cure for this being, then…? | |
| 19:19:12 | dansmith | sdeath: delete one | |
| 19:19:35 | sean-k-mooney | dansmith: wait sdeath do you also have duplicate host names across cells | |
| 19:19:57 | sean-k-mooney | dansmith: how would you get teh same uuid otherwise? | |
| 19:20:13 | sdeath | sean-k: it returns the same rows both times… I can paste what I pasted to #openstack (and got kickbanned, thank you Freenode, I AM NOT A SPAMMER grumble mumble bah) | |
| 19:20:13 | dansmith | sean-k-mooney: two cell mappings pointing at the same db will cause us to list from it twice | |
| 19:20:31 | jaypipes | cfriesen: "it *is* possible to predict what will be consumed by the guest given the host information" <-- but it's not possible to do scheduling that way. | |
| 19:20:32 | dansmith | sdeath: pastebin, yo | |
| 19:20:34 | melwitt | use paste.openstack.org | |
| 19:20:46 | melwitt | or pastebin | |
| 19:20:51 | sean-k-mooney | dansmith: ah ok so its not two compute services with the same uuid its two cell mappings | |
| 19:21:20 | melwitt | all of that said, you might hit the bug sean-k-mooney mentioned after that, and if so, the fix is up for review currently and will be backported https://bugs.launchpad.net/nova/+bug/1746509 | |
| 19:21:21 | openstack | Launchpad bug 1746509 in OpenStack Compute (nova) "TypeError: Can't upgrade a READER transaction to a WRITER mid-transaction" [Medium,In progress] - Assigned to melanie witt (melwitt) | |
| 19:21:22 | sdeath | so OK… I can't kill the cell because it's got hosts in it... | |
| 19:21:28 | sdeath | evidently… | |
| 19:21:36 | cfriesen | jaypipes: right, so what if we ignore siblings for the placement prefiltering, run through the scheduler filters as usual (which will look at siblings properly), then once we pick a host we update the actual allocations in placement based on the knowledge we have of the host. | |
| 19:21:41 | dansmith | sdeath: you'll have to delete it from sql I guess | |
| 19:21:48 | sdeath | DS: is that safe, then? | |
| 19:21:54 | sdeath | if I remove from the nova.cells table? | |
| 19:22:11 | dansmith | sdeath: from nova_api.cell_mappings | |
| 19:22:22 | sean-k-mooney | cfriesen: well that is what we were going to do anywya in jays current spec right? | |
| 19:22:22 | jaypipes | cfriesen: gross. | |
| 19:22:34 | mriedem | dansmith: sdeath: we have --force flag on delete-cell i thought? | |
| 19:22:39 | cfriesen | sean-k-mooney: except for the final allocations part at the end | |
| 19:22:44 | rybridges | mlavalle: melwitt: nova_api.aggregate_hosts is empty in the db. that is likely our problem. But as I said earlier the segmenthostmappings table in the neutron db has the right hosts in it.. | |
| 19:22:48 | dansmith | mriedem: that will delete all the hosts | |
| 19:22:52 | dansmith | mriedem: which he doesn't want | |
| 19:22:53 | jaypipes | sean-k-mooney: we weren't going to dynamically adjust the number of resources requested in the allocation requests depending on which host was picked! | |
| 19:22:59 | sean-k-mooney | cfriesen: the allcoation stil happen in the scheduler right | |
| 19:23:00 | mriedem | the mappings, right | |
| 19:23:01 | mriedem | nvm | |
| 19:23:30 | sean-k-mooney | jaypipes: ah ya but it could be done in the schduler and ask placement to validate it | |
| 19:24:07 | sean-k-mooney | jaypipes: we would have to have a down call to the compute host however to figure out if we need to adjust the claim amount | |
| 19:24:22 | dansmith | sdeath: figure out which one is right right one and delete the other | |
| 19:24:23 | cfriesen | sean-k-mooney: isnt' that already in the host infomration? | |
| 19:24:28 | mlavalle | rybridges: sehgmenthostmappings has a lot of rows that will naver make it to Placement. Only those segments that are part of a routed network will have a RP in Placement | |
| 19:24:30 | jaypipes | sean-k-mooney: no. I'm not willing to change the design of the placement service (and the allocation request atomicity/guarantees) just to meet these wack-o requirements. | |
| 19:24:34 | dansmith | sdeath: hopefully all your instance and host mappings refer to one of the two cell mappings | |
| 19:24:36 | melwitt | rybridges: yeah, so that implies the nova API call to add the host must be failing https://github.com/openstack/neutron/blob/master/neutron/services/segments/plugin.py#L224 else segment_host_mappings is empty there | |
| 19:24:42 | sean-k-mooney | cfriesen: what how many ht the host has | |
| 19:24:48 | sdeath | m: let's see if —force exists... | |
| 19:25:01 | sean-k-mooney | cfriesen: its in the numa toplogy blob kindof but not really | |
| 19:25:03 | dansmith | sdeath: you do not want --force | |
| 19:25:18 | dansmith | sdeath: --force on that is actually --recursive | |
| 19:25:18 | mlavalle | rybridges: in other words. Have you created a routed network yet? | |
| 19:25:25 | cfriesen | jaypipes: once we select a compute node in the scheduler we need to make a call to placement to actually consume the resources, right? | |
| 19:25:29 | sdeath | ds: so maybe we won't do that then | |
| 19:25:48 | sdeath | [backs slowly away from angry bomb with ominous red light glowing on it] | |
| 19:25:50 | dansmith | sdeath: PSA: I dunno what irc client you're using but I bet it supports tab nick completion | |
| 19:25:52 | melwitt | heh | |
| 19:26:05 | cfriesen | jaypipes: oh, but we tell it that we want to actually consume a specific allocation from earlier, | |
| 19:26:08 | sdeath | adium, evidently it does | |
| 19:26:35 | sdeath | so, backtrcing: nova_api.cell_mappings | |
| 19:26:44 | cfriesen | jaypipes: I think I see what you mean, we can't retroactively change the size of an earlier allocation request | |
| 19:27:00 | sean-k-mooney | cfriesen: ya so in this case we would have to make another allocation_candiates request and then find the host n that again and consume that | |
| 19:27:49 | cfriesen | sean-k-mooney: right. Or else we throw out the cpu_thread_policy option in the flavor, or else we model siblings in placement. | |
| 19:28:02 | jaypipes | cfriesen: right. the whole point of the allocation candidates list was "these are the nodes that met your request for resources". if we then just modify that resource amount, we invalidate the result of allocation candidates. :( | |
| 19:28:27 | sdeath | dansmith: OK, rows delete (from host_mappings and from cell_mappings; host_mappings due to foreign_key constraint) | |
| 19:28:30 | sdeath | restarting nova… let's see if this work. | |
| 19:28:33 | sdeath | *wrks | |
| 19:28:51 | sdeath | hey, how about that. only one row per hypervisor! | |
| 19:28:55 | sdeath | OK, so... | |
| 19:29:00 | dansmith | sdeath: so you'll need to do discover_hosts again, but it probably won't find them all because they've been marked as already mapped | |
| 19:29:17 | dansmith | sdeath: you should have changed the id on the others to be the cell you kept | |
| 19:30:06 | sdeath | will it barf if I manually reinsert? | |
| 19:30:13 | dansmith | sdeath: no | |
| 19:30:20 | sean-k-mooney | cfriesen: i dont really see a good way to keep cpu_thread_policy=isolate with out having an allocation on the compute host | |
| 19:30:29 | tblakes | mriedmen: I responded to his comment in bug 1756360. He was seeing that issue because he wasn't passing in a kwarg that nova passes in. | |
| 19:30:31 | sdeath | dansmith: actually, map_cell_and_hosts seems to have picked them back up.. | |
| 19:30:31 | jaypipes | cfriesen: or... we just go with my spec and just track dedicated CPUs as their own thing and deal with the workloads that can't tolerate having their dedicated CPUs be hyperthreads as the snowflakey thing that it is when we get to the destination host the scheduler selected. | |
| 19:30:32 | openstack | bug 1756360 in OpenStack Compute (nova) "Serializer strips Exception kwargs" [Undecided,Incomplete] https://launchpad.net/bugs/1756360 - Assigned to Tyler Blakeslee (tblakes) | |
| 19:30:41 | sdeath | I see fresh IDs and happiness | |
| 19:30:51 | dansmith | sdeath: that's not what you wanted I think | |
| 19:31:25 | cfriesen | jaypipes: so we'd get to the compute node and find out "oops, it doesn't have as many PCPU resources as we thought" and fail the claim? | |
| 19:31:28 | dansmith | sdeath: didn't that create a new cell mapping again? | |
| 19:31:34 | sdeath | dansmith: nope | |
| 19:31:49 | sean-k-mooney | cfriesen: well claim would happen in scuduler before getting to compute host | |
| 19:31:58 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add unit test for non-placement resize https://review.openstack.org/537614 | |
| 19:32:10 | sean-k-mooney | cfriesen: we could at the very end try to extend the claim and only fail if we could not extend | |
| 19:32:19 | sdeath | cell that I wanted to keep is there; new host_mapping IDs mapping the previously-mapped compute nodes to that cell; nova service-list and openstack compute service list show the expected list of hypervisors. | |
| 19:32:24 | dansmith | sdeath: ah, I see where it went, okay | |
| 19:32:52 | dansmith | sdeath: so you're good now? | |