| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-13 | |||
| 18:41:27 | mgagne | mriedem: I guess I need to find a way to heal the allocations for a specific compute instead. | |
| 18:42:52 | mriedem | mgagne: i'm not quite sure i understand, you've got 1 nova-compute service running to service multiple ironic nodes, right? that is totally fine - and actually, as noted earlier, ironic computes should already be posting allocations for their instances anyway | |
| 18:43:13 | mriedem | ironic instances will be 1:1 with compute_nodes records | |
| 18:43:19 | mgagne | mriedem: we figured out that we don't need to heal Ironic because it auto heals. I'm currently testing libvirt. | |
| 18:43:23 | dansmith | mriedem: I know you're asking for it, but it clearly wasn't preloaded otherwise it'd not be lazy loading | |
| 18:43:51 | mgagne | mriedem: we have 100+ nodes in prod but I can't model as much nodes in my dev setup. I only have node01. | |
| 18:44:16 | mriedem | mgagne: ok so you're saying for your other libvirt nodes, they haven't reported resource providers into placement yet b/c they haven't started | |
| 18:44:22 | mgagne | so I only have to make sure I test the healing with a specific node instead. | |
| 18:44:27 | mgagne | yes | |
| 18:44:33 | mriedem | ok that makes more sense | |
| 18:44:45 | mgagne | just talking out loud atm =) | |
| 18:44:55 | mriedem | PUT allocations fails b/c the specified resource provider wasn't found by id | |
| 18:44:57 | mriedem | np | |
| 18:44:58 | mriedem | i get it | |
| 18:45:16 | mgagne | yes | |
| 18:45:17 | mgagne | exactly | |
| 18:45:18 | mgagne | Failed to create allocations for instance XXX against resource provider YYY. | |
| 18:45:59 | mriedem | ok but you got past that lazy-load issue so i can post a fix for that | |
| 18:46:07 | mgagne | oh yes | |
| 18:46:26 | mgagne | just wanted to make sure I didn't hit a similar bug later with a non-deleted instance for reasons | |
| 18:48:44 | mriedem | joining all my coworker friends in the kitchen | |
| 18:49:00 | dansmith | mriedem: the cats? | |
| 18:49:06 | mgagne | are they pugs like jaypipes ones? | |
| 18:49:41 | mriedem | there is the one cat | |
| 18:50:14 | mgagne | https://bugs.launchpad.net/nova/+bug/1776743 | |
| 18:50:15 | openstack | Launchpad bug 1776743 in OpenStack Compute (nova) "Allocation healer should ignore deleted instances" [Undecided,New] | |
| 18:50:15 | mriedem | left those fish assholes back in the office | |
| 18:50:24 | mriedem | thanks | |
| 18:50:44 | mgagne | should I propose a fix or will you take care of it? | |
| 18:52:53 | jaypipes | mgagne: you mean, basically, "cat-like dogs"? :) | |
| 18:53:15 | mriedem | mgagne: you can if you want, i'm going to be updating this other change first - i think we'd also want a functional test wrinkle for that since unit tests aren't really going to cut it | |
| 18:53:17 | mgagne | jaypipes: I'm not a pug expert unfortunately so I can't tell. ¯\_(ツ)_/¯ | |
| 18:53:31 | mgagne | mriedem: ouch =( | |
| 18:53:33 | mriedem | there are existing functional tests for heal_allocatoins | |
| 18:53:41 | mriedem | so adding one that just creates a server and then deletes it should be easy enough | |
| 18:53:51 | mriedem | create server w/o allocs, delete it, run heal_allocations | |
| 18:54:28 | mgagne | ok, I might not work on it right away, still trying to test the actual healer first | |
| 18:56:42 | zcorneli | dansmith: Updated service_version here: https://review.openstack.org/#/c/567876/14/nova/objects/service.py , Want to see that pass the tests before I post the doc changes (just to make sure the rebase is fully passing) | |
| 18:56:53 | dansmith | zcorneli: I had already done that for you this morning | |
| 18:57:12 | dansmith | zcorneli: because I was trying to get a good run on the devstack change, which I guess we need to recheck now | |
| 18:57:49 | zcorneli | Your patch had failed in Zuul, added the service version to the log, but didn't increment the constant | |
| 18:58:07 | dansmith | zcorneli: https://review.openstack.org/#/c/574792 | |
| 18:58:08 | dansmith | zcorneli: ah, okay sorry, hadn't seen, thought maybe you didn't notice | |
| 18:58:16 | dansmith | zcorneli: I think we probably also need to set allocation_ratio=1.0 in that devstack patch now that I think of it | |
| 18:58:43 | zcorneli | dansmith: Yea, without setting ram_allocation_ratio=1.0, it should fail to start once file_backed_memory is enabled. | |
| 18:58:51 | dansmith | yeah | |
| 19:00:40 | dansmith | tssurya: mriedem melwitt: cells meeting today? I don't think I have anything to discuss | |
| 19:01:51 | tssurya | dansmith: me neither, just working on mriedem's improvisations regarding host states in our env, will provide input on it when we have something solid | |
| 19:02:26 | dansmith | cool | |
| 19:03:44 | mriedem | dansmith: get_minimum_version_all_cells doesn't cache does it | |
| 19:04:03 | mriedem | so on multi-create, we query that for each instance in some cases | |
| 19:04:17 | dansmith | mriedem: doesn't cache what? | |
| 19:04:38 | melwitt | dansmith: +1 on a skip | |
| 19:04:43 | mriedem | the result, like nova.objects.Service.get_minimum_version | |
| 19:04:57 | dansmith | oh, does it | |
| 19:05:24 | dansmith | I think it does if so configured | |
| 19:05:52 | dansmith | although........ | |
| 19:05:54 | dansmith | hmm | |
| 19:06:05 | mriedem | not that one | |
| 19:06:43 | dansmith | oh, yep | |
| 19:06:47 | dansmith | it calls the inner | |
| 19:06:50 | dansmith | so yeah, shouldn't | |
| 19:07:17 | mriedem | yeah i'm just leaving a todo about optimizing the calls to that for multicreate in this certs change | |
| 19:07:26 | melwitt | mriedem: +1 on the rebuild logic | |
| 19:07:44 | dansmith | mriedem: we'll have to do something separate so we don't affect the cache when one returns and another hasn't run yet | |
| 19:07:52 | dansmith | (which is what I was hmmming about | |
| 19:08:32 | mriedem | i'm not planning to try and cachify get_minimum_version_all_cells | |
| 19:09:29 | melwitt | jaypipes: yaaay (pugs) | |
| 19:19:29 | jmlowe | Is there something special I have to do to get soft-anti-affinity to work? | |
| 19:20:24 | mgagne | mriedem: so I managed to run it against 2 libvirt instances and so far, the results are good. The allocations are created successfully. Note that I had to hack the inventory values a bit due to dev setup being smaller than production. But allocation values are ok otherwise. Next step for me will be to make sure inventory is fine in prod when migrating to placement (like ratios). But that's a different story. | |
| 19:22:20 | melwitt | jaypipes: http://i0.kym-cdn.com/photos/images/original/001/326/830/821.jpg | |
| 19:23:20 | mgagne | they can learn pretty fast how to spell park... | |
| 19:25:03 | mriedem | jmlowe: using OSC CLI? | |
| 19:25:19 | jmlowe | magnum actually | |
| 19:25:30 | mriedem | jmlowe: is magnum creating the server group? | |
| 19:25:37 | jmlowe | afaik | |
| 19:25:57 | mriedem | jmlowe: then they probably aren't specifying at least microversion 2.15: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id13 | |
| 19:27:55 | jmlowe | I have this https://review.openstack.org/#/c/436447/ | |
| 19:28:08 | jmlowe | so I think they think they are using 2.15 | |
| 19:28:21 | mriedem | mgagne: heh, i remember we used to have to spell everything out when talking about that | |
| 19:28:30 | mriedem | kids and dogs, they are the same | |
| 19:28:51 | mgagne | my parents used english =) | |
| 19:29:23 | mriedem | jmlowe: the testing on that patch is spectacular | |
| 19:29:44 | jmlowe | .... | |
| 19:30:09 | mriedem | having said that, i don't speak heat code so not sure what you're seeing for failures | |
| 19:30:11 | jmlowe | my mother did teach me that if I have nothing nice to say | |
| 19:30:12 | mriedem | check the nova-api logs? | |
| 19:30:25 | mriedem | jmlowe: if the server group was created successfully, then that's not the issue | |
| 19:30:31 | mriedem | do you have the proper scheduler filters enabled? | |
| 19:30:57 | mriedem | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#servergroupaffinityfilter | |
| 19:31:04 | mriedem | note the affinity filters are enabled by default, | |
| 19:31:06 | mriedem | as is the soft weigher | |
| 19:31:21 | mriedem | https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#id15 | |
| 19:31:25 | jaypipes | melwitt: :) | |
| 19:32:11 | mriedem | god our docs are confusing for people new to nova and cells v2 | |
| 19:32:12 | mriedem | "If cells are used, cells are weighted by the scheduler in the same manner as hosts." | |
| 19:32:14 | jmlowe | http://paste.openstack.org/show/723412/ | |
| 19:32:30 | jmlowe | I should have the defaults | |
| 19:32:32 | mriedem | we should asterisk all 'cells' mentions in the docs :) | |
| 19:32:57 | melwitt | bugathon candidate | |