| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-16 | |||
| 16:02:36 | dansmith | maybe yeah | |
| 16:03:29 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/845896 but its not merged so its not that | |
| 16:04:55 | dansmith | okay, so it's not a regression, or at least, not a recent one | |
| 16:05:13 | dansmith | n-cpu memory usage after a single-node run is ~500m, but after a multinode one, it's 1.5G to 2.5G | |
| 16:05:23 | dansmith | so I wonder if we're buffering something in memory due to live migration or something | |
| 16:05:34 | sean-k-mooney | do you have swap awailable | |
| 16:05:55 | dansmith | what does it matter? it' | |
| 16:06:06 | dansmith | it's not OOMing, we're just noticing it's much larger | |
| 16:07:02 | sean-k-mooney | just tha ti saw high memory usage in the past when swap was disabled | |
| 16:07:11 | sean-k-mooney | that did not happen when it was available | |
| 16:07:23 | sean-k-mooney | but i guess you have the saem config in singel vs multinode | |
| 16:07:35 | dansmith | yeah, these are CI workers | |
| 16:07:56 | sean-k-mooney | so is it that somehow adding a second compute node increase memory usage on the first | |
| 16:08:17 | sean-k-mooney | like if you enable a multi node install but idsable nova-cpu on the subnode | |
| 16:08:20 | sean-k-mooney | does it still happen | |
| 16:08:25 | dansmith | I was thinking more like in live migration we read in the configdrive to memory or something | |
| 16:08:36 | dansmith | you know how python increases the water mark but basically never decreases it | |
| 16:09:05 | sean-k-mooney | hum ok maybe | |
| 16:09:20 | sean-k-mooney | you could confirm that by disabling move operations tests | |
| 16:09:47 | sean-k-mooney | the config drive is pretty small but maybe | |
| 16:09:56 | dansmith | just an example | |
| 16:10:16 | sean-k-mooney | ya im jsut trying to think what else is like that | |
| 16:10:41 | sean-k-mooney | is the memory usage the same on both hosts? | |
| 16:10:49 | dansmith | I don't collect the other so dunno | |
| 16:13:35 | sean-k-mooney | would it be hard to add that | |
| 16:14:02 | sean-k-mooney | do you have the link to the ci job by the way | |
| 16:16:17 | dansmith | it's all the jobs, there's no need to look at one specifically | |
| 16:16:36 | dansmith | I thought it was recent, but I compared to some old snaps I had and they were high then too, but only for multinode | |
| 16:36:22 | gibi | sean-k-mooney, bauzas, artom: on master we still reject the evacuate request if the instance is in transient state (i.e. powering-off) https://paste.opendev.org/show/bQphEfOf8eLBnM6XmleQ/ | |
| 16:37:23 | gibi | so for me this seem like a valid request to evacuate the instance in this state | |
| 16:38:47 | sean-k-mooney | its not safe nessisarly to do by default | |
| 16:39:02 | gibi | even if the compute is forced down? | |
| 16:39:18 | sean-k-mooney | if its force down its oke | |
| 16:39:21 | sean-k-mooney | but only because | |
| 16:39:27 | sean-k-mooney | the operator is | |
| 16:39:33 | sean-k-mooney | promising they have fenced it | |
| 16:39:36 | gibi | yepp | |
| 16:39:44 | sean-k-mooney | but normlaly evacuate without force down would not be safe | |
| 16:39:44 | gibi | btw, reset-state --active helps https://paste.opendev.org/show/bVI7D8H5g9Oqjjo4rKfk/ | |
| 16:39:57 | sean-k-mooney | we cant tell that the vm is not running | |
| 16:40:00 | gibi | yepp in generate evacuate without fencing is unsafe in non transient states too | |
| 16:40:10 | gibi | s/generate/general/ | |
| 16:40:22 | sean-k-mooney | so ya if we check both then we can ignore the task state | |
| 16:40:35 | sean-k-mooney | *both-> force-down | |
| 16:40:44 | sean-k-mooney | so question | |
| 16:40:53 | sean-k-mooney | does the decorator support ignoring task state | |
| 16:42:29 | gibi | so on master I'm OK to ignore the task state | |
| 16:42:51 | gibi | downstream I would suggest the customer to use the reset-state, so no code changes is needed there | |
| 16:43:16 | sean-k-mooney | reset_state only resets vm state not task state right | |
| 16:43:27 | sean-k-mooney | or dose it force task_state=None | |
| 16:43:32 | artom | IIRC it can do both | |
| 16:43:44 | sean-k-mooney | well you cant reset taskstate with it | |
| 16:43:57 | sean-k-mooney | configurably | |
| 16:44:25 | sean-k-mooney | i.e. reset-state vm-sate=active task-state=none | |
| 16:44:33 | sean-k-mooney | but i dont know if task-state=none | |
| 16:44:38 | sean-k-mooney | is hardcoded internally | |
| 16:45:27 | gibi | sean-k-mooney: reset-state resets the task state too | |
| 16:45:46 | sean-k-mooney | then ya it should be good | |
| 16:46:07 | sean-k-mooney | it would be nice if we could add task-state to the api in the futrue | |
| 16:46:09 | sean-k-mooney | https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/api/openstack/compute/schemas/reset_server_state.py#L15 | |
| 16:46:21 | sean-k-mooney | but for this case we shoudl be good | |
| 16:47:25 | sean-k-mooney | https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/api/openstack/compute/admin_actions.py#L77-L78= | |
| 16:48:09 | gibi | all good | |
| 16:48:10 | gibi | :)\ | |
| 16:48:28 | sean-k-mooney | gibi: are you going to file an upstream bug for the transient state | |
| 16:49:00 | gibi | sure I can I have the repro ready. But I won't assign it to myself to fix it :) | |
| 16:49:15 | gibi | at least not right now | |
| 16:58:54 | gibi | sean-k-mooney, artom, bauzas https://bugs.launchpad.net/nova/+bug/1978983 | |
| 17:00:12 | gibi | sean-k-mooney: btw, I added the pci inventory reporting to the resource tracker level https://review.opendev.org/c/openstack/nova/+/846187 it does not need to be in the virt layer so I did not force to it | |
| 17:00:14 | artom | Nice | |
| 17:00:27 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: fake: Ensure need_legacy_block_device_info returns False https://review.opendev.org/c/openstack/nova/+/843950 | |
| 17:00:28 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843951 | |
| 17:00:29 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: compute: Ensure updates to bdms during pre_live_migration are saved https://review.opendev.org/c/openstack/nova/+/843952 | |
| 17:00:30 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: fup: Make connection_info returned by CinderFixture unique per attachment https://review.opendev.org/c/openstack/nova/+/844606 | |
| 17:00:31 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: fup: Assert state of connection_info during LM rollback in func tests https://review.opendev.org/c/openstack/nova/+/844607 | |
| 17:00:33 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: functional: Use tempdir for CONF.instances_path https://review.opendev.org/c/openstack/nova/+/846204 | |
| 17:03:09 | sean-k-mooney | gibi: cool just triaged it as medium since you have repoduced it and there is a workaround | |
| 17:03:18 | gibi | sean-k-mooney: thanks | |
| 17:03:58 | sean-k-mooney | by the way by doing it at the compute manager level that shoudl work for all virt drivers which is nice | |
| 17:04:32 | gibi | yepp that also a + | |
| 17:04:44 | sean-k-mooney | we will likely only test with libvirt but if someone was to test with powervm and it worked they could update the docs | |
| 17:04:56 | gibi | yes | |
| 17:11:40 | opendevreview | Gorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (1/3) https://review.opendev.org/c/openstack/nova/+/846207 | |
| 17:11:40 | opendevreview | Gorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (2/3) https://review.opendev.org/c/openstack/nova/+/846208 | |
| 17:11:41 | opendevreview | Gorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (3/3) https://review.opendev.org/c/openstack/nova/+/846209 | |
| 17:12:03 | geguileo | dansmith: sean-k-mooney ^ Those are the test patches to see if we can figure something out... | |
| 17:12:31 | geguileo | fist is doing nothing, next is trimming malloc memory, last is trimming and setting malloc arenas | |
| 17:14:55 | opendevreview | Gorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (1/3) https://review.opendev.org/c/openstack/nova/+/846207 | |
| 17:14:56 | opendevreview | Gorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (2/3) https://review.opendev.org/c/openstack/nova/+/846208 | |
| 17:14:56 | opendevreview | Gorka Eguileor proposed openstack/nova master: DNM: Testing memory usage of n-cpu (3/3) https://review.opendev.org/c/openstack/nova/+/846209 | |
| 17:16:17 | dansmith | geguileo: I believe no human has ever gotten a zuul config change correct on the first try | |
| 17:20:34 | sean-k-mooney | so you are directly calling into mallac to free memory | |
| 17:20:41 | sean-k-mooney | thats interesting | |
| 17:20:51 | sean-k-mooney | sory forgot the ... | |
| 17:56:26 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: functional: Use tempdir for CONF.instances_path https://review.opendev.org/c/openstack/nova/+/846204 | |
| 17:56:27 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: Add a regression test for bug 1939545 https://review.opendev.org/c/openstack/nova/+/843951 | |
| 17:56:28 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: compute: Ensure updates to bdms during pre_live_migration are saved https://review.opendev.org/c/openstack/nova/+/843952 | |
| 17:56:29 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: fup: Make connection_info returned by CinderFixture unique per attachment https://review.opendev.org/c/openstack/nova/+/844606 | |
| 17:56:31 | opendevreview | Artom Lifshitz proposed openstack/nova stable/ussuri: fup: Assert state of connection_info during LM rollback in func tests https://review.opendev.org/c/openstack/nova/+/844607 | |
| 18:29:23 | opendevreview | Balazs Gibizer proposed openstack/nova master: Report PCI inventory to placement basics https://review.opendev.org/c/openstack/nova/+/846187 | |