| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-02 | |||
| 09:43:18 | chengsheng1 | kashyap: This modification has a relatively small impact on the overall code. | |
| 09:44:02 | kashyap | chengsheng1: Yes, I see that. Did you also get to do any real live migration tests? | |
| 09:46:11 | chengsheng1 | kashyap: Yes, I did tests in several real environments, and it also solved the problems I encountered. | |
| 09:48:01 | kashyap | chengsheng1: Ah, cool. | |
| 09:55:50 | openstack | Launchpad bug 1906494 in OpenStack Compute (nova) "Placement error when using GPUs that is utilizing SR-IOV for VGPU" [Undecided,New] | |
| 09:55:50 | LarsErikP | Hi guys. I've just submitted a bug about how nova treats GPUs that's using SR-IOV for VGPUs (mainly the Nvidia A100). It's kinda weird.. https://bugs.launchpad.net/nova/+bug/1906494 hope you can figure something out :-) | |
| 10:06:47 | gibi | LarsErikP: seems like a valid bug. Nova assumes that an inventory total can be set to 0 in placmenet, but placement does not allow that | |
| 10:06:58 | gibi | bauzas: ^^ | |
| 10:07:11 | gibi | bauzas: it is about VGPU:total=0 | |
| 10:09:47 | LarsErikP | gibi: thanks for the quick feedback! | |
| 10:11:19 | gibi | LarsErikP: I need bauzas to confirm it more from vgpu perspective, but If we need to model 0 inventory then either we have to remove the whole resource class from the inventory, or keep the total value positive but increase the reserved value to express that there is no free VGPU resource left | |
| 10:13:24 | LarsErikP | I see. I also whish there was someway to workaround this issue while waiting for a fix.. But I can't really think of anything smart | |
| 10:14:33 | gibi | right now, me neither. I guess if you manually fix the placement inventory be removing the VGPU rc from the RPs then nova-compute tries to put it back with total=0 and still fail | |
| 10:14:48 | gibi | s/be/by | |
| 10:14:52 | LarsErikP | already tried that :P | |
| 10:15:02 | gibi | :) | |
| 10:15:36 | gibi | I don't want to suggest yout to hack the placment code to allow total=0 | |
| 10:16:31 | gibi | but the schema validation that fails is here https://github.com/openstack/placement/blob/c02a073c523d363d7136677ab12884dc4ec03e6f/placement/schemas/inventory.py#L29 | |
| 10:30:04 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: DNM/WIP Remove device from attach_volume request response https://review.opendev.org/c/openstack/nova-specs/+/765097 | |
| 10:47:47 | openstack | Launchpad bug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,Confirmed] | |
| 10:47:47 | stephenfin | man, https://bugs.launchpad.net/tempest/+bug/1906428 appear to have pretty much broken the gate :-( | |
| 10:48:01 | stephenfin | and I can't figure out what's changed | |
| 10:49:18 | lyarwood | stephenfin: yeah it's weird, I don't understand how the test ever worked prior to that change anyway tbh | |
| 10:49:50 | lyarwood | oh wait! | |
| 10:49:53 | stephenfin | What's the idea behind? That you'd create a file and it would still exist after you'd stashed, offloaded and unstashed | |
| 10:50:03 | lyarwood | stephenfin: I don't think we offload | |
| 10:50:21 | lyarwood | stephenfin: doesn't that just pause and unpause | |
| 10:50:31 | lyarwood | stephenfin: so the mount within the guestOS persists | |
| 10:50:43 | lyarwood | stephenfin: but now as we cold migrate we essentially reboot the instance clearing the mount | |
| 10:51:06 | stephenfin | I see "Shelve offloading" followed by "Instance destroyed successfully" in the log I'm looking at | |
| 10:51:12 | lyarwood | ah kk | |
| 10:51:18 | stephenfin | search for tempest-TestShelveInstance-307565458 here https://zuul.opendev.org/t/openstack/build/41da7c0de7994dddb641c0a783b53c70/log/compute1/logs/screen-n-cpu.txt | |
| 10:51:25 | lyarwood | if it's dead then I'm back to not understanding how this ever worked | |
| 10:51:40 | lyarwood | unless verification was off in the job | |
| 10:51:49 | lyarwood | so these checks were skipped | |
| 10:51:58 | openstack | Launchpad bug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,Confirmed] | |
| 10:51:58 | stephenfin | aarents: Have you seen https://bugs.launchpad.net/tempest/+bug/1906428 ? | |
| 10:53:04 | lyarwood | https://review.opendev.org/c/openstack/tempest/+/765079 I've posted a revert | |
| 10:53:14 | lyarwood | brb | |
| 10:53:27 | stephenfin | I see 186 hits. Nice | |
| 10:54:08 | stephenfin | https://review.opendev.org/c/opendev/elastic-recheck/+/765108 | |
| 10:56:43 | stephenfin | Oh, I didn't know that that patch only merged two days ago /o\ | |
| 10:57:02 | stephenfin | I was looking at the author date in Git and thought something had regressed | |
| 10:57:08 | stephenfin | Yeah, let's yank that for sure | |
| 10:58:47 | gibi | lyarwood, stephenfin: nice teamwork on this gate blocker | |
| 11:08:49 | lyarwood | np | |
| 11:11:25 | masterpe | Hi, I try to live migrate a instance to a specific host. But I get the error: "There is not enough capacity on the host for the instance." but on the "openstack hypervisor show" gives me enough resources left. In the allocations table of database nova_api I see some ghost resources. What can I do to fix this? I'm running Train. | |
| 11:15:57 | stephenfin | masterpe: Don't trust os-hypervisors. It lies. Placement is where you want to get your information https://specs.openstack.org/openstack/nova-specs/specs/wallaby/approved/modernize-os-hypervisors-api.html | |
| 11:16:08 | stephenfin | What do you mean by ghost resources? | |
| 11:16:18 | lyarwood | https://review.opendev.org/c/openstack/nova/+/720838 - huh we didn't backport this in the end | |
| 11:16:38 | stephenfin | If placement's inventory is wrong, you'll want to use nova-manage to handle things https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement | |
| 11:16:47 | lyarwood | stephenfin: that isn't in train | |
| 11:16:50 | stephenfin | oh, jinx | |
| 11:16:58 | stephenfin | damn :( | |
| 11:17:22 | lyarwood | brb again | |
| 11:17:28 | stephenfin | topic for discussion in tomorrow's meeting? gibi ^ | |
| 11:23:28 | masterpe | stephenfin: I see UUID of instances that I can not find anymore with "openstack server show" | |
| 11:24:35 | openstackgerrit | Merged openstack/nova-specs master: libvirt: Store and allow the default machine type to be changed https://review.opendev.org/c/openstack/nova-specs/+/762199 | |
| 11:37:20 | gibi | stephenfin: you mean backporting placement audit to stable branches? | |
| 11:37:25 | stephenfin | yes | |
| 11:38:11 | gibi | would be good to know why we abandoned the backports | |
| 11:41:38 | gibi | anyhow I can add it to tomorrow's agenda | |
| 11:58:46 | aarents | stephenfin lyarwood gibi Hi sorry, I'm blocked in a block in some training sessions this week, this issue is odd.. because it is working in tempest-slow-py3 https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/job-output.txt | |
| 12:05:21 | lyarwood | aarents: yeah odd, it's calling cat on the mount and that appears to be passing | |
| 12:05:23 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35793 | |
| 12:05:47 | aarents | for me /mnt is part of '/' there is no mount | |
| 12:05:51 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35623 | |
| 12:05:56 | lyarwood | yeah | |
| 12:06:00 | lyarwood | was about to say | |
| 12:13:57 | aarents | may be patch not working in multi-cell context, need to check logs, may be we can put it in blacklist of nova-multi-cell instead of revert, the time to check. | |
| 12:55:09 | lyarwood | aarents / stephenfin ; right this is a cross cell resize in the context of the job | |
| 12:55:54 | lyarwood | this smells like a bug with the cross cell resize flow tbh | |
| 13:00:30 | aarents | lyarwood: hum something like a rebuild from base image instead scp the disk ? | |
| 13:00:56 | gibi | I think cross cell resize do a snapshot instead of scp | |
| 13:01:22 | lyarwood | yeah it's a cold snapshot | |
| 13:03:34 | openstackgerrit | Takashi Natsume proposed openstack/nova stable/ussuri: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/765082 | |
| 13:04:54 | lyarwood | I don't think it's being used however | |
| 13:11:32 | aarents | I can see rebase is done after unshelve so instance disk should be fine before resize(cold snap) Running cmd (subprocess): qemu-img rebase -b /opt/stack/data/nova/instances/_base/30b9e4f95a51c37d278a2a0a447a270519e2be92 -F raw /opt/stack/data/nova/instances/9616dc4e-c2b8-46be-bf5c-6de9a11330f8/disk | |
| 13:18:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 13:30:28 | openstackgerrit | Aleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout https://review.opendev.org/c/openstack/nova/+/764927 | |
| 13:32:07 | lyarwood | aarents: yeah the spawn is using the original image | |
| 13:32:19 | lyarwood | aarents: this is a bug in the cross cell resize flow somewhere | |
| 13:32:25 | lyarwood | aarents: http://paste.openstack.org/show/800633/ | |
| 13:32:42 | lyarwood | aarents: ^ that's the qemu-img commands we run, we use the same base as the original spawn | |
| 13:33:14 | lyarwood | oh wait because we rebase | |
| 13:33:25 | lyarwood | during unshelve | |
| 13:33:33 | lyarwood | sorry I'm getting all confused | |
| 13:36:02 | lyarwood | ah but it's recreating the overlay | |
| 13:40:54 | aarents | lyarwood how it recreates the overlay ? | |
| 13:41:57 | lyarwood | aarents: I think it's doing it based on the image_meta we pass into driver.spawn here https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L6015-L6019 | |
| 13:42:09 | lyarwood | aarents: but I'm not sure why that isn't the snapshot image_meta at this point | |
| 13:42:21 | lyarwood | aarents: brb | |
| 13:52:01 | aarents | Yeah it should be the same code than in unshelve where it should use image-id from snapshot(stored in system_metada) | |
| 13:52:04 | aarents | https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/conductor/manager.py#L932 | |
| 13:59:52 | aarents | ha it is here where it should take the snapshot_id https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L5906 | |
| 14:01:57 | lyarwood | aarents: right but in this case it looks like snapshot_id wasn't passed and we fallback to the instance.image_meta | |
| 14:02:07 | lyarwood | aarents: at least that's what I'm assuming | |
| 14:02:28 | lyarwood | Let me blacklist this test in the multi cell job for now and throw up another DNM change adding some debug around this flow | |
| 14:03:15 | aarents | lyarwood: agree | |
| 14:06:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |