| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-01 | |||
| 17:14:00 | stephenfin | I can raise one if you haven't | |
| 17:15:51 | lyarwood | no issues, I'll do it now | |
| 17:21:05 | openstack | Launchpad bug 1906428 in tempest "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Undecided,New] | |
| 17:21:05 | lyarwood | stephenfin: https://bugs.launchpad.net/tempest/+bug/1906428 | |
| 18:10:04 | openstackgerrit | Merged openstack/nova master: remove python warnning from tox https://review.opendev.org/c/openstack/nova/+/752190 | |
| 19:14:15 | sean-k-mooney | bauzas: gibi stephenfin by the way now that provider.yaml is a thing we shoudl update https://docs.openstack.org/nova/latest/admin/virtual-gpu.html#optional-provide-custom-traits-for-multiple-gpu-types to descibe how to use that to add teh traits tor the PGPU RPs | |
| 20:00:45 | openstackgerrit | sean mooney proposed openstack/nova-specs master: libvirt: support vdpa kernel networking https://review.opendev.org/c/openstack/nova-specs/+/764999 | |
| 23:53:08 | openstackgerrit | Merged openstack/nova stable/train: Change default num_retries for glance to 3 https://review.opendev.org/c/openstack/nova/+/762610 | |
| 23:53:26 | openstackgerrit | Merged openstack/nova stable/victoria: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/758112 | |
| #openstack-nova - 2020-12-02 | |||
| 00:23:24 | openstackgerrit | Keigo Noha proposed openstack/nova stable/stein: Change default num_retries for glance to 3 https://review.opendev.org/c/openstack/nova/+/764963 | |
| 01:18:49 | openstackgerrit | Jorhson Deng proposed openstack/nova-specs master: Remove RPC call to ensure volume attachment successful https://review.opendev.org/c/openstack/nova-specs/+/758920 | |
| 01:41:19 | openstackgerrit | Jorhson Deng proposed openstack/nova-specs master: Remove RPC call to ensure volume attachment successful https://review.opendev.org/c/openstack/nova-specs/+/758920 | |
| 01:57:50 | chengsheng1 | I am a new contributor. My code has been submitted for a long time, but there is no review. I ask for help if there is a problem with which process. https://review.opendev.org/c/openstack/nova/+/762330/ | |
| 03:57:33 | melwitt | kashyap, sean-k-mooney: patch you have reviewed in the past has been updated recently ^ | |
| 06:51:00 | openstackgerrit | Wenping Song proposed openstack/nova master: Nova supports password encrypted VNC https://review.opendev.org/c/openstack/nova/+/622336 | |
| 08:15:19 | openstackgerrit | Xinran WANG proposed openstack/nova-specs master: Smart-nic Support Specification https://review.opendev.org/c/openstack/nova-specs/+/742785 | |
| 08:45:41 | openstackgerrit | Adit Sarfaty proposed openstack/nova stable/train: Retry on vmware create_vm when it fails https://review.opendev.org/c/openstack/nova/+/764515 | |
| 09:14:10 | kashyap | melwitt: chengsheng1: Hi, looking | |
| 09:21:08 | kashyap | chengsheng1: There's no problem with process; just getting around to it. A quick comment here, but I'll respond in detail on the patch | |
| 09:21:54 | kashyap | chengsheng1: Along with compare_hypervisor_cpu() the spec I wrote also suggests to introduce a baseline_hypervisor_cpu() method | |
| 09:22:24 | kashyap | chengsheng1: I wonder if you looked into it; it's okay if you haven't yet, we can sort out the details on the review. | |
| 09:39:38 | chengsheng1 | kashyap: If the baseline_hypervisor_cpu() method is necessary, I can modify the code. | |
| 09:40:17 | kashyap | chengsheng1: I'll comment on the change. I see you've updated the existing get_capabilities() method w/ the newer baselineHypervisor... I don't think we can just replace the older API just like that. | |
| 09:43:18 | chengsheng1 | kashyap: This modification has a relatively small impact on the overall code. | |
| 09:44:02 | kashyap | chengsheng1: Yes, I see that. Did you also get to do any real live migration tests? | |
| 09:46:11 | chengsheng1 | kashyap: Yes, I did tests in several real environments, and it also solved the problems I encountered. | |
| 09:48:01 | kashyap | chengsheng1: Ah, cool. | |
| 09:55:50 | openstack | Launchpad bug 1906494 in OpenStack Compute (nova) "Placement error when using GPUs that is utilizing SR-IOV for VGPU" [Undecided,New] | |
| 09:55:50 | LarsErikP | Hi guys. I've just submitted a bug about how nova treats GPUs that's using SR-IOV for VGPUs (mainly the Nvidia A100). It's kinda weird.. https://bugs.launchpad.net/nova/+bug/1906494 hope you can figure something out :-) | |
| 10:06:47 | gibi | LarsErikP: seems like a valid bug. Nova assumes that an inventory total can be set to 0 in placmenet, but placement does not allow that | |
| 10:06:58 | gibi | bauzas: ^^ | |
| 10:07:11 | gibi | bauzas: it is about VGPU:total=0 | |
| 10:09:47 | LarsErikP | gibi: thanks for the quick feedback! | |
| 10:11:19 | gibi | LarsErikP: I need bauzas to confirm it more from vgpu perspective, but If we need to model 0 inventory then either we have to remove the whole resource class from the inventory, or keep the total value positive but increase the reserved value to express that there is no free VGPU resource left | |
| 10:13:24 | LarsErikP | I see. I also whish there was someway to workaround this issue while waiting for a fix.. But I can't really think of anything smart | |
| 10:14:33 | gibi | right now, me neither. I guess if you manually fix the placement inventory be removing the VGPU rc from the RPs then nova-compute tries to put it back with total=0 and still fail | |
| 10:14:48 | gibi | s/be/by | |
| 10:14:52 | LarsErikP | already tried that :P | |
| 10:15:02 | gibi | :) | |
| 10:15:36 | gibi | I don't want to suggest yout to hack the placment code to allow total=0 | |
| 10:16:31 | gibi | but the schema validation that fails is here https://github.com/openstack/placement/blob/c02a073c523d363d7136677ab12884dc4ec03e6f/placement/schemas/inventory.py#L29 | |
| 10:30:04 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: DNM/WIP Remove device from attach_volume request response https://review.opendev.org/c/openstack/nova-specs/+/765097 | |
| 10:47:47 | openstack | Launchpad bug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,Confirmed] | |
| 10:47:47 | stephenfin | man, https://bugs.launchpad.net/tempest/+bug/1906428 appear to have pretty much broken the gate :-( | |
| 10:48:01 | stephenfin | and I can't figure out what's changed | |
| 10:49:18 | lyarwood | stephenfin: yeah it's weird, I don't understand how the test ever worked prior to that change anyway tbh | |
| 10:49:50 | lyarwood | oh wait! | |
| 10:49:53 | stephenfin | What's the idea behind? That you'd create a file and it would still exist after you'd stashed, offloaded and unstashed | |
| 10:50:03 | lyarwood | stephenfin: I don't think we offload | |
| 10:50:21 | lyarwood | stephenfin: doesn't that just pause and unpause | |
| 10:50:31 | lyarwood | stephenfin: so the mount within the guestOS persists | |
| 10:50:43 | lyarwood | stephenfin: but now as we cold migrate we essentially reboot the instance clearing the mount | |
| 10:51:06 | stephenfin | I see "Shelve offloading" followed by "Instance destroyed successfully" in the log I'm looking at | |
| 10:51:12 | lyarwood | ah kk | |
| 10:51:18 | stephenfin | search for tempest-TestShelveInstance-307565458 here https://zuul.opendev.org/t/openstack/build/41da7c0de7994dddb641c0a783b53c70/log/compute1/logs/screen-n-cpu.txt | |
| 10:51:25 | lyarwood | if it's dead then I'm back to not understanding how this ever worked | |
| 10:51:40 | lyarwood | unless verification was off in the job | |
| 10:51:49 | lyarwood | so these checks were skipped | |
| 10:51:58 | openstack | Launchpad bug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,Confirmed] | |
| 10:51:58 | stephenfin | aarents: Have you seen https://bugs.launchpad.net/tempest/+bug/1906428 ? | |
| 10:53:04 | lyarwood | https://review.opendev.org/c/openstack/tempest/+/765079 I've posted a revert | |
| 10:53:14 | lyarwood | brb | |
| 10:53:27 | stephenfin | I see 186 hits. Nice | |
| 10:54:08 | stephenfin | https://review.opendev.org/c/opendev/elastic-recheck/+/765108 | |
| 10:56:43 | stephenfin | Oh, I didn't know that that patch only merged two days ago /o\ | |
| 10:57:02 | stephenfin | I was looking at the author date in Git and thought something had regressed | |
| 10:57:08 | stephenfin | Yeah, let's yank that for sure | |
| 10:58:47 | gibi | lyarwood, stephenfin: nice teamwork on this gate blocker | |
| 11:08:49 | lyarwood | np | |
| 11:11:25 | masterpe | Hi, I try to live migrate a instance to a specific host. But I get the error: "There is not enough capacity on the host for the instance." but on the "openstack hypervisor show" gives me enough resources left. In the allocations table of database nova_api I see some ghost resources. What can I do to fix this? I'm running Train. | |
| 11:15:57 | stephenfin | masterpe: Don't trust os-hypervisors. It lies. Placement is where you want to get your information https://specs.openstack.org/openstack/nova-specs/specs/wallaby/approved/modernize-os-hypervisors-api.html | |
| 11:16:08 | stephenfin | What do you mean by ghost resources? | |
| 11:16:18 | lyarwood | https://review.opendev.org/c/openstack/nova/+/720838 - huh we didn't backport this in the end | |
| 11:16:38 | stephenfin | If placement's inventory is wrong, you'll want to use nova-manage to handle things https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement | |
| 11:16:47 | lyarwood | stephenfin: that isn't in train | |
| 11:16:50 | stephenfin | oh, jinx | |
| 11:16:58 | stephenfin | damn :( | |
| 11:17:22 | lyarwood | brb again | |
| 11:17:28 | stephenfin | topic for discussion in tomorrow's meeting? gibi ^ | |
| 11:23:28 | masterpe | stephenfin: I see UUID of instances that I can not find anymore with "openstack server show" | |
| 11:24:35 | openstackgerrit | Merged openstack/nova-specs master: libvirt: Store and allow the default machine type to be changed https://review.opendev.org/c/openstack/nova-specs/+/762199 | |
| 11:37:20 | gibi | stephenfin: you mean backporting placement audit to stable branches? | |
| 11:37:25 | stephenfin | yes | |
| 11:38:11 | gibi | would be good to know why we abandoned the backports | |
| 11:41:38 | gibi | anyhow I can add it to tomorrow's agenda | |
| 11:58:46 | aarents | stephenfin lyarwood gibi Hi sorry, I'm blocked in a block in some training sessions this week, this issue is odd.. because it is working in tempest-slow-py3 https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/job-output.txt | |
| 12:05:21 | lyarwood | aarents: yeah odd, it's calling cat on the mount and that appears to be passing | |
| 12:05:23 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35793 | |
| 12:05:47 | aarents | for me /mnt is part of '/' there is no mount | |
| 12:05:51 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35623 | |
| 12:05:56 | lyarwood | yeah | |
| 12:06:00 | lyarwood | was about to say | |
| 12:13:57 | aarents | may be patch not working in multi-cell context, need to check logs, may be we can put it in blacklist of nova-multi-cell instead of revert, the time to check. | |
| 12:55:09 | lyarwood | aarents / stephenfin ; right this is a cross cell resize in the context of the job | |
| 12:55:54 | lyarwood | this smells like a bug with the cross cell resize flow tbh | |
| 13:00:30 | aarents | lyarwood: hum something like a rebuild from base image instead scp the disk ? | |
| 13:00:56 | gibi | I think cross cell resize do a snapshot instead of scp | |
| 13:01:22 | lyarwood | yeah it's a cold snapshot | |
| 13:03:34 | openstackgerrit | Takashi Natsume proposed openstack/nova stable/ussuri: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/765082 | |
| 13:04:54 | lyarwood | I don't think it's being used however | |