| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-02 | |||
| 11:15:57 | stephenfin | masterpe: Don't trust os-hypervisors. It lies. Placement is where you want to get your information https://specs.openstack.org/openstack/nova-specs/specs/wallaby/approved/modernize-os-hypervisors-api.html | |
| 11:16:08 | stephenfin | What do you mean by ghost resources? | |
| 11:16:18 | lyarwood | https://review.opendev.org/c/openstack/nova/+/720838 - huh we didn't backport this in the end | |
| 11:16:38 | stephenfin | If placement's inventory is wrong, you'll want to use nova-manage to handle things https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement | |
| 11:16:47 | lyarwood | stephenfin: that isn't in train | |
| 11:16:50 | stephenfin | oh, jinx | |
| 11:16:58 | stephenfin | damn :( | |
| 11:17:22 | lyarwood | brb again | |
| 11:17:28 | stephenfin | topic for discussion in tomorrow's meeting? gibi ^ | |
| 11:23:28 | masterpe | stephenfin: I see UUID of instances that I can not find anymore with "openstack server show" | |
| 11:24:35 | openstackgerrit | Merged openstack/nova-specs master: libvirt: Store and allow the default machine type to be changed https://review.opendev.org/c/openstack/nova-specs/+/762199 | |
| 11:37:20 | gibi | stephenfin: you mean backporting placement audit to stable branches? | |
| 11:37:25 | stephenfin | yes | |
| 11:38:11 | gibi | would be good to know why we abandoned the backports | |
| 11:41:38 | gibi | anyhow I can add it to tomorrow's agenda | |
| 11:58:46 | aarents | stephenfin lyarwood gibi Hi sorry, I'm blocked in a block in some training sessions this week, this issue is odd.. because it is working in tempest-slow-py3 https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/job-output.txt | |
| 12:05:21 | lyarwood | aarents: yeah odd, it's calling cat on the mount and that appears to be passing | |
| 12:05:23 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35793 | |
| 12:05:47 | aarents | for me /mnt is part of '/' there is no mount | |
| 12:05:51 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35623 | |
| 12:05:56 | lyarwood | yeah | |
| 12:06:00 | lyarwood | was about to say | |
| 12:13:57 | aarents | may be patch not working in multi-cell context, need to check logs, may be we can put it in blacklist of nova-multi-cell instead of revert, the time to check. | |
| 12:55:09 | lyarwood | aarents / stephenfin ; right this is a cross cell resize in the context of the job | |
| 12:55:54 | lyarwood | this smells like a bug with the cross cell resize flow tbh | |
| 13:00:30 | aarents | lyarwood: hum something like a rebuild from base image instead scp the disk ? | |
| 13:00:56 | gibi | I think cross cell resize do a snapshot instead of scp | |
| 13:01:22 | lyarwood | yeah it's a cold snapshot | |
| 13:03:34 | openstackgerrit | Takashi Natsume proposed openstack/nova stable/ussuri: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/765082 | |
| 13:04:54 | lyarwood | I don't think it's being used however | |
| 13:11:32 | aarents | I can see rebase is done after unshelve so instance disk should be fine before resize(cold snap) Running cmd (subprocess): qemu-img rebase -b /opt/stack/data/nova/instances/_base/30b9e4f95a51c37d278a2a0a447a270519e2be92 -F raw /opt/stack/data/nova/instances/9616dc4e-c2b8-46be-bf5c-6de9a11330f8/disk | |
| 13:18:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 13:30:28 | openstackgerrit | Aleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout https://review.opendev.org/c/openstack/nova/+/764927 | |
| 13:32:07 | lyarwood | aarents: yeah the spawn is using the original image | |
| 13:32:19 | lyarwood | aarents: this is a bug in the cross cell resize flow somewhere | |
| 13:32:25 | lyarwood | aarents: http://paste.openstack.org/show/800633/ | |
| 13:32:42 | lyarwood | aarents: ^ that's the qemu-img commands we run, we use the same base as the original spawn | |
| 13:33:14 | lyarwood | oh wait because we rebase | |
| 13:33:25 | lyarwood | during unshelve | |
| 13:33:33 | lyarwood | sorry I'm getting all confused | |
| 13:36:02 | lyarwood | ah but it's recreating the overlay | |
| 13:40:54 | aarents | lyarwood how it recreates the overlay ? | |
| 13:41:57 | lyarwood | aarents: I think it's doing it based on the image_meta we pass into driver.spawn here https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L6015-L6019 | |
| 13:42:09 | lyarwood | aarents: but I'm not sure why that isn't the snapshot image_meta at this point | |
| 13:42:21 | lyarwood | aarents: brb | |
| 13:52:01 | aarents | Yeah it should be the same code than in unshelve where it should use image-id from snapshot(stored in system_metada) | |
| 13:52:04 | aarents | https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/conductor/manager.py#L932 | |
| 13:59:52 | aarents | ha it is here where it should take the snapshot_id https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L5906 | |
| 14:01:57 | lyarwood | aarents: right but in this case it looks like snapshot_id wasn't passed and we fallback to the instance.image_meta | |
| 14:02:07 | lyarwood | aarents: at least that's what I'm assuming | |
| 14:02:28 | lyarwood | Let me blacklist this test in the multi cell job for now and throw up another DNM change adding some debug around this flow | |
| 14:03:15 | aarents | lyarwood: agree | |
| 14:06:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 14:06:50 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-multi-cell: Skip test_cold_migrate_unshelved_instance https://review.opendev.org/c/openstack/nova/+/765141 | |
| 14:07:18 | lyarwood | stephenfin / gibi ; ^ okay lets try it this way, skipping the test in the job directly as it looks like an issue with cross cell resize. | |
| 14:11:28 | gibi | lyarwood: ack +2 | |
| 14:12:02 | aarents | +1 | |
| 14:18:02 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM - debug test_cold_migrate_unshelved_instance on nova-multi-cell https://review.opendev.org/c/openstack/nova/+/765145 | |
| 14:18:04 | lyarwood | et voilà | |
| 14:18:17 | lyarwood | all because I don't want to build an env for this locally | |
| 14:25:07 | aarents | lyarwood: https://review.opendev.org/c/openstack/nova/+/765145/1/nova/compute/manager.py#5906 | |
| 14:25:57 | lyarwood | aarents: https://realpython.com/python-f-strings/#f-strings-a-new-and-improved-way-to-format-strings-in-python | |
| 14:26:15 | aarents | lyarwood: thks | |
| 14:26:35 | lyarwood | np | |
| 14:32:44 | gibi | lyarwood: just note that fstring in LOG calls does not delay interpolation | |
| 14:33:38 | lyarwood | yup but in this situation I'm just dumping things to troubleshoot, not landing the changes in master :) | |
| 14:33:48 | gibi | sure | |
| 15:17:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 15:18:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py https://review.opendev.org/c/openstack/nova/+/765152 | |
| 15:22:35 | LarsErikP | gibi: found a workaround =) | |
| 15:23:02 | LarsErikP | gibi: I tweak nvidia's sriov-manage script from creating all the VFs the GPU is capable of, to only create two | |
| 15:23:23 | gibi | LarsErikP: nice | |
| 15:23:25 | LarsErikP | gibi: then I cleaned up the resource providers, and now everything is fine.. Kinda dirty, but it works.. | |
| 15:24:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make remove allocation symmetric with add allocation https://review.opendev.org/c/openstack/nova/+/757110 | |
| 15:24:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Factor out port resource deallocation https://review.opendev.org/c/openstack/nova/+/757111 | |
| 15:24:24 | LarsErikP | I've asked nvidia enterprise support if there is a "proper" way of doing this. The specifically say in the grid docs that you should _never_ use anything else than their script to enable the sr-iov functions | |
| 15:24:38 | LarsErikP | but they did no say I shouldn't edit their script... :P | |
| 15:24:50 | gibi | :D | |
| 15:27:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor update_pci_request_spec_with_allocated_interface_name https://review.opendev.org/c/openstack/nova/+/757614 | |
| 15:27:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 15:29:25 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py https://review.opendev.org/c/openstack/nova/+/765152 | |
| 15:35:29 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/752284 | |
| 15:41:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py https://review.opendev.org/c/openstack/nova/+/765152 | |
| 16:33:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Docs: correct cpu_thread_policy explanation https://review.opendev.org/c/openstack/nova/+/765172 | |
| 16:46:50 | lyarwood | gibi / aarents ; I'm going to be out on PTO for the rest of the week btw in regards to the cross cell resize | |
| 16:46:57 | lyarwood | bug* | |
| 16:47:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Docs: correct cpu_thread_policy explanation https://review.opendev.org/c/openstack/nova/+/765172 | |
| 16:48:20 | gibi | lyarwood: thanks for the heads up | |
| 16:48:41 | gibi | I will make sure the backlisting merges | |
| 16:48:51 | lyarwood | ack thanks, I can pick up the fix on Monday | |
| 16:48:59 | lyarwood | if aarents doesn't make progress | |
| 16:49:02 | gibi | have a nice time off | |
| 16:49:43 | lyarwood | lockdown ended today in England and my child is in nursery for the first time tomorrow, so don't worry I'll be having a nice time :D | |
| 16:49:47 | lyarwood | sleeping mostly | |
| 16:50:58 | gibi | sleeping is good | |
| 17:09:07 | kashyap | Yes, it adds years to life. (Cf. https://www.penguin.co.uk/books/295/295665/why-we-sleep/9780141983769.html) | |
| 17:09:34 | kashyap | (Counter-intuitively, enough ;-)) | |
| 17:18:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 17:34:46 | efried | +1 Why We Sleep -- life-changing read. | |
| 18:14:05 | gmann | gate is too slow today, this did not get the node yet. waiting since 4 hrs- https://review.opendev.org/c/openstack/nova/+/765141 | |