| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-12-02 | |||
| 11:41:38 | gibi | anyhow I can add it to tomorrow's agenda | |
| 11:58:46 | aarents | stephenfin lyarwood gibi Hi sorry, I'm blocked in a block in some training sessions this week, this issue is odd.. because it is working in tempest-slow-py3 https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/job-output.txt | |
| 12:05:21 | lyarwood | aarents: yeah odd, it's calling cat on the mount and that appears to be passing | |
| 12:05:23 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35793 | |
| 12:05:47 | aarents | for me /mnt is part of '/' there is no mount | |
| 12:05:51 | lyarwood | https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35623 | |
| 12:05:56 | lyarwood | yeah | |
| 12:06:00 | lyarwood | was about to say | |
| 12:13:57 | aarents | may be patch not working in multi-cell context, need to check logs, may be we can put it in blacklist of nova-multi-cell instead of revert, the time to check. | |
| 12:55:09 | lyarwood | aarents / stephenfin ; right this is a cross cell resize in the context of the job | |
| 12:55:54 | lyarwood | this smells like a bug with the cross cell resize flow tbh | |
| 13:00:30 | aarents | lyarwood: hum something like a rebuild from base image instead scp the disk ? | |
| 13:00:56 | gibi | I think cross cell resize do a snapshot instead of scp | |
| 13:01:22 | lyarwood | yeah it's a cold snapshot | |
| 13:03:34 | openstackgerrit | Takashi Natsume proposed openstack/nova stable/ussuri: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/765082 | |
| 13:04:54 | lyarwood | I don't think it's being used however | |
| 13:11:32 | aarents | I can see rebase is done after unshelve so instance disk should be fine before resize(cold snap) Running cmd (subprocess): qemu-img rebase -b /opt/stack/data/nova/instances/_base/30b9e4f95a51c37d278a2a0a447a270519e2be92 -F raw /opt/stack/data/nova/instances/9616dc4e-c2b8-46be-bf5c-6de9a11330f8/disk | |
| 13:18:17 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 13:30:28 | openstackgerrit | Aleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout https://review.opendev.org/c/openstack/nova/+/764927 | |
| 13:32:07 | lyarwood | aarents: yeah the spawn is using the original image | |
| 13:32:19 | lyarwood | aarents: this is a bug in the cross cell resize flow somewhere | |
| 13:32:25 | lyarwood | aarents: http://paste.openstack.org/show/800633/ | |
| 13:32:42 | lyarwood | aarents: ^ that's the qemu-img commands we run, we use the same base as the original spawn | |
| 13:33:14 | lyarwood | oh wait because we rebase | |
| 13:33:25 | lyarwood | during unshelve | |
| 13:33:33 | lyarwood | sorry I'm getting all confused | |
| 13:36:02 | lyarwood | ah but it's recreating the overlay | |
| 13:40:54 | aarents | lyarwood how it recreates the overlay ? | |
| 13:41:57 | lyarwood | aarents: I think it's doing it based on the image_meta we pass into driver.spawn here https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L6015-L6019 | |
| 13:42:09 | lyarwood | aarents: but I'm not sure why that isn't the snapshot image_meta at this point | |
| 13:42:21 | lyarwood | aarents: brb | |
| 13:52:01 | aarents | Yeah it should be the same code than in unshelve where it should use image-id from snapshot(stored in system_metada) | |
| 13:52:04 | aarents | https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/conductor/manager.py#L932 | |
| 13:59:52 | aarents | ha it is here where it should take the snapshot_id https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L5906 | |
| 14:01:57 | lyarwood | aarents: right but in this case it looks like snapshot_id wasn't passed and we fallback to the instance.image_meta | |
| 14:02:07 | lyarwood | aarents: at least that's what I'm assuming | |
| 14:02:28 | lyarwood | Let me blacklist this test in the multi cell job for now and throw up another DNM change adding some debug around this flow | |
| 14:03:15 | aarents | lyarwood: agree | |
| 14:06:15 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 14:06:50 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-multi-cell: Skip test_cold_migrate_unshelved_instance https://review.opendev.org/c/openstack/nova/+/765141 | |
| 14:07:18 | lyarwood | stephenfin / gibi ; ^ okay lets try it this way, skipping the test in the job directly as it looks like an issue with cross cell resize. | |
| 14:11:28 | gibi | lyarwood: ack +2 | |
| 14:12:02 | aarents | +1 | |
| 14:18:02 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM - debug test_cold_migrate_unshelved_instance on nova-multi-cell https://review.opendev.org/c/openstack/nova/+/765145 | |
| 14:18:04 | lyarwood | et voilà | |
| 14:18:17 | lyarwood | all because I don't want to build an env for this locally | |
| 14:25:07 | aarents | lyarwood: https://review.opendev.org/c/openstack/nova/+/765145/1/nova/compute/manager.py#5906 | |
| 14:25:57 | lyarwood | aarents: https://realpython.com/python-f-strings/#f-strings-a-new-and-improved-way-to-format-strings-in-python | |
| 14:26:15 | aarents | lyarwood: thks | |
| 14:26:35 | lyarwood | np | |
| 14:32:44 | gibi | lyarwood: just note that fstring in LOG calls does not delay interpolation | |
| 14:33:38 | lyarwood | yup but in this situation I'm just dumping things to troubleshoot, not landing the changes in master :) | |
| 14:33:48 | gibi | sure | |
| 15:17:02 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 15:18:26 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py https://review.opendev.org/c/openstack/nova/+/765152 | |
| 15:22:35 | LarsErikP | gibi: found a workaround =) | |
| 15:23:02 | LarsErikP | gibi: I tweak nvidia's sriov-manage script from creating all the VFs the GPU is capable of, to only create two | |
| 15:23:23 | gibi | LarsErikP: nice | |
| 15:23:25 | LarsErikP | gibi: then I cleaned up the resource providers, and now everything is fine.. Kinda dirty, but it works.. | |
| 15:24:11 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Make remove allocation symmetric with add allocation https://review.opendev.org/c/openstack/nova/+/757110 | |
| 15:24:12 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Factor out port resource deallocation https://review.opendev.org/c/openstack/nova/+/757111 | |
| 15:24:24 | LarsErikP | I've asked nvidia enterprise support if there is a "proper" way of doing this. The specifically say in the grid docs that you should _never_ use anything else than their script to enable the sr-iov functions | |
| 15:24:38 | LarsErikP | but they did no say I shouldn't edit their script... :P | |
| 15:24:50 | gibi | :D | |
| 15:27:35 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Refactor update_pci_request_spec_with_allocated_interface_name https://review.opendev.org/c/openstack/nova/+/757614 | |
| 15:27:42 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 15:29:25 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py https://review.opendev.org/c/openstack/nova/+/765152 | |
| 15:35:29 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/752284 | |
| 15:41:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py https://review.opendev.org/c/openstack/nova/+/765152 | |
| 16:33:08 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Docs: correct cpu_thread_policy explanation https://review.opendev.org/c/openstack/nova/+/765172 | |
| 16:46:50 | lyarwood | gibi / aarents ; I'm going to be out on PTO for the rest of the week btw in regards to the cross cell resize | |
| 16:46:57 | lyarwood | bug* | |
| 16:47:03 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: Docs: correct cpu_thread_policy explanation https://review.opendev.org/c/openstack/nova/+/765172 | |
| 16:48:20 | gibi | lyarwood: thanks for the heads up | |
| 16:48:41 | gibi | I will make sure the backlisting merges | |
| 16:48:51 | lyarwood | ack thanks, I can pick up the fix on Monday | |
| 16:48:59 | lyarwood | if aarents doesn't make progress | |
| 16:49:02 | gibi | have a nice time off | |
| 16:49:43 | lyarwood | lockdown ended today in England and my child is in nursery for the first time tomorrow, so don't worry I'll be having a nice time :D | |
| 16:49:47 | lyarwood | sleeping mostly | |
| 16:50:58 | gibi | sleeping is good | |
| 17:09:07 | kashyap | Yes, it adds years to life. (Cf. https://www.penguin.co.uk/books/295/295665/why-we-sleep/9780141983769.html) | |
| 17:09:34 | kashyap | (Counter-intuitively, enough ;-)) | |
| 17:18:07 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530 | |
| 17:34:46 | efried | +1 Why We Sleep -- life-changing read. | |
| 18:14:05 | gmann | gate is too slow today, this did not get the node yet. waiting since 4 hrs- https://review.opendev.org/c/openstack/nova/+/765141 | |
| 18:55:30 | openstackgerrit | melanie witt proposed openstack/nova master: WIP Omit resource inventories from placement update if zero https://review.opendev.org/c/openstack/nova/+/759348 | |
| 22:28:19 | rm_work | Is there a way to NOT have a default AZ, and to *require* the user specify one in their server-create command? or would that require a patch? | |
| 22:29:04 | rm_work | it seems we just removed the section from our config without thinking about it very much, but that had the effect of just causing the default AZ to be "nova" (default default) which does not exist in our cloud | |
| 22:50:01 | sean-k-mooney | not that i can see no | |
| 22:50:21 | rm_work | yeah looks like it has to be a custom patch | |
| 22:50:27 | sean-k-mooney | vms always need a AZ | |
| 22:50:27 | rm_work | ah well | |
| 22:50:54 | sean-k-mooney | rm_work: well you can propose it upstream or do it with api middelware | |
| 22:51:17 | sean-k-mooney | e.g. you coudl write middleware to reject all server creates that dont have an az | |
| 22:51:50 | rm_work | yeah that might be an interesting approach | |
| 22:51:56 | sean-k-mooney | this is config driven api bevhiaor and will break interoprablity between your cloud and others however | |
| 22:52:21 | sean-k-mooney | e.g. by default you should be able to boot without specifying an az | |
| 22:52:39 | sean-k-mooney | so it will be a qurik of your cloud that its always required | |
| 22:52:50 | rm_work | yeah, unfortunate but documentable | |