Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-12-02
10:16:31 gibi but the schema validation that fails is here https://github.com/openstack/placement/blob/c02a073c523d363d7136677ab12884dc4ec03e6f/placement/schemas/inventory.py#L29
10:30:04 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: DNM/WIP Remove device from attach_volume request response https://review.opendev.org/c/openstack/nova-specs/+/765097
10:47:47 stephenfin man, https://bugs.launchpad.net/tempest/+bug/1906428 appear to have pretty much broken the gate :-(
10:47:47 openstack Launchpad bug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,Confirmed]
10:48:01 stephenfin and I can't figure out what's changed
10:49:18 lyarwood stephenfin: yeah it's weird, I don't understand how the test ever worked prior to that change anyway tbh
10:49:50 lyarwood oh wait!
10:49:53 stephenfin What's the idea behind? That you'd create a file and it would still exist after you'd stashed, offloaded and unstashed
10:50:03 lyarwood stephenfin: I don't think we offload
10:50:21 lyarwood stephenfin: doesn't that just pause and unpause
10:50:31 lyarwood stephenfin: so the mount within the guestOS persists
10:50:43 lyarwood stephenfin: but now as we cold migrate we essentially reboot the instance clearing the mount
10:51:06 stephenfin I see "Shelve offloading" followed by "Instance destroyed successfully" in the log I'm looking at
10:51:12 lyarwood ah kk
10:51:18 stephenfin search for tempest-TestShelveInstance-307565458 here https://zuul.opendev.org/t/openstack/build/41da7c0de7994dddb641c0a783b53c70/log/compute1/logs/screen-n-cpu.txt
10:51:25 lyarwood if it's dead then I'm back to not understanding how this ever worked
10:51:40 lyarwood unless verification was off in the job
10:51:49 lyarwood so these checks were skipped
10:51:58 stephenfin aarents: Have you seen https://bugs.launchpad.net/tempest/+bug/1906428 ?
10:51:58 openstack Launchpad bug 1906428 in OpenStack Compute (nova) "test_cold_migrate_unshelved_instance failing with cat: can't open '/mnt/timestamp': No such file or directory" [Critical,Confirmed]
10:53:04 lyarwood https://review.opendev.org/c/openstack/tempest/+/765079 I've posted a revert
10:53:14 lyarwood brb
10:53:27 stephenfin I see 186 hits. Nice
10:54:08 stephenfin https://review.opendev.org/c/opendev/elastic-recheck/+/765108
10:56:43 stephenfin Oh, I didn't know that that patch only merged two days ago /o\
10:57:02 stephenfin I was looking at the author date in Git and thought something had regressed
10:57:08 stephenfin Yeah, let's yank that for sure
10:58:47 gibi lyarwood, stephenfin: nice teamwork on this gate blocker
11:08:49 lyarwood np
11:11:25 masterpe Hi, I try to live migrate a instance to a specific host. But I get the error: "There is not enough capacity on the host for the instance." but on the "openstack hypervisor show" gives me enough resources left. In the allocations table of database nova_api I see some ghost resources. What can I do to fix this? I'm running Train.
11:15:57 stephenfin masterpe: Don't trust os-hypervisors. It lies. Placement is where you want to get your information https://specs.openstack.org/openstack/nova-specs/specs/wallaby/approved/modernize-os-hypervisors-api.html
11:16:08 stephenfin What do you mean by ghost resources?
11:16:18 lyarwood https://review.opendev.org/c/openstack/nova/+/720838 - huh we didn't backport this in the end
11:16:38 stephenfin If placement's inventory is wrong, you'll want to use nova-manage to handle things https://docs.openstack.org/nova/latest/cli/nova-manage.html#placement
11:16:47 lyarwood stephenfin: that isn't in train
11:16:50 stephenfin oh, jinx
11:16:58 stephenfin damn :(
11:17:22 lyarwood brb again
11:17:28 stephenfin topic for discussion in tomorrow's meeting? gibi ^
11:23:28 masterpe stephenfin: I see UUID of instances that I can not find anymore with "openstack server show"
11:24:35 openstackgerrit Merged openstack/nova-specs master: libvirt: Store and allow the default machine type to be changed https://review.opendev.org/c/openstack/nova-specs/+/762199
11:37:20 gibi stephenfin: you mean backporting placement audit to stable branches?
11:37:25 stephenfin yes
11:38:11 gibi would be good to know why we abandoned the backports
11:41:38 gibi anyhow I can add it to tomorrow's agenda
11:58:46 aarents stephenfin lyarwood gibi Hi sorry, I'm blocked in a block in some training sessions this week, this issue is odd.. because it is working in tempest-slow-py3 https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/job-output.txt
12:05:21 lyarwood aarents: yeah odd, it's calling cat on the mount and that appears to be passing
12:05:23 lyarwood https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35793
12:05:47 aarents for me /mnt is part of '/' there is no mount
12:05:51 lyarwood https://zuul.opendev.org/t/openstack/build/41a4f9aa2b084025b4f928b6a8187802/log/controller/logs/tempest_log.txt#35623
12:05:56 lyarwood yeah
12:06:00 lyarwood was about to say
12:13:57 aarents may be patch not working in multi-cell context, need to check logs, may be we can put it in blacklist of nova-multi-cell instead of revert, the time to check.
12:55:09 lyarwood aarents / stephenfin ; right this is a cross cell resize in the context of the job
12:55:54 lyarwood this smells like a bug with the cross cell resize flow tbh
13:00:30 aarents lyarwood: hum something like a rebuild from base image instead scp the disk ?
13:00:56 gibi I think cross cell resize do a snapshot instead of scp
13:01:22 lyarwood yeah it's a cold snapshot
13:03:34 openstackgerrit Takashi Natsume proposed openstack/nova stable/ussuri: Fix a hacking test https://review.opendev.org/c/openstack/nova/+/765082
13:04:54 lyarwood I don't think it's being used however
13:11:32 aarents I can see rebase is done after unshelve so instance disk should be fine before resize(cold snap) Running cmd (subprocess): qemu-img rebase -b /opt/stack/data/nova/instances/_base/30b9e4f95a51c37d278a2a0a447a270519e2be92 -F raw /opt/stack/data/nova/instances/9616dc4e-c2b8-46be-bf5c-6de9a11330f8/disk
13:18:17 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530
13:30:28 openstackgerrit Aleksey Ulyanov proposed openstack/nova master: Fix graceful shutdown timeout https://review.opendev.org/c/openstack/nova/+/764927
13:32:07 lyarwood aarents: yeah the spawn is using the original image
13:32:19 lyarwood aarents: this is a bug in the cross cell resize flow somewhere
13:32:25 lyarwood aarents: http://paste.openstack.org/show/800633/
13:32:42 lyarwood aarents: ^ that's the qemu-img commands we run, we use the same base as the original spawn
13:33:14 lyarwood oh wait because we rebase
13:33:25 lyarwood during unshelve
13:33:33 lyarwood sorry I'm getting all confused
13:36:02 lyarwood ah but it's recreating the overlay
13:40:54 aarents lyarwood how it recreates the overlay ?
13:41:57 lyarwood aarents: I think it's doing it based on the image_meta we pass into driver.spawn here https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L6015-L6019
13:42:09 lyarwood aarents: but I'm not sure why that isn't the snapshot image_meta at this point
13:42:21 lyarwood aarents: brb
13:52:01 aarents Yeah it should be the same code than in unshelve where it should use image-id from snapshot(stored in system_metada)
13:52:04 aarents https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/conductor/manager.py#L932
13:59:52 aarents ha it is here where it should take the snapshot_id https://github.com/openstack/nova/blob/f0efcae6975a99044ef7052453f905f60fcecac6/nova/compute/manager.py#L5906
14:01:57 lyarwood aarents: right but in this case it looks like snapshot_id wasn't passed and we fallback to the instance.image_meta
14:02:07 lyarwood aarents: at least that's what I'm assuming
14:02:28 lyarwood Let me blacklist this test in the multi cell job for now and throw up another DNM change adding some debug around this flow
14:03:15 aarents lyarwood: agree
14:06:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530
14:06:50 openstackgerrit Lee Yarwood proposed openstack/nova master: nova-multi-cell: Skip test_cold_migrate_unshelved_instance https://review.opendev.org/c/openstack/nova/+/765141
14:07:18 lyarwood stephenfin / gibi ; ^ okay lets try it this way, skipping the test in the job directly as it looks like an issue with cross cell resize.
14:11:28 gibi lyarwood: ack +2
14:12:02 aarents +1
14:18:02 openstackgerrit Lee Yarwood proposed openstack/nova master: DNM - debug test_cold_migrate_unshelved_instance on nova-multi-cell https://review.opendev.org/c/openstack/nova/+/765145
14:18:04 lyarwood et voilà
14:18:17 lyarwood all because I don't want to build an env for this locally
14:25:07 aarents lyarwood: https://review.opendev.org/c/openstack/nova/+/765145/1/nova/compute/manager.py#5906
14:25:57 lyarwood aarents: https://realpython.com/python-f-strings/#f-strings-a-new-and-improved-way-to-format-strings-in-python
14:26:15 aarents lyarwood: thks
14:26:35 lyarwood np
14:32:44 gibi lyarwood: just note that fstring in LOG calls does not delay interpolation
14:33:38 lyarwood yup but in this situation I'm just dumping things to troubleshoot, not landing the changes in master :)
14:33:48 gibi sure
15:17:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Support interface attach with qos ports https://review.opendev.org/c/openstack/nova/+/756530
15:18:26 openstackgerrit Balazs Gibizer proposed openstack/nova master: Enable mypy on scheduler/report.py https://review.opendev.org/c/openstack/nova/+/765152
15:22:35 LarsErikP gibi: found a workaround =)

Earlier   Later