| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-22 | |||
| 11:59:56 | gibi | uups | |
| 12:00:58 | sean-k-mooney | if i boot a ubuntu imagae with local sotorage and then try to do a rescue it also fails | |
| 12:01:00 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [req-a5b97168-4a74-4314-ba7f-9b1af4a8508d 0eb84fc371204ddb8c1dd829464d9284 468fb37037924eaeaf7104b34c4c077d - default default] [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] Failed to attach 858fdfc4-632d-47d3-bc60-d7f6aa76dc7b at /dev/sdb: libvirt.libvirtError: internal error: child reported (status=125): unable to open /dev/sdg: No such device | |
| 12:01:02 | sean-k-mooney | or address | |
| 12:01:04 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] Traceback (most recent call last): | |
| 12:01:06 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/compute/manager.py", line 6915, in _attach_volume | |
| 12:01:08 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] do_driver_attach=True) | |
| 12:01:10 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/block_device.py", line 46, in wrapped | |
| 12:01:12 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] ret_val = method(obj, context, *args, **kwargs) | |
| 12:01:14 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/block_device.py", line 673, in attach | |
| 12:01:16 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] virt_driver, do_driver_attach) | |
| 12:01:18 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/block_device.py", line 660, in _do_attach | |
| 12:01:20 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] do_driver_attach) | |
| 12:01:22 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/block_device.py", line 607, in _volume_attach | |
| 12:01:24 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] attachment_id) | |
| 12:01:26 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ | |
| 12:01:28 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] self.force_reraise() | |
| 12:01:30 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise | |
| 12:01:32 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] six.reraise(self.type_, self.value, self.tb) | |
| 12:01:34 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise | |
| 12:01:36 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] raise value | |
| 12:01:38 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/block_device.py", line 598, in _volume_attach | |
| 12:01:40 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] device_type=self['device_type'], encryption=encryption) | |
| 12:01:42 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1763, in attach_volume | |
| 12:01:44 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] encryption=encryption) | |
| 12:01:46 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ | |
| 12:01:48 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] self.force_reraise() | |
| 12:01:50 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise | |
| 12:01:52 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] six.reraise(self.type_, self.value, self.tb) | |
| 12:01:54 | gibi | ups again :) | |
| 12:01:54 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise | |
| 12:01:56 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] raise value | |
| 12:01:58 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 1736, in attach_volume | |
| 12:02:00 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] guest.attach_device(conf, persistent=True, live=live) | |
| 12:02:02 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 293, in attach_device | |
| 12:02:04 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] self._domain.attachDeviceFlags(device_xml, flags=flags) | |
| 12:02:06 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit | |
| 12:02:08 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] result = proxy_call(self._autowrap, f, *args, **kwargs) | |
| 12:02:10 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call | |
| 12:02:12 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] rv = execute(f, *args, **kwargs) | |
| 12:02:14 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute | |
| 12:02:16 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] six.reraise(c, e, tb) | |
| 12:02:18 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise | |
| 12:02:20 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] raise value | |
| 12:02:22 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/var/lib/kolla/venv/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker | |
| 12:02:24 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] rv = meth(*args, **kwargs) | |
| 12:02:26 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 614, in attachDeviceFlags | |
| 12:02:28 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self) | |
| 12:02:30 | sean-k-mooney | 2020-05-22 00:28:20.355 6 ERROR nova.compute.manager [instance: 8cfaf7ce-13ae-4b50-9bd8-20c957a8dfb5] libvirt.libvirtError: internal error: child reported (status=125): unable to open /dev/sdg: No such device or address | |
| 12:02:32 | sean-k-mooney | ops | |
| 12:02:34 | sean-k-mooney | http://paste.openstack.org/show/793881/ | |
| 12:02:36 | sean-k-mooney | ment to past the link | |
| 12:02:38 | sean-k-mooney | lyarwood: are you still working on stable rescue? | |
| 12:02:50 | gibi | I think stable rescue was finished in ussuri | |
| 12:03:06 | sean-k-mooney | this might be related or might not. i have different disk_bus and cdrom_bus image proterties set in my orginal image and the rescue image | |
| 12:03:51 | sean-k-mooney | i might do an upgrade of my cloud and see if the issue still exists. i have a random commit off master deployed | |
| 12:06:12 | lyarwood | sean-k-mooney: finished in U, what's up? | |
| 12:06:51 | sean-k-mooney | if we use q35 i have found at least two cases where attach volume is broken | |
| 12:06:58 | sean-k-mooney | one is when you boot form an iso | |
| 12:07:10 | sean-k-mooney | the other is if you try to use an iso for rescue | |
| 12:07:29 | lyarwood | sean-k-mooney: in both cases is it trying to use an IDE bus? | |
| 12:07:40 | sean-k-mooney | no | |
| 12:08:05 | sean-k-mooney | the iso image is using sata | |
| 12:08:18 | sean-k-mooney | i set cdrom_bus=sata in the image meta | |
| 12:09:19 | openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM: Test m1.nano flavor with 96MB to avoid tmpfs exhaustion https://review.opendev.org/730308 | |
| 12:09:30 | sean-k-mooney | lyarwood: i do not have rescue_bus set in any of the cases | |
| 12:10:18 | lyarwood | sean-k-mooney: kk can you write up some bugs for that? | |
| 12:10:44 | lyarwood | I've had my head in cirros sysvinit scripts all morning | |
| 12:10:45 | sean-k-mooney | yep i found it last night after trying to create a windows vm from an iso | |
| 12:13:51 | sean-k-mooney | the code is now different on master https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1416-L1425 | |
| 12:14:01 | sean-k-mooney | but it might stil have the bug | |
| 12:14:18 | openstackgerrit | Merged openstack/nova stable/stein: Update scheduler instance info at confirm resize https://review.opendev.org/729538 | |
| 12:14:37 | sean-k-mooney | max([]) | |
| 12:14:39 | sean-k-mooney | Traceback (most recent call last): | |
| 12:14:41 | sean-k-mooney | File "<stdin>", line 1, in <module> | |
| 12:14:43 | sean-k-mooney | ValueError: max() arg is an empty sequence | |
| 12:14:45 | sean-k-mooney | yep it does | |
| 12:14:55 | sean-k-mooney | oh no its fixed | |
| 12:15:02 | sean-k-mooney | max(ret) + 1 if ret else 0 | |
| 12:15:15 | sean-k-mooney | the if will prevent the ValueError | |
| 12:20:30 | sean-k-mooney | https://github.com/openstack/nova/commit/c8d6767cf8baaf3cc81496c83db10c8ae72fce06 is what im missing | |
| 12:21:56 | sean-k-mooney | so we should backport https://review.opendev.org/#/c/712607/ | |
| 12:24:55 | gibi | sean-k-mooney: +1 on that to backport | |
| 12:25:39 | sean-k-mooney | ill check how far back we had the old logic and then start propsoing them | |
| 12:26:58 | gibi | cool | |
| 12:33:50 | sean-k-mooney | apparently forever or at last to ocata. ill backport to queens | |
| 12:35:42 | openstackgerrit | sean mooney proposed openstack/nova stable/train: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730310 | |
| 12:36:33 | openstackgerrit | sean mooney proposed openstack/nova stable/stein: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730311 | |
| 12:36:55 | openstackgerrit | sean mooney proposed openstack/nova stable/rocky: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730312 | |
| 12:37:15 | openstackgerrit | sean mooney proposed openstack/nova stable/queens: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/730313 | |
| 12:51:46 | openstackgerrit | Elod Illes proposed openstack/nova stable/ussuri: remove support of oslo.messaging 9.8.0 warning message https://review.opendev.org/728055 | |
| 12:56:26 | jsuchome | lyarwood: hey, just a humble ping with a request for a new review... | |
| 12:59:36 | lyarwood | jsuchome: ack looking | |
| 13:08:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Make barbican-simple-crypto-devstack-tempest a non-voting check job https://review.opendev.org/730319 | |
| 13:08:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Make devstack-plugin-ceph-tempest-py3 a voting check job again https://review.opendev.org/730320 | |
| 13:10:40 | sean-k-mooney | gibi: dansmith added https://github.com/openstack/nova/commit/04e79b5b7e18f033053d4057edc768281b91a4d7 to reserve migration for ussuri but we also merged https://github.com/openstack/nova/commit/75a2d600fc26beda78bdafd209e92f0bc9802927 | |
| 13:10:55 | sean-k-mooney | https://github.com/openstack/nova/commit/75a2d600fc26beda78bdafd209e92f0bc9802927 is not on stable/ussuri however | |
| 13:11:12 | sean-k-mooney | do you know why we have 2 set of placeholder migration for ussuri | |
| 13:11:22 | sean-k-mooney | and why only one of them is in stable ussuri | |
| 13:12:12 | sean-k-mooney | is the commit message in https://github.com/openstack/nova/commit/75a2d600fc26beda78bdafd209e92f0bc9802927 just wrong and those are for victoria | |