| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-17 | |||
| 18:54:07 | melwitt | this is right up your alley | |
| 18:54:16 | melwitt | thanks. I will try that too | |
| 18:55:11 | prometheanfire | hmm, I don't see fixtures itself calling mock, so I guess it's in how nova calls fixtures? | |
| 18:56:17 | mriedem | my guess is it's this down in here https://github.com/testing-cabal/fixtures/blob/41774d58ed86b92dd60d727f00f1040f862c8ed1/fixtures/_fixtures/mockpatch.py#L22 | |
| 18:56:46 | melwitt | mriedem: yup, that's exactly it | |
| 18:57:12 | mriedem | could probably also avoid using those fixtures, there are other ways of doing those | |
| 18:57:17 | mriedem | but their usage might be pervasive | |
| 18:57:26 | melwitt | yeah, fair | |
| 18:57:49 | mriedem | not sure how you _prevent_ their usage later without a hacking check | |
| 18:58:10 | prometheanfire | hmm, ya, guess I was looking at what pypi linked to (launchpad) and not testing-cabal's org | |
| 18:59:16 | mriedem | i remember quite a bit of this in nova which isn't using mock https://github.com/openstack/nova/blob/master/nova/test.py#L352 but _can_ be replaced with mock | |
| 18:59:36 | mriedem | and stuff like https://github.com/openstack/nova/blob/master/nova/test.py#L423 could be replaced to just be mock rather than fixtures | |
| 18:59:51 | mriedem | i smell a 3 year all encompassing blueprint coming on | |
| 18:59:54 | melwitt | we could chance it and just fix the stuck cases (by not using fixtures in those spots) and then once third-party mock is gone from reqs, we are in the clear for future even with fixtures | |
| 19:00:58 | melwitt | lol @ 3 year blueprint. yes, I think we all want that in our life | |
| 19:01:09 | mriedem | i've had to ween myself off the fixtures library because i don't think it works with pytest, and i'm in pytest land now | |
| 19:01:17 | melwitt | ah | |
| 21:23:22 | openstack | bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [Undecided,New] https://launchpad.net/bugs/1888395 - Assigned to Sergey Galas' (shrike742) | |
| 21:23:22 | openstackgerrit | Kirill Egorov proposed openstack/nova master: fixes broken shared live migration of a vm with a vif aka Bug #1888395 https://review.opendev.org/742180 | |
| 22:28:29 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Few todo fixes for API new policies https://review.opendev.org/746610 | |
| 23:00:18 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/746210 | |
| #openstack-nova - 2020-08-18 | |||
| 01:01:52 | openstackgerrit | Merged openstack/nova master: Removes the delta file once image is extracted https://review.opendev.org/734421 | |
| 01:01:59 | openstackgerrit | Merged openstack/nova stable/rocky: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/703115 | |
| 02:28:17 | openstackgerrit | Merged openstack/nova stable/rocky: Block rebuild when NUMA topology changed https://review.opendev.org/703116 | |
| 02:30:05 | openstackgerrit | Merged openstack/nova stable/rocky: Disable NUMATopologyFilter on rebuild https://review.opendev.org/703117 | |
| 02:30:25 | openstackgerrit | Merged openstack/nova stable/rocky: FUP for in-place numa rebuild https://review.opendev.org/703118 | |
| 02:36:36 | openstackgerrit | Merged openstack/nova stable/pike: Fix os-simple-tenant-usage result order https://review.opendev.org/709600 | |
| 03:16:41 | openstackgerrit | Xinliang Liu proposed openstack/nova master: Always pass image source info to ironic https://review.opendev.org/746626 | |
| 07:44:15 | openstackgerrit | Luigi Toscano proposed openstack/python-novaclient master: zuul functional job: drop the custom playbooks https://review.opendev.org/746649 | |
| 09:20:48 | admin0 | hi all .. in one new setup with ceph, in the logs, i see the claim successful .. but the instance is just kind of stuck in the spawn/build state .. i don't see any errors in nova .. this is the log from the hypervisor .. https://gist.github.com/a1git/c14bc9268f429f48169c6c050a605d0c | |
| 09:21:20 | admin0 | the instance stays in spawning state for hours like this . and is never created | |
| 09:23:02 | admin0 | system is all ubuntu 18.04 .. | |
| 09:35:11 | lyarwood | admin0: Aug 18 04:09:12 h3 nova-compute[75006]: 2020-08-18 04:09:12.789 75006 DEBUG oslo_concurrency.processutils [req-61d78385-6bb2-4af2-aea9-f0942ef497b7 1be712988728425ea0060d3bf7ed7a1e 594a25f5acc346518cd6340ec8c36945 - default default] Running cmd (subprocess): rbd import --pool vms /var/lib/nova/instances/_base/ | |
| 09:35:13 | lyarwood | 85e55585a75fa020bfd46e25ab7aad9c7d360a41 fd14724b-8e52-49c9-b906-9f9a5ab5a651_disk --image-format=2 --id cinder --conf /etc/ceph/ceph.conf execute /openstack/venvs/nova-20.1.4/lib/python3.6/site-packages/oslo_concurrency/processutils.py: | |
| 09:35:24 | lyarwood | ^ that's where it appears to get stuck | |
| 09:35:48 | lyarwood | importing the image into the local imagecache | |
| 09:36:22 | lyarwood | the env isn't using rbd for images FWIW, I'm not sure if you assume it is | |
| 09:36:34 | lyarwood | ah wait it is | |
| 09:37:20 | lyarwood | so it's importing into rbd from the imagecache | |
| 09:37:36 | admin0 | and the files are present in base | |
| 09:37:52 | lyarwood | stupid question but how big is /var/lib/nova/instances/_base/85e55585a75fa020bfd46e25ab7aad9c7d360a41 | |
| 09:38:17 | admin0 | 1.1G | |
| 09:38:22 | lyarwood | so nothing huge | |
| 09:38:35 | admin0 | nothing .. i even tried it with a cirros image . stuck in the same way | |
| 09:39:30 | lyarwood | admin0: is glance using ceph as well? | |
| 09:39:36 | admin0 | it is | |
| 09:39:47 | admin0 | i have no issues uploading images to glance .. which i validated | |
| 09:39:56 | lyarwood | admin0: try converting this image or the cirros image to RAW from qcow2 | |
| 09:39:57 | admin0 | creating volumes is also having no issues | |
| 09:40:09 | admin0 | hmm.. | |
| 09:40:36 | admin0 | ok .. let me try that way | |
| 09:40:41 | admin0 | i found a snippet : https://gist.github.com/winggundamth/7fe69a99ffcd1dfd14471101497b1407 | |
| 09:41:24 | lyarwood | yup so with RAW we can clone directly between rbd pools | |
| 09:41:55 | lyarwood | with qcow2 in stable/train that you appear to be using we download the image to the compute and upload to the vms pool | |
| 09:46:18 | openstackgerrit | Mamduh proposed openstack/os-vif master: Update linux_net - Refactor code of linux_net to more cleaner and increase performace https://review.opendev.org/746673 | |
| 09:54:36 | admin0 | lyarwood, https://gist.githubusercontent.com/a1git/78adef462d99eb88829006f36b988f6b/raw/03ffff0d17a94200f6ce4d2594c76f9c2f130890/gistfile1.txt -- this is using a 112mb cirros raw image | |
| 09:54:41 | admin0 | still in spawning state | |
| 09:56:41 | admin0 | i can confirm that from this server, if i copy a command like ceph df --format=json --id cinder --conf /etc/ceph/ceph.conf .. it works | |
| 09:59:36 | lyarwood | admin0: looks like the final thing it's doing again is a rbd import | |
| 09:59:48 | lyarwood | admin0: Aug 18 05:47:41 h1 nova-compute[74659]: 2020-08-18 05:47:41.882 74659 DEBUG oslo_concurrency.processutils [req-f72a63da-329e-45c4-b4dc-7728bb8d2b93 1be712988728425ea0060d3bf7ed7a1e 594a25f5acc346518cd6340ec8c36945 - default default] Running cmd (subprocess): rbd import --pool vms /var/lib/nova/instances/9ea7d4 | |
| 10:00:24 | lyarwood | admin0: I think that's the config drive | |
| 10:00:41 | lyarwood | hmmm odd | |
| 10:00:52 | admin0 | so removing config drive will fix this ? | |
| 10:01:06 | admin0 | i can give that a try also | |
| 10:01:22 | lyarwood | well it will allow an instance to spawn but there's an underlying issue with rbd import to address | |
| 10:03:33 | lyarwood | btw to keep the log snippets sane try grepping for the request-id listed in `openstack server event list $instance` next time | |
| 10:04:08 | lyarwood | it should limit the logs you need to look at to that instance and request to spawn etc | |
| 10:04:25 | lyarwood | brb | |
| 10:07:29 | admin0 | i only saw delete and create in the event | |
| 10:09:13 | admin0 | events: [{'event': 'compute__do_build_and_run_instance', 'start_time': '2020-08-18T09:47:36.000000', 'finish_time': None, 'result': None, 'traceback': None}] | |
| 10:20:17 | admin0 | got it | |
| 10:20:27 | admin0 | how to grep using that specific instanceid | |
| 10:28:28 | lyarwood | admin0: it's the request-id that you want | |
| 10:28:37 | lyarwood | openstack server event list $instance | grep create | awk '{ print $2 }' | |
| 10:42:18 | admin0 | i am making change to remove the config drive to retry again | |
| 10:42:23 | admin0 | so config drives are not compatible with ceph ? | |
| 10:43:26 | lyarwood | admin0: they are but there's something broken in your env with regards to importing disks into rbd | |
| 10:43:56 | admin0 | is there any manual command i can try to validate this ? | |
| 10:44:48 | lyarwood | I'm not sure how you've deployed your env but try using rbd --import as the same user as is running nova-compute | |
| 10:44:59 | admin0 | its done using openstack-ansible | |
| 10:45:12 | admin0 | so the standard osa install | |
| 10:45:44 | admin0 | which has quiet frankly worked well in all past deployments .. but in this new one, it suddenly does not work and i am unable to figure out why | |
| 10:46:38 | lyarwood | kk I've never actually used OSA but that would be my way forward here, run the command as the same user and in the same place as nova-compute is running it | |
| 10:46:52 | lyarwood | and then throw on some additional -v flags (or whatever rbd uses) to debug | |
| 10:47:22 | lyarwood | sean-k-mooney: ^ when you're about | |
| 10:47:24 | admin0 | so basically will be to change nova.conf debug = true, grep the subprocess to capture the commands and then run the same commands again .. | |
| 10:47:41 | admin0 | i will give it a shot | |
| 10:47:48 | lyarwood | admin0: looks like you're already logging in DEBUG, but yeah. | |
| 10:48:06 | admin0 | its because it failed here :D | |
| 10:48:07 | lyarwood | admin0: you just need to be sure you're running the commands under the same user and path (assuming OSA isn't deploying containers?) | |
| 10:48:16 | admin0 | it is not | |
| 10:48:28 | lyarwood | kk | |
| 10:48:37 | admin0 | the containers are only in the controllers . but as far as i have seen the api does its job and its the hypervisor where its stuck | |
| 10:48:55 | lyarwood | yeah it's just the rbd import that's borked | |
| 10:52:55 | admin0 | is this the file which is the source of all these commands? https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/storage/rbd_utils.py | |
| 10:58:18 | lyarwood | admin0: https://opendev.org/openstack/nova/src/branch/stable/train/nova/virt/libvirt/storage/rbd_utils.py for the version you are running | |
| 10:59:39 | admin0 | i removed the configdrive option .. checking once more | |
| 11:00:15 | admin0 | and all images are raw | |
| 11:02:25 | admin0 | lyarwood, it worked | |
| 11:02:27 | admin0 | finally its active now | |