| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-17 | |||
| 15:11:54 | openstackgerrit | Luyao Zhong proposed openstack/nova master: track error migrations in resource tracker https://review.opendev.org/745281 | |
| 15:14:17 | openstackgerrit | Luyao Zhong proposed openstack/nova master: improve orphans tracking in resource tracker https://review.opendev.org/745282 | |
| 15:29:47 | openstackgerrit | Merged openstack/nova master: Remove monotonic usage https://review.opendev.org/725425 | |
| 15:29:55 | openstackgerrit | Merged openstack/nova master: Rebase qcow2 images when unshelving an instance https://review.opendev.org/696084 | |
| 15:31:35 | prometheanfire | any work nova side to support mock 4.0.2? https://review.opendev.org/712713 | |
| 15:35:26 | melwitt | prometheanfire: I spent a lot of time on it, didn't get anywhere, got discouraged. working up the will to try again | |
| 15:37:03 | prometheanfire | fair | |
| 17:45:06 | kegorov35 | hey! my supervisor told me that this review https://review.opendev.org/#/c/742180/ should be merged. can you please merge it or explain to me why can't you merge it | |
| 17:56:04 | kegorov35 | #openstack-nova urgent question | |
| 17:58:52 | prometheanfire | kegorov35: better to ask for a review than a merge (a review allows the review to say no, a merge kinda expects a yes) | |
| 17:58:53 | dansmith | kegorov35: it can't be merged because it hasn't been reviewed, and definitely because it has no tests, and also because it claims there's a bug being fixed, with non referenced and no reproducer information | |
| 18:01:38 | dansmith | oh I see it's jammed in the subject, but still.. | |
| 18:10:13 | openstackgerrit | Luigi Toscano proposed openstack/nova master: zuul: use the new barbican simple-crypto job https://review.opendev.org/746583 | |
| 18:46:16 | prometheanfire | melwitt: for mock, my stance is that if nova is not gonna use the python builtin unittest.mock then it needs to update :| | |
| 18:49:05 | melwitt | library without importing third-party mock | |
| 18:49:05 | melwitt | prometheanfire: we want to and we can, but the snag is that the fixtures library we use (https://pypi.org/project/fixtures/) prefers the third-party mock library when it imports mock, and then it creates mocks using that library, and so when we "switch" to unittest.mock in some places, we end up with third-party mocks + unittest.mocks that don't know about each other and we can't "stop" a third-party mock embedded inside the fixtures | |
| 18:49:53 | prometheanfire | melwitt: understood, I wasn't knocking you at all :D | |
| 18:50:07 | melwitt | everything works fine if the third-party mock library is not installed in the venv at all but if it's present, fixtures prefers it and causes problems | |
| 18:50:41 | melwitt | I tried a bunch of workarounds and none worked, I ran out of ideas and I haven't been back yet to see if I could find some other way around | |
| 18:50:43 | prometheanfire | eww, uses launchpad :P | |
| 18:51:48 | melwitt | maybe if I did something like fixtures.<whatever>.mock.stopall() that might work | |
| 18:53:28 | mriedem | hack idea: monkey patch the 3rd party lib with the std lib if found | |
| 18:53:57 | melwitt | yesss mriedem | |
| 18:54:07 | melwitt | this is right up your alley | |
| 18:54:16 | melwitt | thanks. I will try that too | |
| 18:55:11 | prometheanfire | hmm, I don't see fixtures itself calling mock, so I guess it's in how nova calls fixtures? | |
| 18:56:17 | mriedem | my guess is it's this down in here https://github.com/testing-cabal/fixtures/blob/41774d58ed86b92dd60d727f00f1040f862c8ed1/fixtures/_fixtures/mockpatch.py#L22 | |
| 18:56:46 | melwitt | mriedem: yup, that's exactly it | |
| 18:57:12 | mriedem | could probably also avoid using those fixtures, there are other ways of doing those | |
| 18:57:17 | mriedem | but their usage might be pervasive | |
| 18:57:26 | melwitt | yeah, fair | |
| 18:57:49 | mriedem | not sure how you _prevent_ their usage later without a hacking check | |
| 18:58:10 | prometheanfire | hmm, ya, guess I was looking at what pypi linked to (launchpad) and not testing-cabal's org | |
| 18:59:16 | mriedem | i remember quite a bit of this in nova which isn't using mock https://github.com/openstack/nova/blob/master/nova/test.py#L352 but _can_ be replaced with mock | |
| 18:59:36 | mriedem | and stuff like https://github.com/openstack/nova/blob/master/nova/test.py#L423 could be replaced to just be mock rather than fixtures | |
| 18:59:51 | mriedem | i smell a 3 year all encompassing blueprint coming on | |
| 18:59:54 | melwitt | we could chance it and just fix the stuck cases (by not using fixtures in those spots) and then once third-party mock is gone from reqs, we are in the clear for future even with fixtures | |
| 19:00:58 | melwitt | lol @ 3 year blueprint. yes, I think we all want that in our life | |
| 19:01:09 | mriedem | i've had to ween myself off the fixtures library because i don't think it works with pytest, and i'm in pytest land now | |
| 19:01:17 | melwitt | ah | |
| 21:23:22 | openstack | bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [Undecided,New] https://launchpad.net/bugs/1888395 - Assigned to Sergey Galas' (shrike742) | |
| 21:23:22 | openstackgerrit | Kirill Egorov proposed openstack/nova master: fixes broken shared live migration of a vm with a vif aka Bug #1888395 https://review.opendev.org/742180 | |
| 22:28:29 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Few todo fixes for API new policies https://review.opendev.org/746610 | |
| 23:00:18 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Do not reference VIR_ERR_DEVICE_MISSING when libvirt is < v4.1.0 https://review.opendev.org/746210 | |
| #openstack-nova - 2020-08-18 | |||
| 01:01:52 | openstackgerrit | Merged openstack/nova master: Removes the delta file once image is extracted https://review.opendev.org/734421 | |
| 01:01:59 | openstackgerrit | Merged openstack/nova stable/rocky: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/703115 | |
| 02:28:17 | openstackgerrit | Merged openstack/nova stable/rocky: Block rebuild when NUMA topology changed https://review.opendev.org/703116 | |
| 02:30:05 | openstackgerrit | Merged openstack/nova stable/rocky: Disable NUMATopologyFilter on rebuild https://review.opendev.org/703117 | |
| 02:30:25 | openstackgerrit | Merged openstack/nova stable/rocky: FUP for in-place numa rebuild https://review.opendev.org/703118 | |
| 02:36:36 | openstackgerrit | Merged openstack/nova stable/pike: Fix os-simple-tenant-usage result order https://review.opendev.org/709600 | |
| 03:16:41 | openstackgerrit | Xinliang Liu proposed openstack/nova master: Always pass image source info to ironic https://review.opendev.org/746626 | |
| 07:44:15 | openstackgerrit | Luigi Toscano proposed openstack/python-novaclient master: zuul functional job: drop the custom playbooks https://review.opendev.org/746649 | |
| 09:20:48 | admin0 | hi all .. in one new setup with ceph, in the logs, i see the claim successful .. but the instance is just kind of stuck in the spawn/build state .. i don't see any errors in nova .. this is the log from the hypervisor .. https://gist.github.com/a1git/c14bc9268f429f48169c6c050a605d0c | |
| 09:21:20 | admin0 | the instance stays in spawning state for hours like this . and is never created | |
| 09:23:02 | admin0 | system is all ubuntu 18.04 .. | |
| 09:35:11 | lyarwood | admin0: Aug 18 04:09:12 h3 nova-compute[75006]: 2020-08-18 04:09:12.789 75006 DEBUG oslo_concurrency.processutils [req-61d78385-6bb2-4af2-aea9-f0942ef497b7 1be712988728425ea0060d3bf7ed7a1e 594a25f5acc346518cd6340ec8c36945 - default default] Running cmd (subprocess): rbd import --pool vms /var/lib/nova/instances/_base/ | |
| 09:35:13 | lyarwood | 85e55585a75fa020bfd46e25ab7aad9c7d360a41 fd14724b-8e52-49c9-b906-9f9a5ab5a651_disk --image-format=2 --id cinder --conf /etc/ceph/ceph.conf execute /openstack/venvs/nova-20.1.4/lib/python3.6/site-packages/oslo_concurrency/processutils.py: | |
| 09:35:24 | lyarwood | ^ that's where it appears to get stuck | |
| 09:35:48 | lyarwood | importing the image into the local imagecache | |
| 09:36:22 | lyarwood | the env isn't using rbd for images FWIW, I'm not sure if you assume it is | |
| 09:36:34 | lyarwood | ah wait it is | |
| 09:37:20 | lyarwood | so it's importing into rbd from the imagecache | |
| 09:37:36 | admin0 | and the files are present in base | |
| 09:37:52 | lyarwood | stupid question but how big is /var/lib/nova/instances/_base/85e55585a75fa020bfd46e25ab7aad9c7d360a41 | |
| 09:38:17 | admin0 | 1.1G | |
| 09:38:22 | lyarwood | so nothing huge | |
| 09:38:35 | admin0 | nothing .. i even tried it with a cirros image . stuck in the same way | |
| 09:39:30 | lyarwood | admin0: is glance using ceph as well? | |
| 09:39:36 | admin0 | it is | |
| 09:39:47 | admin0 | i have no issues uploading images to glance .. which i validated | |
| 09:39:56 | lyarwood | admin0: try converting this image or the cirros image to RAW from qcow2 | |
| 09:39:57 | admin0 | creating volumes is also having no issues | |
| 09:40:09 | admin0 | hmm.. | |
| 09:40:36 | admin0 | ok .. let me try that way | |
| 09:40:41 | admin0 | i found a snippet : https://gist.github.com/winggundamth/7fe69a99ffcd1dfd14471101497b1407 | |
| 09:41:24 | lyarwood | yup so with RAW we can clone directly between rbd pools | |
| 09:41:55 | lyarwood | with qcow2 in stable/train that you appear to be using we download the image to the compute and upload to the vms pool | |
| 09:46:18 | openstackgerrit | Mamduh proposed openstack/os-vif master: Update linux_net - Refactor code of linux_net to more cleaner and increase performace https://review.opendev.org/746673 | |
| 09:54:36 | admin0 | lyarwood, https://gist.githubusercontent.com/a1git/78adef462d99eb88829006f36b988f6b/raw/03ffff0d17a94200f6ce4d2594c76f9c2f130890/gistfile1.txt -- this is using a 112mb cirros raw image | |
| 09:54:41 | admin0 | still in spawning state | |
| 09:56:41 | admin0 | i can confirm that from this server, if i copy a command like ceph df --format=json --id cinder --conf /etc/ceph/ceph.conf .. it works | |
| 09:59:36 | lyarwood | admin0: looks like the final thing it's doing again is a rbd import | |
| 09:59:48 | lyarwood | admin0: Aug 18 05:47:41 h1 nova-compute[74659]: 2020-08-18 05:47:41.882 74659 DEBUG oslo_concurrency.processutils [req-f72a63da-329e-45c4-b4dc-7728bb8d2b93 1be712988728425ea0060d3bf7ed7a1e 594a25f5acc346518cd6340ec8c36945 - default default] Running cmd (subprocess): rbd import --pool vms /var/lib/nova/instances/9ea7d4 | |
| 10:00:24 | lyarwood | admin0: I think that's the config drive | |
| 10:00:41 | lyarwood | hmmm odd | |
| 10:00:52 | admin0 | so removing config drive will fix this ? | |
| 10:01:06 | admin0 | i can give that a try also | |
| 10:01:22 | lyarwood | well it will allow an instance to spawn but there's an underlying issue with rbd import to address | |
| 10:03:33 | lyarwood | btw to keep the log snippets sane try grepping for the request-id listed in `openstack server event list $instance` next time | |
| 10:04:08 | lyarwood | it should limit the logs you need to look at to that instance and request to spawn etc | |
| 10:04:25 | lyarwood | brb | |
| 10:07:29 | admin0 | i only saw delete and create in the event | |
| 10:09:13 | admin0 | events: [{'event': 'compute__do_build_and_run_instance', 'start_time': '2020-08-18T09:47:36.000000', 'finish_time': None, 'result': None, 'traceback': None}] | |
| 10:20:17 | admin0 | got it | |
| 10:20:27 | admin0 | how to grep using that specific instanceid | |
| 10:28:28 | lyarwood | admin0: it's the request-id that you want | |
| 10:28:37 | lyarwood | openstack server event list $instance | grep create | awk '{ print $2 }' | |
| 10:42:18 | admin0 | i am making change to remove the config drive to retry again | |
| 10:42:23 | admin0 | so config drives are not compatible with ceph ? | |
| 10:43:26 | lyarwood | admin0: they are but there's something broken in your env with regards to importing disks into rbd | |