| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-07 | |||
| 21:31:04 | eandersson | melwitt, is that a bug in rocky as well? | |
| 21:31:20 | melwitt | no, stein is the earliest it shows up | |
| 21:36:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement update_provider_tree for mocked driver in test_resource_tracker https://review.opendev.org/693431 | |
| 21:37:42 | mriedem | eandersson: you should see this in the conductor logs https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/migrate.py#L51 | |
| 21:38:17 | mriedem | i would expect the scheduler to create allocations for the instance against the dest node | |
| 21:38:29 | mriedem | and those to stick if the migration completes successfully | |
| 21:38:48 | eandersson | We basically moved a site from one DC to another | |
| 21:38:58 | eandersson | and migrated all VMs | |
| 21:39:18 | ccstone | we didn't have nova in debug so we wouldn't have seen that message :/ | |
| 21:39:21 | eandersson | all of them was "healed" using the heal_allocations | |
| 21:40:29 | mriedem | ok, i don't know what would be going on there, but i'd expect the migration to heal the instance allocations as well | |
| 21:40:37 | mriedem | s/heal/recreate on dest node/ | |
| 21:41:44 | eandersson | Of course we are basing this on the heal_allocations output | |
| 21:45:14 | openstackgerrit | François Palin proposed openstack/nova master: Add retry to cinder api calls related to volume detach https://review.opendev.org/669674 | |
| 21:46:17 | mriedem | so the vm didn't have allocations, you migrated it successfully (and confirmed the migration), and ran heal allocations and it healed it rather than saying there was nothing to do? | |
| 21:47:03 | eandersson | pretty much yes | |
| 21:47:14 | mriedem | fake news! | |
| 21:47:16 | eandersson | haha | |
| 21:47:58 | eandersson | ignorance is probably the leading factor to fake news :p | |
| 21:48:49 | mriedem | no dude it's all ukraine | |
| 21:48:58 | eandersson | lol | |
| 21:49:01 | mriedem | so i wonder if on confirm we're doing something dumb and wiping out the allocations that the scheduler created | |
| 21:49:15 | mriedem | do you see any messages in the compute logs for that operation? | |
| 21:50:15 | mriedem | so i bet you don't get here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L4143 | |
| 21:50:35 | mriedem | which means you'd see this in the logs i think (on the source node) | |
| 21:50:36 | mriedem | https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L4143 | |
| 21:50:47 | mriedem | and i bet we drop the allocations here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L4143 | |
| 21:51:55 | mriedem | it's a guess, but i'd expect to see some messages in the compute logs related to that | |
| 21:53:07 | eandersson | I would have to dig in our logs as this was probably a month ago | |
| 21:53:15 | eandersson | when we actually performed the migrations | |
| 21:54:51 | mriedem | do your logs go that far back? | |
| 21:54:59 | mriedem | and do you have debug enabled on the compute service? | |
| 21:55:10 | eandersson | We never got debug enabled :'( | |
| 21:55:33 | mriedem | ok, it might not be something at debug, i'd expect to see log messages about how it expected to find allocations but found none, doing old style thingies | |
| 21:56:00 | mriedem | would probably also be pretty easy to recreate with a test vm, just create it, drop its allocations in placement, migrate and confirm and then check if the allocations are still gone | |
| 21:56:43 | eandersson | It's probably one of those things that only affect users that do crazy things like upgrade from mitaka to rocky :p | |
| 22:02:09 | efried | sean-k-mooney: for some reason n-cpu still thinks I'm using libvirt 4.0.0. How do I get it to pick up the thing I built? | |
| 22:02:33 | mriedem | efried: you likely need to recompile libvirt-python | |
| 22:02:47 | efried | I've got a recent libvirt-python | |
| 22:02:54 | efried | uhh | |
| 22:02:57 | mriedem | but is it built against the version of libvirt you're running now? | |
| 22:03:03 | efried | yeah | |
| 22:03:13 | efried | what do you mean 'built against'? | |
| 22:03:19 | efried | I didn't build it locally against the libvirt I built locally. | |
| 22:03:22 | efried | But the version matches. | |
| 22:03:40 | mriedem | the package version on libvirt python != the version of libvirt it thinks you have | |
| 22:03:51 | mriedem | it's just a python shim over the libvirt c api | |
| 22:04:12 | mriedem | so from what i remember you have to make sure your libvirt-python is built from the same libvirt source as the service | |
| 22:04:20 | mriedem | otherwise you think you're using x but you're getting y | |
| 22:04:23 | mriedem | and it's confusing | |
| 22:05:19 | efried | Hm. When I was trying to install libvirt-python 5.7 with libvirt 5.8 on my system, it wouldn't even install, complaining of symbols. I had to have libvirt-python >= libvirt for it to work. | |
| 22:05:30 | efried | So *something* was checking that I had the right libvirt. | |
| 22:07:10 | mriedem | ok at this point you're going to have to phone a red hat friend | |
| 22:11:07 | efried | so I disabled sean-k-mooney's plugin before unstack, clean, stack | |
| 22:11:07 | efried | but I didn't want to have to recompile libvirt/qemu again | |
| 22:11:07 | efried | and I wanted to clean because the problem was in the db | |
| 22:11:07 | efried | I restacked because I needed a newer nova branch and it wasn't working to just pull and restart services | |
| 22:11:07 | efried | I have a thing to try. | |
| 22:11:08 | efried | so I'm going to re-enable the plugin and restack (without cleaning)... | |
| 22:11:08 | efried | so I suspect the `stack` installed and set up stock libvirt again | |
| 22:12:04 | efried | yeesh, it's probably as simple as `make install` in those dirs, but too late, I already unstacked... | |
| 22:17:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Implement update_provider_tree for mocked driver in test_resource_tracker https://review.opendev.org/693431 | |
| 22:17:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop compat for non-update_provider_tree code paths https://review.opendev.org/693435 | |
| 22:20:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop compat for non-update_provider_tree code paths https://review.opendev.org/693435 | |
| 22:25:49 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete _normalize_inventory_from_cn_obj https://review.opendev.org/693436 | |
| 22:31:52 | efried | that was it | |
| 22:32:23 | efried | oooo, I just saw those last few patch titles mriedem, I'm excited. | |
| 22:33:34 | mriedem | i knew you would be | |
| 22:35:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Move compute_node_to_inventory_dict to test-only code https://review.opendev.org/693438 | |
| 22:38:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove get_minimum_version mocks from test_resource_tracker https://review.opendev.org/693439 | |
| 22:54:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Use named kwargs in compute.API.resize https://review.opendev.org/693425 | |
| #openstack-nova - 2019-11-08 | |||
| 00:06:30 | openstackgerrit | melanie witt proposed openstack/nova master: Use wrapper class for NeutronFixture get_client https://review.opendev.org/693453 | |
| 00:08:28 | openstackgerrit | melanie witt proposed openstack/nova master: Use wrapper class for NeutronFixture get_client https://review.opendev.org/693453 | |
| 00:26:26 | openstackgerrit | Archit Modi proposed openstack/nova stable/rocky: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693240 | |
| 00:29:09 | openstackgerrit | Archit Modi proposed openstack/nova stable/queens: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693239 | |
| 00:30:53 | openstackgerrit | Archit Modi proposed openstack/nova stable/pike: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693236 | |
| 00:45:37 | openstackgerrit | Dustin Cowles proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree https://review.opendev.org/676522 | |
| 00:45:38 | openstackgerrit | Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Enable loading and merging of provider configs https://review.opendev.org/693460 | |
| 00:49:56 | openstackgerrit | melanie witt proposed openstack/nova master: Use wrapper class for NeutronFixture get_client https://review.opendev.org/693453 | |
| 01:23:31 | openstackgerrit | ya.wang proposed openstack/nova-specs master: Re-propose expose-auto-converge-post-copy spec for Ussuri https://review.opendev.org/687199 | |
| 03:05:11 | eandersson | gibi_ptg, still having an minor issue with that patch | |
| 03:05:12 | eandersson | https://github.com/openstack/nova/commit/a5269012a3b442a9e4055a7d523faff45f105f2b#diff-77f9348ab09642ba46409b6828af4af0R1327 | |
| 03:05:27 | eandersson | That error message is thrown when an empty compute is restarted. | |
| 03:05:56 | eandersson | Since an empty compute has no allocations | |
| 04:05:57 | gibi_ptg | eandersson: could you please file a bug? | |
| 04:23:34 | eandersson | Sure gibi_ptg I'll do it tomorrow | |
| 04:39:29 | gibi_ptg | eandersson: thanks, I will try to look at it next week | |
| 04:39:53 | gibi_ptg | my brain is failing after all the ptg discussions | |
| 04:47:06 | eandersson | No worries - It's minor | |
| 05:41:50 | openstackgerrit | melanie witt proposed openstack/nova master: Replace time.sleep(10) with service forced_down in tests https://review.opendev.org/692252 | |
| 06:15:05 | gibi_ptg | melwitt: I'm +2 on ^^ thanks for fixing | |
| 06:16:08 | melwitt | thanks gibi_ptg. I wanted to delete that service.kill() too but was worried someone would complain it's not related xD | |
| 06:16:23 | melwitt | haha, understandable | |
| 06:20:08 | gibi_ptg | melwitt: don't worry about that kill() call too much. It does not cause any trouble either | |
| 08:18:08 | openstackgerrit | Merged openstack/nova master: Remove the TODO about using OSC for BFV in test_evacuate.sh https://review.opendev.org/693406 | |
| 08:36:25 | kashyap | sean-k-mooney: efried Just catching up with stuff after a week-ish of work travel. | |
| 08:37:01 | kashyap | sean-k-mooney: On RNG: We have this patch sitting, and waiting for reviews: https://review.opendev.org/#/c/616603/ ("libvirt: Add a default Virtio RNG device to guests") | |
| 11:48:52 | openstackgerrit | Akira KAMIO proposed openstack/nova master: VMware: disk_io_limits settings are not reflected when resize https://review.opendev.org/680296 | |
| 12:04:00 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove native LUKS compat code https://review.opendev.org/669121 | |
| 12:04:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Take an instance.uuid lock when rebooting https://review.opendev.org/673463 | |