Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-07
19:48:47 eandersson but a lot of these issues are so specific to us I feel stupid writing the reports lol
19:49:08 eandersson like some of these "bad vms" are from like Juno
19:49:34 sean-k-mooney did any of the vms "disapear" after upgrading
19:49:35 eandersson or maybe even due to bad database surgery at some point
19:50:03 eandersson none disapeared
19:50:21 eandersson I think because we run NUMA a lot of issues are hidden from us.
19:50:26 sean-k-mooney ok there was a change around rocky where we start cleaning up falid evaucted instnace
19:50:38 eandersson Like the above would have been a major issue if it wasn't for NUMA, because when it fails due to the above it just reschedules.
19:51:07 eandersson We only notice it when it hits like 10+ failed reschedules and throws an exception
19:51:34 eandersson and ironically we bumped it from 10 -> 20 failures because we didn't realize that placement was in an inconsistent state.
19:51:34 sean-k-mooney so numa made it better or worse?
19:51:51 eandersson well it prevented computes from unintentially getting oversubscribed
19:52:13 sean-k-mooney i think that is a first. numa making the failure mode better :)
19:52:41 sean-k-mooney but ya the addtional tracking in the numa toplogy blob in the db would help with that
20:02:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove service_uuids_online_data_migration https://review.opendev.org/693421
20:12:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove dead HostAPI.service_delete code https://review.opendev.org/693422
20:35:22 openstackgerrit Matt Riedemann proposed openstack/nova master: Use named kwargs in compute.API.resize https://review.opendev.org/693425
20:36:14 mriedem god py38 non-voting talk already?
20:36:20 mriedem can we just skip to py500 already
20:45:12 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove now invalid TODO from ComputeManager._confirm_resize https://review.opendev.org/693427
20:54:16 openstackgerrit Merged openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668
20:54:23 openstackgerrit Merged openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012
21:24:14 mriedem man test_detach_volume_shelved_or_offload_server seems to fail a lot
21:26:45 openstackgerrit François Palin proposed openstack/nova master: Add retry to cinder api calls related to volume detach https://review.opendev.org/669674
21:28:26 eandersson Interesting - a VM without an allocation that is migrated also does not get an allocation
21:28:38 eandersson Would expected it to get "repaired" :p
21:28:41 openstackgerrit Merged openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906
21:28:41 openstack bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt)
21:28:44 openstackgerrit Merged openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907
21:31:04 eandersson melwitt, is that a bug in rocky as well?
21:31:20 melwitt no, stein is the earliest it shows up
21:36:13 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement update_provider_tree for mocked driver in test_resource_tracker https://review.opendev.org/693431
21:37:42 mriedem eandersson: you should see this in the conductor logs https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/migrate.py#L51
21:38:17 mriedem i would expect the scheduler to create allocations for the instance against the dest node
21:38:29 mriedem and those to stick if the migration completes successfully
21:38:48 eandersson We basically moved a site from one DC to another
21:38:58 eandersson and migrated all VMs
21:39:18 ccstone we didn't have nova in debug so we wouldn't have seen that message :/
21:39:21 eandersson all of them was "healed" using the heal_allocations
21:40:29 mriedem ok, i don't know what would be going on there, but i'd expect the migration to heal the instance allocations as well
21:40:37 mriedem s/heal/recreate on dest node/
21:41:44 eandersson Of course we are basing this on the heal_allocations output
21:45:14 openstackgerrit François Palin proposed openstack/nova master: Add retry to cinder api calls related to volume detach https://review.opendev.org/669674
21:46:17 mriedem so the vm didn't have allocations, you migrated it successfully (and confirmed the migration), and ran heal allocations and it healed it rather than saying there was nothing to do?
21:47:03 eandersson pretty much yes
21:47:14 mriedem fake news!
21:47:16 eandersson haha
21:47:58 eandersson ignorance is probably the leading factor to fake news :p
21:48:49 mriedem no dude it's all ukraine
21:48:58 eandersson lol
21:49:01 mriedem so i wonder if on confirm we're doing something dumb and wiping out the allocations that the scheduler created
21:49:15 mriedem do you see any messages in the compute logs for that operation?
21:50:15 mriedem so i bet you don't get here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L4143
21:50:35 mriedem which means you'd see this in the logs i think (on the source node)
21:50:36 mriedem https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L4143
21:50:47 mriedem and i bet we drop the allocations here https://github.com/openstack/nova/blob/stable/rocky/nova/compute/manager.py#L4143
21:51:55 mriedem it's a guess, but i'd expect to see some messages in the compute logs related to that
21:53:07 eandersson I would have to dig in our logs as this was probably a month ago
21:53:15 eandersson when we actually performed the migrations
21:54:51 mriedem do your logs go that far back?
21:54:59 mriedem and do you have debug enabled on the compute service?
21:55:10 eandersson We never got debug enabled :'(
21:55:33 mriedem ok, it might not be something at debug, i'd expect to see log messages about how it expected to find allocations but found none, doing old style thingies
21:56:00 mriedem would probably also be pretty easy to recreate with a test vm, just create it, drop its allocations in placement, migrate and confirm and then check if the allocations are still gone
21:56:43 eandersson It's probably one of those things that only affect users that do crazy things like upgrade from mitaka to rocky :p
22:02:09 efried sean-k-mooney: for some reason n-cpu still thinks I'm using libvirt 4.0.0. How do I get it to pick up the thing I built?
22:02:33 mriedem efried: you likely need to recompile libvirt-python
22:02:47 efried I've got a recent libvirt-python
22:02:54 efried uhh
22:02:57 mriedem but is it built against the version of libvirt you're running now?
22:03:03 efried yeah
22:03:13 efried what do you mean 'built against'?
22:03:19 efried I didn't build it locally against the libvirt I built locally.
22:03:22 efried But the version matches.
22:03:40 mriedem the package version on libvirt python != the version of libvirt it thinks you have
22:03:51 mriedem it's just a python shim over the libvirt c api
22:04:12 mriedem so from what i remember you have to make sure your libvirt-python is built from the same libvirt source as the service
22:04:20 mriedem otherwise you think you're using x but you're getting y
22:04:23 mriedem and it's confusing
22:05:19 efried Hm. When I was trying to install libvirt-python 5.7 with libvirt 5.8 on my system, it wouldn't even install, complaining of symbols. I had to have libvirt-python >= libvirt for it to work.
22:05:30 efried So *something* was checking that I had the right libvirt.
22:07:10 mriedem ok at this point you're going to have to phone a red hat friend
22:11:07 efried I have a thing to try.
22:11:07 efried I restacked because I needed a newer nova branch and it wasn't working to just pull and restart services
22:11:07 efried and I wanted to clean because the problem was in the db
22:11:07 efried but I didn't want to have to recompile libvirt/qemu again
22:11:07 efried so I disabled sean-k-mooney's plugin before unstack, clean, stack
22:11:08 efried so I suspect the `stack` installed and set up stock libvirt again
22:11:08 efried so I'm going to re-enable the plugin and restack (without cleaning)...
22:12:04 efried yeesh, it's probably as simple as `make install` in those dirs, but too late, I already unstacked...
22:17:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement update_provider_tree for mocked driver in test_resource_tracker https://review.opendev.org/693431
22:17:58 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop compat for non-update_provider_tree code paths https://review.opendev.org/693435
22:20:44 openstackgerrit Matt Riedemann proposed openstack/nova master: Drop compat for non-update_provider_tree code paths https://review.opendev.org/693435
22:25:49 openstackgerrit Matt Riedemann proposed openstack/nova master: Delete _normalize_inventory_from_cn_obj https://review.opendev.org/693436
22:31:52 efried that was it
22:32:23 efried oooo, I just saw those last few patch titles mriedem, I'm excited.
22:33:34 mriedem i knew you would be
22:35:16 openstackgerrit Matt Riedemann proposed openstack/nova master: Move compute_node_to_inventory_dict to test-only code https://review.opendev.org/693438
22:38:23 openstackgerrit Matt Riedemann proposed openstack/nova master: Remove get_minimum_version mocks from test_resource_tracker https://review.opendev.org/693439
22:54:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Use named kwargs in compute.API.resize https://review.opendev.org/693425

Earlier   Later