| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-25 | |||
| 05:38:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299 | |
| 05:38:44 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299 | |
| 05:39:43 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344 | |
| 05:39:54 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344 | |
| 05:40:29 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673 | |
| 05:40:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673 | |
| 05:41:14 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676 | |
| 05:41:25 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676 | |
| 05:42:11 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689 | |
| 05:42:20 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689 | |
| 05:42:39 | openstackgerrit | Tony Breeds proposed openstack/nova stable/pike: Document unset/reset wrinkle for *_allocation_ratio options https://review.openstack.org/647292 | |
| 05:43:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709 | |
| 05:43:13 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709 | |
| 05:43:33 | openstackgerrit | Tony Breeds proposed openstack/nova stable/pike: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/613263 | |
| 05:43:51 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.openstack.org/576712 | |
| 05:43:54 | openstackgerrit | Tony Breeds proposed openstack/nova stable/pike: Update resources once in update_available_resource https://review.openstack.org/612295 | |
| 05:44:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.openstack.org/576712 | |
| 06:19:58 | openstackgerrit | Abhishek Kekane proposed openstack/nova-specs master: Support multiple backend of Glance https://review.openstack.org/641210 | |
| 06:58:11 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.openstack.org/641200 | |
| 06:59:58 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add TODO note for mox removal https://review.openstack.org/576758 | |
| 07:00:04 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add TODO note for mox removal https://review.openstack.org/576758 | |
| 08:22:35 | openstackgerrit | Martin Schuppert proposed openstack/nova stable/stein: Eventlet monkey patching should be as early as possible https://review.openstack.org/647310 | |
| 09:40:01 | openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Support adding the reason behind a server lock https://review.openstack.org/638629 | |
| 10:57:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add falvor to requested_resources in RequestSpec https://review.openstack.org/647396 | |
| 11:34:58 | openstackgerrit | Merged openstack/nova-specs master: Replace openstack.org git:// URLs with https:// https://review.openstack.org/646695 | |
| 12:36:46 | shilpasd | Hi: openstacksdk returning 400 for 'force_service_down' API with version as 2.72, pl refer https://bugs.launchpad.net/ubuntu/+source/python-openstacksdk/+bug/1821584 | |
| 12:36:47 | openstack | Launchpad bug 1821584 in python-openstacksdk (Ubuntu) "Compute 'force_service_down' API returns 400 with current version 2.72 " [Undecided,New] | |
| 12:36:49 | tssurya | melwitt, mriedem: I feel we should fallback to legacy if the queued_for_delete is not migrated and is still NULL like its done for user_id : https://review.openstack.org/#/c/638072/14/nova/objects/instance_mapping.py@386 | |
| 12:37:25 | shilpasd | Hi All: please confirm the same | |
| 12:58:38 | efried | shilpasd: Thanks for the bug report. | |
| 12:59:00 | efried | FYI there's a community-wide goal getting set up for Train to try to get more of the CLIs up to parity | |
| 12:59:27 | efried | If you have the means, it would be great if you wanted to propose a fix for this. | |
| 13:00:07 | shilpasd | <efried> sure | |
| 13:00:16 | shilpasd | efried: sure | |
| 13:00:18 | sean-k-mooney | efried: in this case its an openstacsdk limitation not a cli issue | |
| 13:00:33 | efried | sean-k-mooney: Yes, I agree. shilpasd opened the bug against the right component. | |
| 13:00:41 | sean-k-mooney | nova cli and osc do not use the sdk for the api currently | |
| 13:00:56 | sean-k-mooney | ah yes they did | |
| 13:02:45 | sean-k-mooney | efried: stephenfin and i were talking internally about ho to improve cli partity | |
| 13:04:00 | sean-k-mooney | efried: we were thinking a phased approch make the most sence. first make osc have partity with nova cli by just delegating to it internall. then port osc to openstack sdk once that is done | |
| 13:04:28 | sean-k-mooney | so form a user point of view we get to cli parity first then we can get teh sdk to partity and swap over to useing it | |
| 13:05:10 | sean-k-mooney | but there are still big gaps in that plan. e.g. should clint only things like nova host-evac be ported and how to deal with micorverions | |
| 13:07:57 | ganso | the Placement API via curl. After several migration attempts that fail in the same spot, the source node is filled with these allocations that prevent new instances from being created or instances migrated to this node. | |
| 13:07:57 | ganso | Hi folks. I am about to log a bug and would like to confirm if this is an intentional behavior or not: If an exception is raised by the driver during confirm_migration, the migration record is stuck in confirming state and the allocations are not removed. The instance is fine at the destination in this stage, but the source host has allocations that is not possible to clear without going to the database or invoking | |
| 13:08:39 | sean-k-mooney | efried: but ya in anycase i do hope we can make progress on that goal in train | |
| 13:16:18 | efried | ganso: What release are you running? | |
| 13:16:41 | ganso | efried: I tried pike, queens and stein | |
| 13:17:07 | ganso | efried: same problem in all of them ^ | |
| 13:18:50 | efried | ganso: Well, that doesn't sound great. mriedem is probably your best bet to say whether this is a known issue. If he doesn't chime in here I think you can safely open a bug. | |
| 13:19:34 | ganso | efried: thanks! I will wait for mriedem's confirmation first. | |
| 13:20:58 | mriedem | looking, but i wouldn't be at all surprised | |
| 13:21:58 | mriedem | ack on the migration.status not being set to error because the @errors_out_migration_ctx decorator is not used here https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L3933 | |
| 13:22:11 | mriedem | so you never get this far https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L4011 | |
| 13:22:22 | mriedem | nor do the allocations get cleaned up https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L4017 | |
| 13:22:32 | mriedem | so yes it's a big | |
| 13:22:38 | mriedem | setting the migration status to error is easy | |
| 13:22:48 | mriedem | knowing whether or not the allocations should be changed is harder | |
| 13:23:22 | mriedem | ganso: the instance status goes to ERROR right? https://github.com/openstack/nova/blob/926753ee6bebee5bacc29073e123ed5c1a0bb647/nova/compute/manager.py#L3993 | |
| 13:23:32 | ganso | mriedem: yes the instance goes to error | |
| 13:23:33 | mriedem | which means you can't revert the resize after confirm fails | |
| 13:23:45 | mriedem | unless you manually reset the vm_state to 'resized' in the db | |
| 13:24:16 | mriedem | so at this point, the instance is on the dest, the db points the instance at the dest, and confirm fails on the dest and you're stuck in error state, | |
| 13:24:17 | ganso | mriedem: I didn't try that. I noticed I could only change the status to either error or active, so I changed the status to active and could use the instance fine after that, even migrate it again. | |
| 13:24:25 | mriedem | i would say we probably just consider the instance is stuck there now and cleanup the allocations | |
| 13:24:39 | mriedem | ganso: yeah reset status only changes to active/errror | |
| 13:24:40 | mriedem | *error | |
| 13:24:53 | mriedem | so you could have tried hard rebooting the instance to fix it also since you can do that with an error instance | |
| 13:25:06 | mriedem | but that's not going to cleanup the allocations as you found | |
| 13:25:17 | mriedem | ganso: anyway, it's definitely a bug | |
| 13:25:24 | ganso | mriedem: hmmm seems to me the migration_confirm runs on the source, and fails there. The libvirt driver only cleans up on that method's implementation | |
| 13:25:51 | mriedem | oh right, yeah confirm_resize runs on the source host | |
| 13:25:54 | mriedem | to cleanup the source host | |
| 13:26:07 | efried | no confirm_resize --force? | |
| 13:26:18 | efried | "I know this thing is copacetic on the destination, make it so" | |
| 13:26:29 | mriedem | probably another good reason to just cleanup the guest on the hypervisor as best effort but still cleanup the allocations | |
| 13:26:43 | ganso | mriedem: thanks! I will log the bug =) | |
| 13:26:47 | mriedem | efried: that would assume you know you need to force it before the instance gets stuck in error status | |
| 13:27:01 | mriedem | you can't confirm resize on an error instance | |
| 13:27:10 | efried | That's what the force would be | |
| 13:27:20 | mriedem | ew | |
| 13:27:21 | efried | "I done checked the instance, and it's good. Make it look good in the databases and stuff" | |
| 13:27:30 | efried | shrug, just spitballing. | |
| 13:27:38 | mriedem | you can fix it up with a hard reboot | |
| 13:27:41 | mriedem | on the dest | |
| 13:27:44 | efried | except the allocations | |
| 13:27:48 | mriedem | but that won't clean the allocations | |
| 13:28:05 | mriedem | right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way | |
| 13:28:12 | ganso | efried, mriedem: yea the main problem are the allocations. | |
| 13:28:27 | efried | does nova manage heal_allocations work on something like this? | |
| 13:28:28 | mriedem | these giant compute / conductor manager methods are so convoluted that it gets hard to deal with errors in isolation | |
| 13:28:31 | mriedem | and if they should be fatal or not | |
| 13:28:53 | mriedem | efried: no, that only creates missing allocations, doesn't cleanup ones that shouldn't exist | |
| 13:29:01 | mriedem | mnaser has a script for that though | |
| 13:29:02 | mriedem | sec | |
| 13:29:16 | purplerbot | <mriedem> right, so what i'm saying is we should suck less and care for where errors can happen in a more granular way [2019-03-25 13:28:05.178818] [n c6v1] | |
| 13:29:16 | cdent | I'm gonna get a tattoo of [t c6v1] | |
| 13:29:22 | efried | What I'm getting at is, if there's some way to clean up the allocations other than curl'ing placement, I'm less concerned about this than otherwise | |
| 13:29:42 | mriedem | not built into nova no | |
| 13:29:52 | mriedem | there are various operator scripts floating around | |
| 13:31:59 | mriedem | https://bugs.launchpad.net/nova/+bug/1793569 | |