Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
21:24:53 mriedem i wonder if you maybe have duplicate compute nodes?
21:24:57 mriedem but with different uuids
21:25:49 mriedem although unless the hostname changed i'm not sure how that could happen since there is a unique constraint on that table for host/hypervisor_hostname
21:26:27 mriedem melwitt: ^ this kind of problem is why switching counting quota usage from placement by default worries me
21:26:31 ceryx Yeah, all migration attempts were post-rocky upgrade. Each one of the resource provides does match a different compute_node that are all still online, they were just past targets for the failed migration.
21:27:10 mriedem ceryx: hmm, so maybe the scheduler created the allocations for that instance and each of those providers, but then the migration failed and we failed to cleanup the allocations created by the scheduler
21:27:13 ceryx In the allocations DB they were all created when we ran heal_allocations though according to the created_at date, so these aren't old allocations that were failed
21:27:48 ceryx All the allocations for this consumer_id across all 6 resource providers were created at 2019-04-11 20:25:57
21:27:57 mriedem like i said, heal_allocations should only create allocations if the instance doesn't have any and even then should only be against the same node for the instance.host/node values
21:28:07 mriedem do you have the output of the command?
21:28:34 melwitt mriedem: ack. the more examples I see, the more I lean toward making it opt-in by default for now. until we have allocations issues more sorted out
21:29:00 dansmith also why I want to make the image filter (and other prefilters) opt-in
21:29:06 dansmith until we're sure they're 100% right for everyone
21:29:44 mriedem ceryx: this is the method that does the actual work per instance https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1811
21:30:01 mriedem so the instance has to be on a node here https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1838
21:30:13 mriedem then we check to see if it already has allocations https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1848
21:30:52 mriedem if it doesn't, we get the compute node uuid which should match the provider https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1883
21:31:18 mriedem so i'm not sure why that would create allocations for the same instance against 6 different providers
21:31:48 mriedem what would be more likely to me is what i said before - the scheduler created the allocations during the migration, it failed, and then we didn't cleanup somewher
21:32:07 ceryx mriedem: I don't have the full output, was not expecting this many allocations to get created and it ate up all my scrollback.
21:32:47 ceryx What would the process be for cleaning up allocations that exist but shouldn't? Would it be relatively safe to delete the allocations for this one consumer_id, then rerun heal_allocations and confirm what was added back?
21:32:57 mriedem sec
21:33:06 mriedem mnaser has a script i think
21:33:30 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
21:33:30 mriedem https://bugs.launchpad.net/nova/+bug/1793569
21:35:00 mriedem ceryx: so that bug has a link to a script from mnaser which it looks like just dumps commands to run,
21:35:15 mriedem and links to another tool from larsks
21:36:27 mriedem ceryx: "Would it be relatively safe to delete the allocations for this one consumer_id, then rerun heal_allocations and confirm what was added back?" - i think so, but i'd also like it better if you had a --dry-run option when doing that with heal_allocations as well,
21:36:31 mriedem which i could probably wip up real quick
21:38:18 ceryx That would be awesome :D
21:38:48 mriedem ok will crank something out here
21:40:24 imacdonn_ so I seem to have a problem with Stein .. haven't fully diagnosed yet, but if I run online_data_migrations a second time, fill_virtual_interface_list fails with:
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage TypeError: 'NoneType' object has no attribute '__getitem__'
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage default_group = _security_group_get_by_names(context, ['default'])[0]
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4050, in _security_group_ensure_default
21:40:54 imacdonn_ ring any bells ?
21:42:23 imacdonn_ http://paste.openstack.org/show/73aAO3bB23d62wBjt5nL/
21:42:31 mriedem imacdonn_: not for me
21:43:20 imacdonn_ k. I'll try to dig into it a bit. Tnx.
21:46:21 efried imacdonn_: Looking at that method, that should be impossible
21:47:21 efried oh
21:47:30 efried imacdonn_: Do you have more than one security group named 'default'?
21:47:51 imacdonn_ well, each project has one.....
21:48:55 efried It oughtta be filtering by project ID
21:49:01 efried but that's the only way that method can return None
21:49:01 imacdonn_ I do see two rows in the security_groups table with name="default" but project_id NULL .. not sure if that should be
21:49:10 efried mhm, that'd do it.
21:49:25 efried if you called with project_id NULL somehow
21:49:27 efried in your context
21:49:46 efried imacdonn_: Repeatable?
21:49:58 mriedem ah,
21:50:05 mriedem the online_data_migratoin is using an admin context,
21:50:08 mriedem which doesn't have a project_id
21:50:42 imacdonn_ I'm seeing it in two different installations - one was a fresh install, the other upgraded from Rocky
21:51:26 efried mriedem: but that works fine unless there's multiple default security groups with project_id NULL, yah?
21:51:54 efried imacdonn_: So I can WIP a patch that ought to make the problem go away; or you can manually delete the extra row from your security groups table.
21:52:03 mriedem efried: he said he's running it twice
21:52:05 melwitt I really hope this isn't related to the user_id instance mapping migration somehow
21:52:05 efried course it'd be nice to know how it got there
21:52:06 mriedem and blows up the 2nd time right?
21:52:15 mriedem melwitt: he said it was the vifs one
21:52:22 mriedem "if I run online_data_migrations a second time, fill_virtual_interface_list fails with:"
21:52:32 melwitt yeah, but I added user_id to that, which shouldn't hurt
21:53:19 imacdonn_ yeah, it seemed to work OK the first time, but blows on subsequent attempts ... not sure where these NULL security groups are coming from
21:54:11 imacdonn_ in the frest install case, the projects probably didn't exist when migrations were run the first time
21:55:29 melwitt ok, I added the user_id for the fill virtual interface list instance mapping marker record. so shouldn't be related but just wanted to mention there was a change in stein there
22:07:33 mriedem the vifs migration was also new in stein
22:09:45 melwitt oh. nevermind me
22:10:20 mriedem imacdonn_: w/o looking at the code i think the migration is creating a marker instance record
22:10:29 mriedem which is why it's using an empty admin context with no project,
22:10:34 mriedem it should be using a sentinel for the project_id probably
22:10:59 mriedem should be fairly easy to reproduce that by just modifying an existing test to run the command twice
22:10:59 melwitt it uses a sentinel of all zeros uuid
22:11:45 melwitt https://github.com/openstack/nova/blob/master/nova/objects/virtual_interface.py#L303
22:11:50 mriedem melwitt: but the context doesn't have that
22:11:54 mriedem and thta's what the db api is looking for i think
22:12:33 melwitt oh, other direction
22:12:38 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/api.py#n4037
22:13:03 melwitt I see, ok
22:13:18 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP/PoC: Introduces the openstacksdk to nova https://review.openstack.org/643664
22:13:26 mriedem imacdonn_: report a bug
22:13:53 mriedem i'm glad we didn't backport that data migration yet...i was worried about just backporting it before anyone was using it (besides ovh) since it's pretty complicated
22:14:03 mriedem maciejjozefczyk: ^
22:14:10 imacdonn_ OK. Are we still using launchpad? I've been a bit out of the loop
22:14:20 mriedem ceryx: i've got this --dry-run patch coming, just building docs and running tests locally first
22:14:25 mriedem imacdonn_: of course
22:14:30 mriedem only crazy projects move to SB :)
22:14:35 imacdonn_ heh ok
22:14:37 melwitt imacdonn_: yes launchpad for nova. it's placement that has moved to storyboard
22:16:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Add --dry-run option to heal_allocations CLI https://review.openstack.org/651932
22:16:20 mriedem ceryx: eandersson: ^ should be backportable to rocky i think, that code hasn't changed much
22:16:26 mriedem or run it in a container or something
22:18:52 imacdonn_ https://bugs.launchpad.net/nova/+bug/1824435
22:18:54 openstack Launchpad bug 1824435 in OpenStack Compute (nova) "fill_virtual_interface_list migration fails on second attempt" [Undecided,New]
22:24:31 mriedem imacdonn_: thanks triaged - are you working a fix?
22:25:13 imacdonn_ mriedem, negative .. I don't think I understand the problem well though (yet?)
22:29:03 efried mriedem: I still don't get how the duplicate row is getting created.
22:29:14 efried Shouldn't https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/db/sqlalchemy/api.py#L4037 only happen the first time?
22:32:19 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:33:06 mriedem hmm yeah i'm not sure how https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/db/sqlalchemy/api.py#L3874 can't either return at least 1 or raise

Earlier   Later