Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-11
21:27:13 ceryx In the allocations DB they were all created when we ran heal_allocations though according to the created_at date, so these aren't old allocations that were failed
21:27:48 ceryx All the allocations for this consumer_id across all 6 resource providers were created at 2019-04-11 20:25:57
21:27:57 mriedem like i said, heal_allocations should only create allocations if the instance doesn't have any and even then should only be against the same node for the instance.host/node values
21:28:07 mriedem do you have the output of the command?
21:28:34 melwitt mriedem: ack. the more examples I see, the more I lean toward making it opt-in by default for now. until we have allocations issues more sorted out
21:29:00 dansmith also why I want to make the image filter (and other prefilters) opt-in
21:29:06 dansmith until we're sure they're 100% right for everyone
21:29:44 mriedem ceryx: this is the method that does the actual work per instance https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1811
21:30:01 mriedem so the instance has to be on a node here https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1838
21:30:13 mriedem then we check to see if it already has allocations https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1848
21:30:52 mriedem if it doesn't, we get the compute node uuid which should match the provider https://github.com/openstack/nova/blob/stable/rocky/nova/cmd/manage.py#L1883
21:31:18 mriedem so i'm not sure why that would create allocations for the same instance against 6 different providers
21:31:48 mriedem what would be more likely to me is what i said before - the scheduler created the allocations during the migration, it failed, and then we didn't cleanup somewher
21:32:07 ceryx mriedem: I don't have the full output, was not expecting this many allocations to get created and it ate up all my scrollback.
21:32:47 ceryx What would the process be for cleaning up allocations that exist but shouldn't? Would it be relatively safe to delete the allocations for this one consumer_id, then rerun heal_allocations and confirm what was added back?
21:32:57 mriedem sec
21:33:06 mriedem mnaser has a script i think
21:33:30 mriedem https://bugs.launchpad.net/nova/+bug/1793569
21:33:30 openstack Launchpad bug 1793569 in OpenStack Compute (nova) "Add placement audit commands" [Wishlist,Confirmed]
21:35:00 mriedem ceryx: so that bug has a link to a script from mnaser which it looks like just dumps commands to run,
21:35:15 mriedem and links to another tool from larsks
21:36:27 mriedem ceryx: "Would it be relatively safe to delete the allocations for this one consumer_id, then rerun heal_allocations and confirm what was added back?" - i think so, but i'd also like it better if you had a --dry-run option when doing that with heal_allocations as well,
21:36:31 mriedem which i could probably wip up real quick
21:38:18 ceryx That would be awesome :D
21:38:48 mriedem ok will crank something out here
21:40:24 imacdonn_ so I seem to have a problem with Stein .. haven't fully diagnosed yet, but if I run online_data_migrations a second time, fill_virtual_interface_list fails with:
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 4050, in _security_group_ensure_default
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage default_group = _security_group_get_by_names(context, ['default'])[0]
21:40:27 imacdonn_ 2019-04-11 03:51:27.632 22147 ERROR nova.cmd.manage TypeError: 'NoneType' object has no attribute '__getitem__'
21:40:54 imacdonn_ ring any bells ?
21:42:23 imacdonn_ http://paste.openstack.org/show/73aAO3bB23d62wBjt5nL/
21:42:31 mriedem imacdonn_: not for me
21:43:20 imacdonn_ k. I'll try to dig into it a bit. Tnx.
21:46:21 efried imacdonn_: Looking at that method, that should be impossible
21:47:21 efried oh
21:47:30 efried imacdonn_: Do you have more than one security group named 'default'?
21:47:51 imacdonn_ well, each project has one.....
21:48:55 efried It oughtta be filtering by project ID
21:49:01 imacdonn_ I do see two rows in the security_groups table with name="default" but project_id NULL .. not sure if that should be
21:49:01 efried but that's the only way that method can return None
21:49:10 efried mhm, that'd do it.
21:49:25 efried if you called with project_id NULL somehow
21:49:27 efried in your context
21:49:46 efried imacdonn_: Repeatable?
21:49:58 mriedem ah,
21:50:05 mriedem the online_data_migratoin is using an admin context,
21:50:08 mriedem which doesn't have a project_id
21:50:42 imacdonn_ I'm seeing it in two different installations - one was a fresh install, the other upgraded from Rocky
21:51:26 efried mriedem: but that works fine unless there's multiple default security groups with project_id NULL, yah?
21:51:54 efried imacdonn_: So I can WIP a patch that ought to make the problem go away; or you can manually delete the extra row from your security groups table.
21:52:03 mriedem efried: he said he's running it twice
21:52:05 efried course it'd be nice to know how it got there
21:52:05 melwitt I really hope this isn't related to the user_id instance mapping migration somehow
21:52:06 mriedem and blows up the 2nd time right?
21:52:15 mriedem melwitt: he said it was the vifs one
21:52:22 mriedem "if I run online_data_migrations a second time, fill_virtual_interface_list fails with:"
21:52:32 melwitt yeah, but I added user_id to that, which shouldn't hurt
21:53:19 imacdonn_ yeah, it seemed to work OK the first time, but blows on subsequent attempts ... not sure where these NULL security groups are coming from
21:54:11 imacdonn_ in the frest install case, the projects probably didn't exist when migrations were run the first time
21:55:29 melwitt ok, I added the user_id for the fill virtual interface list instance mapping marker record. so shouldn't be related but just wanted to mention there was a change in stein there
22:07:33 mriedem the vifs migration was also new in stein
22:09:45 melwitt oh. nevermind me
22:10:20 mriedem imacdonn_: w/o looking at the code i think the migration is creating a marker instance record
22:10:29 mriedem which is why it's using an empty admin context with no project,
22:10:34 mriedem it should be using a sentinel for the project_id probably
22:10:59 melwitt it uses a sentinel of all zeros uuid
22:10:59 mriedem should be fairly easy to reproduce that by just modifying an existing test to run the command twice
22:11:45 melwitt https://github.com/openstack/nova/blob/master/nova/objects/virtual_interface.py#L303
22:11:50 mriedem melwitt: but the context doesn't have that
22:11:54 mriedem and thta's what the db api is looking for i think
22:12:33 melwitt oh, other direction
22:12:38 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/db/sqlalchemy/api.py#n4037
22:13:03 melwitt I see, ok
22:13:18 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP/PoC: Introduces the openstacksdk to nova https://review.openstack.org/643664
22:13:26 mriedem imacdonn_: report a bug
22:13:53 mriedem i'm glad we didn't backport that data migration yet...i was worried about just backporting it before anyone was using it (besides ovh) since it's pretty complicated
22:14:03 mriedem maciejjozefczyk: ^
22:14:10 imacdonn_ OK. Are we still using launchpad? I've been a bit out of the loop
22:14:20 mriedem ceryx: i've got this --dry-run patch coming, just building docs and running tests locally first
22:14:25 mriedem imacdonn_: of course
22:14:30 mriedem only crazy projects move to SB :)
22:14:35 imacdonn_ heh ok
22:14:37 melwitt imacdonn_: yes launchpad for nova. it's placement that has moved to storyboard
22:16:19 openstackgerrit Matt Riedemann proposed openstack/nova master: Add --dry-run option to heal_allocations CLI https://review.openstack.org/651932
22:16:20 mriedem ceryx: eandersson: ^ should be backportable to rocky i think, that code hasn't changed much
22:16:26 mriedem or run it in a container or something
22:18:52 imacdonn_ https://bugs.launchpad.net/nova/+bug/1824435
22:18:54 openstack Launchpad bug 1824435 in OpenStack Compute (nova) "fill_virtual_interface_list migration fails on second attempt" [Undecided,New]
22:24:31 mriedem imacdonn_: thanks triaged - are you working a fix?
22:25:13 imacdonn_ mriedem, negative .. I don't think I understand the problem well though (yet?)
22:29:03 efried mriedem: I still don't get how the duplicate row is getting created.
22:29:14 efried Shouldn't https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/db/sqlalchemy/api.py#L4037 only happen the first time?
22:32:19 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP/PoC: Use SDK instead of ironicclient for node.get https://review.openstack.org/642899
22:33:06 mriedem hmm yeah i'm not sure how https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/db/sqlalchemy/api.py#L3874 can't either return at least 1 or raise
22:35:45 efried oh, that part is because there's two rows in the database with the "right" project_id (NULL)
22:36:03 efried mriedem: So the initial check (==) fails because there's *more* db rows than expected
22:36:16 efried and then the for loop doesn't hit because all the names match (because they're the same)
22:36:21 mriedem ah yup
22:36:33 mriedem i don't know why the unique constraint doesn't blow up - because NULL isn't considered unique?
22:36:37 efried so the problem is that we're somehow creating two rows, and I don't know how that ....

Earlier   Later