Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-20
18:04:56 dansmith but yeah, rabbit split brain could have done something I guess
18:05:27 dansmith mriedem: what if, in bury_in_cell0, we check last minute to see if there's already a host mapping and if so, we log and abort?
18:05:37 dansmith I mean, I would think that's where we fail in that routine anyway,
18:05:50 dansmith so there should be some log fallout from us failing to create the cell0 mapping anyway right?
18:06:03 mriedem is this one of the many reasons people say to not cluster rabbit?
18:06:07 mriedem beyond performance?
18:06:27 mnaser rabbitmq is the worlds biggest pita :(
18:07:55 mnaser ok so at least to clean things up i will update the instance_mappings to point that the right cell
18:08:01 mnaser and then at least the instance will be delete-able
18:08:12 mnaser and then the cell0 will just get archived and disappear
18:08:52 mriedem dansmith: i'm not sure what you mean by "already a host mapping" in bury_in_cell0
18:09:09 dansmith mriedem: when we bury in cell0, we also have to create a mapping for it to be there
18:09:22 dansmith mriedem: we should fail to do that if we're late to the party because we're a failed reschedule right?
18:09:37 mnaser hmm yeah, that's true, i guess the bury doesn't check if there's an existing one there
18:09:43 mriedem dansmith: so here https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1282 ?
18:09:52 mriedem before doing that, double check that instance_mapping.cell_mapping is None, right?
18:11:03 mriedem i mean we'd want to check way before that though, before this https://github.com/openstack/nova/blob/stable/stein/nova/conductor/manager.py#L1260
18:11:41 mriedem i'm not opposed to adding a sanity check
18:11:48 mriedem worst case is it doesn't hurt anything,
18:11:58 mriedem best case is it avoids crazy rabbit clustering split brain weirdness
18:12:10 dansmith mriedem: well, I'm thinking more like direct hit the database to make sure
18:12:21 dansmith mriedem: but again, we should fail to create a duplicate mapping there
18:13:10 mnaser is this just a matter of `if inst_mapping.cell_mapping is not None:` before updating it?
18:14:22 dansmith no
18:19:52 dansmith mriedem: oh, hmm
18:20:02 dansmith mriedem: we look up the mapping and set it to cell0 there and then save, not a create
18:20:05 dansmith that's why no dupe
18:20:21 dansmith so maybe the cell0 save happens first and then the cell1 one comes in later?
18:20:31 dansmith so, mnaser maybe yes to your above question
18:20:44 dansmith I was forgetting how this worked, that we create the mapping early and then update it later
18:21:34 mriedem mnaser: yes that's what i was thinking
18:21:37 mriedem dansmith: the create happens in the api
18:21:45 mriedem atomically with the build request and request spec
18:21:55 mriedem yeah that (sorry was catching up)
18:22:36 mriedem mnaser: so i'd move the code that gets the instance mapping earlier before we create the instance record in cell0, check if instance_mapping.cell_mapping is not None and if not log an error or something and return
18:22:57 mriedem note we still have a "remove after ocata" thing in that method :)
18:23:30 dansmith mriedem: move the getting of the mapping? I would think you'd want it very tight like it is now so you had the best opportunity to see that it has been set by a competitor
18:37:35 mriedem dansmith: as long as it's before we create the instance record in cell0
18:38:13 mriedem b/c otherwise you have to roll that back
18:38:24 mriedem anyway, i think the shed should be bronw
18:38:26 mriedem *brown
18:43:01 dansmith maybe we're talking about different things
18:47:17 mriedem mnaser: were you going to push a patch for this sanity check?
18:47:24 mriedem it'd be easier to discuss in review
19:18:32 openstackgerrit Artom Lifshitz proposed openstack/nova master: Poison netifaces.interfaces() in tests https://review.opendev.org/671773
19:37:45 mnaser mriedem: i was *if* i was talking about the right thing
19:38:05 mnaser even if i pushed the initial patch tbh i dont know if i have enough bandwidth to drive it all the way through review and all right now
19:38:11 mnaser but i can do the initial if statement before the update there..
19:38:17 mnaser with a test
19:40:11 openstackgerrit Artom Lifshitz proposed openstack/nova master: Rename Claims resources to compute_node https://review.opendev.org/679470
19:45:37 mriedem travis --com, travis --org, what a mess
19:46:31 mriedem mnaser: that's good enough to start, it shouldn't be a difficult change, i think the complexity is mostly in the test since we want to assert that (1) we fail if the instance mapping already has a cell mapping set in _bury_in_cell0 and (2) we check that before calling instance.create
20:24:45 openstackgerrit Matt Riedemann proposed openstack/nova master: Clear instance.launched_on when build fails https://review.opendev.org/683725
21:01:44 mriedem maybe we should finally fix that bug where nova-compute creates volumes but doesn't name them...
21:10:44 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: Sanity check instance mapping in _bury_in_cell0 https://review.opendev.org/683730
21:11:04 mriedem mnaser: dansmith: ^ this is what i was thinking
21:11:10 mriedem it's a bit fuglier than i expected
21:12:15 dansmith mriedem: okay I thought you meant adding it quite a bit earlier
21:12:19 dansmith er, moving
21:12:23 mriedem so did i,
21:12:26 mriedem but realized we have a list here
21:12:31 mriedem so it has to go in the loop
21:12:42 dansmith mriedem: the other thing is, we should probably do the same sanity check when we set it for a non cell0 cell,
21:12:51 dansmith because if cell0 is faster (likely) then we'll create it there and map it,
21:12:56 dansmith then remap it to cell1
21:13:05 dansmith and we should at least warn that that happened for the forensic value
21:13:08 mriedem yeah - can you leave a comment so i don't forget?
21:13:12 dansmith I mean...if we think this is what is happening
21:14:05 dansmith done
21:14:09 mriedem danke
21:18:37 mriedem replied with a question,
21:18:50 mriedem mostly about leaving 2 copies in different dbs which could mess up listing,
21:19:16 mriedem i guess the warning could just be "uh oh spaghettios this is in cell0 and now it's in cell1 too - we're going to trust the cell1 version but you'll need to cleanup cell0"
21:20:11 dansmith it's going to be in both places anyway, apparently, so mapped appropriately and logged is minimal best case I think
21:20:28 dansmith you could also nuke the cell0 version, but I kinda want to see evidence that this is really happening first
21:21:28 mriedem right i don't want to overengineer this
21:21:35 mriedem anyway, not going to happen today, i'm taking off
21:21:55 dansmith ack
22:07:12 mlavalle I am working on a devstack built with stable rockey. I modified and restarted the nova api with the following changes: https://review.opendev.org/#/c/674038 and https://review.opendev.org/#/c/645452. I also modified https://github.com/openstack/nova/blob/master/nova/policies/servers.py#L58 to base.RULE_ADMIN_API or base.SYSTEM_READER
22:08:37 mlavalle granted alt_demo user the reader role with system(all) scope. alt_demo is still unbale to do list server --all-users. what am I missing?
22:09:18 mlavalle granted alt_demo user the reader role with system(all) scope. alt_demo is still unbale to do list server --all-projects. what am I missing?
22:09:57 mlavalle lbragstad: ^^^^ any advice?
22:14:31 lbragstad mlavalle what kind of token are you using to make the request to nova?
22:19:05 lbragstad mlavalle you could put something like https://pasted.tech/pastes/7d06348fdea072ad4784fa75940c142dc3d63f86.raw in your clouds.yaml and then export OS_CLOUD=devstack-alt-system to ensure you're using a system-scoped token in your request
22:19:36 lbragstad i assume you're using openstackclient
22:19:37 mlavalle lbragstad: duuh. that was the problem. I was getting a project scoped token. I just tested with a system scoped token and it worked
22:19:47 lbragstad mlavalle nice
22:19:51 mlavalle thanks!
22:19:59 mlavalle have a nice weekend
22:20:00 lbragstad no problem
22:20:02 lbragstad you, too
#openstack-nova - 2019-09-21
00:47:44 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP: Provider Config File: Merge provider configs to provider tree https://review.opendev.org/676522
01:41:41 openstackgerrit Merged openstack/nova master: libvirt: Start reporting 'HW_CPU_HYPERTHREADING' trait https://review.opendev.org/675571
01:41:49 openstackgerrit Merged openstack/nova master: tests: Additional functional tests for pinned instances https://review.opendev.org/681750
01:53:01 openstackgerrit Merged openstack/nova master: Include both VCPU and PCPU in core quota count https://review.opendev.org/681374
02:07:11 openstackgerrit Merged openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801
02:07:19 openstackgerrit Merged openstack/nova master: fakelibvirt: Make 'Connection.getHostname' unique https://review.opendev.org/681060
02:07:29 openstackgerrit Merged openstack/nova master: libvirt: Mock 'libvirt_utils.file_open' properly https://review.opendev.org/681061
02:07:38 openstackgerrit Merged openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895
03:40:34 openstackgerrit Merged openstack/nova master: libvirt: Enable driver configuring PMEM namespaces https://review.opendev.org/679640

Earlier   Later