Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-20
16:44:39 dansmith right, but once that gap is more than N-1 it's technically out of scope,
16:44:45 dansmith even if we leave it for longer
16:45:12 dansmith if we really need to be defensive in the api about something, we should probably be looking at service version,
16:45:24 dansmith but we can't just look for those forever, which is why we have rules about the gaps
16:46:09 mriedem i don't know if that rule is written down anywhere
16:46:11 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Add functional tests to ensure BDM removal on delete https://review.openstack.org/546227
16:46:12 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Clean up ports and volumes when deleting ERROR instance https://review.openstack.org/546228
16:46:16 mriedem which is a question that gets asked every 6 months or so
16:46:31 dansmith the rule that we only support N-1?
16:46:34 mriedem in other words,
16:46:56 mriedem i think one could upgrade the api to queens while they have pike computes and it's fine, but when they upgrade the api to rocky, their computes have to all be queens
16:46:59 mriedem so it's a bit different behavior
16:47:34 mriedem but then multinode grenade wouldn't work...so that doesn't make sense
16:47:35 sean-k-mooney QQ anyone know if the regex support in the pci whitelist allow negation? basical i want to say whitlist vf where vendor_id=x product_id=y and pci_adress is not z
16:47:47 cfriesen what's the criteria for when we expect to create a Migration record on a rebuild?
16:48:10 mriedem cfriesen: a rebuild isn't a migration so never
16:48:48 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Add functional recreate test of deleting a BFV server pre-scheduling https://review.openstack.org/546230
16:49:06 cfriesen mriedem: what about evacuate or rebuild-to-new-server?
16:49:07 mriedem dansmith: here is an example of one of these things that i'd like to remove from the api https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L1138
16:49:18 mriedem cfriesen: the api creates the migration record for evacuate
16:49:23 mriedem cfriesen: you could just look at the code
16:49:57 mnaser i'm sure i messed up a conflict somewhere, i have to
16:50:10 cfriesen I am, but it wasn't obvious who was creating it...the ComputeManager code just does a "look it up if it exists and log if it doesn't"
16:50:20 dansmith mriedem: yeah, so if your api is on rocky past this commit, you (obviously) can't hit that, nor can you detect that it won't be supported unless you go by service version
16:50:24 cfriesen or rather, that's the ConductorManager code
16:50:48 dansmith mriedem: if you have configured your upgrade_levels to attempt to allow talking to older than queens nodes, you would have failed to start the api service entirely because rpc client setup will fail
16:55:06 openstackgerrit Mohammed Naser proposed openstack/nova stable/pike: Add functional tests to ensure BDM removal on delete https://review.openstack.org/546220
16:56:17 mriedem dansmith: so wondering if we should have an upgrade release note saying you can do rolling upgrades of your computes, but don't upgrade your api until all computes are at least rocky
16:56:48 dansmith mriedem: well, I don't think that we do because I feel like that is the rule, so calling it out makes this sound special when it's really not
16:57:15 dansmith mriedem: I'd rather clarify it in the docs somewhere if you think that's necessary
16:57:41 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Add functional tests to ensure BDM removal on delete https://review.openstack.org/546227
16:57:52 doude Hi jaypipes, could you have a look to that fix https://review.openstack.org/#/c/533212/ ? It's about a bug appeared when Contrail VIF vrouter plugging was moved under privsep. Not sure you are the person I should ping for that but I don't know how to proceed now. I proposed the fix mid-January and I think I'll miss the Queens release :(
16:57:59 dansmith we've had other cases before where if you spanned the N-1 gap you'd have much less obvious issues, and certainly the last rpc bump was the same
16:58:52 mriedem dansmith: i don't think https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process is very clear about it
16:59:04 mriedem 3rd bullet under #2
16:59:50 mriedem if i'm reading that correctly, it's saying bring down the control plane services, with nova-api last (ok), but then the next bullet is talking about rolling through the computes
17:00:10 dansmith mriedem: "This results in a condition where both old (N) and new (N+1) nova-compute services co-exist for a certain time period."
17:00:44 dansmith that's all that promises
17:00:53 dansmith but, if it's not clear, let's be clear about it
17:01:30 dansmith but I really don't want to get into the habit of trying to call out problematic releases more than others so that people think if it's not called out for a given release then they should expect it to work
17:01:53 mriedem that's fine,
17:02:11 mriedem but then i need to understand if what some of the api compat code was doing via the compute rpcapi client was wrong, so we don't continue that pattern
17:02:18 mriedem because it's in several different compute rpcapi methods
17:02:53 dansmith mriedem: I don't think it's wrong,
17:03:03 dansmith mriedem: at some point the api needs to assume it can make a call and it will work
17:03:31 mriedem ok, so that means, any of these compat checks in compute rpcapi made *before* queens are OK to remove
17:03:33 dansmith if the operator has set up something wrong or created some unsupportable scenario, a 5xx error code from the api is correct rihgt?
17:03:54 dansmith I think any that were made *in* queens are good to remove
17:04:08 dansmith we can't detect or honor any of them made for scenarios before 5.0
17:04:22 dansmith (in or before, in case it's not clear)
17:04:44 mriedem yeah, so then what the api could gracefully handle in queens, it cannot in rocky
17:04:54 dansmith right
17:04:58 mriedem but,
17:05:03 cfriesen In stable/pike if we do a rebuild-to-same-node with a new image, it results in ComputeManager.rebuild_instance() being called with "scheduled_node=<hostname>" and "recreate=False". This results in a new Claim, which seems wrong since we're not changing the flavor and that claim could fail if the compute node is already full.
17:05:05 mriedem we don't explicitly say the api can do that
17:05:19 mriedem is what you're saying, right?
17:05:47 mriedem you can have N control services and N and N-1 compute services, fin.
17:05:55 dansmith mriedem: we've said the api can be one rev newer than the computes, so our api code needs to handle the case where some computes don't support a function because they're N-1, and we do
17:06:01 mriedem no guarantees that the API will support the N-1 computes
17:06:08 dansmith as of this moment, api code for rocky can't have any computes that don't support things
17:06:44 dansmith no, we guarantee the api will support N-1 computes
17:06:45 dansmith but not N-2
17:06:46 dansmith and at the moment, N-1 computes can do all the things of the N ones, so no checks are needed
17:12:22 mriedem cfriesen: that's a regression
17:12:25 mriedem yet another one
17:12:45 dansmith gawd
17:13:27 mriedem i don't know why this isn't just "if recreate:
17:13:28 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2866
17:13:36 mriedem recreate == evacuate == always do a claim
17:14:15 mriedem this is stale: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2854
17:14:25 mriedem since the new crap to consult the scheduler if the image changes
17:15:30 mriedem cfriesen: open a bug
17:16:03 mriedem it's a regression from I11746d1ea996a0f18b7c54b4c9c21df58cc4714b
17:16:20 cfriesen mriedem: will do
17:20:40 openstack Launchpad bug 1750618 in OpenStack Compute (nova) "rebuild to same host with a different image results in erroneously doing a Claim" [Undecided,New]
17:20:40 cfriesen mriedem: https://bugs.launchpad.net/nova/+bug/1750618
17:21:00 mriedem cfriesen: are you going to post a patch?
17:21:05 mriedem i think it's just changing that one line
17:22:52 cfriesen just change it to only check "recreate" and ignore "scheduled_node"? yeah, I can post that
17:23:07 cfriesen I guess we'd need a new unit test?
17:23:29 cfriesen or a tweak to one of the existing ones to check if we're doing a real claim
17:23:37 mriedem i guess, there might already be one that has to change,
17:23:45 mriedem the problem here isn't in unit tests,
17:24:06 mriedem the problem is the fact that rebuild/evacuate is so tightly coupled between api, conductor and compute, and what the assumptions are between all of them about what the other is doing
17:24:21 cfriesen fair enough
17:24:22 mriedem so changing anything in one breaks the others
17:24:43 mriedem a solid test would be a functional rebuild test with a new image that asserts that we never do a claim
17:24:54 mriedem i know we have functional tests that do a rebuild with a new image,
17:25:04 mriedem but it's using the fake driver so likely the claim stuff was never a problem
17:25:15 mriedem there might be a way to poison the claim such that if it's called, it will fail
17:25:35 openstackgerrit Merged openstack/nova master: Check quota before creating volume snapshots https://review.openstack.org/520158
17:27:43 mriedem cfriesen: functional test 'test_rebuild_with_new_image' should probably be tweaked
17:27:56 mriedem such that if nova.compute.resource_tracker.ResourceTracker.rebuild_claim is called, it blows up
17:28:53 cfriesen will take a look.
17:30:13 mriedem bauzas: melwitt was +2 on https://review.openstack.org/#/c/545132/ before i updated the commit message, can you just proxy her +2 and approve so we can do backports?
17:33:15 dansmith jaypipes: cdent which of you was going to work on the placement api change to allow passing aggregates?
17:33:40 cfriesen mriedem: for the rebuild-on-same-host-with-different-image case, if it runs through the scheduler could it fail due to placement if there isn't room for another one of itself on its current compute node?
17:35:10 mriedem cfriesen: we don't claim in that case
17:35:27 cdent dansmith: I probably _can_ do it, but I haven't made plans to do it
17:35:53 mriedem although i guess it's possible that placement could return 0 allocation candidates and we'd get a novalidhost

Earlier   Later