Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-02-20
16:39:43 mnaser bauzas: no worries. i'll get them all in and make sure they pass CI.. i think some might need some test changes because some function signatures are different
16:40:08 mriedem dansmith: checking my understanding here https://review.openstack.org/#/c/543580/1/nova/compute/rpcapi.py@a1265 - we have some REST API handler code for compat exceptions raised from the compute rpcapi client; but we can't remove that handling in the api in queens because we could still have queens computes sending those errors?
16:40:34 dansmith mriedem: s/queens/rocky?
16:41:08 mriedem i'm lost
16:41:11 dansmith mriedem: anything queens could handle, but was being compat for pike should be remove-able in rocky
16:41:30 mriedem some of these are compat for things added in queens
16:41:42 mriedem like the multiattach exception in reserve_block_device_name
16:42:02 dansmith compat for pike.5 right?
16:42:05 dansmith meaning,
16:42:06 mriedem https://review.openstack.org/#/c/543580/1/nova/compute/rpcapi.py@a1044
16:42:08 dansmith queens could handle it
16:42:14 dansmith but something between pike and queens could not
16:42:44 dansmith right, so you can't handle 4.x clients at all anymore in rocky after this,
16:42:55 dansmith but all queens (release) nodes can, and that's as far back as this supports
16:43:17 dansmith if you tried to pin to 4.15 in that case, you'd get the failure we saw in the early grenade run anyway, well before you got to this point
16:43:39 mriedem so i can't upgrade my api to rocky until all of my computes have been upgraded to queens right?
16:43:55 dansmith right, but that's the case anyway, per the rules
16:44:01 mriedem well,
16:44:13 mriedem we've had compat code in the api for lots of different cases
16:44:17 mriedem which is why i was wondering about this
16:44:23 mriedem the api has code to handle mixed version computes
16:44:39 dansmith right, but once that gap is more than N-1 it's technically out of scope,
16:44:45 dansmith even if we leave it for longer
16:45:12 dansmith if we really need to be defensive in the api about something, we should probably be looking at service version,
16:45:24 dansmith but we can't just look for those forever, which is why we have rules about the gaps
16:46:09 mriedem i don't know if that rule is written down anywhere
16:46:11 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Add functional tests to ensure BDM removal on delete https://review.openstack.org/546227
16:46:12 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Clean up ports and volumes when deleting ERROR instance https://review.openstack.org/546228
16:46:16 mriedem which is a question that gets asked every 6 months or so
16:46:31 dansmith the rule that we only support N-1?
16:46:34 mriedem in other words,
16:46:56 mriedem i think one could upgrade the api to queens while they have pike computes and it's fine, but when they upgrade the api to rocky, their computes have to all be queens
16:46:59 mriedem so it's a bit different behavior
16:47:34 mriedem but then multinode grenade wouldn't work...so that doesn't make sense
16:47:35 sean-k-mooney QQ anyone know if the regex support in the pci whitelist allow negation? basical i want to say whitlist vf where vendor_id=x product_id=y and pci_adress is not z
16:47:47 cfriesen what's the criteria for when we expect to create a Migration record on a rebuild?
16:48:10 mriedem cfriesen: a rebuild isn't a migration so never
16:48:48 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Add functional recreate test of deleting a BFV server pre-scheduling https://review.openstack.org/546230
16:49:06 cfriesen mriedem: what about evacuate or rebuild-to-new-server?
16:49:07 mriedem dansmith: here is an example of one of these things that i'd like to remove from the api https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L1138
16:49:18 mriedem cfriesen: the api creates the migration record for evacuate
16:49:23 mriedem cfriesen: you could just look at the code
16:49:57 mnaser i'm sure i messed up a conflict somewhere, i have to
16:50:10 cfriesen I am, but it wasn't obvious who was creating it...the ComputeManager code just does a "look it up if it exists and log if it doesn't"
16:50:20 dansmith mriedem: yeah, so if your api is on rocky past this commit, you (obviously) can't hit that, nor can you detect that it won't be supported unless you go by service version
16:50:24 cfriesen or rather, that's the ConductorManager code
16:50:48 dansmith mriedem: if you have configured your upgrade_levels to attempt to allow talking to older than queens nodes, you would have failed to start the api service entirely because rpc client setup will fail
16:55:06 openstackgerrit Mohammed Naser proposed openstack/nova stable/pike: Add functional tests to ensure BDM removal on delete https://review.openstack.org/546220
16:56:17 mriedem dansmith: so wondering if we should have an upgrade release note saying you can do rolling upgrades of your computes, but don't upgrade your api until all computes are at least rocky
16:56:48 dansmith mriedem: well, I don't think that we do because I feel like that is the rule, so calling it out makes this sound special when it's really not
16:57:15 dansmith mriedem: I'd rather clarify it in the docs somewhere if you think that's necessary
16:57:41 openstackgerrit Mohammed Naser proposed openstack/nova stable/ocata: Add functional tests to ensure BDM removal on delete https://review.openstack.org/546227
16:57:52 doude Hi jaypipes, could you have a look to that fix https://review.openstack.org/#/c/533212/ ? It's about a bug appeared when Contrail VIF vrouter plugging was moved under privsep. Not sure you are the person I should ping for that but I don't know how to proceed now. I proposed the fix mid-January and I think I'll miss the Queens release :(
16:57:59 dansmith we've had other cases before where if you spanned the N-1 gap you'd have much less obvious issues, and certainly the last rpc bump was the same
16:58:52 mriedem dansmith: i don't think https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process is very clear about it
16:59:04 mriedem 3rd bullet under #2
16:59:50 mriedem if i'm reading that correctly, it's saying bring down the control plane services, with nova-api last (ok), but then the next bullet is talking about rolling through the computes
17:00:10 dansmith mriedem: "This results in a condition where both old (N) and new (N+1) nova-compute services co-exist for a certain time period."
17:00:44 dansmith that's all that promises
17:00:53 dansmith but, if it's not clear, let's be clear about it
17:01:30 dansmith but I really don't want to get into the habit of trying to call out problematic releases more than others so that people think if it's not called out for a given release then they should expect it to work
17:01:53 mriedem that's fine,
17:02:11 mriedem but then i need to understand if what some of the api compat code was doing via the compute rpcapi client was wrong, so we don't continue that pattern
17:02:18 mriedem because it's in several different compute rpcapi methods
17:02:53 dansmith mriedem: I don't think it's wrong,
17:03:03 dansmith mriedem: at some point the api needs to assume it can make a call and it will work
17:03:31 mriedem ok, so that means, any of these compat checks in compute rpcapi made *before* queens are OK to remove
17:03:33 dansmith if the operator has set up something wrong or created some unsupportable scenario, a 5xx error code from the api is correct rihgt?
17:03:54 dansmith I think any that were made *in* queens are good to remove
17:04:08 dansmith we can't detect or honor any of them made for scenarios before 5.0
17:04:22 dansmith (in or before, in case it's not clear)
17:04:44 mriedem yeah, so then what the api could gracefully handle in queens, it cannot in rocky
17:04:54 dansmith right
17:04:58 mriedem but,
17:05:03 cfriesen In stable/pike if we do a rebuild-to-same-node with a new image, it results in ComputeManager.rebuild_instance() being called with "scheduled_node=<hostname>" and "recreate=False". This results in a new Claim, which seems wrong since we're not changing the flavor and that claim could fail if the compute node is already full.
17:05:05 mriedem we don't explicitly say the api can do that
17:05:19 mriedem is what you're saying, right?
17:05:47 mriedem you can have N control services and N and N-1 compute services, fin.
17:05:55 dansmith mriedem: we've said the api can be one rev newer than the computes, so our api code needs to handle the case where some computes don't support a function because they're N-1, and we do
17:06:01 mriedem no guarantees that the API will support the N-1 computes
17:06:08 dansmith as of this moment, api code for rocky can't have any computes that don't support things
17:06:44 dansmith no, we guarantee the api will support N-1 computes
17:06:45 dansmith but not N-2
17:06:46 dansmith and at the moment, N-1 computes can do all the things of the N ones, so no checks are needed
17:12:22 mriedem cfriesen: that's a regression
17:12:25 mriedem yet another one
17:12:45 dansmith gawd
17:13:27 mriedem i don't know why this isn't just "if recreate:
17:13:28 mriedem https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2866
17:13:36 mriedem recreate == evacuate == always do a claim
17:14:15 mriedem this is stale: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2854
17:14:25 mriedem since the new crap to consult the scheduler if the image changes
17:15:30 mriedem cfriesen: open a bug
17:16:03 mriedem it's a regression from I11746d1ea996a0f18b7c54b4c9c21df58cc4714b
17:16:20 cfriesen mriedem: will do
17:20:40 openstack Launchpad bug 1750618 in OpenStack Compute (nova) "rebuild to same host with a different image results in erroneously doing a Claim" [Undecided,New]
17:20:40 cfriesen mriedem: https://bugs.launchpad.net/nova/+bug/1750618
17:21:00 mriedem cfriesen: are you going to post a patch?
17:21:05 mriedem i think it's just changing that one line
17:22:52 cfriesen just change it to only check "recreate" and ignore "scheduled_node"? yeah, I can post that

Earlier   Later