Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-16
15:31:30 dansmith mriedem: we might benefit from a "needs upgrade" flag on the instance, that would be shown in detailed list, to admins only,
15:31:44 dansmith where I could list all tenants and see instances that have "needs upgrade"
15:32:16 sean-k-mooney so the old usecase woulould be achive with a member_of request in the flavor extra spec
15:32:25 dansmith anything could set that flag, like compute manager when it notices some legacy data, or even libvirt when it notices something like a disk format that needs to be upgraded or something
15:32:51 mriedem dansmith: if the thing setting the flag has to already calculate that it will set the flag, why not just do the online migration right then?
15:33:31 dansmith mriedem: well, because things like the resource topology thing would need to be done to all instances on the compute node at the same time, and after a config change
15:33:45 dansmith mriedem: referring to the thing stephenfin and jaypipes and bauzas and I were discussing last week
15:34:04 bauzas FWIW, on a 1.5h meeting atm
15:34:15 bauzas but listening
15:34:45 mriedem so instances a,b,c need an upgrade, but the admin doesn't know how to upgrade them, i.e. is it migrating them, running the online_data_migrations cli, restarting the compute they are on, etc
15:35:22 mriedem if the flag were an enum that's one way
15:35:41 dansmith mriedem: yeah, so it'd be nice if we also tagged "issues" to the instance that you whittle down until the upgrade flag goes away, but I think that's too heavy for the moment.. but if we LOG.warning() for each instance that we were setting the flag on, then you would at least have a record
15:35:55 dansmith mriedem: yeah, could be iterative, like service_version
15:36:01 dansmith if we always did those things in order
15:36:22 sean-k-mooney dansmith: could you tie it into a nova status check or something
15:36:42 mriedem i don't think i'd rely on someone catching that warning
15:36:49 openstackgerrit Merged openstack/nova stable/rocky: Delete allocations even if _confirm_resize raises https://review.openstack.org/652146
15:36:50 mriedem before their logs wrap or something
15:36:57 mriedem depends on how long they retain stuff in es
15:37:08 mriedem we already blast the logs with warnings that aren't useful
15:37:43 dansmith mriedem: sure, it just makes it lighter for the first rev of the idea, but if we made it a "schema version" sort of iterative "level" like service_version that could be easy, as you would look up the reason in the decoder ring
15:38:21 cdent Is there something up with the gate today? queue seems long and slow
15:38:22 dansmith and we could translate it in the api to "is current or not" if the version is backlevel (and show the version I guess)
15:38:33 mriedem cdent: it was like that yesterday too
15:38:53 mriedem dansmith: i'm not sure what service_version you're referring to
15:38:59 mriedem just the Service.version?
15:39:05 dansmith no,
15:39:34 openstackgerrit Merged openstack/nova stable/rocky: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/651797
15:39:37 dansmith SERVICE_VERSION is a global counter of things we've done to services, so we can tell if they're up to date or not
15:40:23 dansmith we could have a similar global "instance version" which was a little more like a schema version, where we record whether instance records had been modified for a specific transition
15:41:04 dansmith this is not important right now, I was thinking you'd latch onto this so we could expose more info to the operators about upgraded-ness, but I'll just bring it up the next time it's appropriate, like in those specs
15:41:34 mriedem ack
15:42:17 bauzas sean-k-mooney: so, about the upgrade impact of the cpu-resources thing, I had an idea
15:42:38 bauzas what we could do is potentially leave operators upgrading to Train without any impact
15:42:54 bauzas ie. config options act exactly like Stein
15:43:02 bauzas (for the existing ones)
15:43:07 mriedem cdent: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?panelId=3&fullscreen&orgId=1
15:43:27 bauzas but, before you would like to use new config options, you'd have to say "hey, reshape my stuff"
15:43:29 sean-k-mooney bauzas: yes. i suggested not enableing any of the new functionality if the new option was not set
15:43:31 bauzas for this host
15:43:40 sean-k-mooney yep
15:43:44 sean-k-mooney so they would upgrade
15:43:49 bauzas and in this case, we would model the new inventories
15:43:51 cdent mriedem: "data without context can never be information"
15:44:01 sean-k-mooney then move vms if need and then enable the new funcitonaltiy
15:44:10 bauzas from an operator pov, it would mean "upgrade your cloud, upgrade your hosts"
15:44:14 mriedem cdent: just saying there is a spike in the check queue so things are maybe slow as a result
15:44:14 sean-k-mooney so they could to a rooling config update after the upgrade
15:44:21 bauzas when you're done, then upgrade your config when you want
15:44:32 sean-k-mooney yep
15:44:42 bauzas a nova-status check could do the pre-flight check
15:44:45 sean-k-mooney i think we are on the same page for that workflow
15:44:49 cdent mriedem: sure, but we knew that already (or at least I did)
15:45:13 sean-k-mooney bauzas: it requires use to support both codepaths in train
15:45:22 sean-k-mooney then remove teh old one in U
15:45:29 bauzas sean-k-mooney: yup, this was my concern
15:45:31 sean-k-mooney but i think that will help with upgrades significantly
15:45:31 bauzas but
15:45:37 mriedem cdent: well you didn't tell me that, so who's misinformed now?!
15:45:52 bauzas with addition to be : trigger a reshape by the operator
15:45:58 bauzas or
15:46:05 bauzas trigger a reshape by the config change
15:46:07 cdent heh. touche
15:48:20 sean-k-mooney bauzas: yep i was going to suggest that too.
15:48:52 sean-k-mooney stephenfin: ^ when you get a chance after this call does the above make sense
15:49:04 mriedem stephenfin: https://review.openstack.org/#/c/651291/3
15:49:12 mriedem you've got a pep8 failure now in your bottom cells v1 removal series
15:49:23 mriedem i pulled it down since i didn't want to wait for zuul
15:50:49 openstackgerrit Merged openstack/nova master: Add minimum value in max_concurrent_live_migrations https://review.openstack.org/648302
15:50:56 openstackgerrit Merged openstack/nova stable/rocky: libvirt: disconnect volume when encryption fails https://review.openstack.org/651796
15:53:00 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: libvirt: disconnect volume when encryption fails https://review.openstack.org/653033
15:53:43 melwitt dansmith: the other day when you mentioned about how ideally quota usage counting from placement would take max(old, new) flavor for a resize, were you meaning only a same host resize? or also for a move?
15:54:55 dansmith melwitt: well, definitely for same-host, but I think it's probably reasonable for either too, depending on your view of it
15:55:04 dansmith melwitt: and depending on the resource type
15:56:24 dansmith melwitt: as a user, I would probably think it's kinda silly to consume sum(old, new) of my resources during a resize of any kind because I'm not able to use both resources simultaneously, and the fact that they're counted against my old instance for a while is just an artifact of how nova does the two-phase resize, you know?
15:56:56 melwitt dansmith: ok, thanks. just wanted to make sure I understood. I've been thinking about "how would we do that?" eventually and was thinking, would that be done on the placement side /usages API? when it detects allocations for the same 'instance' type consumer for the same resource class? or would this be something we do on the nova side somehow?
15:57:06 dansmith as an operator, I might want to "charge" the user for those resources until they confirm/revert, but that too is an artifact of how nova behaves
15:57:33 dansmith melwitt: that would be super nova-specific behavior, so I would not think that should ever go into placement
15:57:50 mnaser can someone point me to where network_info is updated from?
15:58:02 melwitt dansmith: yeah, I agree from a user perspective definitely sum(old, new) is weird. but I was feeling unsure because of how we really are holding resources in two places
15:58:05 mnaser I have unsuccessfully been digging the code, and I have a cloud here where a bunch of instances have network_info=[]
15:58:31 mriedem mnaser: for neutron, nova.network.neutronv2.api.API._get_instance_network_info or something like that
15:58:38 dansmith melwitt: right, that's the operator argument, but it still seems silly to me
15:58:50 dansmith melwitt: that's kinda overhead and transient
15:58:59 melwitt dansmith: ok. that was my assumption but like.... we wouldn't be able to just use the /usages API simply anymore. how would we get all the info about what /usages are part of a resize etc. I can't even think about it right now
15:59:01 dansmith melwitt: "cost of business"
15:59:34 mriedem melwitt: i don't think you can unless placement has consumer types
15:59:40 melwitt dansmith: yeah, I agree with that too, overhead and transient
15:59:46 dansmith melwitt: well, the reserved resources are owned by a migration and not an instance, but yeah, consumer types :)
15:59:55 mriedem mnaser: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L1824
16:00:07 melwitt mriedem: yes, definitely true, need consumer types but even still, I'm not 100% sure that would be enough
16:00:16 mriedem mnaser: that's called from here https://github.com/openstack/nova/blob/master/nova/network/base_api.py#L253
16:00:30 melwitt and what would the calls look like to work that out on the nova side
16:01:15 mriedem GET /usages?project_id=foo&consumer_type=instance
16:01:19 mnaser mriedem: it doesn't seem like there's a task that just calls get_instance_nw_info() from time to time ,gr
16:01:27 mriedem mnaser: oh but there is
16:01:38 mriedem mnaser: question is, which release is this cloud on?
16:01:42 mnaser rocky
16:01:47 melwitt mriedem: yeah, but then you get a sum of everything. how to pick apart and remove the min(old, new)?
16:01:58 dansmith melwitt: no

Earlier   Later