Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-16
15:30:21 dansmith mriedem: when I was reading the two numa specs today I was reminded of something I was thinking about earlier, related to cases where we have instances which store old-format data, like something stuck in their flavor which needs to be migrated
15:30:22 sean-k-mooney dansmith: ok i had tought we still supported "bring your own" metadata key for image too i guess not
15:31:06 dansmith mriedem: things that we would need to online_migrate or something, and things that we would be tempted to resolve with "meh, just migrate all your instances left one rack to clean those up".. kinda like the recent discussion about reshape for that stuff
15:31:30 dansmith mriedem: we might benefit from a "needs upgrade" flag on the instance, that would be shown in detailed list, to admins only,
15:31:44 dansmith where I could list all tenants and see instances that have "needs upgrade"
15:32:16 sean-k-mooney so the old usecase woulould be achive with a member_of request in the flavor extra spec
15:32:25 dansmith anything could set that flag, like compute manager when it notices some legacy data, or even libvirt when it notices something like a disk format that needs to be upgraded or something
15:32:51 mriedem dansmith: if the thing setting the flag has to already calculate that it will set the flag, why not just do the online migration right then?
15:33:31 dansmith mriedem: well, because things like the resource topology thing would need to be done to all instances on the compute node at the same time, and after a config change
15:33:45 dansmith mriedem: referring to the thing stephenfin and jaypipes and bauzas and I were discussing last week
15:34:04 bauzas FWIW, on a 1.5h meeting atm
15:34:15 bauzas but listening
15:34:45 mriedem so instances a,b,c need an upgrade, but the admin doesn't know how to upgrade them, i.e. is it migrating them, running the online_data_migrations cli, restarting the compute they are on, etc
15:35:22 mriedem if the flag were an enum that's one way
15:35:41 dansmith mriedem: yeah, so it'd be nice if we also tagged "issues" to the instance that you whittle down until the upgrade flag goes away, but I think that's too heavy for the moment.. but if we LOG.warning() for each instance that we were setting the flag on, then you would at least have a record
15:35:55 dansmith mriedem: yeah, could be iterative, like service_version
15:36:01 dansmith if we always did those things in order
15:36:22 sean-k-mooney dansmith: could you tie it into a nova status check or something
15:36:42 mriedem i don't think i'd rely on someone catching that warning
15:36:49 openstackgerrit Merged openstack/nova stable/rocky: Delete allocations even if _confirm_resize raises https://review.openstack.org/652146
15:36:50 mriedem before their logs wrap or something
15:36:57 mriedem depends on how long they retain stuff in es
15:37:08 mriedem we already blast the logs with warnings that aren't useful
15:37:43 dansmith mriedem: sure, it just makes it lighter for the first rev of the idea, but if we made it a "schema version" sort of iterative "level" like service_version that could be easy, as you would look up the reason in the decoder ring
15:38:21 cdent Is there something up with the gate today? queue seems long and slow
15:38:22 dansmith and we could translate it in the api to "is current or not" if the version is backlevel (and show the version I guess)
15:38:33 mriedem cdent: it was like that yesterday too
15:38:53 mriedem dansmith: i'm not sure what service_version you're referring to
15:38:59 mriedem just the Service.version?
15:39:05 dansmith no,
15:39:34 openstackgerrit Merged openstack/nova stable/rocky: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/651797
15:39:37 dansmith SERVICE_VERSION is a global counter of things we've done to services, so we can tell if they're up to date or not
15:40:23 dansmith we could have a similar global "instance version" which was a little more like a schema version, where we record whether instance records had been modified for a specific transition
15:41:04 dansmith this is not important right now, I was thinking you'd latch onto this so we could expose more info to the operators about upgraded-ness, but I'll just bring it up the next time it's appropriate, like in those specs
15:41:34 mriedem ack
15:42:17 bauzas sean-k-mooney: so, about the upgrade impact of the cpu-resources thing, I had an idea
15:42:38 bauzas what we could do is potentially leave operators upgrading to Train without any impact
15:42:54 bauzas ie. config options act exactly like Stein
15:43:02 bauzas (for the existing ones)
15:43:07 mriedem cdent: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?panelId=3&fullscreen&orgId=1
15:43:27 bauzas but, before you would like to use new config options, you'd have to say "hey, reshape my stuff"
15:43:29 sean-k-mooney bauzas: yes. i suggested not enableing any of the new functionality if the new option was not set
15:43:31 bauzas for this host
15:43:40 sean-k-mooney yep
15:43:44 sean-k-mooney so they would upgrade
15:43:49 bauzas and in this case, we would model the new inventories
15:43:51 cdent mriedem: "data without context can never be information"
15:44:01 sean-k-mooney then move vms if need and then enable the new funcitonaltiy
15:44:10 bauzas from an operator pov, it would mean "upgrade your cloud, upgrade your hosts"
15:44:14 sean-k-mooney so they could to a rooling config update after the upgrade
15:44:14 mriedem cdent: just saying there is a spike in the check queue so things are maybe slow as a result
15:44:21 bauzas when you're done, then upgrade your config when you want
15:44:32 sean-k-mooney yep
15:44:42 bauzas a nova-status check could do the pre-flight check
15:44:45 sean-k-mooney i think we are on the same page for that workflow
15:44:49 cdent mriedem: sure, but we knew that already (or at least I did)
15:45:13 sean-k-mooney bauzas: it requires use to support both codepaths in train
15:45:22 sean-k-mooney then remove teh old one in U
15:45:29 bauzas sean-k-mooney: yup, this was my concern
15:45:31 bauzas but
15:45:31 sean-k-mooney but i think that will help with upgrades significantly
15:45:37 mriedem cdent: well you didn't tell me that, so who's misinformed now?!
15:45:52 bauzas with addition to be : trigger a reshape by the operator
15:45:58 bauzas or
15:46:05 bauzas trigger a reshape by the config change
15:46:07 cdent heh. touche
15:48:20 sean-k-mooney bauzas: yep i was going to suggest that too.
15:48:52 sean-k-mooney stephenfin: ^ when you get a chance after this call does the above make sense
15:49:04 mriedem stephenfin: https://review.openstack.org/#/c/651291/3
15:49:12 mriedem you've got a pep8 failure now in your bottom cells v1 removal series
15:49:23 mriedem i pulled it down since i didn't want to wait for zuul
15:50:49 openstackgerrit Merged openstack/nova master: Add minimum value in max_concurrent_live_migrations https://review.openstack.org/648302
15:50:56 openstackgerrit Merged openstack/nova stable/rocky: libvirt: disconnect volume when encryption fails https://review.openstack.org/651796
15:53:00 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: libvirt: disconnect volume when encryption fails https://review.openstack.org/653033
15:53:43 melwitt dansmith: the other day when you mentioned about how ideally quota usage counting from placement would take max(old, new) flavor for a resize, were you meaning only a same host resize? or also for a move?
15:54:55 dansmith melwitt: well, definitely for same-host, but I think it's probably reasonable for either too, depending on your view of it
15:55:04 dansmith melwitt: and depending on the resource type
15:56:24 dansmith melwitt: as a user, I would probably think it's kinda silly to consume sum(old, new) of my resources during a resize of any kind because I'm not able to use both resources simultaneously, and the fact that they're counted against my old instance for a while is just an artifact of how nova does the two-phase resize, you know?
15:56:56 melwitt dansmith: ok, thanks. just wanted to make sure I understood. I've been thinking about "how would we do that?" eventually and was thinking, would that be done on the placement side /usages API? when it detects allocations for the same 'instance' type consumer for the same resource class? or would this be something we do on the nova side somehow?
15:57:06 dansmith as an operator, I might want to "charge" the user for those resources until they confirm/revert, but that too is an artifact of how nova behaves
15:57:33 dansmith melwitt: that would be super nova-specific behavior, so I would not think that should ever go into placement
15:57:50 mnaser can someone point me to where network_info is updated from?
15:58:02 melwitt dansmith: yeah, I agree from a user perspective definitely sum(old, new) is weird. but I was feeling unsure because of how we really are holding resources in two places
15:58:05 mnaser I have unsuccessfully been digging the code, and I have a cloud here where a bunch of instances have network_info=[]
15:58:31 mriedem mnaser: for neutron, nova.network.neutronv2.api.API._get_instance_network_info or something like that
15:58:38 dansmith melwitt: right, that's the operator argument, but it still seems silly to me
15:58:50 dansmith melwitt: that's kinda overhead and transient
15:58:59 melwitt dansmith: ok. that was my assumption but like.... we wouldn't be able to just use the /usages API simply anymore. how would we get all the info about what /usages are part of a resize etc. I can't even think about it right now
15:59:01 dansmith melwitt: "cost of business"
15:59:34 mriedem melwitt: i don't think you can unless placement has consumer types
15:59:40 melwitt dansmith: yeah, I agree with that too, overhead and transient
15:59:46 dansmith melwitt: well, the reserved resources are owned by a migration and not an instance, but yeah, consumer types :)
15:59:55 mriedem mnaser: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L1824
16:00:07 melwitt mriedem: yes, definitely true, need consumer types but even still, I'm not 100% sure that would be enough
16:00:16 mriedem mnaser: that's called from here https://github.com/openstack/nova/blob/master/nova/network/base_api.py#L253
16:00:30 melwitt and what would the calls look like to work that out on the nova side
16:01:15 mriedem GET /usages?project_id=foo&consumer_type=instance
16:01:19 mnaser mriedem: it doesn't seem like there's a task that just calls get_instance_nw_info() from time to time ,gr
16:01:27 mriedem mnaser: oh but there is
16:01:38 mriedem mnaser: question is, which release is this cloud on?

Earlier   Later