Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-12
15:23:23 cdent orphans
15:23:23 mriedem the compute manager explicitly does stuff with migration-based allocations
15:23:54 mriedem cdent: that's about it since all of the other allocation "healing" code was removed
15:23:54 cdent ever single allocation for instance.uuid is compared to each instance on the host
15:24:13 cdent that chunk can get really loud becuase of https://github.com/openstack/nova/blob/52b9359d6c6f387b8b9728c723c9a2501136d605/nova/compute/resource_tracker.py#L1340
15:24:22 cdent commenting out that line can save a bunch of log noise
15:24:51 mriedem debug("things are working normally")
15:24:55 cdent and 1372 there is one last remove
15:25:53 sean-k-mooney mriedem: i dont know im suspios that we normally would print that :P
15:26:14 sean-k-mooney but yes that does not really add much value
15:26:30 cdent if you both agree, I can put up a patch to remove it?
15:26:45 cdent meanwhile, I will create 1000 fake instances again
15:27:17 mriedem there used to be an audit level which is lower than debug, i'm not sure i'm totally in favor of removing it,
15:27:21 mriedem as the comment at the top says,
15:27:22 mriedem # NOTE(jaypipes): All of this code sucks.
15:27:34 mriedem but when i'm debugging rt issues, the more logging i have the better
15:27:39 cdent true
15:27:52 mriedem i frequently have to add logging to the rt when i'm debugging functional test fialures that involve it (like i was on friday for at least 2 hours)
15:27:59 cdent me too
15:28:11 cdent i have at least 6 additional LOG.debug in there right now
15:28:12 sean-k-mooney ya that is fair
15:28:32 mriedem https://review.opendev.org/#/c/366422/
15:28:49 sean-k-mooney we do have a way to filter out log messages via config too right
15:28:50 mriedem i said AUDIT but i was thinking TRACE
15:29:40 redkrieg lyarwood: Are you around by chance? I found your blueprint for stable disk devices in rescue mode and it's exactly what I need to offer my customers reliable ISO support. Just wanted to know what the status is and whether there's anything I can work on related to it to get it in a release in the future: https://blueprints.launchpad.net/nova/+spec/virt-rescue-stable-disk-devices
15:30:42 sean-k-mooney just based on the ordering in the gate logs http://logs.openstack.org/38/671338/5/check/tempest-full-py3/12e1ea1/controller/logs/screen-n-cpu.txt.gz
15:31:00 sean-k-mooney i would have guess audit and trace were both higher then info/debug
15:31:30 mriedem i don't know how to filter specific messages from logging config
15:32:21 sean-k-mooney i think we can do it at the module level but i dont know if we can do it at the function level. we cant do it on indigiual message via config so it proably too course for this usecase
15:32:55 sean-k-mooney i.e. we dont want to jsut make the RT be info and everything else debug
15:33:57 sean-k-mooney i guess that message does serve a use in confiming which branch is taken
15:34:22 sean-k-mooney but for ops that run at debug level in production its proably noisier then they would like
15:36:22 openstackgerrit Balazs Gibizer proposed openstack/nova master: Avoid timeout from service update notification tests https://review.opendev.org/675935
15:40:48 openstack Launchpad bug 1839853 in OpenStack Compute (nova) "Misuse of nova.objects.base.obj_equal_prims in tests" [Medium,Confirmed]
15:40:48 mriedem low-hanging-fruit test-only bug https://bugs.launchpad.net/nova/+bug/1839853
15:41:07 openstackgerrit Jing Zhang proposed openstack/nova master: Remove the current host from cold migration target host list https://review.opendev.org/675025
15:51:30 openstackgerrit Balazs Gibizer proposed openstack/nova master: Avoid timeout from service update notification tests https://review.opendev.org/675935
15:56:40 mriedem cdent: up your vmware alley https://review.opendev.org/#/c/675025/2
15:56:52 cdent sounds a bit rude, that
15:57:00 mriedem ha
15:58:06 mriedem dansmith: efried: melwitt: what do you think about the [workarounds]/allow_cold_migrate_to_same_host idea in https://review.opendev.org/#/c/675025/ as a backportable thing until we can be smarter about scheduling for cold migrate based on compute capabilities / traits filtering?
16:00:50 dansmith mriedem: meaning different behavior for cold migrate vs. resize?
16:01:11 mriedem yeah
16:01:35 mriedem the existing option conflates them which is bad for everything except vmware
16:01:41 dansmith the intent of cold migration is always to land it on a different host, so I guess I'm not sure why the patch is wrong (in intent, I haven't looked at the reasoning)
16:02:04 mriedem that's true for everything except vcenter
16:02:29 dansmith okay so the workaround config knob would only need tweaking for vmware?
16:02:32 cdent yet another raeson for vogan
16:02:42 mriedem dansmith: yeah
16:02:50 melwitt I'm not opposed to the patch but isn't this a super latent thing or? surprised this hasn't come up before
16:02:55 mriedem and is a backportable thing until we can be smarter about filtering based on traits
16:03:00 mriedem melwitt: it is,
16:03:06 mriedem there are a bunch of related bugs
16:03:07 dansmith mriedem: and that workaround would be read/honored in api or conductor when assembling the request to scheduler/
16:03:12 mriedem we just never do anything about fixing it :)
16:03:18 melwitt heh, wow
16:03:19 mriedem dansmith: api
16:03:29 mriedem dansmith: same place that allow_resize_to_same_host is read
16:03:47 dansmith yeah
16:04:00 dansmith for some reason I feel like workarounds are usually per-host and thus read by the compute
16:04:02 dansmith and further,
16:04:15 dansmith if you have one vmware compute and a bunch of libvirt computes, you likely don't want that to be a global policy
16:04:36 mriedem sure, but we already do,
16:04:47 dansmith for resize you mean
16:04:51 mriedem and that's why i think long-term we replace the workaround option with traits-based compute capability filtering,
16:05:03 mriedem yes, but we treat reaize == cold migrate in the api from this regard
16:05:23 dansmith sure, but the stated goal of allowing resize to same host makes more sense as a global I think
16:05:27 dansmith but anyway, what I'm saying is:
16:06:07 dansmith maybe the knob should be read by compute and factor in to what traits it exposes and then let the control side throw some required or forbidden trait in there
16:06:19 mriedem you can't backport that
16:06:29 mriedem but yes, that's what i'm saying we do with https://review.opendev.org/#/c/666604/
16:06:29 dansmith sure
16:06:48 mriedem compute exposes a trait for if it can do a thing, api reads that and configures the request spec for scheduler filtering appropriately,
16:06:54 dansmith but if we add it as honored by control, then backport that, then we kinda need deprecation or a dance to move to the per-compute way
16:06:56 mriedem which replaces the workaround thing i'm suggesting for backports
16:07:04 mriedem correct
16:07:15 mriedem workaround option -> backports -> eventual deprecation with the new thing https://review.opendev.org/#/c/666604/
16:07:21 dansmith right, so point is, since this is latent since the beginning of time, I don't care so much about the backport
16:07:38 dansmith but if it's super important to you then, I guess whatever
16:08:03 mriedem it's not super important to me, it's just i've dealt with it more than once and it continues to come up as a bug for *us*
16:08:27 mriedem my fear is the longer we don't do something, the more hack patches like this show up
16:08:55 dansmith ack
16:09:34 sean-k-mooney mriedem: speaking about https://review.opendev.org/#/c/666604/?
16:10:02 mriedem related yes
16:10:11 mriedem prompted by https://review.opendev.org/#/c/675025/
16:10:14 sean-k-mooney we spoke about it a bit on irc and understand the motivaition more now
16:10:25 mriedem the royal we?
16:10:38 sean-k-mooney as in i pinging you about it last week
16:10:43 sean-k-mooney or the week before
16:10:51 sean-k-mooney and you explained what it was for
16:11:32 sean-k-mooney it is not a trait i would expect to see in a flavor or image
16:11:46 sean-k-mooney but i can see how it could be useful in this specific case
16:12:02 melwitt I'm ok with the wokraround conceptually, I'm not sure why we didn't just do that years ago. I'm assuming it's a really simple change to consider resize different than cold migrate for "allow same host"
16:12:57 mriedem sean-k-mooney: nor are several of our compute-capability based traits
16:13:04 mriedem melwitt: yeah i think it would be a small change
16:13:12 mriedem if allow_resize_to_same_host:
16:13:32 mriedem if cold_migrate and allow_cold_migrate_to_same_host:
16:13:33 sean-k-mooney a workaround config option has the benifit that it could be backported were as the trait could not
16:13:39 mriedem dude
16:13:53 melwitt lol

Earlier   Later