Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-08-12
15:18:26 sean-k-mooney i think it woudl only be an issue for resouce that are not tracked in placmenet
15:18:29 mriedem since we just removed the cpu/ram/disk claims
15:20:05 mriedem tracked_migrations could also be wonky but on a smaller scale / tighter race
15:20:29 cdent go RT
15:21:13 sean-k-mooney well anythin relying on the perodic task to fix up allocation could be off but it should get fixed on the next run
15:21:27 sean-k-mooney missing frees is not that concuerning but missing claims would be
15:21:33 cdent sean-k-mooney: yeah, which ought to be fine
15:21:54 mriedem sean-k-mooney: the periodic doesn't fix up allocations...
15:21:59 mriedem not since ocata/pike anyway
15:22:36 sean-k-mooney out side of the migration ones? i though it healed allcoation for instace once the migration was conrimed
15:23:09 mriedem the rt does not do that
15:23:12 cdent sure it does: it spends lots of time in here: https://github.com/openstack/nova/blob/52b9359d6c6f387b8b9728c723c9a2501136d605/nova/compute/resource_tracker.py#L1314
15:23:23 cdent orphans
15:23:23 mriedem the compute manager explicitly does stuff with migration-based allocations
15:23:54 mriedem cdent: that's about it since all of the other allocation "healing" code was removed
15:23:54 cdent ever single allocation for instance.uuid is compared to each instance on the host
15:24:13 cdent that chunk can get really loud becuase of https://github.com/openstack/nova/blob/52b9359d6c6f387b8b9728c723c9a2501136d605/nova/compute/resource_tracker.py#L1340
15:24:22 cdent commenting out that line can save a bunch of log noise
15:24:51 mriedem debug("things are working normally")
15:24:55 cdent and 1372 there is one last remove
15:25:53 sean-k-mooney mriedem: i dont know im suspios that we normally would print that :P
15:26:14 sean-k-mooney but yes that does not really add much value
15:26:30 cdent if you both agree, I can put up a patch to remove it?
15:26:45 cdent meanwhile, I will create 1000 fake instances again
15:27:17 mriedem there used to be an audit level which is lower than debug, i'm not sure i'm totally in favor of removing it,
15:27:21 mriedem as the comment at the top says,
15:27:22 mriedem # NOTE(jaypipes): All of this code sucks.
15:27:34 mriedem but when i'm debugging rt issues, the more logging i have the better
15:27:39 cdent true
15:27:52 mriedem i frequently have to add logging to the rt when i'm debugging functional test fialures that involve it (like i was on friday for at least 2 hours)
15:27:59 cdent me too
15:28:11 cdent i have at least 6 additional LOG.debug in there right now
15:28:12 sean-k-mooney ya that is fair
15:28:32 mriedem https://review.opendev.org/#/c/366422/
15:28:49 sean-k-mooney we do have a way to filter out log messages via config too right
15:28:50 mriedem i said AUDIT but i was thinking TRACE
15:29:40 redkrieg lyarwood: Are you around by chance? I found your blueprint for stable disk devices in rescue mode and it's exactly what I need to offer my customers reliable ISO support. Just wanted to know what the status is and whether there's anything I can work on related to it to get it in a release in the future: https://blueprints.launchpad.net/nova/+spec/virt-rescue-stable-disk-devices
15:30:42 sean-k-mooney just based on the ordering in the gate logs http://logs.openstack.org/38/671338/5/check/tempest-full-py3/12e1ea1/controller/logs/screen-n-cpu.txt.gz
15:31:00 sean-k-mooney i would have guess audit and trace were both higher then info/debug
15:31:30 mriedem i don't know how to filter specific messages from logging config
15:32:21 sean-k-mooney i think we can do it at the module level but i dont know if we can do it at the function level. we cant do it on indigiual message via config so it proably too course for this usecase
15:32:55 sean-k-mooney i.e. we dont want to jsut make the RT be info and everything else debug
15:33:57 sean-k-mooney i guess that message does serve a use in confiming which branch is taken
15:34:22 sean-k-mooney but for ops that run at debug level in production its proably noisier then they would like
15:36:22 openstackgerrit Balazs Gibizer proposed openstack/nova master: Avoid timeout from service update notification tests https://review.opendev.org/675935
15:40:48 openstack Launchpad bug 1839853 in OpenStack Compute (nova) "Misuse of nova.objects.base.obj_equal_prims in tests" [Medium,Confirmed]
15:40:48 mriedem low-hanging-fruit test-only bug https://bugs.launchpad.net/nova/+bug/1839853
15:41:07 openstackgerrit Jing Zhang proposed openstack/nova master: Remove the current host from cold migration target host list https://review.opendev.org/675025
15:51:30 openstackgerrit Balazs Gibizer proposed openstack/nova master: Avoid timeout from service update notification tests https://review.opendev.org/675935
15:56:40 mriedem cdent: up your vmware alley https://review.opendev.org/#/c/675025/2
15:56:52 cdent sounds a bit rude, that
15:57:00 mriedem ha
15:58:06 mriedem dansmith: efried: melwitt: what do you think about the [workarounds]/allow_cold_migrate_to_same_host idea in https://review.opendev.org/#/c/675025/ as a backportable thing until we can be smarter about scheduling for cold migrate based on compute capabilities / traits filtering?
16:00:50 dansmith mriedem: meaning different behavior for cold migrate vs. resize?
16:01:11 mriedem yeah
16:01:35 mriedem the existing option conflates them which is bad for everything except vmware
16:01:41 dansmith the intent of cold migration is always to land it on a different host, so I guess I'm not sure why the patch is wrong (in intent, I haven't looked at the reasoning)
16:02:04 mriedem that's true for everything except vcenter
16:02:29 dansmith okay so the workaround config knob would only need tweaking for vmware?
16:02:32 cdent yet another raeson for vogan
16:02:42 mriedem dansmith: yeah
16:02:50 melwitt I'm not opposed to the patch but isn't this a super latent thing or? surprised this hasn't come up before
16:02:55 mriedem and is a backportable thing until we can be smarter about filtering based on traits
16:03:00 mriedem melwitt: it is,
16:03:06 mriedem there are a bunch of related bugs
16:03:07 dansmith mriedem: and that workaround would be read/honored in api or conductor when assembling the request to scheduler/
16:03:12 mriedem we just never do anything about fixing it :)
16:03:18 melwitt heh, wow
16:03:19 mriedem dansmith: api
16:03:29 mriedem dansmith: same place that allow_resize_to_same_host is read
16:03:47 dansmith yeah
16:04:00 dansmith for some reason I feel like workarounds are usually per-host and thus read by the compute
16:04:02 dansmith and further,
16:04:15 dansmith if you have one vmware compute and a bunch of libvirt computes, you likely don't want that to be a global policy
16:04:36 mriedem sure, but we already do,
16:04:47 dansmith for resize you mean
16:04:51 mriedem and that's why i think long-term we replace the workaround option with traits-based compute capability filtering,
16:05:03 mriedem yes, but we treat reaize == cold migrate in the api from this regard
16:05:23 dansmith sure, but the stated goal of allowing resize to same host makes more sense as a global I think
16:05:27 dansmith but anyway, what I'm saying is:
16:06:07 dansmith maybe the knob should be read by compute and factor in to what traits it exposes and then let the control side throw some required or forbidden trait in there
16:06:19 mriedem you can't backport that
16:06:29 mriedem but yes, that's what i'm saying we do with https://review.opendev.org/#/c/666604/
16:06:29 dansmith sure
16:06:48 mriedem compute exposes a trait for if it can do a thing, api reads that and configures the request spec for scheduler filtering appropriately,
16:06:54 dansmith but if we add it as honored by control, then backport that, then we kinda need deprecation or a dance to move to the per-compute way
16:06:56 mriedem which replaces the workaround thing i'm suggesting for backports
16:07:04 mriedem correct
16:07:15 mriedem workaround option -> backports -> eventual deprecation with the new thing https://review.opendev.org/#/c/666604/
16:07:21 dansmith right, so point is, since this is latent since the beginning of time, I don't care so much about the backport
16:07:38 dansmith but if it's super important to you then, I guess whatever
16:08:03 mriedem it's not super important to me, it's just i've dealt with it more than once and it continues to come up as a bug for *us*
16:08:27 mriedem my fear is the longer we don't do something, the more hack patches like this show up
16:08:55 dansmith ack
16:09:34 sean-k-mooney mriedem: speaking about https://review.opendev.org/#/c/666604/?
16:10:02 mriedem related yes
16:10:11 mriedem prompted by https://review.opendev.org/#/c/675025/
16:10:14 sean-k-mooney we spoke about it a bit on irc and understand the motivaition more now
16:10:25 mriedem the royal we?
16:10:38 sean-k-mooney as in i pinging you about it last week

Earlier   Later