Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-24
19:29:11 efried aspiers: Sure, I would be fine splitting that up.
19:29:36 aspiers efried: well, shouldn't be too hard to do that and then move the SEV bit in there
19:30:06 aspiers efried: other than that and maybe some trait taxonomy thing which I don't yet know about, hopefully everything else is in order with this patch
19:30:09 efried aspiers: https://review.opendev.org/#/c/538498/6/nova/compute/resource_tracker.py@904 (patch look familiar?)
19:31:14 aspiers efried: yeah, I do remember reading that and vaguely understanding
19:31:23 aspiers I'm sure a re-read will make a lot more sense now
19:47:20 mriedem ganso: ok so i think things in queens are ok except the functional test is failing because of some missing test handling stuff on teardown in queens, working on that now
19:48:17 ganso mriedem: thanks Matt! =D
19:52:34 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add functional confirm_migration_error test https://review.opendev.org/658136
19:52:50 mriedem ganso: i think you can drop the -1 on https://review.opendev.org/#/c/652153/ now
19:53:27 ganso mriedem: yup, dropped =)
20:07:56 kaiokmo hey nova folks! I was wondering if something changed regarding how metadata are defined/handled by nova-scheduler
20:08:10 kaiokmo specifically on the aggregate_instance_extra_specs filter
20:08:56 kaiokmo I have a Pike deployment in which this filter works properly
20:10:15 kaiokmo last week I deployed a environment on Rocky (tag is 18.1.7), and this filters is not working as expected
20:10:43 kaiokmo nova-scheduler is ignoring the filter and all hosts are passing the filtering phase
20:12:06 kaiokmo nova.conf is configured as follows (only the aggregate_instance_extra_specs is enabled): http://paste.openstack.org/show/752054/
20:12:21 kaiokmo I appreciate any help.
20:13:41 kaiokmo thanks in advance
20:22:49 aspiers alright, I'm off. happy weekend o/
20:28:59 melwitt kaiokmo: not aware of any change off the top of my head. will look around the code to see if anything has changed in that area
20:32:04 kaiokmo melwitt: thank you very much
20:36:50 melwitt I'm not seeing that anything has changed in the area
20:37:40 melwitt kaiokmo: have you enabled debug logging in nova-scheduler and look at what's going on? if so, do you see DEBUG messages like these? https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_instance_extra_specs.py#L64
20:39:33 melwitt interesting, my devstack change started running on a node within 14 minutes of uploading it
20:40:13 melwitt reminds me of something in a ML thread where things in the nova queue have to wait longer because of something to do with zuul scheduling?
20:41:34 efried kaiokmo: Does this happen to be on a rebuild?
20:42:21 kaiokmo melwitt: debug is set to True on nova.conf but can't see messages like that on nova-scheduler log
20:43:05 kaiokmo efried: no. RUN_ON_REBUILD is set to false on the filter, and I'm only spawning new instances, not rebuilding an existing one
20:44:46 melwitt kaiokmo: hm, did you restart nova-scheduler after adding debug=True? that config should work. we'll probably need to look at the debug logs to get to the bottom of this because I didn't find any open bugs mentioning the AggregateInstanceExtraSpecsFilter and no recent code changes
20:45:02 efried kaiokmo: I'm looking specifically at https://review.opendev.org/#/c/523212/ which introduced RUN_ON_REBUILD and wondering if you're running afoul of the logic at https://review.opendev.org/#/c/523212/2/nova/scheduler/host_manager.py
20:46:11 efried that's some complex stuff with check type and force hosts/nodes, I don't really understand it.
20:46:37 efried kaiokmo: If you felt like experimenting, you could try reverting (pieces of) that change and see if the problem goes away.
20:47:09 kaiokmo melwitt: yes, I did restart the service. right now I can see some DEBUG messages on the logs
20:48:03 melwitt oh, good. argh, I just realized you had said the problem is all hosts are included, not the other way around
20:48:08 kaiokmo efried: I can test reverting this change, although I don't feel like it is going to change anything for me
20:48:18 melwitt so you wouldn't see messages like that if it thinks the hosts are good when they should not be
20:50:00 kaiokmo I see. I'm wondering if even the host_passes function is being called. I managed to put some LOG.debug myself inside the function, but none of my messages are displayed on the logs
20:50:01 efried kaiokmo: It's just the only thing that changed on pike in that particular filter, so it's the first suspect for root cause.
20:50:12 efried right
20:50:25 melwitt kaiokmo: does your flavor have extra_specs in it? because the filter code is showing it will "pass" the host if the flavor does not have any extra_specs
20:50:57 melwitt oh really, ok that's not good. lemme see...
20:51:57 kaiokmo btw, the is being loaded correctly by the scheduler. LOG.debug placed before host_passes appear on the logs
20:53:28 melwitt kaiokmo: this might be a problem with your config, the enabled_filters and available_filters are supposed to be lists. I wonder if that's what's wrong?
20:56:00 melwitt I can never remember how to format lists in ini conf
20:57:29 melwitt looks like it's just comma separated: enabled_filters = RetryFilter,AvailabilityZoneFilter
20:58:38 efried iiuc you're using the same config file as before?
20:59:13 efried kaiokmo: Oh, disregard the thing about reverting that patch. I misread your original issue, you said this was working on pike and broke on rocky. That patch went back to pike, so probably isn't to blame.
20:59:33 melwitt I was wondering the same thing. because according to this, it should be the class name only, not the whole path. but I don't know whether having the whole path would hurt or not
20:59:57 kaiokmo melwitt: yes. metadata on aggregate is custom: "aggregate" with "test" value
21:00:29 kaiokmo metadata on flavor is "aggregate_instance_extra_specs:aggregate" with "test" value
21:01:39 melwitt kaiokmo: when you added your own debug messages, did you put one here? https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_instance_extra_specs.py#L48 wondering if you would see that one?
21:02:52 kaiokmo melwitt: yes, it is LOG.debug("***OMG***") :)
21:03:24 melwitt haha xD
21:04:20 melwitt did you put one right at the beginning of host_passes before anything else? because if that's not showing, then you're right the filter isn't running
21:05:22 melwitt if the filter isn't running, I'd try to use config options like available_filters = AggregateInstanceExtraSpecsFilter enabled_filters = AggregateInstanceExtraSpecsFilter and then see if you see the filter run
21:06:25 kaiokmo yes. I put one on the beginning of the function, and it is not showing. but, I also did put one right before the function and this one appears on the logs when the nova-scheduler service is restarted (which I think means that the filter was loaded)
21:07:27 kaiokmo that's what I did: this is the filter section on nova.conf for nova-scheduler http://paste.openstack.org/show/752054/
21:07:58 melwitt yeah, I agree, the message you do see indicates the class was loaded by something
21:08:16 melwitt I'd try also changing available_filters to match enabled_filters, i.e. use class name only
21:08:39 kaiokmo ok. trying that now
21:08:50 melwitt because available_filters is a superset of enabled_filters
21:09:55 melwitt if that doesn't work to get the filter to run, then I will be lost again
21:10:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix hard-delete of instance with soft-deleted referential constraints https://review.opendev.org/661398
21:14:07 kaiokmo melwitt: well, if available_filter is class only, oslo importutils (called by nova-scheduler) doesn't seem to be able to load the filter
21:14:18 kaiokmo ValueError: Empty module name
21:14:34 melwitt :\
21:16:03 melwitt ok, then I guess the config help on that one is wrong. these available_filters and enabled_filters are supposed to work the same way
21:16:18 melwitt next I would try commenting out the available_filters setting altogether and let it default to all filters
21:16:36 melwitt it will still only use what you have in enabled_filters and won't use any additional filters
21:17:27 melwitt and if that enables the filter to run, then it sounds like we have some kind of bug or config help doc problem with the available_filters option
21:18:39 melwitt I can't find any examples of it being used and not being left as the default, so I don't know it's intended to be set as a list
21:19:04 melwitt *how it's intended
21:34:33 kaiokmo melwitt: did as you said. the filter was loaded (like before), but it did not run (also like before)
21:36:40 melwitt ... I don't understand what is going on
21:37:11 melwitt I'm going to try on my devstack
21:38:03 kaiokmo me neither
21:40:56 melwitt ok, when I appended AggregateInstanceExtraSpecsFilter to enabled_filters in devstack it runs the filter
21:42:24 kaiokmo runs it or loads it? did you created a host aggregate and a flavor (both with the proper metadata)?
21:42:43 melwitt runs it, I added a log message right at the beginning inside the host_passes method
21:43:41 melwitt no, I didn't do anything with aggregate and flavor. the filter not running would be unrelated to aggregate/flavor AFAIK
21:44:16 melwitt I'll try it
21:44:46 kaiokmo doesn't seem to work for me. I configured nova.conf as you suggested, with enabled_filters=AggregateInstanceExtraSpecsFilter and I left available_filters commented to load all filters by default
21:45:12 kaiokmo in which tag are your nova/devstack on?
21:45:32 openstackgerrit Dustin Cowles proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899
21:45:32 openstackgerrit Dustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for node.list https://review.opendev.org/656027
21:45:33 openstackgerrit Dustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for validating instance and node https://review.opendev.org/656028
21:45:34 openstackgerrit Dustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for setting instance id https://review.opendev.org/659690
21:45:34 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP: Use SDK instead of ironicclient for add/remove instance info from node https://review.opendev.org/659691
21:46:00 melwitt I'm at a5e3054e1d6df248fc4c00b9abd7289dde160393 in train
21:47:58 melwitt when I set enabled_filters = AggregateInstanceExtraSpecsFilter I get only the one filter running
21:52:46 melwitt gonna try now with aggregate and flavor
21:53:37 kaiokmo ok, thank you. it is probably gonna work, since the filter is running
21:54:36 kaiokmo I'm at 18.0.0, deployed with OSA
22:00:09 melwitt ok, booting with matching metadata works. now I'll try no match
22:01:41 kaiokmo my knowledge about nova behaviou (api and etcetera) is limited. could this be related with the placement service or something?
22:01:42 melwitt nonmatch fails to boot
22:01:58 kaiokmo s/behaviou/behavior
22:02:11 melwitt I thought about whether placement could be involved, but that would *not* affect whether your filter runs
22:03:05 melwitt when something changes related to placement, it would a change where the host candidates coming back from placement are problematic in some way

Earlier   Later