Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-24
20:43:05 kaiokmo efried: no. RUN_ON_REBUILD is set to false on the filter, and I'm only spawning new instances, not rebuilding an existing one
20:44:46 melwitt kaiokmo: hm, did you restart nova-scheduler after adding debug=True? that config should work. we'll probably need to look at the debug logs to get to the bottom of this because I didn't find any open bugs mentioning the AggregateInstanceExtraSpecsFilter and no recent code changes
20:45:02 efried kaiokmo: I'm looking specifically at https://review.opendev.org/#/c/523212/ which introduced RUN_ON_REBUILD and wondering if you're running afoul of the logic at https://review.opendev.org/#/c/523212/2/nova/scheduler/host_manager.py
20:46:11 efried that's some complex stuff with check type and force hosts/nodes, I don't really understand it.
20:46:37 efried kaiokmo: If you felt like experimenting, you could try reverting (pieces of) that change and see if the problem goes away.
20:47:09 kaiokmo melwitt: yes, I did restart the service. right now I can see some DEBUG messages on the logs
20:48:03 melwitt oh, good. argh, I just realized you had said the problem is all hosts are included, not the other way around
20:48:08 kaiokmo efried: I can test reverting this change, although I don't feel like it is going to change anything for me
20:48:18 melwitt so you wouldn't see messages like that if it thinks the hosts are good when they should not be
20:50:00 kaiokmo I see. I'm wondering if even the host_passes function is being called. I managed to put some LOG.debug myself inside the function, but none of my messages are displayed on the logs
20:50:01 efried kaiokmo: It's just the only thing that changed on pike in that particular filter, so it's the first suspect for root cause.
20:50:12 efried right
20:50:25 melwitt kaiokmo: does your flavor have extra_specs in it? because the filter code is showing it will "pass" the host if the flavor does not have any extra_specs
20:50:57 melwitt oh really, ok that's not good. lemme see...
20:51:57 kaiokmo btw, the is being loaded correctly by the scheduler. LOG.debug placed before host_passes appear on the logs
20:53:28 melwitt kaiokmo: this might be a problem with your config, the enabled_filters and available_filters are supposed to be lists. I wonder if that's what's wrong?
20:56:00 melwitt I can never remember how to format lists in ini conf
20:57:29 melwitt looks like it's just comma separated: enabled_filters = RetryFilter,AvailabilityZoneFilter
20:58:38 efried iiuc you're using the same config file as before?
20:59:13 efried kaiokmo: Oh, disregard the thing about reverting that patch. I misread your original issue, you said this was working on pike and broke on rocky. That patch went back to pike, so probably isn't to blame.
20:59:33 melwitt I was wondering the same thing. because according to this, it should be the class name only, not the whole path. but I don't know whether having the whole path would hurt or not
20:59:57 kaiokmo melwitt: yes. metadata on aggregate is custom: "aggregate" with "test" value
21:00:29 kaiokmo metadata on flavor is "aggregate_instance_extra_specs:aggregate" with "test" value
21:01:39 melwitt kaiokmo: when you added your own debug messages, did you put one here? https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_instance_extra_specs.py#L48 wondering if you would see that one?
21:02:52 kaiokmo melwitt: yes, it is LOG.debug("***OMG***") :)
21:03:24 melwitt haha xD
21:04:20 melwitt did you put one right at the beginning of host_passes before anything else? because if that's not showing, then you're right the filter isn't running
21:05:22 melwitt if the filter isn't running, I'd try to use config options like available_filters = AggregateInstanceExtraSpecsFilter enabled_filters = AggregateInstanceExtraSpecsFilter and then see if you see the filter run
21:06:25 kaiokmo yes. I put one on the beginning of the function, and it is not showing. but, I also did put one right before the function and this one appears on the logs when the nova-scheduler service is restarted (which I think means that the filter was loaded)
21:07:27 kaiokmo that's what I did: this is the filter section on nova.conf for nova-scheduler http://paste.openstack.org/show/752054/
21:07:58 melwitt yeah, I agree, the message you do see indicates the class was loaded by something
21:08:16 melwitt I'd try also changing available_filters to match enabled_filters, i.e. use class name only
21:08:39 kaiokmo ok. trying that now
21:08:50 melwitt because available_filters is a superset of enabled_filters
21:09:55 melwitt if that doesn't work to get the filter to run, then I will be lost again
21:10:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Fix hard-delete of instance with soft-deleted referential constraints https://review.opendev.org/661398
21:14:07 kaiokmo melwitt: well, if available_filter is class only, oslo importutils (called by nova-scheduler) doesn't seem to be able to load the filter
21:14:18 kaiokmo ValueError: Empty module name
21:14:34 melwitt :\
21:16:03 melwitt ok, then I guess the config help on that one is wrong. these available_filters and enabled_filters are supposed to work the same way
21:16:18 melwitt next I would try commenting out the available_filters setting altogether and let it default to all filters
21:16:36 melwitt it will still only use what you have in enabled_filters and won't use any additional filters
21:17:27 melwitt and if that enables the filter to run, then it sounds like we have some kind of bug or config help doc problem with the available_filters option
21:18:39 melwitt I can't find any examples of it being used and not being left as the default, so I don't know it's intended to be set as a list
21:19:04 melwitt *how it's intended
21:34:33 kaiokmo melwitt: did as you said. the filter was loaded (like before), but it did not run (also like before)
21:36:40 melwitt ... I don't understand what is going on
21:37:11 melwitt I'm going to try on my devstack
21:38:03 kaiokmo me neither
21:40:56 melwitt ok, when I appended AggregateInstanceExtraSpecsFilter to enabled_filters in devstack it runs the filter
21:42:24 kaiokmo runs it or loads it? did you created a host aggregate and a flavor (both with the proper metadata)?
21:42:43 melwitt runs it, I added a log message right at the beginning inside the host_passes method
21:43:41 melwitt no, I didn't do anything with aggregate and flavor. the filter not running would be unrelated to aggregate/flavor AFAIK
21:44:16 melwitt I'll try it
21:44:46 kaiokmo doesn't seem to work for me. I configured nova.conf as you suggested, with enabled_filters=AggregateInstanceExtraSpecsFilter and I left available_filters commented to load all filters by default
21:45:12 kaiokmo in which tag are your nova/devstack on?
21:45:32 openstackgerrit Dustin Cowles proposed openstack/nova master: Introduces SDK to IronicDriver and uses for node.get https://review.opendev.org/642899
21:45:32 openstackgerrit Dustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for node.list https://review.opendev.org/656027
21:45:33 openstackgerrit Dustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for validating instance and node https://review.opendev.org/656028
21:45:34 openstackgerrit Dustin Cowles proposed openstack/nova master: Use SDK instead of ironicclient for setting instance id https://review.opendev.org/659690
21:45:34 openstackgerrit Dustin Cowles proposed openstack/nova master: WIP: Use SDK instead of ironicclient for add/remove instance info from node https://review.opendev.org/659691
21:46:00 melwitt I'm at a5e3054e1d6df248fc4c00b9abd7289dde160393 in train
21:47:58 melwitt when I set enabled_filters = AggregateInstanceExtraSpecsFilter I get only the one filter running
21:52:46 melwitt gonna try now with aggregate and flavor
21:53:37 kaiokmo ok, thank you. it is probably gonna work, since the filter is running
21:54:36 kaiokmo I'm at 18.0.0, deployed with OSA
22:00:09 melwitt ok, booting with matching metadata works. now I'll try no match
22:01:41 kaiokmo my knowledge about nova behaviou (api and etcetera) is limited. could this be related with the placement service or something?
22:01:42 melwitt nonmatch fails to boot
22:01:58 kaiokmo s/behaviou/behavior
22:02:11 melwitt I thought about whether placement could be involved, but that would *not* affect whether your filter runs
22:03:05 melwitt when something changes related to placement, it would a change where the host candidates coming back from placement are problematic in some way
22:03:17 melwitt placement will pre-filter hosts before they run through the nova scheduler filters
22:03:55 kaiokmo ah. I see
22:04:25 melwitt the fact that your filter isn't running at all is bizarre and I can't think of how that could be happening other than a config problem
22:14:02 melwitt kaiokmo: when the nova-scheduler service starts up, the DEBUG log will show what values it picked up for available_filters and enabled_filters. you might take a look at that to see if it yields any clues
22:14:08 openstackgerrit Merged openstack/nova master: Move get_pci_mapping_for_migration to MigrationContext https://review.opendev.org/643023
22:14:16 openstackgerrit Merged openstack/nova master: Allow driver to properly unplug VIFs on destination on confirm resize https://review.opendev.org/643024
22:14:57 melwitt for example May 24 21:46:45 ubuntu-xenial nova-scheduler[15474]: DEBUG oslo_service.service [None req-3351fca8-ca06-4931-bebb-e1f40771a478 None None] filter_scheduler.enabled_filters = ['AggregateInstanceExtraSpecsFilter']
22:15:26 melwitt May 24 21:46:45 ubuntu-xenial nova-scheduler[15474]: DEBUG oslo_service.service [None req-3351fca8-ca06-4931-bebb-e1f40771a478 None None] filter_scheduler.available_filters = ['nova.scheduler.filters.all_
22:15:26 melwitt filters']
22:16:25 kaiokmo mine is filter_scheduler.available_filters = ['nova.scheduler.filters.all_filters']
22:16:33 kaiokmo and filter_scheduler.enabled_filters = ['AggregateInstanceExtraSpecs Filter']
22:17:00 kaiokmo which seems correct
22:17:36 melwitt is the space in the middle of ['AggregateInstanceExtraSpecs Filter'] just a typo in chat or?
22:17:48 melwitt because it should not have a space
22:19:53 kaiokmo typo in chat.
22:20:04 kaiokmo for a moment I thought "omg, it can't be" :)
22:23:11 melwitt scheduler.driver = filter_scheduler should be in the log too
22:25:55 kaiokmo yeah, it is here
22:26:28 melwitt w t f
22:27:23 kaiokmo doesn't make sense for me either
22:28:01 kaiokmo been trying to get this working since yesterday.
22:30:02 melwitt yeah, I dunno what to tell you. I'd be putting prints all over the code and just try to trace it at this point
22:30:09 melwitt do you see a log like this at least? DEBUG nova.filters [None req-4d2d1a33-c0da-4285-a500-3806a3313c4d admin admin] Starting with 1 host(s)
22:30:42 melwitt that's what it says before it starts running filters
22:33:13 kaiokmo not really. only a bunch of "Lock acquired...", "Running periodic tasks...", and "Succesfully synced instances...
22:33:28 kaiokmo so, the filters are not running at all? that's not good
22:34:27 melwitt backing up, you are seeing a server successfully boot right? and it's going onto hosts you don't want?
22:35:23 melwitt if so, are you running one scheduler or multiple? at least one of them should be showing filtering related messages if debug=True in the nova-scheduler config

Earlier   Later