| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-05-24 | |||
| 19:05:15 | sean-k-mooney | ah ok i taught you ment in the nova code | |
| 19:05:30 | melwitt | heh, yeah no | |
| 19:05:42 | openstackgerrit | Adam Spiers proposed openstack/nova master: Reduce logging of host hypervisor capabilities to DEBUG level https://review.opendev.org/661379 | |
| 19:09:07 | efried | aspiers: I was partway through PS12, was going to suggest moving your trait setting into _get_cpu_traits itself. And also, where's the bit where the SEV trait was going to be moved under the X86 namespace? | |
| 19:10:23 | aspiers | efried: I did consider putting it in _get_cpu_traits but that already seemed pretty over-grown | |
| 19:11:04 | openstackgerrit | Merged openstack/nova stable/rocky: [stable-only] Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/659338 | |
| 19:11:12 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional confirm_migration_error test https://review.opendev.org/658834 | |
| 19:11:18 | aspiers | efried: maybe it can be split up | |
| 19:11:46 | aspiers | efried: oh, I remember now - that wasn't the only reason | |
| 19:13:08 | aspiers | efried: I'm still behind with the latest news on the great CPU trait taxonomy debate | |
| 19:14:00 | aspiers | efried: this None check gave me pause https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L6777 | |
| 19:14:37 | aspiers | efried: *theoretically* all the traits could vanish from a host, and then this code would fail to remove them | |
| 19:15:04 | aspiers | but if that actually happened, maybe it would be a sure sign things have gone very badly wrong, in which case maybe that's a desirable accident anyway | |
| 19:16:11 | dansmith | efried: mriedem I'll admit I had my head on backwards with what I was expecting, but: http://logs.openstack.org/02/661002/4/check/nova-next/39819ea/job-output.txt.gz#_2019-05-24_18_14_23_954469 | |
| 19:16:20 | dansmith | seems to work to make sure we don't archive recent stuff | |
| 19:16:35 | mriedem | i just -1ed that :) | |
| 19:16:38 | mriedem | we are sympatico | |
| 19:16:45 | dansmith | my intent was to make the job actually call this with "tomorrow" or something | |
| 19:16:50 | mriedem | heh yeah | |
| 19:18:22 | efried | okay, that was clear to me, thought it was intentional. But I guess if this is the only place we test the script, it makes sense to invoke it in a way that does something. | |
| 19:18:52 | dansmith | efried: I was expecting to actually land this, so it needs to not not run | |
| 19:19:07 | dansmith | I was expecting to make it do what we do today, but with a date, which would be tomorrow not yesterday | |
| 19:19:30 | dansmith | but in reality, making sure it doesn't eff up today's records when called for yesterday is what we wanted to sanity check | |
| 19:19:31 | mriedem | plus note that purge failed | |
| 19:19:33 | mriedem | as a result | |
| 19:19:38 | mriedem | but b/c no set -e the job doesn't fail | |
| 19:19:46 | efried | dansmith: What about running it twice, once like this to make sure it outputs "Nothing was archived" and then the second time without --before (or with --before $(date -d tomorrow)) to make sure it works? | |
| 19:19:49 | mriedem | dansmith: you could run it both ways | |
| 19:19:53 | mriedem | jinx | |
| 19:19:57 | dansmith | yep, doing that now | |
| 19:20:22 | efried | what went wrong with purge? | |
| 19:20:30 | efried | And is there a fup change to turn on -e? | |
| 19:20:39 | mriedem | purge fails with rc=3 if nothing was purged | |
| 19:20:43 | mriedem | "fails" | |
| 19:20:59 | mriedem | i'm all fup'ed out | |
| 19:22:02 | efried | yeah, so 3 shouldn't be a failure condition, from the pov of the calling script, right? | |
| 19:24:17 | openstackgerrit | Dan Smith proposed openstack/nova master: Make nova-next archive using --before https://review.opendev.org/661002 | |
| 19:24:34 | dansmith | I'm just about done for the holiday, we can finish this up next week | |
| 19:25:15 | efried | aspiers: If the traits all vanished, that dict would still not be None. It would just have all False values. | |
| 19:28:23 | aspiers | efried: OK, because CPU_TRAITS_MAPPING is hardcoded. Makes sense | |
| 19:28:32 | efried | yah | |
| 19:28:54 | aspiers | _get_cpu_traits() is still too big though | |
| 19:28:56 | efried | aspiers: When mriedem was doing the trait reporting in the RT, I made sure we were covering that corner case | |
| 19:29:02 | aspiers | nice | |
| 19:29:11 | efried | aspiers: Sure, I would be fine splitting that up. | |
| 19:29:36 | aspiers | efried: well, shouldn't be too hard to do that and then move the SEV bit in there | |
| 19:30:06 | aspiers | efried: other than that and maybe some trait taxonomy thing which I don't yet know about, hopefully everything else is in order with this patch | |
| 19:30:09 | efried | aspiers: https://review.opendev.org/#/c/538498/6/nova/compute/resource_tracker.py@904 (patch look familiar?) | |
| 19:31:14 | aspiers | efried: yeah, I do remember reading that and vaguely understanding | |
| 19:31:23 | aspiers | I'm sure a re-read will make a lot more sense now | |
| 19:47:20 | mriedem | ganso: ok so i think things in queens are ok except the functional test is failing because of some missing test handling stuff on teardown in queens, working on that now | |
| 19:48:17 | ganso | mriedem: thanks Matt! =D | |
| 19:52:34 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add functional confirm_migration_error test https://review.opendev.org/658136 | |
| 19:52:50 | mriedem | ganso: i think you can drop the -1 on https://review.opendev.org/#/c/652153/ now | |
| 19:53:27 | ganso | mriedem: yup, dropped =) | |
| 20:07:56 | kaiokmo | hey nova folks! I was wondering if something changed regarding how metadata are defined/handled by nova-scheduler | |
| 20:08:10 | kaiokmo | specifically on the aggregate_instance_extra_specs filter | |
| 20:08:56 | kaiokmo | I have a Pike deployment in which this filter works properly | |
| 20:10:15 | kaiokmo | last week I deployed a environment on Rocky (tag is 18.1.7), and this filters is not working as expected | |
| 20:10:43 | kaiokmo | nova-scheduler is ignoring the filter and all hosts are passing the filtering phase | |
| 20:12:06 | kaiokmo | nova.conf is configured as follows (only the aggregate_instance_extra_specs is enabled): http://paste.openstack.org/show/752054/ | |
| 20:12:21 | kaiokmo | I appreciate any help. | |
| 20:13:41 | kaiokmo | thanks in advance | |
| 20:22:49 | aspiers | alright, I'm off. happy weekend o/ | |
| 20:28:59 | melwitt | kaiokmo: not aware of any change off the top of my head. will look around the code to see if anything has changed in that area | |
| 20:32:04 | kaiokmo | melwitt: thank you very much | |
| 20:36:50 | melwitt | I'm not seeing that anything has changed in the area | |
| 20:37:40 | melwitt | kaiokmo: have you enabled debug logging in nova-scheduler and look at what's going on? if so, do you see DEBUG messages like these? https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_instance_extra_specs.py#L64 | |
| 20:39:33 | melwitt | interesting, my devstack change started running on a node within 14 minutes of uploading it | |
| 20:40:13 | melwitt | reminds me of something in a ML thread where things in the nova queue have to wait longer because of something to do with zuul scheduling? | |
| 20:41:34 | efried | kaiokmo: Does this happen to be on a rebuild? | |
| 20:42:21 | kaiokmo | melwitt: debug is set to True on nova.conf but can't see messages like that on nova-scheduler log | |
| 20:43:05 | kaiokmo | efried: no. RUN_ON_REBUILD is set to false on the filter, and I'm only spawning new instances, not rebuilding an existing one | |
| 20:44:46 | melwitt | kaiokmo: hm, did you restart nova-scheduler after adding debug=True? that config should work. we'll probably need to look at the debug logs to get to the bottom of this because I didn't find any open bugs mentioning the AggregateInstanceExtraSpecsFilter and no recent code changes | |
| 20:45:02 | efried | kaiokmo: I'm looking specifically at https://review.opendev.org/#/c/523212/ which introduced RUN_ON_REBUILD and wondering if you're running afoul of the logic at https://review.opendev.org/#/c/523212/2/nova/scheduler/host_manager.py | |
| 20:46:11 | efried | that's some complex stuff with check type and force hosts/nodes, I don't really understand it. | |
| 20:46:37 | efried | kaiokmo: If you felt like experimenting, you could try reverting (pieces of) that change and see if the problem goes away. | |
| 20:47:09 | kaiokmo | melwitt: yes, I did restart the service. right now I can see some DEBUG messages on the logs | |
| 20:48:03 | melwitt | oh, good. argh, I just realized you had said the problem is all hosts are included, not the other way around | |
| 20:48:08 | kaiokmo | efried: I can test reverting this change, although I don't feel like it is going to change anything for me | |
| 20:48:18 | melwitt | so you wouldn't see messages like that if it thinks the hosts are good when they should not be | |
| 20:50:00 | kaiokmo | I see. I'm wondering if even the host_passes function is being called. I managed to put some LOG.debug myself inside the function, but none of my messages are displayed on the logs | |
| 20:50:01 | efried | kaiokmo: It's just the only thing that changed on pike in that particular filter, so it's the first suspect for root cause. | |
| 20:50:12 | efried | right | |
| 20:50:25 | melwitt | kaiokmo: does your flavor have extra_specs in it? because the filter code is showing it will "pass" the host if the flavor does not have any extra_specs | |
| 20:50:57 | melwitt | oh really, ok that's not good. lemme see... | |
| 20:51:57 | kaiokmo | btw, the is being loaded correctly by the scheduler. LOG.debug placed before host_passes appear on the logs | |
| 20:53:28 | melwitt | kaiokmo: this might be a problem with your config, the enabled_filters and available_filters are supposed to be lists. I wonder if that's what's wrong? | |
| 20:56:00 | melwitt | I can never remember how to format lists in ini conf | |
| 20:57:29 | melwitt | looks like it's just comma separated: enabled_filters = RetryFilter,AvailabilityZoneFilter | |
| 20:58:38 | efried | iiuc you're using the same config file as before? | |
| 20:59:13 | efried | kaiokmo: Oh, disregard the thing about reverting that patch. I misread your original issue, you said this was working on pike and broke on rocky. That patch went back to pike, so probably isn't to blame. | |
| 20:59:33 | melwitt | I was wondering the same thing. because according to this, it should be the class name only, not the whole path. but I don't know whether having the whole path would hurt or not | |
| 20:59:57 | kaiokmo | melwitt: yes. metadata on aggregate is custom: "aggregate" with "test" value | |
| 21:00:29 | kaiokmo | metadata on flavor is "aggregate_instance_extra_specs:aggregate" with "test" value | |
| 21:01:39 | melwitt | kaiokmo: when you added your own debug messages, did you put one here? https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_instance_extra_specs.py#L48 wondering if you would see that one? | |
| 21:02:52 | kaiokmo | melwitt: yes, it is LOG.debug("***OMG***") :) | |
| 21:03:24 | melwitt | haha xD | |
| 21:04:20 | melwitt | did you put one right at the beginning of host_passes before anything else? because if that's not showing, then you're right the filter isn't running | |
| 21:05:22 | melwitt | if the filter isn't running, I'd try to use config options like available_filters = AggregateInstanceExtraSpecsFilter enabled_filters = AggregateInstanceExtraSpecsFilter and then see if you see the filter run | |