Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-24
19:01:20 mriedem most people are probably not going to be using extracted placement at stein ga
19:01:21 sean-k-mooney my comment still applies then
19:01:44 mriedem your comment about threads=1?
19:01:55 sean-k-mooney ya https://bugs.launchpad.net/nova/+bug/1829062/comments/7
19:01:56 openstack Launchpad bug 1829062 in StarlingX "nova placement api non-responsive due to eventlet error" [Critical,In progress] - Assigned to Gerry Kopec (gerry-kopec)
19:02:04 mriedem if so, then my comment about a known issue release note applies :)
19:02:27 sean-k-mooney if they set wsgi to use 1 thread per process then they wont get the context swtich error
19:02:35 sean-k-mooney *thread switch
19:02:57 melwitt mriedem: I'm actually thinking that in devstack lib/nova, we could use NOVA_CPU_CELL to mimic the other offset code to set the ports properly in the one nova-cpu.conf it writes. I'll give it a try and see if it could work
19:03:22 sean-k-mooney also i spend the last 5 minutes trying to figure out how the starling x repos work and i am more confused then when i started
19:03:37 openstackgerrit Adam Spiers proposed openstack/nova master: Provide HW_CPU_AMD_SEV trait when SEV is supported https://review.opendev.org/638680
19:03:56 melwitt I didn't realize there was a variable that would tell us what cell we (nova-compute) we are
19:04:04 melwitt -we
19:04:08 aspiers efried: this might be ready to go now ^^^
19:04:24 sean-k-mooney melwitt: we do?
19:04:30 mriedem melwitt: ok, yeah that's probably easy to test with a nova patch depending on it since nova-multi-cell doesn't run on devstack changes (we could add it to the devstack experimental queue for testing stuff like this on-demand)
19:04:50 mriedem i'd just add nova-multi-cell to devstack experimental, i can push a change for that
19:04:51 melwitt sean-k-mooney: yeah NOVA_CPU_CELL, zuul job configs can set it
19:04:54 sean-k-mooney melwitt: oh a devstack variable
19:04:57 melwitt yes
19:05:14 melwitt mriedem: ok, sounds good
19:05:15 sean-k-mooney ah ok i taught you ment in the nova code
19:05:30 melwitt heh, yeah no
19:05:42 openstackgerrit Adam Spiers proposed openstack/nova master: Reduce logging of host hypervisor capabilities to DEBUG level https://review.opendev.org/661379
19:09:07 efried aspiers: I was partway through PS12, was going to suggest moving your trait setting into _get_cpu_traits itself. And also, where's the bit where the SEV trait was going to be moved under the X86 namespace?
19:10:23 aspiers efried: I did consider putting it in _get_cpu_traits but that already seemed pretty over-grown
19:11:04 openstackgerrit Merged openstack/nova stable/rocky: [stable-only] Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/659338
19:11:12 openstackgerrit Merged openstack/nova stable/rocky: Add functional confirm_migration_error test https://review.opendev.org/658834
19:11:18 aspiers efried: maybe it can be split up
19:11:46 aspiers efried: oh, I remember now - that wasn't the only reason
19:13:08 aspiers efried: I'm still behind with the latest news on the great CPU trait taxonomy debate
19:14:00 aspiers efried: this None check gave me pause https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/driver.py#L6777
19:14:37 aspiers efried: *theoretically* all the traits could vanish from a host, and then this code would fail to remove them
19:15:04 aspiers but if that actually happened, maybe it would be a sure sign things have gone very badly wrong, in which case maybe that's a desirable accident anyway
19:16:11 dansmith efried: mriedem I'll admit I had my head on backwards with what I was expecting, but: http://logs.openstack.org/02/661002/4/check/nova-next/39819ea/job-output.txt.gz#_2019-05-24_18_14_23_954469
19:16:20 dansmith seems to work to make sure we don't archive recent stuff
19:16:35 mriedem i just -1ed that :)
19:16:38 mriedem we are sympatico
19:16:45 dansmith my intent was to make the job actually call this with "tomorrow" or something
19:16:50 mriedem heh yeah
19:18:22 efried okay, that was clear to me, thought it was intentional. But I guess if this is the only place we test the script, it makes sense to invoke it in a way that does something.
19:18:52 dansmith efried: I was expecting to actually land this, so it needs to not not run
19:19:07 dansmith I was expecting to make it do what we do today, but with a date, which would be tomorrow not yesterday
19:19:30 dansmith but in reality, making sure it doesn't eff up today's records when called for yesterday is what we wanted to sanity check
19:19:31 mriedem plus note that purge failed
19:19:33 mriedem as a result
19:19:38 mriedem but b/c no set -e the job doesn't fail
19:19:46 efried dansmith: What about running it twice, once like this to make sure it outputs "Nothing was archived" and then the second time without --before (or with --before $(date -d tomorrow)) to make sure it works?
19:19:49 mriedem dansmith: you could run it both ways
19:19:53 mriedem jinx
19:19:57 dansmith yep, doing that now
19:20:22 efried what went wrong with purge?
19:20:30 efried And is there a fup change to turn on -e?
19:20:39 mriedem purge fails with rc=3 if nothing was purged
19:20:43 mriedem "fails"
19:20:59 mriedem i'm all fup'ed out
19:22:02 efried yeah, so 3 shouldn't be a failure condition, from the pov of the calling script, right?
19:24:17 openstackgerrit Dan Smith proposed openstack/nova master: Make nova-next archive using --before https://review.opendev.org/661002
19:24:34 dansmith I'm just about done for the holiday, we can finish this up next week
19:25:15 efried aspiers: If the traits all vanished, that dict would still not be None. It would just have all False values.
19:28:23 aspiers efried: OK, because CPU_TRAITS_MAPPING is hardcoded. Makes sense
19:28:32 efried yah
19:28:54 aspiers _get_cpu_traits() is still too big though
19:28:56 efried aspiers: When mriedem was doing the trait reporting in the RT, I made sure we were covering that corner case
19:29:02 aspiers nice
19:29:11 efried aspiers: Sure, I would be fine splitting that up.
19:29:36 aspiers efried: well, shouldn't be too hard to do that and then move the SEV bit in there
19:30:06 aspiers efried: other than that and maybe some trait taxonomy thing which I don't yet know about, hopefully everything else is in order with this patch
19:30:09 efried aspiers: https://review.opendev.org/#/c/538498/6/nova/compute/resource_tracker.py@904 (patch look familiar?)
19:31:14 aspiers efried: yeah, I do remember reading that and vaguely understanding
19:31:23 aspiers I'm sure a re-read will make a lot more sense now
19:47:20 mriedem ganso: ok so i think things in queens are ok except the functional test is failing because of some missing test handling stuff on teardown in queens, working on that now
19:48:17 ganso mriedem: thanks Matt! =D
19:52:34 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add functional confirm_migration_error test https://review.opendev.org/658136
19:52:50 mriedem ganso: i think you can drop the -1 on https://review.opendev.org/#/c/652153/ now
19:53:27 ganso mriedem: yup, dropped =)
20:07:56 kaiokmo hey nova folks! I was wondering if something changed regarding how metadata are defined/handled by nova-scheduler
20:08:10 kaiokmo specifically on the aggregate_instance_extra_specs filter
20:08:56 kaiokmo I have a Pike deployment in which this filter works properly
20:10:15 kaiokmo last week I deployed a environment on Rocky (tag is 18.1.7), and this filters is not working as expected
20:10:43 kaiokmo nova-scheduler is ignoring the filter and all hosts are passing the filtering phase
20:12:06 kaiokmo nova.conf is configured as follows (only the aggregate_instance_extra_specs is enabled): http://paste.openstack.org/show/752054/
20:12:21 kaiokmo I appreciate any help.
20:13:41 kaiokmo thanks in advance
20:22:49 aspiers alright, I'm off. happy weekend o/
20:28:59 melwitt kaiokmo: not aware of any change off the top of my head. will look around the code to see if anything has changed in that area
20:32:04 kaiokmo melwitt: thank you very much
20:36:50 melwitt I'm not seeing that anything has changed in the area
20:37:40 melwitt kaiokmo: have you enabled debug logging in nova-scheduler and look at what's going on? if so, do you see DEBUG messages like these? https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_instance_extra_specs.py#L64
20:39:33 melwitt interesting, my devstack change started running on a node within 14 minutes of uploading it
20:40:13 melwitt reminds me of something in a ML thread where things in the nova queue have to wait longer because of something to do with zuul scheduling?
20:41:34 efried kaiokmo: Does this happen to be on a rebuild?
20:42:21 kaiokmo melwitt: debug is set to True on nova.conf but can't see messages like that on nova-scheduler log
20:43:05 kaiokmo efried: no. RUN_ON_REBUILD is set to false on the filter, and I'm only spawning new instances, not rebuilding an existing one
20:44:46 melwitt kaiokmo: hm, did you restart nova-scheduler after adding debug=True? that config should work. we'll probably need to look at the debug logs to get to the bottom of this because I didn't find any open bugs mentioning the AggregateInstanceExtraSpecsFilter and no recent code changes
20:45:02 efried kaiokmo: I'm looking specifically at https://review.opendev.org/#/c/523212/ which introduced RUN_ON_REBUILD and wondering if you're running afoul of the logic at https://review.opendev.org/#/c/523212/2/nova/scheduler/host_manager.py
20:46:11 efried that's some complex stuff with check type and force hosts/nodes, I don't really understand it.
20:46:37 efried kaiokmo: If you felt like experimenting, you could try reverting (pieces of) that change and see if the problem goes away.
20:47:09 kaiokmo melwitt: yes, I did restart the service. right now I can see some DEBUG messages on the logs
20:48:03 melwitt oh, good. argh, I just realized you had said the problem is all hosts are included, not the other way around

Earlier   Later