Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-04-16
12:56:40 sean-k-mooney my preference would be make the emulator pinning work for floating cpus too
12:56:54 sean-k-mooney and have a seperate pin set for it
12:57:00 stephenfin What would the alternative be? Always offload emulator threads if 'cpu_shared_set' is defined?
12:57:01 stephenfin Why?
12:57:13 stephenfin The reason you're offloading this stuff is performance, no?
12:57:30 stephenfin More specifically, real-time performance
12:57:32 stephenfin If you're using floating CPUs, you've already given up on that
12:57:35 sean-k-mooney yes but perfroamce still maters for non pinned instance
12:57:48 sean-k-mooney the main reason however is to simplfy the code
12:57:50 sean-k-mooney and the config
12:58:20 sean-k-mooney basicaly if you define emulator_pin_set in the libvirt section we will always use it
12:58:43 sean-k-mooney for all vms and if not then it overlap with the vm cores
12:58:59 stephenfin I was thinking a similar thing but only for instances with PCPUs and only if 'cpu_shared_set' is defined
12:59:13 stephenfin (As opposed to all instances and only if 'emulator_pin_set' is defined)
12:59:15 sean-k-mooney in either case if emulator_pin_set is not overlapping with the dedicated and floating pin sets then you dont need any accounting in placement
12:59:45 sean-k-mooney i would do it for all instance as i would liek to try and share more code between pinned and floating instance
13:00:24 stephenfin that makes sense
13:00:32 sean-k-mooney like we do with numa i would like to soft pin all vms to a floating_pin_set so you can properly reserve cores on the host for OS or vswith use
13:00:55 stephenfin Well, I think we'll be doing that going forward regardless
13:00:59 sean-k-mooney we need to do that anyway when we mix pinned and floating vm on the same host
13:01:03 stephenfin Yeah
13:01:04 sean-k-mooney yep
13:01:16 stephenfin Adding another configuration option would be yet another breaking change though
13:01:48 stephenfin So many breaking changes
13:03:16 sean-k-mooney it does not need to be
13:03:18 sean-k-mooney https://review.openstack.org/#/c/555081/23/specs/train/approved/cpu-resources.rst@157
13:03:35 sean-k-mooney [compute]
13:03:37 sean-k-mooney pinned_cpu_set
13:03:39 sean-k-mooney floating_cpu_set
13:03:41 sean-k-mooney [libvirt]
13:03:43 sean-k-mooney emulator_cpu_set
13:04:11 sean-k-mooney i suggested deprecating cpu_shared_set and renaming/moving it to [libvirt]/emulator_cpu_set
13:04:46 sean-k-mooney so it will not need config change initally althong they will get a deprecation warning
13:06:06 stephenfin Hmm, I need to think about this
13:07:44 NewBruce sean-k-mooney / mnaser … we seem to actually have made this worse - RDO - RDO which was working previously is now also broken
13:08:21 sean-k-mooney is it broken the same way?
13:08:39 NewBruce seems to be
13:09:06 NewBruce same 500 internal error; same duplicate port entries in ml2_port_bindings
13:09:08 sean-k-mooney and all RDO nodes have the same code deployed and config
13:10:45 sean-k-mooney the is begining to feel like it might be related to the specific configuration of some of the compute nodes rather than a gloabl configuration issue but that is a feeling rather then anything based on fact
13:11:27 NewBruce not identical (due to the time span taken to update), we have mostly 18.2.0, but some 18.1.0 and an 18.0.3
13:11:47 NewBruce we did try rolling an 18.2.0 back to 18.1.0 - but that didn’t seem to help
13:12:25 sean-k-mooney can you try migration between two nodes of the same version on the rdo side for each of the 3 versions you have deployed
13:12:28 NewBruce yeah, i agree its starting to feel something odd (random)… but as you say, feeling - not fact
13:12:35 NewBruce Yeah, will do now
13:12:50 sean-k-mooney perhaps we can narrow it donw to a version and then do a git biset for the changes
13:17:20 mnaser hmm
13:17:23 mnaser this is strange
13:17:38 mnaser NewBruce: so that cloud is 100% rocky at this point?
13:22:54 NewBruce mnaser yeah, 100% rocky
13:23:01 NewBruce … in all senses of the word :D
13:25:01 sean-k-mooney this is the delta in termes of ptaches https://github.com/openstack/nova/compare/18.0.3...18.1.0
13:25:57 mnaser NewBruce: curl -H "X-Auth-Token: `openstack token issue -c id -f value`" http://network-sjc1.vexxhost.us/v2.0/extensions | python -mjson.tool | grep binding-extended
13:26:03 mnaser change network-sjc1 with your neutron endpoint
13:26:21 mnaser can you run that many several times and see if there is every a time that it doesn't return binding-extended?
13:26:35 sean-k-mooney i wonder if it could be realated to https://github.com/openstack/nova/commit/4a12c9c298913f99570f2f8e93500db687e98dc9
13:28:31 sean-k-mooney hum although that is only on revert
13:29:11 NewBruce so an RDO 18.1.0 -> RDO 18.1.0 does the same thing; fails on duplicate ports
13:30:20 NewBruce "description": "Expose port bindings of a virtual port to external application",
13:30:20 NewBruce "alias": "binding-extended",
13:30:20 NewBruce "extension": {
13:30:20 NewBruce {
13:30:20 NewBruce mnaser : curl -s -H "X-Auth-Token: $OS_TOKEN" https://kna1.citycloud.com:9696/v2.0/extensions/binding-extended | python -m json.tool
13:30:21 NewBruce "name": "Port Bindings Extended",
13:30:21 NewBruce "links": [],
13:30:22 NewBruce }
13:30:22 NewBruce "updated": "2017-07-17T10:00:00-00:00"
13:30:23 NewBruce }
13:31:16 NewBruce (#SoccerDad duties - back online in about 30 min)
13:31:17 mnaser NewBruce: but did you hit it many different times, and it always gave a response?
13:31:28 mnaser im wondering if one backend might be acting weird
13:33:49 sean-k-mooney mnaser: so you resuggesting runnint it with "watch -d -n 1 ..." and seing if it changes
13:34:14 mnaser sean-k-mooney: well just knowing that there is probably multiple network servers, maybe keep hitting it because *maybe* one of them is responding without that extension
13:34:21 mnaser https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L282 just trying to eliminate all those things here
13:34:48 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Change a log level for overwriting allocation https://review.openstack.org/652994
13:35:04 sean-k-mooney mnaser: i think i asked NewBruce to check that against each of the neutron api enpoints directly in the past bypassing the loadblancer
13:35:52 sean-k-mooney it would cause this issue espically if there was a network partion or some other factor cause different nodes to prefer diffeent api nodes
13:36:14 sean-k-mooney but if that was the case we would expect the same resules for osa to osa too right?
13:38:23 mnaser that is true as well
13:38:36 mnaser sean-k-mooney: but this is one of those issues where you're just desperately trying to do whatever works lol
13:38:59 efried dansmith: When you get a chance, would you please do the channel topic thing with https://etherpad.openstack.org/p/nova-spec-review-day ?
13:39:11 openstackgerrit Merged openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458
13:39:39 sean-k-mooney ya i know from what we can tell it should be working if it wasnt for the evidence e.g. its not that would be my respocne if some one asked shoudl this work
13:43:28 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Add missing libvirt exception during device detach https://review.openstack.org/651639
13:43:38 openstackgerrit sean mooney proposed openstack/os-traits master: add libvirt image metadata traits https://review.openstack.org/652996
13:52:04 bauzas mriedem: sorry, finished earlier yesterday
13:52:19 bauzas mriedem: could you please ping me again which stable changes I could review ?
13:53:09 mriedem bauzas: at this point, probably queens https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/queens
13:53:28 bauzas ack
13:53:38 mriedem bauzas: and https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/pike+topic:bug/1669054 for pike
13:57:34 dansmith efried: as a placement and request groups expert, I hope you can look at this one and provide a short path to being able to make request groups expressive enough that we don't have to keep adding legacy stuff like this: https://review.openstack.org/#/c/650963/2
13:59:01 efried dansmith: on my list. I had suggested https://review.openstack.org/#/c/650476/ which is generating lots of discussion both on the spec and on the ML http://lists.openstack.org/pipermail/openstack-discuss/2019-April/thread.html#4782
13:59:37 dansmith efried: ack, glad to see that
13:59:42 efried unfortunately it's not an easy problem.
13:59:54 dansmith stephenfin: bauzas: I trust you will be all over that ^ spec to make sure it will solve your problems
14:00:02 efried which is why we've been talking about it for a couple years
14:02:52 mriedem melwitt: are consoles/console pools in the db model a xen only thing?
14:04:09 mriedem ah yes, "The nova-console service is deprecated as it is Xen specific"
14:05:17 melwitt cool. I don't recall the term "console pools" off the top of my head
14:07:28 NewBruce mnaser have run it in a loop and no changes

Earlier   Later