| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-16 | |||
| 12:53:49 | sean-k-mooney | i kind of feel we should require the dedicated_cpu_set to be in the config before enableing any of this | |
| 12:54:03 | sean-k-mooney | stephenfin: ya that i was original not sure about but i think i agree | |
| 12:54:20 | stephenfin | Because I can't think of anyway to account for the extra core without mangling the request | |
| 12:54:32 | sean-k-mooney | hw:emulator_threads_policy=share shoudl be sufficent and more efficent | |
| 12:54:50 | sean-k-mooney | well the acounting is trivial | |
| 12:54:53 | stephenfin | I really, really wish we'd overloaded 'isolate' instead of 'share' to do this offload to 'shared_cpu_set' | |
| 12:55:25 | stephenfin | because I don't see a reason why anyone would want to use 'isolate' as it's implemented | |
| 12:55:35 | sean-k-mooney | we already generate teh resouce:vcpu part fo teh placement request for the flavor.vcpu and add 1 core to it for isolate | |
| 12:55:44 | sean-k-mooney | we can easilly just add a pcpu instead | |
| 12:56:19 | sean-k-mooney | well personally i would like to kill the option entirly. | |
| 12:56:22 | stephenfin | True, but rewriting the request like that feels rotten | |
| 12:56:25 | stephenfin | So would I | |
| 12:56:40 | sean-k-mooney | my preference would be make the emulator pinning work for floating cpus too | |
| 12:56:54 | sean-k-mooney | and have a seperate pin set for it | |
| 12:57:00 | stephenfin | What would the alternative be? Always offload emulator threads if 'cpu_shared_set' is defined? | |
| 12:57:01 | stephenfin | Why? | |
| 12:57:13 | stephenfin | The reason you're offloading this stuff is performance, no? | |
| 12:57:30 | stephenfin | More specifically, real-time performance | |
| 12:57:32 | stephenfin | If you're using floating CPUs, you've already given up on that | |
| 12:57:35 | sean-k-mooney | yes but perfroamce still maters for non pinned instance | |
| 12:57:48 | sean-k-mooney | the main reason however is to simplfy the code | |
| 12:57:50 | sean-k-mooney | and the config | |
| 12:58:20 | sean-k-mooney | basicaly if you define emulator_pin_set in the libvirt section we will always use it | |
| 12:58:43 | sean-k-mooney | for all vms and if not then it overlap with the vm cores | |
| 12:58:59 | stephenfin | I was thinking a similar thing but only for instances with PCPUs and only if 'cpu_shared_set' is defined | |
| 12:59:13 | stephenfin | (As opposed to all instances and only if 'emulator_pin_set' is defined) | |
| 12:59:15 | sean-k-mooney | in either case if emulator_pin_set is not overlapping with the dedicated and floating pin sets then you dont need any accounting in placement | |
| 12:59:45 | sean-k-mooney | i would do it for all instance as i would liek to try and share more code between pinned and floating instance | |
| 13:00:24 | stephenfin | that makes sense | |
| 13:00:32 | sean-k-mooney | like we do with numa i would like to soft pin all vms to a floating_pin_set so you can properly reserve cores on the host for OS or vswith use | |
| 13:00:55 | stephenfin | Well, I think we'll be doing that going forward regardless | |
| 13:00:59 | sean-k-mooney | we need to do that anyway when we mix pinned and floating vm on the same host | |
| 13:01:03 | stephenfin | Yeah | |
| 13:01:04 | sean-k-mooney | yep | |
| 13:01:16 | stephenfin | Adding another configuration option would be yet another breaking change though | |
| 13:01:48 | stephenfin | So many breaking changes | |
| 13:03:16 | sean-k-mooney | it does not need to be | |
| 13:03:18 | sean-k-mooney | https://review.openstack.org/#/c/555081/23/specs/train/approved/cpu-resources.rst@157 | |
| 13:03:35 | sean-k-mooney | [compute] | |
| 13:03:37 | sean-k-mooney | pinned_cpu_set | |
| 13:03:39 | sean-k-mooney | floating_cpu_set | |
| 13:03:41 | sean-k-mooney | [libvirt] | |
| 13:03:43 | sean-k-mooney | emulator_cpu_set | |
| 13:04:11 | sean-k-mooney | i suggested deprecating cpu_shared_set and renaming/moving it to [libvirt]/emulator_cpu_set | |
| 13:04:46 | sean-k-mooney | so it will not need config change initally althong they will get a deprecation warning | |
| 13:06:06 | stephenfin | Hmm, I need to think about this | |
| 13:07:44 | NewBruce | sean-k-mooney / mnaser … we seem to actually have made this worse - RDO - RDO which was working previously is now also broken | |
| 13:08:21 | sean-k-mooney | is it broken the same way? | |
| 13:08:39 | NewBruce | seems to be | |
| 13:09:06 | NewBruce | same 500 internal error; same duplicate port entries in ml2_port_bindings | |
| 13:09:08 | sean-k-mooney | and all RDO nodes have the same code deployed and config | |
| 13:10:45 | sean-k-mooney | the is begining to feel like it might be related to the specific configuration of some of the compute nodes rather than a gloabl configuration issue but that is a feeling rather then anything based on fact | |
| 13:11:27 | NewBruce | not identical (due to the time span taken to update), we have mostly 18.2.0, but some 18.1.0 and an 18.0.3 | |
| 13:11:47 | NewBruce | we did try rolling an 18.2.0 back to 18.1.0 - but that didn’t seem to help | |
| 13:12:25 | sean-k-mooney | can you try migration between two nodes of the same version on the rdo side for each of the 3 versions you have deployed | |
| 13:12:28 | NewBruce | yeah, i agree its starting to feel something odd (random)… but as you say, feeling - not fact | |
| 13:12:35 | NewBruce | Yeah, will do now | |
| 13:12:50 | sean-k-mooney | perhaps we can narrow it donw to a version and then do a git biset for the changes | |
| 13:17:20 | mnaser | hmm | |
| 13:17:23 | mnaser | this is strange | |
| 13:17:38 | mnaser | NewBruce: so that cloud is 100% rocky at this point? | |
| 13:22:54 | NewBruce | mnaser yeah, 100% rocky | |
| 13:23:01 | NewBruce | … in all senses of the word :D | |
| 13:25:01 | sean-k-mooney | this is the delta in termes of ptaches https://github.com/openstack/nova/compare/18.0.3...18.1.0 | |
| 13:25:57 | mnaser | NewBruce: curl -H "X-Auth-Token: `openstack token issue -c id -f value`" http://network-sjc1.vexxhost.us/v2.0/extensions | python -mjson.tool | grep binding-extended | |
| 13:26:03 | mnaser | change network-sjc1 with your neutron endpoint | |
| 13:26:21 | mnaser | can you run that many several times and see if there is every a time that it doesn't return binding-extended? | |
| 13:26:35 | sean-k-mooney | i wonder if it could be realated to https://github.com/openstack/nova/commit/4a12c9c298913f99570f2f8e93500db687e98dc9 | |
| 13:28:31 | sean-k-mooney | hum although that is only on revert | |
| 13:29:11 | NewBruce | so an RDO 18.1.0 -> RDO 18.1.0 does the same thing; fails on duplicate ports | |
| 13:30:20 | NewBruce | "description": "Expose port bindings of a virtual port to external application", | |
| 13:30:20 | NewBruce | "alias": "binding-extended", | |
| 13:30:20 | NewBruce | "extension": { | |
| 13:30:20 | NewBruce | { | |
| 13:30:20 | NewBruce | mnaser : curl -s -H "X-Auth-Token: $OS_TOKEN" https://kna1.citycloud.com:9696/v2.0/extensions/binding-extended | python -m json.tool | |
| 13:30:21 | NewBruce | "name": "Port Bindings Extended", | |
| 13:30:21 | NewBruce | "links": [], | |
| 13:30:22 | NewBruce | } | |
| 13:30:22 | NewBruce | "updated": "2017-07-17T10:00:00-00:00" | |
| 13:30:23 | NewBruce | } | |
| 13:31:16 | NewBruce | (#SoccerDad duties - back online in about 30 min) | |
| 13:31:17 | mnaser | NewBruce: but did you hit it many different times, and it always gave a response? | |
| 13:31:28 | mnaser | im wondering if one backend might be acting weird | |
| 13:33:49 | sean-k-mooney | mnaser: so you resuggesting runnint it with "watch -d -n 1 ..." and seing if it changes | |
| 13:34:14 | mnaser | sean-k-mooney: well just knowing that there is probably multiple network servers, maybe keep hitting it because *maybe* one of them is responding without that extension | |
| 13:34:21 | mnaser | https://github.com/openstack/nova/blob/stable/rocky/nova/conductor/tasks/live_migrate.py#L282 just trying to eliminate all those things here | |
| 13:34:48 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Change a log level for overwriting allocation https://review.openstack.org/652994 | |
| 13:35:04 | sean-k-mooney | mnaser: i think i asked NewBruce to check that against each of the neutron api enpoints directly in the past bypassing the loadblancer | |
| 13:35:52 | sean-k-mooney | it would cause this issue espically if there was a network partion or some other factor cause different nodes to prefer diffeent api nodes | |
| 13:36:14 | sean-k-mooney | but if that was the case we would expect the same resules for osa to osa too right? | |
| 13:38:23 | mnaser | that is true as well | |
| 13:38:36 | mnaser | sean-k-mooney: but this is one of those issues where you're just desperately trying to do whatever works lol | |
| 13:38:59 | efried | dansmith: When you get a chance, would you please do the channel topic thing with https://etherpad.openstack.org/p/nova-spec-review-day ? | |
| 13:39:11 | openstackgerrit | Merged openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | |
| 13:39:39 | sean-k-mooney | ya i know from what we can tell it should be working if it wasnt for the evidence e.g. its not that would be my respocne if some one asked shoudl this work | |
| 13:43:28 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Add missing libvirt exception during device detach https://review.openstack.org/651639 | |
| 13:43:38 | openstackgerrit | sean mooney proposed openstack/os-traits master: add libvirt image metadata traits https://review.openstack.org/652996 | |
| 13:52:04 | bauzas | mriedem: sorry, finished earlier yesterday | |
| 13:52:19 | bauzas | mriedem: could you please ping me again which stable changes I could review ? | |
| 13:53:09 | mriedem | bauzas: at this point, probably queens https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/queens | |