| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-16 | |||
| 12:28:46 | sean-k-mooney | it will be useful for project like kolla that need to figureout what to install | |
| 12:48:42 | openstackgerrit | Merged openstack/nova master: conf: Undeprecate and move the 'dhcp_domain' option https://review.openstack.org/480616 | |
| 12:49:00 | stephenfin | sean-k-mooney: We're going to have to make a call on https://review.openstack.org/#/c/555081/23/specs/train/approved/cpu-resources.rst@180 | |
| 12:49:21 | bauzas | stephenfin: I still need to provide my comments | |
| 12:49:41 | stephenfin | sean-k-mooney: On one hand, I totally get where you're coming from and agree that it's a valid concern | |
| 12:50:01 | sean-k-mooney | yes that is one of the bits im most uncomfortable with. | |
| 12:50:01 | stephenfin | On the other though, if we don't use 'vcpu_pin_set' to populate this stuff, the user will be left in a state where they can no longer boot any instances with 'hw:cpu_policy' configured until they do additional configuration on each host | |
| 12:50:20 | stephenfin | That's as much a breaking change as anything else we've discussed | |
| 12:51:19 | stephenfin | bauzas: Ack. Just focussing on that one point atm | |
| 12:51:32 | sean-k-mooney | yes its not simple. if we use the totally new config options i suggest we could have tehm pre set them and possible provide an upstread check in nova status? | |
| 12:52:07 | sean-k-mooney | vcpu_pin_set was never intended to be "pinned cpus" alther it was assumed to be that in some cases | |
| 12:52:23 | sean-k-mooney | so i dont know what the right way forward is | |
| 12:52:28 | kashyap | lyarwood: Okay, aside: the 'nvram' stanza in /etc/libvirt/qemu.conf will be depreacted. So we can ignore that | |
| 12:52:32 | stephenfin | Hmm, neither do I :\ | |
| 12:53:18 | stephenfin | I'm also thinking we need to kill 'hw:emulator_threads_policy=isolate' | |
| 12:53:49 | sean-k-mooney | i kind of feel we should require the dedicated_cpu_set to be in the config before enableing any of this | |
| 12:54:03 | sean-k-mooney | stephenfin: ya that i was original not sure about but i think i agree | |
| 12:54:20 | stephenfin | Because I can't think of anyway to account for the extra core without mangling the request | |
| 12:54:32 | sean-k-mooney | hw:emulator_threads_policy=share shoudl be sufficent and more efficent | |
| 12:54:50 | sean-k-mooney | well the acounting is trivial | |
| 12:54:53 | stephenfin | I really, really wish we'd overloaded 'isolate' instead of 'share' to do this offload to 'shared_cpu_set' | |
| 12:55:25 | stephenfin | because I don't see a reason why anyone would want to use 'isolate' as it's implemented | |
| 12:55:35 | sean-k-mooney | we already generate teh resouce:vcpu part fo teh placement request for the flavor.vcpu and add 1 core to it for isolate | |
| 12:55:44 | sean-k-mooney | we can easilly just add a pcpu instead | |
| 12:56:19 | sean-k-mooney | well personally i would like to kill the option entirly. | |
| 12:56:22 | stephenfin | True, but rewriting the request like that feels rotten | |
| 12:56:25 | stephenfin | So would I | |
| 12:56:40 | sean-k-mooney | my preference would be make the emulator pinning work for floating cpus too | |
| 12:56:54 | sean-k-mooney | and have a seperate pin set for it | |
| 12:57:00 | stephenfin | What would the alternative be? Always offload emulator threads if 'cpu_shared_set' is defined? | |
| 12:57:01 | stephenfin | Why? | |
| 12:57:13 | stephenfin | The reason you're offloading this stuff is performance, no? | |
| 12:57:30 | stephenfin | More specifically, real-time performance | |
| 12:57:32 | stephenfin | If you're using floating CPUs, you've already given up on that | |
| 12:57:35 | sean-k-mooney | yes but perfroamce still maters for non pinned instance | |
| 12:57:48 | sean-k-mooney | the main reason however is to simplfy the code | |
| 12:57:50 | sean-k-mooney | and the config | |
| 12:58:20 | sean-k-mooney | basicaly if you define emulator_pin_set in the libvirt section we will always use it | |
| 12:58:43 | sean-k-mooney | for all vms and if not then it overlap with the vm cores | |
| 12:58:59 | stephenfin | I was thinking a similar thing but only for instances with PCPUs and only if 'cpu_shared_set' is defined | |
| 12:59:13 | stephenfin | (As opposed to all instances and only if 'emulator_pin_set' is defined) | |
| 12:59:15 | sean-k-mooney | in either case if emulator_pin_set is not overlapping with the dedicated and floating pin sets then you dont need any accounting in placement | |
| 12:59:45 | sean-k-mooney | i would do it for all instance as i would liek to try and share more code between pinned and floating instance | |
| 13:00:24 | stephenfin | that makes sense | |
| 13:00:32 | sean-k-mooney | like we do with numa i would like to soft pin all vms to a floating_pin_set so you can properly reserve cores on the host for OS or vswith use | |
| 13:00:55 | stephenfin | Well, I think we'll be doing that going forward regardless | |
| 13:00:59 | sean-k-mooney | we need to do that anyway when we mix pinned and floating vm on the same host | |
| 13:01:03 | stephenfin | Yeah | |
| 13:01:04 | sean-k-mooney | yep | |
| 13:01:16 | stephenfin | Adding another configuration option would be yet another breaking change though | |
| 13:01:48 | stephenfin | So many breaking changes | |
| 13:03:16 | sean-k-mooney | it does not need to be | |
| 13:03:18 | sean-k-mooney | https://review.openstack.org/#/c/555081/23/specs/train/approved/cpu-resources.rst@157 | |
| 13:03:35 | sean-k-mooney | [compute] | |
| 13:03:37 | sean-k-mooney | pinned_cpu_set | |
| 13:03:39 | sean-k-mooney | floating_cpu_set | |
| 13:03:41 | sean-k-mooney | [libvirt] | |
| 13:03:43 | sean-k-mooney | emulator_cpu_set | |
| 13:04:11 | sean-k-mooney | i suggested deprecating cpu_shared_set and renaming/moving it to [libvirt]/emulator_cpu_set | |
| 13:04:46 | sean-k-mooney | so it will not need config change initally althong they will get a deprecation warning | |
| 13:06:06 | stephenfin | Hmm, I need to think about this | |
| 13:07:44 | NewBruce | sean-k-mooney / mnaser … we seem to actually have made this worse - RDO - RDO which was working previously is now also broken | |
| 13:08:21 | sean-k-mooney | is it broken the same way? | |
| 13:08:39 | NewBruce | seems to be | |
| 13:09:06 | NewBruce | same 500 internal error; same duplicate port entries in ml2_port_bindings | |
| 13:09:08 | sean-k-mooney | and all RDO nodes have the same code deployed and config | |
| 13:10:45 | sean-k-mooney | the is begining to feel like it might be related to the specific configuration of some of the compute nodes rather than a gloabl configuration issue but that is a feeling rather then anything based on fact | |
| 13:11:27 | NewBruce | not identical (due to the time span taken to update), we have mostly 18.2.0, but some 18.1.0 and an 18.0.3 | |
| 13:11:47 | NewBruce | we did try rolling an 18.2.0 back to 18.1.0 - but that didn’t seem to help | |
| 13:12:25 | sean-k-mooney | can you try migration between two nodes of the same version on the rdo side for each of the 3 versions you have deployed | |
| 13:12:28 | NewBruce | yeah, i agree its starting to feel something odd (random)… but as you say, feeling - not fact | |
| 13:12:35 | NewBruce | Yeah, will do now | |
| 13:12:50 | sean-k-mooney | perhaps we can narrow it donw to a version and then do a git biset for the changes | |
| 13:17:20 | mnaser | hmm | |
| 13:17:23 | mnaser | this is strange | |
| 13:17:38 | mnaser | NewBruce: so that cloud is 100% rocky at this point? | |
| 13:22:54 | NewBruce | mnaser yeah, 100% rocky | |
| 13:23:01 | NewBruce | … in all senses of the word :D | |
| 13:25:01 | sean-k-mooney | this is the delta in termes of ptaches https://github.com/openstack/nova/compare/18.0.3...18.1.0 | |
| 13:25:57 | mnaser | NewBruce: curl -H "X-Auth-Token: `openstack token issue -c id -f value`" http://network-sjc1.vexxhost.us/v2.0/extensions | python -mjson.tool | grep binding-extended | |
| 13:26:03 | mnaser | change network-sjc1 with your neutron endpoint | |
| 13:26:21 | mnaser | can you run that many several times and see if there is every a time that it doesn't return binding-extended? | |
| 13:26:35 | sean-k-mooney | i wonder if it could be realated to https://github.com/openstack/nova/commit/4a12c9c298913f99570f2f8e93500db687e98dc9 | |
| 13:28:31 | sean-k-mooney | hum although that is only on revert | |
| 13:29:11 | NewBruce | so an RDO 18.1.0 -> RDO 18.1.0 does the same thing; fails on duplicate ports | |
| 13:30:20 | NewBruce | "description": "Expose port bindings of a virtual port to external application", | |
| 13:30:20 | NewBruce | "alias": "binding-extended", | |
| 13:30:20 | NewBruce | "extension": { | |
| 13:30:20 | NewBruce | { | |
| 13:30:20 | NewBruce | mnaser : curl -s -H "X-Auth-Token: $OS_TOKEN" https://kna1.citycloud.com:9696/v2.0/extensions/binding-extended | python -m json.tool | |
| 13:30:21 | NewBruce | "name": "Port Bindings Extended", | |
| 13:30:21 | NewBruce | "links": [], | |
| 13:30:22 | NewBruce | } | |
| 13:30:22 | NewBruce | "updated": "2017-07-17T10:00:00-00:00" | |
| 13:30:23 | NewBruce | } | |
| 13:31:16 | NewBruce | (#SoccerDad duties - back online in about 30 min) | |
| 13:31:17 | mnaser | NewBruce: but did you hit it many different times, and it always gave a response? | |
| 13:31:28 | mnaser | im wondering if one backend might be acting weird | |
| 13:33:49 | sean-k-mooney | mnaser: so you resuggesting runnint it with "watch -d -n 1 ..." and seing if it changes | |
| 13:34:14 | mnaser | sean-k-mooney: well just knowing that there is probably multiple network servers, maybe keep hitting it because *maybe* one of them is responding without that extension | |