| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-16 | |||
| 11:09:52 | aspiers | lpetrut: https://docs.openstack.org/releasenotes/nova/stein.html#new-features | |
| 11:14:53 | openstackgerrit | Merged openstack/nova master: Change a log level for overwriting allocation https://review.openstack.org/649788 | |
| 12:17:11 | NewBruce | mnaser / sean-k-mooney howdy… | |
| 12:17:54 | sean-k-mooney | NewBruce: o/ | |
| 12:18:00 | NewBruce | update on https://bugs.launchpad.net/nova/+bug/1822884 - we’ve finished upgrading all nodes on our site to rocky (service level 35), nut the issue still exists... | |
| 12:18:03 | openstack | NewBruce: Error: malone bug 1822884 not found | |
| 12:18:36 | NewBruce | (we have 3 rows in the services table not on level 35, but they have deleted > 0, so im hoping they are ignored) | |
| 12:19:31 | NewBruce | so im officially out of ideas on this one…. | |
| 12:19:33 | sean-k-mooney | if deleted = id then its a soft delete service record | |
| 12:19:57 | sean-k-mooney | that bug link does not work form me by the way but i remember the bug you were having | |
| 12:20:06 | NewBruce | Nova-compute deleted = 10 | |
| 12:20:11 | NewBruce | deleted = 60 | |
| 12:20:14 | NewBruce | deleted = 61 | |
| 12:20:56 | NewBruce | yeah i moved the bug to private for now; some logs got i want to remove; but haven’t heard back from the nova admin on that yet | |
| 12:22:45 | sean-k-mooney | ah ok | |
| 12:22:46 | NewBruce | cleaned up and public again; i can mail you the log files if you are interested | |
| 12:22:56 | NewBruce | https://bugs.launchpad.net/nova/+bug/1822884 | |
| 12:22:58 | openstack | Launchpad bug 1822884 in OpenStack Compute (nova) "live migration fails due to port binding duplicate key entry in post_live_migrate" [Undecided,New] | |
| 12:23:51 | NewBruce | posted some more to the launchpad now | |
| 12:24:50 | sean-k-mooney | so at this point all compute nodes are running rocky. the neutorn contol plane is entirely on osa | |
| 12:25:22 | sean-k-mooney | and all compute services are using the same compute service version | |
| 12:25:51 | NewBruce | correct | |
| 12:27:28 | kashyap | lyarwood: sean-k-mooney: I forgot to mention: the key enrollment tool is already part of EDK2 package (at least in Fedora), as a sub-RPM, look for: 'edk2-qosb' | |
| 12:27:33 | sean-k-mooney | honestly i dont know either. cold miragtion should work as a last resort but i dont quite understand why this would happen | |
| 12:27:35 | kashyap | (Updating the spec) | |
| 12:28:07 | sean-k-mooney | ok yes that is good to add to the reference section | |
| 12:28:18 | kashyap | Yes, adding, actually | |
| 12:28:32 | kashyap | I totally forgot that we did it some 8 months ago. My memory is like goldfish's | |
| 12:28:46 | sean-k-mooney | it will be useful for project like kolla that need to figureout what to install | |
| 12:48:42 | openstackgerrit | Merged openstack/nova master: conf: Undeprecate and move the 'dhcp_domain' option https://review.openstack.org/480616 | |
| 12:49:00 | stephenfin | sean-k-mooney: We're going to have to make a call on https://review.openstack.org/#/c/555081/23/specs/train/approved/cpu-resources.rst@180 | |
| 12:49:21 | bauzas | stephenfin: I still need to provide my comments | |
| 12:49:41 | stephenfin | sean-k-mooney: On one hand, I totally get where you're coming from and agree that it's a valid concern | |
| 12:50:01 | sean-k-mooney | yes that is one of the bits im most uncomfortable with. | |
| 12:50:01 | stephenfin | On the other though, if we don't use 'vcpu_pin_set' to populate this stuff, the user will be left in a state where they can no longer boot any instances with 'hw:cpu_policy' configured until they do additional configuration on each host | |
| 12:50:20 | stephenfin | That's as much a breaking change as anything else we've discussed | |
| 12:51:19 | stephenfin | bauzas: Ack. Just focussing on that one point atm | |
| 12:51:32 | sean-k-mooney | yes its not simple. if we use the totally new config options i suggest we could have tehm pre set them and possible provide an upstread check in nova status? | |
| 12:52:07 | sean-k-mooney | vcpu_pin_set was never intended to be "pinned cpus" alther it was assumed to be that in some cases | |
| 12:52:23 | sean-k-mooney | so i dont know what the right way forward is | |
| 12:52:28 | kashyap | lyarwood: Okay, aside: the 'nvram' stanza in /etc/libvirt/qemu.conf will be depreacted. So we can ignore that | |
| 12:52:32 | stephenfin | Hmm, neither do I :\ | |
| 12:53:18 | stephenfin | I'm also thinking we need to kill 'hw:emulator_threads_policy=isolate' | |
| 12:53:49 | sean-k-mooney | i kind of feel we should require the dedicated_cpu_set to be in the config before enableing any of this | |
| 12:54:03 | sean-k-mooney | stephenfin: ya that i was original not sure about but i think i agree | |
| 12:54:20 | stephenfin | Because I can't think of anyway to account for the extra core without mangling the request | |
| 12:54:32 | sean-k-mooney | hw:emulator_threads_policy=share shoudl be sufficent and more efficent | |
| 12:54:50 | sean-k-mooney | well the acounting is trivial | |
| 12:54:53 | stephenfin | I really, really wish we'd overloaded 'isolate' instead of 'share' to do this offload to 'shared_cpu_set' | |
| 12:55:25 | stephenfin | because I don't see a reason why anyone would want to use 'isolate' as it's implemented | |
| 12:55:35 | sean-k-mooney | we already generate teh resouce:vcpu part fo teh placement request for the flavor.vcpu and add 1 core to it for isolate | |
| 12:55:44 | sean-k-mooney | we can easilly just add a pcpu instead | |
| 12:56:19 | sean-k-mooney | well personally i would like to kill the option entirly. | |
| 12:56:22 | stephenfin | True, but rewriting the request like that feels rotten | |
| 12:56:25 | stephenfin | So would I | |
| 12:56:40 | sean-k-mooney | my preference would be make the emulator pinning work for floating cpus too | |
| 12:56:54 | sean-k-mooney | and have a seperate pin set for it | |
| 12:57:00 | stephenfin | What would the alternative be? Always offload emulator threads if 'cpu_shared_set' is defined? | |
| 12:57:01 | stephenfin | Why? | |
| 12:57:13 | stephenfin | The reason you're offloading this stuff is performance, no? | |
| 12:57:30 | stephenfin | More specifically, real-time performance | |
| 12:57:32 | stephenfin | If you're using floating CPUs, you've already given up on that | |
| 12:57:35 | sean-k-mooney | yes but perfroamce still maters for non pinned instance | |
| 12:57:48 | sean-k-mooney | the main reason however is to simplfy the code | |
| 12:57:50 | sean-k-mooney | and the config | |
| 12:58:20 | sean-k-mooney | basicaly if you define emulator_pin_set in the libvirt section we will always use it | |
| 12:58:43 | sean-k-mooney | for all vms and if not then it overlap with the vm cores | |
| 12:58:59 | stephenfin | I was thinking a similar thing but only for instances with PCPUs and only if 'cpu_shared_set' is defined | |
| 12:59:13 | stephenfin | (As opposed to all instances and only if 'emulator_pin_set' is defined) | |
| 12:59:15 | sean-k-mooney | in either case if emulator_pin_set is not overlapping with the dedicated and floating pin sets then you dont need any accounting in placement | |
| 12:59:45 | sean-k-mooney | i would do it for all instance as i would liek to try and share more code between pinned and floating instance | |
| 13:00:24 | stephenfin | that makes sense | |
| 13:00:32 | sean-k-mooney | like we do with numa i would like to soft pin all vms to a floating_pin_set so you can properly reserve cores on the host for OS or vswith use | |
| 13:00:55 | stephenfin | Well, I think we'll be doing that going forward regardless | |
| 13:00:59 | sean-k-mooney | we need to do that anyway when we mix pinned and floating vm on the same host | |
| 13:01:03 | stephenfin | Yeah | |
| 13:01:04 | sean-k-mooney | yep | |
| 13:01:16 | stephenfin | Adding another configuration option would be yet another breaking change though | |
| 13:01:48 | stephenfin | So many breaking changes | |
| 13:03:16 | sean-k-mooney | it does not need to be | |
| 13:03:18 | sean-k-mooney | https://review.openstack.org/#/c/555081/23/specs/train/approved/cpu-resources.rst@157 | |
| 13:03:35 | sean-k-mooney | [compute] | |
| 13:03:37 | sean-k-mooney | pinned_cpu_set | |
| 13:03:39 | sean-k-mooney | floating_cpu_set | |
| 13:03:41 | sean-k-mooney | [libvirt] | |
| 13:03:43 | sean-k-mooney | emulator_cpu_set | |
| 13:04:11 | sean-k-mooney | i suggested deprecating cpu_shared_set and renaming/moving it to [libvirt]/emulator_cpu_set | |
| 13:04:46 | sean-k-mooney | so it will not need config change initally althong they will get a deprecation warning | |
| 13:06:06 | stephenfin | Hmm, I need to think about this | |
| 13:07:44 | NewBruce | sean-k-mooney / mnaser … we seem to actually have made this worse - RDO - RDO which was working previously is now also broken | |
| 13:08:21 | sean-k-mooney | is it broken the same way? | |
| 13:08:39 | NewBruce | seems to be | |
| 13:09:06 | NewBruce | same 500 internal error; same duplicate port entries in ml2_port_bindings | |
| 13:09:08 | sean-k-mooney | and all RDO nodes have the same code deployed and config | |
| 13:10:45 | sean-k-mooney | the is begining to feel like it might be related to the specific configuration of some of the compute nodes rather than a gloabl configuration issue but that is a feeling rather then anything based on fact | |
| 13:11:27 | NewBruce | not identical (due to the time span taken to update), we have mostly 18.2.0, but some 18.1.0 and an 18.0.3 | |
| 13:11:47 | NewBruce | we did try rolling an 18.2.0 back to 18.1.0 - but that didn’t seem to help | |
| 13:12:25 | sean-k-mooney | can you try migration between two nodes of the same version on the rdo side for each of the 3 versions you have deployed | |
| 13:12:28 | NewBruce | yeah, i agree its starting to feel something odd (random)… but as you say, feeling - not fact | |
| 13:12:35 | NewBruce | Yeah, will do now | |