| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-12 | |||
| 13:30:59 | sean-k-mooney | https://github.com/openstack/oslo.context/blob/0d02866365bc8b779aef9ebd0b79a52c96ae40e5/oslo_context/context.py#L508-L513 | |
| 13:31:08 | sean-k-mooney | so it can retrun None | |
| 13:31:34 | sean-k-mooney | _request_store is a thread local | |
| 13:31:36 | sean-k-mooney | https://github.com/openstack/oslo.context/blob/0d02866365bc8b779aef9ebd0b79a52c96ae40e5/oslo_context/context.py#L40 | |
| 13:32:10 | sean-k-mooney | so if the green thread resoumes on a different thread becaue we are usign a thread pool it might be empty | |
| 13:32:36 | lyarwood | kk makes sense, both connect and disconnect wait behind a lock | |
| 13:32:45 | lyarwood | so I assume that's where we end up losing it | |
| 13:32:49 | lyarwood | ew | |
| 13:33:07 | sean-k-mooney | do i want to know what you just saw | |
| 13:33:37 | lyarwood | that was more at the logging changing because of thread local storage | |
| 13:34:03 | lyarwood | I'm looking at yet another example failure of the detach thing | |
| 13:34:11 | lyarwood | another test attaches a volume | |
| 13:34:19 | lyarwood | the failing test attaches a volume | |
| 13:34:23 | lyarwood | the other test detaches | |
| 13:34:31 | lyarwood | and then our failing test attempts to detach | |
| 13:34:45 | sean-k-mooney | thats also ew :) | |
| 13:35:08 | lyarwood | well it's all locked and serialised so that's fine | |
| 13:35:18 | lyarwood | but the failing test just times out trying to detach the device | |
| 13:35:19 | sean-k-mooney | can we just chnage this locally in nova to workaround the issue | |
| 13:35:22 | lyarwood | there's some stuff in syslog | |
| 13:35:33 | lyarwood | but I can't get my head around it tbh | |
| 13:36:16 | sean-k-mooney | you still think this is due to conflicting WWNs right | |
| 13:37:14 | lyarwood | yeah I don't think that's helping but I can't find a smoking gun here | |
| 13:47:17 | belmoreira | hi, we are live migrating thousands of instances because hardware retirement and hit an old issue :) this was discussed in the past for instance creation, but now I'm facing it at live migration, when the migration allocation is created. https://bugs.launchpad.net/nova/+bug/1918419 | |
| 13:47:19 | openstack | Launchpad bug 1918419 in OpenStack Compute (nova) "vCPU resource max_unit is hardcoded" [Undecided,New] | |
| 13:48:48 | sean-k-mooney | belmoreira: it would be incorrect to factor in allocation ration when setting max_unit | |
| 13:48:49 | belmoreira | I feel that for the vcpu particular case the operator could take the responsibility to defined the max_unit value. | |
| 13:49:27 | sean-k-mooney | i dont think i agree | |
| 13:49:40 | sean-k-mooney | you might be abel to use provider.yaml to cahnge it | |
| 13:49:53 | sean-k-mooney | but its not valid for a vm to oversubsribe against itself | |
| 13:50:06 | sean-k-mooney | so the max_unit should never exceed total | |
| 13:50:36 | sean-k-mooney | if you are disabling hyperthread thats havlfing your total cpus so the max_unit should also be reduced | |
| 13:50:48 | belmoreira | "its not valid for a vm to oversubsribe against itself" I agree with that | |
| 13:51:12 | sean-k-mooney | max_unit in placment is there to prevent that | |
| 13:51:42 | sean-k-mooney | belmoreira: the correct approch in your case would be to resize the vms before the migration | |
| 13:51:53 | sean-k-mooney | form the 32 core flavor to 16 i assume | |
| 13:52:03 | sean-k-mooney | or 32 but across 2 numa nodes | |
| 13:52:11 | sean-k-mooney | likely depending on the vm | |
| 13:52:50 | belmoreira | of course in the general I agree with you. And it's a good default. What I'm talking about is to give the operator the freedom to change this sensible default for very particular operations. | |
| 13:53:08 | sean-k-mooney | that the thing its all or nothing | |
| 13:53:23 | sean-k-mooney | if you change the max_unit in the placment inventory | |
| 13:53:29 | sean-k-mooney | it will apply to all operations | |
| 13:53:41 | sean-k-mooney | which could allow vms to boot that were over subscibing againt themselves | |
| 13:54:47 | sean-k-mooney | we dont suport setting max_unit for indivigual resouce in the allocation candiates request | |
| 13:55:33 | sean-k-mooney | belmoreira: if you realy need too you can use https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/provider-config-file.html | |
| 13:55:43 | sean-k-mooney | to orverride the max_unit | |
| 13:55:58 | sean-k-mooney | assuming you have at least ussuri | |
| 13:57:27 | belmoreira | I'm not familiar with that spec. Let me have a look. (still in Stein for Nova) | |
| 13:57:31 | sean-k-mooney | you would basically do | |
| 13:57:33 | sean-k-mooney | meta: | |
| 13:57:35 | sean-k-mooney | schema_version: 1.0 | |
| 13:57:37 | sean-k-mooney | providers: | |
| 13:57:39 | sean-k-mooney | # List of dicts | |
| 13:57:41 | sean-k-mooney | - identification: | |
| 13:57:43 | sean-k-mooney | uuid: $COMPUTE_NODE | |
| 13:57:45 | sean-k-mooney | inventories: | |
| 13:57:47 | sean-k-mooney | additional: | |
| 13:57:49 | sean-k-mooney | VCPU: | |
| 13:57:51 | sean-k-mooney | max_unit: 32 | |
| 13:59:04 | belmoreira | If we can do that in Ussuri it looks good enough to me for any special case | |
| 13:59:23 | belmoreira | thanks sean-k-mooney | |
| 13:59:25 | sean-k-mooney | actully it wont work for your usecase | |
| 13:59:41 | sean-k-mooney | i forgot we can only use this for CUSTOM_ resouces | |
| 14:00:09 | sean-k-mooney | we intentionally blocked overwriding sthe standard ones form the virt driver | |
| 14:05:25 | belmoreira | my thinking on all of this is how an operator should move forward when (for whatever reason) the resource inventory changes. | |
| 14:06:12 | sean-k-mooney | well form a downstream and i thnk upstream point of vew we do not supprot changing the number of hyper treads on a host with vms | |
| 14:06:13 | belmoreira | For my particular case, vCPUs were cut in half, but we need to keep the oversubcribed instances running. Now to live migrate then we need "patch" the compute nodes | |
| 14:06:40 | sean-k-mooney | the same aslo gos for numa nodes e.g. by enable cluster on die ot changeing numa per socket in the bios | |
| 14:07:00 | sean-k-mooney | so today the only supported way it to do a resize to a differnt flavor | |
| 14:07:46 | sean-k-mooney | belmoreira: yep i understand unfortuetly the senario your are attempting to do is not currenlty supported by nova | |
| 14:07:52 | belmoreira | sean-k-mooney :) fair enough. I didn't think we would ever disable SMT in production nodes. | |
| 14:08:34 | sean-k-mooney | the "keep vms alive requirement is really the toughest" | |
| 14:09:16 | sean-k-mooney | belmoreira: for live migration we cant really change the toplogy of the guest | |
| 14:10:07 | sean-k-mooney | we can change teh mapping the host but putting that 32 core vm on a 16 core host would not be a valid schudlign decision based on our current oversubscpiton rules | |
| 14:10:33 | sean-k-mooney | belmoreira: actully i hate to say this but did you attempt a force live migration | |
| 14:10:57 | sean-k-mooney | i assume that still fails because placment will block it | |
| 14:11:11 | sean-k-mooney | when we try to update the allcoations | |
| 14:11:21 | belmoreira | sean-k-mooney it's fine. And in most part I agree with you, I just wanted to raise this issue here because other operators may suffer from the same thing. And these are the "use cases" that we didn't think about... Like I said I never thought we would need to disable SMT in production nodes. | |
| 14:12:01 | sean-k-mooney | it is certenly something we could put in the docs somewhwere | |
| 14:12:09 | belmoreira | sean-k-mooney actually I didn't... but in the case is placement, so it should fail | |
| 14:12:55 | sean-k-mooney | i think the allocation update will fail but i have never tried this | |
| 14:13:16 | sean-k-mooney | nova with the old microversion will skip the schduler fileter if you force it | |
| 14:13:27 | sean-k-mooney | but i think we always do the placment update | |
| 14:14:19 | sean-k-mooney | belmoreira: if you want to hack around it without changing the code there is one thing you could try | |
| 14:14:49 | sean-k-mooney | you could increase the interval taht we update placment at in the config on a node temporally so say once an hour | |
| 14:15:09 | sean-k-mooney | and you could manully chagne the value in the placment inventory with osc-placment | |
| 14:15:18 | sean-k-mooney | then migrate | |
| 14:15:48 | sean-k-mooney | that will end up with an invalidly placed vm but you could force it that way | |
| 14:17:10 | belmoreira | sean-k-mooney yeah, that would work. What i'm doing is just putting a fake value for max_int in the update placement. These nodes will be removed anyway. | |
| 14:17:52 | sean-k-mooney | ya its basiclaly the same just via the api for those that cant hack the code direcly in prodcution | |
| 14:18:33 | belmoreira | sean-k-mooney thanks a lot for your comments. | |
| 14:18:49 | belmoreira | Having something in the docs may help others. I think I can move this bug forward and update the docs. | |
| 14:19:39 | sean-k-mooney | i hope you dont mind that i marked it as invalid but you could bring it up at the ptg | |
| 14:19:59 | sean-k-mooney | or in a nova team meeting/mailing list to get more input form others | |
| 14:20:53 | belmoreira | sean-k-mooney I think having something in the docs is reasonable | |
| 14:21:21 | sean-k-mooney | yep i agree if you wanted to convert that to a docs bug i think it would be good | |
| 14:24:08 | belmoreira | I need some guidance... should I mentioned in the bug that we discussed this and we agreed that a docs change could be enough? or this bug needs to be submitted somewhere else? | |
| 14:24:33 | sean-k-mooney | you can link to the irc convo | |
| 14:25:04 | sean-k-mooney | http://eavesdrop.openstack.org/irclogs/%23openstack-nova/latest.log.html#t2021-03-12T13:47:17 | |
| 14:25:34 | sean-k-mooney | then we can triage it as valid | |