Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-02-22
13:18:15 artom It won't be set if there's no emulator thread policy
13:18:36 sean-k-mooney if emulatorpin is not found you need to default it to vcpupin
13:19:08 sean-k-mooney so ya you need an if here https://github.com/openstack/nova/blob/3209f6551652cff7bef0b9d9719ab940dd05a0f8/nova/virt/libvirt/migration.py#L127
13:19:27 sean-k-mooney well
13:19:33 sean-k-mooney actully proably not there
13:19:45 sean-k-mooney but where we buidl teh dst_numa_info in the first place
13:19:59 sean-k-mooney well either works i guess
13:20:39 sean-k-mooney for backport reason we proably want to have _update_numa_xml have the fallback if its not set in dst_numa_info
13:20:52 sean-k-mooney but we should fix where we are creating the numa info to also set it properly
13:22:02 sean-k-mooney so add somethign like
13:23:18 sean-k-mooney emulator_pin = info.emulator_pins if info.emulator_pins else info.cpu_pins.values()
13:23:38 sean-k-mooney emulatorpin.set('cpuset',hardware.format_cpu_spec(emulator_pin))
13:42:44 opendevreview Amit Uniyal proposed openstack/nova master: enforce remote console shutdown https://review.opendev.org/c/openstack/nova/+/901824
14:53:39 opendevreview Pavlo Shchelokovskyy proposed openstack/nova master: Auto set heartbeat_in_pthread for wsgi services https://review.opendev.org/c/openstack/nova/+/909880
15:31:48 mnaser It seems `nova-grenade-multinode` is broken in `stable/zed`. Does anyone know of anything that sticks out
15:32:13 mnaser https://review.opendev.org/c/openstack/nova/+/909098 we're eating up resources with folks rechecking it, we're at 3x, i think the job is deffo broken
15:33:13 mnaser ok, that's neutron being broken
15:33:19 mnaser https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_39b/909098/2/check/nova-grenade-multinode/39bc664/controller/logs/screen-q-svc.txt
15:34:05 frickler iiuc grenade jobs should be dropped from zed now that yoga is unmaintained?
15:43:15 dansmith idk that we actually said that (removing grenade if the source is unmaintained) but that make sense to me
15:43:58 clarkb historically we've dropped grenade jobs for N+1 when N is no longer maintained
15:44:10 clarkb beacuse the grenade jobs start at N and upgrade to N+1 and you very quickly bitrot
15:46:01 dansmith you mean historically...removed right? I meant in response to the recent policy to have branches that are specifically _unmaintained_ but still around
15:46:11 clarkb correct historically
15:47:05 dansmith in the past, IIRC, we'd remove a grenade when it upgrades from a branch that gets deleted because we can't land a fix to make it installable, but it's slightly less clear when the branch exists but is in the newly-minted unmaintained state
15:47:25 opendevreview Artom Lifshitz proposed openstack/nova master: POC: power up dedicated cores during pre_live_migration https://review.opendev.org/c/openstack/nova/+/909806
15:48:05 artom Let's see if the emulator_pins fix is enough
15:53:15 frickler those grenade runs seem also have been in a weird situation where the stable/yoga branches were already removed in the repos, but still present in our in-image-git-cache. but I also don't understand why n-cond seems to get started, but doesn't produce any log
15:57:34 melwitt I dunno if yall have seen but it looks like nova-next is at a 100% fail rate, POST_FAILURE
15:57:41 melwitt Deleting allocation key from the binding:profile of the bandwidth aware port
15:57:41 melwitt + openstack port unset --binding-profile allocation port-normal-qos
15:57:41 melwitt ForbiddenException: 403: Client Error for url: https://213.32.74.123:9696/networking/v2.0/ports/f5b273ab-1c98-44f6-9664-8c1a16d5abda, (rule:update_port and rule:update_port:binding:profile) is disallowed by policy
15:58:36 melwitt not sure what changed that could have affected policy. I'll look around
16:02:18 sean-k-mooney melwitt: i have a fix for it
16:02:40 melwitt ok thank goodness
16:02:43 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/909859
16:02:50 sean-k-mooney i will readd the test coverage later
16:03:17 sean-k-mooney tldr modifying binding_profile now needd a service token
16:03:20 sean-k-mooney hence the 403
16:03:25 melwitt ahh
16:14:39 melwitt sean-k-mooney: looks like there's at least one more thing that should be removed https://review.opendev.org/c/openstack/nova/+/909859/1/gate/post_test_hook.sh#174
16:15:29 sean-k-mooney oh right
16:15:41 sean-k-mooney yep unset woudl also not work
16:16:11 sean-k-mooney im conflicted if we shoudl also remove that coverage
16:16:27 sean-k-mooney or if i shoudl bite the bullet and try and repalce it with a python script
16:18:08 melwitt yeah, I wondered that too. not sure how hard it would be to get something with service token working
16:18:27 sean-k-mooney the my-key stuff is not really that imporant
16:18:36 sean-k-mooney but the unset there is a key part of that test
16:18:52 sean-k-mooney melwitt: well we have nova avialabe in this env
16:19:06 sean-k-mooney so i was hoping we could just import some nova code for hat
16:19:16 sean-k-mooney this is running on the contoler
16:19:27 sean-k-mooney so the nova.conf should have all the setting set correctly
16:19:38 sean-k-mooney so in theory its improt the nvoa.conf module
16:19:40 melwitt yeah, it would
16:19:48 sean-k-mooney create the neutron clinet form our clinet module
16:19:52 sean-k-mooney and do port update
16:19:58 melwitt I should know bc I added it but 😛
16:20:09 melwitt hm k
16:20:16 bauzas JayF: are you around ?
16:20:59 JayF What's up?
16:21:21 bauzas JayF: about https://review.opendev.org/c/openstack/nova/+/903915/ I wonder if you could provide a new revision due to sean-k-mooney's -1 ?
16:21:34 bauzas also, do we now have a Tempest job testing it ?
16:22:24 JayF Sean and I talked in depth about a testing plan earlier this week, and today and tomorrow is the time I set aside to implement it. At which point I'll re-stack all the sharding changes, make sure they pass CI, and do some manual testing
16:23:08 JayF We have implemented significant sharding testing in ironic for ensuring the API works properly, the second step I'm working on today is the scenario tests
16:23:09 sean-k-mooney cool just an fyi tomorrow is a "rechage day" at redhat meaning none of us will be here tomorow
16:23:14 sean-k-mooney but we can take a look on monday
16:23:35 JayF Yeah I think I have all the information I need based on our conversation earlier, I just need to not have my day stolen by a thousand tiny conversations
16:32:37 melwitt sean-k-mooney: the new_websocket_client method is what gets called when a new request to the proxy comes in, so I also dunno why that would need to be called to close a connection ... ?
16:37:17 sean-k-mooney auniyal: can you try https://review.opendev.org/c/openstack/nova/+/901824/18/nova/console/websocketproxy.py#154
16:46:38 auniyal sean-k-mooney, ack thanks
16:48:51 auniyal sean-k-mooney, can we raise an Exception as invalid target token, inside
16:49:00 auniyal reason I am saying is https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L265
16:50:59 sean-k-mooney that is coming from self.do_proxy
16:51:01 sean-k-mooney right
16:51:28 auniyal yes, so there some exception will be raised, like something happend in network or at client
16:52:00 sean-k-mooney so if we think about the reason why that might happen
16:52:22 sean-k-mooney the only reason i can think of is the client already disconnect before the time expired
16:52:43 sean-k-mooney in which case i dont think we should be raising an excption as that is an expected scenairo
16:53:06 sean-k-mooney so im not convinced raising an excption is correct
16:53:14 auniyal ack,
16:53:20 sean-k-mooney a debug level log woudl be fine
16:53:32 sean-k-mooney but do you have any other edge case in mind
16:53:51 auniyal okay, also if something really happend at client side https://review.opendev.org/c/openstack/nova/+/901824/18/nova/console/websocketproxy.py#294
16:54:35 auniyal no, so as you said, everytime we come to close connection we will have valid tsock
16:55:03 auniyal ack, will remove if-else then, thanks
16:56:22 sean-k-mooney you have already protected agasint the socket being cloased with the OSError and the fileno being -1
16:56:37 sean-k-mooney so i htink you have handled all the edgecase that are required there
16:59:06 auniyal ack, I'll just run tests locally and respin
16:59:55 opendevreview sean mooney proposed openstack/nova master: [S-RBAC] adapt nova-next for port's binding:profile field change https://review.opendev.org/c/openstack/nova/+/909859
17:06:43 sean-k-mooney melwitt: by the way using blockcopy instead of blockrebase is a nice find
17:07:04 sean-k-mooney melwitt: it remiened me of something for future us
17:07:31 sean-k-mooney we currenly have some legacy code in hwo we do live snapshots to workaround some bugs with older version fo qemu/libvirt
17:07:42 sean-k-mooney my understanding is those issues have been fixed
17:07:57 sean-k-mooney so we may be able to simply that code alot in the future
17:08:08 sean-k-mooney by removing the legacy workaround
17:08:16 melwitt sean-k-mooney: yeah, I couldn't stop thinking about live snapshot not working so I asked in #virt and they advised to use blockcopy in order to provide encryption options
17:08:39 melwitt yeah, I think we likely could
17:09:23 sean-k-mooney we do this complciated dance where we start a rebase or copy jobs afte we copy a base iamge then abort it to do the live snapshot
17:09:55 sean-k-mooney i.e. we copy the backing file, start a jobs to sysnc the chagnes, briefly freese the guest file system and then abort
17:10:30 sean-k-mooney this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3412

Earlier   Later