Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-11-19
16:37:50 opendevreview Merged openstack/nova stable/2024.2: Route shared storage RPC to evac dest at startup https://review.opendev.org/c/openstack/nova/+/934302
16:37:52 sean-k-mooney there is an older one on the nova side somewhere
16:38:18 sean-k-mooney related to amd cpus? and numa?
16:38:23 gibi luca_: sean-k-mooney: yeah that bug seems to report the same thing
16:38:56 luca_ i see, by any chanche using the old PciPasstroughFilter in the scheduler instead of using placement will solve this issue?
16:39:11 sean-k-mooney it will work around it yes
16:39:26 sean-k-mooney at the cost of the inablity to have quota
16:39:36 sean-k-mooney to do do some more advanced schduilgn
16:39:57 sean-k-mooney if you disable pci in placmenement then it will remove the nested resouce providers
16:40:09 gibi this is GPU so GPU is using placement without the PCI in Placement feature enabled
16:40:10 sean-k-mooney and the allcoation candiate epxlotion will be mitigated
16:40:19 gibi so this logic cannot be turned on I think
16:40:29 gibi turned off
16:40:34 sean-k-mooney is it gpu or vgpu
16:40:40 luca_ GPU
16:40:42 sean-k-mooney gpu passthough is just pci passthough
16:40:52 luca_ yeah i am passing pci-devices
16:41:01 tkajinam gibi, sorry for asking a favor again but do you mind adding +a, too ?
16:41:09 gibi luca_: did you enabled pci_in_placement ?
16:41:13 luca_ yes
16:41:17 gibi luca_: ahh I see.
16:41:18 tkajinam I wonder if adding an option to limit maximum candidates to placement resolves the issue ?
16:41:29 gibi luca_: then yes, turning off pci_in_placement will help
16:41:29 luca_ wwhich i read is not possible to disable after enabling it :D
16:41:34 tkajinam (I don't have full context, though)
16:41:39 sean-k-mooney luca_: this is the main reason that is tech preview downstream for what its worth
16:42:00 sean-k-mooney luca_: kind of not with workload at least
16:42:01 luca_ iyeah no problem this is a testing enviroment we can rebuild it if it is needed
16:42:09 gibi luca_: not out of the box, you probably need to hack around I'm affraid
16:42:16 gibi tkajinam: done, sorry I missed it
16:42:24 tkajinam gibi, np and thanks again !
16:42:31 sean-k-mooney luca_: you can do a few things
16:42:41 gibi luca_: wondering about the limit tkajinam suggests
16:43:02 sean-k-mooney gibi: that kind of works but not well
16:43:22 tkajinam one downside of that limit is that it may not work well with randomize_allocation_candidates
16:43:23 sean-k-mooney if you reduce the limit you will likely only get one or two chosts
16:43:44 tkajinam yeah
16:43:49 sean-k-mooney tkajinam: i think by defualt we generate all the permutiaon on a per host basis
16:44:06 sean-k-mooney we have a downstream issue to optimise this in the future where we woudl do a barbar pole stirp
16:44:20 gibi luca_: this is the limit config https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_placement_results but maybe it limits after the candidates are generated :/
16:44:26 sean-k-mooney i.e. take the first candate for each host until we get to the limit in a round robbing fashion
16:45:01 sean-k-mooney the default limti is 1000 canidates i belive
16:45:15 tkajinam gibi, it is used to determine the limit query parameter in HTTP Request
16:45:27 tkajinam so the limit may be applied in placement (if I read the code correctly)
16:45:45 gibi tkajinam: yeah applied in placement but maybe after all the permutation is generated :/
16:45:49 tkajinam yeah
16:46:56 sean-k-mooney luca_: so what i would do is the following if this is a test env, deleate any insances usign pci devices, remove all pci devices form the pci devspec, restart the comptue agents and then disble the config oprtion for pci in placmeent after they have remove the nested resouce providers
16:47:04 sean-k-mooney then add the pci devspecs back
16:47:10 gibi tkajinam: I don't see the limit applied directly in the permutation generaton in https://github.com/openstack/placement/blob/723da65faf66cc9b8d02f3756387dc58437e62af/placement/objects/allocation_candidate.py#L371
16:47:11 sean-k-mooney and try again
16:47:46 luca_ since i am asking, is it normale that a single resource provider is generated for each GPU?
16:48:10 sean-k-mooney gibi i think its later
16:48:11 luca_ sean-k-mooney_ thanks for the possible solution i will try it
16:48:23 sean-k-mooney but to fix this properly we need to adjust the nestign of those loops
16:48:26 ishanwar Hi Team,
16:49:12 tkajinam gibi, yeah, I see limit_results applies the limit but it is applied to the full list...
16:49:14 sean-k-mooney luca_: yes becasue we create 1 RP per PF
16:49:24 sean-k-mooney ishanwar: o/
16:49:49 sean-k-mooney tkajinam: gibi we basically need to make the permuation generateion lazy using generators
16:49:49 gibi luca_: as you can assign different traits to different PFs via the dev_spec we need to model them separately
16:50:15 tkajinam sean-k-mooney, yeah I totally agree
16:50:25 sean-k-mooney ya that or use diffent resouce classes, physnets ectra
16:50:27 luca_ i see, does this mean that if the device type is type-PCI instead of type-PF you just create 1 rp per kind?
16:50:28 ishanwar I currently facing some issue with the dev stack setup for nova.
16:51:00 gibi luca_: we create one RP for each device basically where the name of the PR contains the PCI address of the device
16:51:03 sean-k-mooney luca_: no if tis type-PF or type-PCI we create on resouce provider per phasical device
16:51:25 luca_ oh ok
16:51:31 sean-k-mooney luca_: for type-VF they are all on a single RP based on the parent PF
16:51:37 gibi https://specs.openstack.org/openstack/nova-specs/specs/zed/approved/pci-device-tracking-in-placement.html#modeling-pci-devices-in-placement
16:51:41 gibi details are here ^^
16:54:17 ishanwar ./stack.sh gets stuck here. LOGS: +functions:wait_for_compute:504 [[ 124 != 0 ]] +functions:wait_for_compute:505 echo 'Didn'\''t find service registered by hostname after 60 seconds' Didn't find service registered by hostname after 60 seconds +functions:wait_for_compute:506 openstack --os-cloud devstack-admin --os-region RegionOne compute service list
16:54:22 gibi luca_: that remainds me that you can trick nova to allow disabling pci_in_placement by stoping nova-compute and removing COMPUTE_MANAGED_PCI_DEVICE traits from the rsource providers
16:55:01 sean-k-mooney ishanwar: that typically means that there was an error that shoudl be visable in "journalctl -u devstack@n-cpu"
16:55:07 luca_ i see, tomorrow we wil try to disable pci in plcaement as sean-k-mooney_ suggested, hopefully it will work, thanks for the help guys!
16:55:29 gibi luca_: keep us posted.
16:55:35 gibi and good luck
16:55:39 luca_ sure, will let you know! thanks!
16:56:33 ishanwar sean-k-mooney: let me check. Can I ping you directly ? The docs seems to lack troubleshooting
16:57:05 sean-k-mooney ishanwar: so devstack pings are proably better in #openstack-qa
16:57:50 opendevreview Stephen Finucane proposed openstack/nova master: Fix typo in release note https://review.opendev.org/c/openstack/nova/+/935681
16:57:50 opendevreview Stephen Finucane proposed openstack/nova master: zuul: Add missing context comments for nova-next https://review.opendev.org/c/openstack/nova/+/935682
16:57:54 sean-k-mooney ishanwar: devstack is written in bash so that its easy to read and debug. it does require some knolage of openstack but for the most part it ties ot keep the code readable and simple
16:59:25 bauzas well, devstack is clearly intended for dev purposes
16:59:26 ishanwar sean-k-mooney: I see. thanks. another question. Are ppl active on #openstack-* channels only during the meetings, or are there people active during other times as well ?
16:59:41 bauzas people are expected to have a bit of knowledge about the services already
16:59:58 sean-k-mooney ishanwar: we generally hang out here most of the time
17:00:03 bauzas and that error self-explains : the nova-compute service didn't show up in time
17:00:17 sean-k-mooney most of use will hae irc open in a window while working on other stuff and check in perodically
17:00:25 bauzas which means that something prevented that service to register itself
17:00:45 bauzas and as a reminder, this channel is not intended for debugging purposes
17:02:47 ishanwar bauzas: Thanks I see. I have some knowledge of nova only from the operational side. I am still figuring out the openstack community, as I am used to slack channels and zoom meetings for opensource in general. Will not use this channel for debugging purposes. Thanks
17:04:32 bauzas tbc we're here to help, for sure
17:05:20 bauzas so, for example, some operators discuss with us their pain points
17:05:39 bauzas sometimes, someone is stuck installing a devstack env and we try to also give him some help
17:06:07 bauzas but here, you should first look at nova-compute logs and see, most of the time, this is a config issue :)
17:06:39 bauzas ishanwar : HTH : https://docs.openstack.org/devstack/latest/systemd.html
17:55:23 opendevreview Merged openstack/nova master: Drop remaining logic for websockify < 0.9.0 https://review.opendev.org/c/openstack/nova/+/934517
17:55:31 opendevreview Merged openstack/nova master: Replace deprecated constant_time_compare https://review.opendev.org/c/openstack/nova/+/931146
18:45:41 opendevreview Antony Messerli proposed openstack/nova master: docs: Corrects a typo on a placement audit command https://review.opendev.org/c/openstack/nova/+/922850
19:01:44 yuriy Please correct me if I'm wrong; In a scenario where an LVM ephemeral based instance needs more ram... there doesn't seem to be a way to do that other than remake the entire VM with a flavor that has more ram. Resizing doesn't appear to be supported for LVM based instances.

Earlier   Later