Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2024-11-19
16:36:58 opendevreview ribaudr proposed openstack/nova master: Support rescuing an instance with shares https://review.opendev.org/c/openstack/nova/+/860287
16:36:58 opendevreview ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862
16:37:00 opendevreview ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751
16:37:00 opendevreview ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830
16:37:02 opendevreview ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753
16:37:02 opendevreview ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501
16:37:04 opendevreview ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028
16:37:04 opendevreview ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029
16:37:06 opendevreview ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282
16:37:06 opendevreview ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283
16:37:08 opendevreview ribaudr proposed openstack/nova master: Reports instance events to the DB regarding attaching and detaching a share https://review.opendev.org/c/openstack/nova/+/927088
16:37:08 opendevreview ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086
16:37:10 opendevreview ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087
16:37:10 opendevreview ribaudr proposed openstack/nova master: Manila shares admin guide documentation https://review.opendev.org/c/openstack/nova/+/871642
16:37:18 bauzas sean-k-mooney: ah I was doing too many things at once, I just briefly looked and saw GPU passthrough"
16:37:43 sean-k-mooney luca_: gibi: i tought we had a diffent one but https://bugs.launchpad.net/placement/+bug/2070257 is the same porblem
16:37:50 opendevreview Merged openstack/nova stable/2024.2: Route shared storage RPC to evac dest at startup https://review.opendev.org/c/openstack/nova/+/934302
16:37:52 sean-k-mooney there is an older one on the nova side somewhere
16:38:18 sean-k-mooney related to amd cpus? and numa?
16:38:23 gibi luca_: sean-k-mooney: yeah that bug seems to report the same thing
16:38:56 luca_ i see, by any chanche using the old PciPasstroughFilter in the scheduler instead of using placement will solve this issue?
16:39:11 sean-k-mooney it will work around it yes
16:39:26 sean-k-mooney at the cost of the inablity to have quota
16:39:36 sean-k-mooney to do do some more advanced schduilgn
16:39:57 sean-k-mooney if you disable pci in placmenement then it will remove the nested resouce providers
16:40:09 gibi this is GPU so GPU is using placement without the PCI in Placement feature enabled
16:40:10 sean-k-mooney and the allcoation candiate epxlotion will be mitigated
16:40:19 gibi so this logic cannot be turned on I think
16:40:29 gibi turned off
16:40:34 sean-k-mooney is it gpu or vgpu
16:40:40 luca_ GPU
16:40:42 sean-k-mooney gpu passthough is just pci passthough
16:40:52 luca_ yeah i am passing pci-devices
16:41:01 tkajinam gibi, sorry for asking a favor again but do you mind adding +a, too ?
16:41:09 gibi luca_: did you enabled pci_in_placement ?
16:41:13 luca_ yes
16:41:17 gibi luca_: ahh I see.
16:41:18 tkajinam I wonder if adding an option to limit maximum candidates to placement resolves the issue ?
16:41:29 gibi luca_: then yes, turning off pci_in_placement will help
16:41:29 luca_ wwhich i read is not possible to disable after enabling it :D
16:41:34 tkajinam (I don't have full context, though)
16:41:39 sean-k-mooney luca_: this is the main reason that is tech preview downstream for what its worth
16:42:00 sean-k-mooney luca_: kind of not with workload at least
16:42:01 luca_ iyeah no problem this is a testing enviroment we can rebuild it if it is needed
16:42:09 gibi luca_: not out of the box, you probably need to hack around I'm affraid
16:42:16 gibi tkajinam: done, sorry I missed it
16:42:24 tkajinam gibi, np and thanks again !
16:42:31 sean-k-mooney luca_: you can do a few things
16:42:41 gibi luca_: wondering about the limit tkajinam suggests
16:43:02 sean-k-mooney gibi: that kind of works but not well
16:43:22 tkajinam one downside of that limit is that it may not work well with randomize_allocation_candidates
16:43:23 sean-k-mooney if you reduce the limit you will likely only get one or two chosts
16:43:44 tkajinam yeah
16:43:49 sean-k-mooney tkajinam: i think by defualt we generate all the permutiaon on a per host basis
16:44:06 sean-k-mooney we have a downstream issue to optimise this in the future where we woudl do a barbar pole stirp
16:44:20 gibi luca_: this is the limit config https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.max_placement_results but maybe it limits after the candidates are generated :/
16:44:26 sean-k-mooney i.e. take the first candate for each host until we get to the limit in a round robbing fashion
16:45:01 sean-k-mooney the default limti is 1000 canidates i belive
16:45:15 tkajinam gibi, it is used to determine the limit query parameter in HTTP Request
16:45:27 tkajinam so the limit may be applied in placement (if I read the code correctly)
16:45:45 gibi tkajinam: yeah applied in placement but maybe after all the permutation is generated :/
16:45:49 tkajinam yeah
16:46:56 sean-k-mooney luca_: so what i would do is the following if this is a test env, deleate any insances usign pci devices, remove all pci devices form the pci devspec, restart the comptue agents and then disble the config oprtion for pci in placmeent after they have remove the nested resouce providers
16:47:04 sean-k-mooney then add the pci devspecs back
16:47:10 gibi tkajinam: I don't see the limit applied directly in the permutation generaton in https://github.com/openstack/placement/blob/723da65faf66cc9b8d02f3756387dc58437e62af/placement/objects/allocation_candidate.py#L371
16:47:11 sean-k-mooney and try again
16:47:46 luca_ since i am asking, is it normale that a single resource provider is generated for each GPU?
16:48:10 sean-k-mooney gibi i think its later
16:48:11 luca_ sean-k-mooney_ thanks for the possible solution i will try it
16:48:23 sean-k-mooney but to fix this properly we need to adjust the nestign of those loops
16:48:26 ishanwar Hi Team,
16:49:12 tkajinam gibi, yeah, I see limit_results applies the limit but it is applied to the full list...
16:49:14 sean-k-mooney luca_: yes becasue we create 1 RP per PF
16:49:24 sean-k-mooney ishanwar: o/
16:49:49 sean-k-mooney tkajinam: gibi we basically need to make the permuation generateion lazy using generators
16:49:49 gibi luca_: as you can assign different traits to different PFs via the dev_spec we need to model them separately
16:50:15 tkajinam sean-k-mooney, yeah I totally agree
16:50:25 sean-k-mooney ya that or use diffent resouce classes, physnets ectra
16:50:27 luca_ i see, does this mean that if the device type is type-PCI instead of type-PF you just create 1 rp per kind?
16:50:28 ishanwar I currently facing some issue with the dev stack setup for nova.
16:51:00 gibi luca_: we create one RP for each device basically where the name of the PR contains the PCI address of the device
16:51:03 sean-k-mooney luca_: no if tis type-PF or type-PCI we create on resouce provider per phasical device
16:51:25 luca_ oh ok
16:51:31 sean-k-mooney luca_: for type-VF they are all on a single RP based on the parent PF
16:51:37 gibi https://specs.openstack.org/openstack/nova-specs/specs/zed/approved/pci-device-tracking-in-placement.html#modeling-pci-devices-in-placement
16:51:41 gibi details are here ^^
16:54:17 ishanwar ./stack.sh gets stuck here. LOGS: +functions:wait_for_compute:504 [[ 124 != 0 ]] +functions:wait_for_compute:505 echo 'Didn'\''t find service registered by hostname after 60 seconds' Didn't find service registered by hostname after 60 seconds +functions:wait_for_compute:506 openstack --os-cloud devstack-admin --os-region RegionOne compute service list
16:54:22 gibi luca_: that remainds me that you can trick nova to allow disabling pci_in_placement by stoping nova-compute and removing COMPUTE_MANAGED_PCI_DEVICE traits from the rsource providers
16:55:01 sean-k-mooney ishanwar: that typically means that there was an error that shoudl be visable in "journalctl -u devstack@n-cpu"
16:55:07 luca_ i see, tomorrow we wil try to disable pci in plcaement as sean-k-mooney_ suggested, hopefully it will work, thanks for the help guys!
16:55:29 gibi luca_: keep us posted.
16:55:35 gibi and good luck
16:55:39 luca_ sure, will let you know! thanks!
16:56:33 ishanwar sean-k-mooney: let me check. Can I ping you directly ? The docs seems to lack troubleshooting
16:57:05 sean-k-mooney ishanwar: so devstack pings are proably better in #openstack-qa
16:57:50 opendevreview Stephen Finucane proposed openstack/nova master: Fix typo in release note https://review.opendev.org/c/openstack/nova/+/935681
16:57:50 opendevreview Stephen Finucane proposed openstack/nova master: zuul: Add missing context comments for nova-next https://review.opendev.org/c/openstack/nova/+/935682
16:57:54 sean-k-mooney ishanwar: devstack is written in bash so that its easy to read and debug. it does require some knolage of openstack but for the most part it ties ot keep the code readable and simple
16:59:25 bauzas well, devstack is clearly intended for dev purposes
16:59:26 ishanwar sean-k-mooney: I see. thanks. another question. Are ppl active on #openstack-* channels only during the meetings, or are there people active during other times as well ?

Earlier   Later