| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-09-13 | |||
| 18:45:44 | dansmith | er cool even | |
| 18:54:02 | gmann | dansmith: you doing so much work and on gate stability makes you always better and not bad :) | |
| 18:54:24 | dansmith | heh | |
| 19:03:44 | bauzas | dansmith: fwiw, also seeing more kernel crashes like https://bb94d2825af897cfcd12-f6f1806a4829a343b7540be166a34ea9.ssl.cf5.rackcdn.com/860829/4/check/nova-next/bf91602/testr_results.html | |
| 19:04:05 | bauzas | again an already known issue but more failures | |
| 19:04:53 | dansmith | bauzas: always with no space left on device? | |
| 19:05:08 | bauzas | don't think so | |
| 19:05:18 | bauzas | I have to doublecheck tho | |
| 19:05:24 | dansmith | that's during boot, so it looks more like maybe a broken image creation or if it's on ceph, maybe a backend problem | |
| 19:06:05 | dansmith | oh volume boot, so yeah something wrong with the actual volume | |
| 19:06:21 | bauzas | yeah probably a ceph RC | |
| 19:06:26 | dansmith | it never mounted and thus couldn't pivot over and the ENOSPC comes from having not mounted something writable on the target | |
| 19:20:52 | bauzas | hmpfff | |
| 19:21:35 | bauzas | Sep 13 13:49:37.042893 np0035238841 nova-compute[40581]: FileNotFoundError: [Errno 2] No such file or directory: 'multipathd' | |
| 19:21:54 | dansmith | always logged by brick AFAIK :( | |
| 19:23:17 | bauzas | I wish I would be Neo 'operator, get me all the knowledge about chopters, err. volume bindings in OpenStack" | |
| 19:24:40 | bauzas | "Tank, I need a pilot program for B-212 helicopter. Hurry.� | |
| 19:24:40 | bauzas | "Tank, I need a pilot program for B-212 helicopter. Hurry.” | |
| 19:24:54 | bauzas | (found the right catchphrase) | |
| 19:25:11 | opendevreview | Dan Smith proposed openstack/nova master: Make our nova-ovs-hybrid-plug job omit cinder https://review.opendev.org/c/openstack/nova/+/893540 | |
| 19:25:16 | dansmith | gmann: looks like it worked this time, but one fix on the cleanup part | |
| 19:25:55 | dansmith | "choppers" or "copters" not "chopters" :P | |
| 19:25:57 | bauzas | so the backref thing is probably about to be punted | |
| 19:26:13 | bauzas | yeah, damn me, it's late | |
| 19:26:27 | bauzas | and when I was young, this was dubbed in French | |
| 19:27:08 | bauzas | so, the multipathd error is just a "normal error" ? | |
| 19:27:14 | bauzas | that's fun | |
| 19:29:32 | JayF | multipathd is only needed for some wacky hardware; we have a similar setup in IPA where we opportunistically load it | |
| 19:31:04 | dansmith | by wacky you mean almost anyone using FC :) | |
| 19:34:18 | JayF | heh, I wasn't exactly sure what storage tech so you know, wacky hardware ;) | |
| 19:34:29 | JayF | nothing we have in CI/gate was more the point :D | |
| 20:08:46 | opendevreview | Merged openstack/nova master: Update compute rpc alias for bobcat https://review.opendev.org/c/openstack/nova/+/893744 | |
| 21:32:21 | opendevreview | Merged openstack/nova master: Revert "Make compute node rebalance safter" https://review.opendev.org/c/openstack/nova/+/894944 | |
| #openstack-nova - 2023-09-14 | |||
| 05:53:41 | opendevreview | Nobuhiro MIKI proposed openstack/nova-specs master: Re-propose "Add maxphysaddr support for Libvirt" for 2024.1 Caracal https://review.opendev.org/c/openstack/nova-specs/+/895135 | |
| 07:41:27 | sahid | sean-k-mooney: o/ we have discussed about using tenant isolation, but that is not really responding our use case an now all vms scheduled for that given tenant are targeted to that az | |
| 07:42:51 | sahid | we would like that, the scheduler consider the new az for some specified tenants | |
| 07:43:03 | sahid | any idea about how we can achieve that? | |
| 07:44:11 | sahid | basically for a given tenant allowed for az3 the return used by the scheduler will be [az1, az2, az3], and for the other tenants only [az1, az2] | |
| 07:46:05 | sahid | bauzas: o/ you may have some idea ^ | |
| 07:46:57 | sahid | i'm thinking about to fix a bit multitenantisolation to acheive that | |
| 08:24:45 | opendevreview | Merged openstack/nova master: Add service version for Bobcat https://review.opendev.org/c/openstack/nova/+/893749 | |
| 08:27:20 | bauzas | sahid: looking (I was afk) | |
| 08:28:22 | bauzas | so you would like to return different AZs based on the tenant ? | |
| 08:28:55 | bauzas | if so, yeah, it's https://github.com/openstack/nova/blob/master/nova/scheduler/request_filter.py#L95 | |
| 08:37:08 | sahid | hum... | |
| 09:09:19 | sahid | that does not really work, we will have to duplicate our aggregates and set filter_tenant_id to all the duplicate ones | |
| 09:11:01 | sahid | I guess in that way, for a given tenant it will be possible to schedule on as1, az2, az3 using filter_tenant_id and the other will continue on az1 and az2 | |
| 09:49:40 | zigo | stephenfin: FYI, I also found that freezer-api probably needs some sqlalchemy 2.x love too... | |
| 09:49:49 | zigo | TypeError: LegacyEngineFacade.from_config() got an unexpected keyword argument 'autocommit' | |
| 09:49:50 | zigo | ... | |
| 09:50:17 | zigo | failures=511 | |
| 09:50:38 | zigo | Is it possible that just removing the autocommit argument is enough? | |
| 09:52:23 | sean-k-mooney | is freezer still supported | |
| 09:52:29 | zigo | Wooops... s/freezer/watcher/ | |
| 09:52:44 | zigo | My bad. | |
| 09:53:05 | sean-k-mooney | i have almost the same qustion for watcher :P although i had seen that mention at least this side of the pandemic | |
| 09:53:28 | zigo | To the contrary of freezer, there was some commits for this cycle. | |
| 09:54:30 | sean-k-mooney | there was like 8 commit in watcher most procedural | |
| 09:54:43 | sean-k-mooney | but i guess its stil offically supported | |
| 09:56:06 | sean-k-mooney | tobias-urdin: you were making some changes to watcher eiarler in the year. is that a project you have use of personaly | |
| 09:56:50 | sean-k-mooney | just wondering if patches were created how likely is it that they would be merged/reviewd | |
| 11:31:14 | opendevreview | Amit Uniyal proposed openstack/nova master: WIP:Adds device tagging functional tests https://review.opendev.org/c/openstack/nova/+/895162 | |
| 11:51:09 | tobias-urdin | sean-k-mooney: we do not use it, nothing i actively work on, i just did some drive-by fixes for oslo.messaging or fixing ci probably | |
| 11:57:50 | sean-k-mooney | tobias-urdin: ack ya that is what i was assumign i was just wondering if there were any active contibutors left | |
| 12:24:30 | bauzas | ralonsoh: while https://review.opendev.org/c/openstack/neutron/+/893447 is now merged (the neutron revert), we still have CI failures today https://zuul.opendev.org/t/openstack/build/01dd3befff5e42cfb0c0f667b8e7e30b | |
| 12:28:54 | bauzas | elodilles: I guess you know the current situation in Nova | |
| 12:29:06 | bauzas | we still need to await a few merges before RC1 | |
| 12:30:53 | bauzas | hmpfff https://zuul.opendev.org/t/openstack/builds?job_name=nova-live-migration&skip=0 | |
| 12:31:05 | bauzas | 30% of failures I'd say | |
| 12:44:09 | mgariepy | is it possible to force nova a device_type for pci devices ? | |
| 12:44:55 | sean-k-mooney | no | |
| 12:45:51 | mgariepy | i wiant to only do some pci passthrough on a couple of A100 | |
| 12:46:14 | sean-k-mooney | you can filter by the pci adress | |
| 12:46:39 | sean-k-mooney | with support for regxes or bash globs if you dont want to just use the adress | |
| 12:46:51 | mgariepy | https://github.com/openstack/nova/blob/stable/2023.1/nova/pci/stats.py#L682C1-L683C1 | |
| 12:47:52 | sean-k-mooney | yes if the alais is not device_type pf then we filter them out when considering candiates | |
| 12:48:10 | sean-k-mooney | that does not mean you cant reqeust PF for passthrough | |
| 12:48:35 | sean-k-mooney | you jsut have to set the device_type to type-PF in the pci ailais | |
| 12:49:22 | sean-k-mooney | so inestead of type-pci or type-vf https://github.com/openstack/nova/blob/stable/2023.1/nova/pci/request.py#L34 you woudl set type-PF | |
| 12:50:50 | sean-k-mooney | the alias and device list work toghtere to selec tthe device. the alias is how you describe the request and the device_list (pci whitelist) is how you declar what device may be allocated | |
| 12:51:36 | mgariepy | alias = { "name": "nvidia-a100-smx4-40gb", "product_id": "20b0", "vendor_id": "10de", 'device_type': 'type-PF'} | |
| 12:51:46 | mgariepy | restart the scheduler, and still get the same error. | |
| 12:52:25 | sean-k-mooney | you need to set this on all the compute too just an fyi hte have to match | |
| 12:52:40 | sean-k-mooney | but what is the error | |
| 12:53:10 | mgariepy | on it's also set on the gpu compute. | |
| 12:53:49 | sean-k-mooney | have you checked the pci devices tabel to ensure the a100s are correctly tracked | |
| 12:54:51 | sean-k-mooney | wether they show up as type-PF or type-pci will depend on if they report supprot for creating VF | |
| 12:55:05 | sean-k-mooney | that may depend on if you have enbale mig mode of not | |
| 12:55:27 | sean-k-mooney | for t4 it depeneded on the firmware version | |
| 12:55:48 | sean-k-mooney | i think a100 woudl alays report sriov supprot but its worth checking | |
| 12:56:12 | mgariepy | Placement PCI resource view: Placement PCI view on gpu-hpc21145: RP(gpu-hpc21145_0000:01:00.0, CUSTOM_PCI_10DE_20B0=1, traits=COMPUTE_MANAGED_PCI_DEVICE) | |
| 12:56:38 | sean-k-mooney | oh your using the pci in placemnt feature | |
| 12:57:01 | mgariepy | yep all i tried was also failing :) haha | |
| 12:57:08 | mgariepy | on 2023.1 | |
| 12:57:19 | sean-k-mooney | well this should be fine | |
| 12:57:32 | sean-k-mooney | so is it passign placment and failing in the pci filter? | |
| 12:57:40 | sean-k-mooney | or is it failing to get results form placemnt | |
| 12:59:23 | elodilles | bauzas: ACK. you can -1 the patch until it'll be updated: https://review.opendev.org/c/openstack/releases/+/894693 | |
| 12:59:52 | bauzas | oh missed it from my mails | |
| 13:00:01 | mgariepy | fails at running : _filter_pools_for_unrequested_pfs() | |