| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-14 | |||
| 02:41:45 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to require mulitpathd for volume attachment https://review.opendev.org/c/openstack/nova/+/845660 | |
| 02:43:18 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to require mulitpathd for volume attachment https://review.opendev.org/c/openstack/nova/+/845660 | |
| 02:46:22 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to require mulitpathd for volume attachment https://review.opendev.org/c/openstack/nova/+/845660 | |
| 03:26:29 | opendevreview | Merged openstack/nova stable/xena: Fix eventlet.tpool import https://review.opendev.org/c/openstack/nova/+/840733 | |
| 03:26:35 | opendevreview | Merged openstack/nova stable/xena: Gracefull recovery when attaching volume fails https://review.opendev.org/c/openstack/nova/+/829433 | |
| 04:07:49 | opendevreview | Merged openstack/nova stable/yoga: Fix segment-aware scheduling permissions error https://review.opendev.org/c/openstack/nova/+/840732 | |
| 04:07:55 | opendevreview | Merged openstack/nova stable/yoga: Isolate PCI tracker unit tests https://review.opendev.org/c/openstack/nova/+/840830 | |
| 04:22:43 | opendevreview | Merged openstack/nova stable/yoga: Remove unavailable but not reported PCI devices at startup https://review.opendev.org/c/openstack/nova/+/840831 | |
| 04:24:54 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to require mulitpathd for volume attachment https://review.opendev.org/c/openstack/nova/+/845660 | |
| 04:26:24 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to require mulitpathd for volume attachment https://review.opendev.org/c/openstack/nova/+/845660 | |
| 04:35:13 | opendevreview | Merged openstack/nova stable/wallaby: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/811809 | |
| 06:45:30 | opendevreview | Takashi Kajinami proposed openstack/nova master: libvirt: Add new option to require mulitpathd for volume attachment https://review.opendev.org/c/openstack/nova/+/845660 | |
| 07:32:55 | gibi | good morning | |
| 07:47:35 | bauzas | good morning | |
| 07:54:18 | gibi | sean-k-mooney, artom: I have concerns in https://review.opendev.org/c/openstack/nova/+/824048 | |
| 08:34:36 | bauzas | gibi: following the discussion we had last week about VGPU allocations https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8203-L8206 | |
| 08:37:18 | bauzas | https://images.squarespace-cdn.com/content/518f5d62e4b075248d6a3f90/1385326046909-827X09Z2RML6LYHWQBVI/git-blame2.jpg?format=1500w&content-type=image%2Fjpeg | |
| 08:42:50 | gibi | bauzas: ahh, it was wishfull thingkin | |
| 08:42:55 | gibi | thinking | |
| 08:43:13 | bauzas | gibi: I just wrote a quick functest and this is weird | |
| 08:43:23 | bauzas | I asked for VGPU=2 and I got only one allocation | |
| 08:43:31 | bauzas | so I need to dig into the code | |
| 08:43:50 | gibi | one allocation of two vgpus from the same pgpu? | |
| 08:44:13 | bauzas | just a plain simple flavor with resources:VGPU=µ2 | |
| 08:44:13 | bauzas | just a plain simple flavor with resources:VGPU=µ2 | |
| 08:44:45 | bauzas | which should have created a single allocation of VGPU=2 | |
| 08:44:56 | bauzas | when arriving before the above code | |
| 08:45:04 | bauzas | against one single RP | |
| 08:45:09 | gibi | yepp placement does not split allocation against a single RC between RPs | |
| 08:45:26 | bauzas | but I don't know why when I introspect, I see : | |
| 08:45:34 | bauzas | (Pdb) vgpu_allocations | |
| 08:45:34 | bauzas | {'1df35fc7-41f4-4ef4-b995-7fd561c6a391': {'resources': {'VGPU': 1}}} | |
| 08:45:39 | gibi | ohh, that is strange | |
| 08:45:46 | gibi | you should have 2 in the allocation | |
| 08:45:46 | bauzas | yeah | |
| 08:45:49 | bauzas | correct | |
| 08:46:01 | gibi | but then you have a nice problem to debug :) | |
| 08:46:14 | bauzas | I'm checking the max limit of the inventory | |
| 08:46:27 | bauzas | that could be the reason | |
| 08:46:47 | bauzas | if we cap to 1, then there are no ways to create an allocation of 2 | |
| 08:46:58 | bauzas | but... the scheduler should have failed, right? | |
| 08:47:01 | gibi | but then you should get no allocation candidates | |
| 08:47:05 | bauzas | yeah that | |
| 08:47:40 | bauzas | something got messed somewhere and I still need to investigate whether this is just a fixture issue | |
| 08:48:00 | gibi | do you have the placement log from the a_c query? | |
| 08:48:26 | gibi | does nova requested 1 VGPU in the a_c query? | |
| 08:49:31 | bauzas | I'll set the DEBUG level | |
| 08:50:55 | bauzas | shit OS_DEBUG=1 doesn't seem to work with functional tests | |
| 08:51:37 | gibi | bauzas: yep it does not, I was not able to track that down last time | |
| 08:52:00 | bauzas | pdb'ing a bit further down then | |
| 08:52:16 | bauzas | anyway, looks like a nice bone to snag | |
| 08:53:01 | bauzas | oh f*** | |
| 08:53:03 | bauzas | forget | |
| 08:53:08 | bauzas | it's PEBKAC | |
| 08:53:44 | bauzas | I wrote stupid code | |
| 08:54:07 | bauzas | https://paste.opendev.org/show/bgjrdJhdinSqXXIY6EWm/ | |
| 08:55:20 | bauzas | way better now this is fixed | |
| 08:55:47 | bauzas | let's pretend this whole conversation never existed | |
| 08:56:12 | bauzas | but my original point remains | |
| 08:56:37 | bauzas | we could let operators to isolate the allocations between different named groups | |
| 08:56:43 | bauzas | in their flavors | |
| 08:57:01 | bauzas | the point is, we will just swallow all of them but one | |
| 08:57:09 | gibi | ahh different falvor :) | |
| 08:58:04 | bauzas | I'm just about modifying the flavor to ask for one VGPU per group | |
| 08:58:18 | bauzas | and I'm pretty sure we'll end up with only one mdev | |
| 09:14:06 | ygk_12345 | hi all can anyone help me with this https://bugs.launchpad.net/nova/+bug/1978065 | |
| 09:18:09 | gibi | ygk_12345: please check the request-id in the conductor and scheduler logs as well | |
| 09:21:15 | ygk_12345 | gibi: All I can find are those messages from all nova logs. They are still stuck in scheduling and building state. Even now out of 10 vms, only 8 are created fine. rest two are in building state | |
| 09:22:15 | ygk_12345 | gibi: i have added those logs now to the case., pls check them | |
| 09:22:47 | gibi | I saw. that is awful small amount of log for an instance boot. do you see more logs for those VMs that booted successfully? | |
| 09:23:15 | ygk_12345 | gibi: let me check that | |
| 09:25:30 | opendevreview | Rajat Dhasmana proposed openstack/python-novaclient master: Add support to rebuild boot volume https://review.opendev.org/c/openstack/python-novaclient/+/827163 | |
| 09:28:58 | ygk_12345 | gibi: i have added the log | |
| 09:30:37 | gibi | I don't think you are actually having / finding all the logs. for a successfull boot you should see many log lines in the conductor / scheduler and compute service | |
| 09:32:18 | bauzas | gibi: sorry to interupt you but my a_c language is a bit rusty | |
| 09:32:30 | gibi | bauzas: no worries | |
| 09:32:38 | bauzas | gibi: /placement/allocation_candidates?group_policy=isolate&in_tree=adbdb144-d84b-4b80-b03b-a0bc520d91ba&limit=1000&resources=DISK_GB%3A20%2CMEMORY_MB%3A2048%2CVCPU%3A2&resources1=VGPU%3A1&resources2=VGPU%3A1&root_required=%21COMPUTE_STATUS_DISABLED gives me no valid candidates and I wonder why | |
| 09:33:06 | bauzas | the two RPs are supporting different types but I don't ask them | |
| 09:33:21 | bauzas | I just ask for one VGPU per RP | |
| 09:35:44 | gibi | do you have the PGPU RP in the tree of in_tree=adbdb144-d84b-4b80-b03b-a0bc520d91ba ? | |
| 09:36:29 | gibi | so you requests two vgpus from two different pGPUs (isolate) | |
| 09:36:42 | gibi | I mean one vgpu from each pgpu | |
| 09:37:53 | gibi | that should work if you have to pGPU RPs | |
| 09:39:04 | gibi | could you paste the output of https://github.com/gibizer/osc-placement-tree ? (or publish the test you are running so I can reproduce it?) | |
| 09:41:33 | bauzas | sorry was doing other thing | |
| 09:42:10 | bauzas | gibi: I can install it in the func venv | |
| 09:42:37 | bauzas | oh, you said it in the README :) | |
| 09:42:45 | bauzas | lemme pip it | |
| 09:48:11 | bauzas | gibi: I need to disappear for gym reasons but I'll work on it this afternoon | |
| 09:48:29 | bauzas | I just installed, I just need to add the placement api module in my test class | |
| 09:48:39 | gibi | bauzas: have a nice workout. feel free to ping me later | |
| 09:51:18 | bauzas | gibi: <3 | |
| 09:51:21 | bauzas | gibi: found my problem | |
| 09:51:30 | bauzas | I was only having one child RP | |
| 09:51:50 | gibi | ack, good to hear that | |
| 09:55:14 | gibi | bauzas, sean-k-mooney: an interesting performance bug https://bugs.launchpad.net/nova/+bug/1978372 | |
| 09:59:49 | sean-k-mooney | first tought is they dont know how to use the hw:numa_* extra specs | |
| 10:00:38 | sean-k-mooney | its very bad pratcit to use hw:numa_cpus or hw:numa_mem with symetric numa toplogies | |
| 10:00:46 | sean-k-mooney | its explictly an anti pattern | |
| 10:00:59 | sean-k-mooney | and should not be doen | |