| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-06-14 | |||
| 08:45:34 | bauzas | (Pdb) vgpu_allocations | |
| 08:45:34 | bauzas | {'1df35fc7-41f4-4ef4-b995-7fd561c6a391': {'resources': {'VGPU': 1}}} | |
| 08:45:39 | gibi | ohh, that is strange | |
| 08:45:46 | gibi | you should have 2 in the allocation | |
| 08:45:46 | bauzas | yeah | |
| 08:45:49 | bauzas | correct | |
| 08:46:01 | gibi | but then you have a nice problem to debug :) | |
| 08:46:14 | bauzas | I'm checking the max limit of the inventory | |
| 08:46:27 | bauzas | that could be the reason | |
| 08:46:47 | bauzas | if we cap to 1, then there are no ways to create an allocation of 2 | |
| 08:46:58 | bauzas | but... the scheduler should have failed, right? | |
| 08:47:01 | gibi | but then you should get no allocation candidates | |
| 08:47:05 | bauzas | yeah that | |
| 08:47:40 | bauzas | something got messed somewhere and I still need to investigate whether this is just a fixture issue | |
| 08:48:00 | gibi | do you have the placement log from the a_c query? | |
| 08:48:26 | gibi | does nova requested 1 VGPU in the a_c query? | |
| 08:49:31 | bauzas | I'll set the DEBUG level | |
| 08:50:55 | bauzas | shit OS_DEBUG=1 doesn't seem to work with functional tests | |
| 08:51:37 | gibi | bauzas: yep it does not, I was not able to track that down last time | |
| 08:52:00 | bauzas | pdb'ing a bit further down then | |
| 08:52:16 | bauzas | anyway, looks like a nice bone to snag | |
| 08:53:01 | bauzas | oh f*** | |
| 08:53:03 | bauzas | forget | |
| 08:53:08 | bauzas | it's PEBKAC | |
| 08:53:44 | bauzas | I wrote stupid code | |
| 08:54:07 | bauzas | https://paste.opendev.org/show/bgjrdJhdinSqXXIY6EWm/ | |
| 08:55:20 | bauzas | way better now this is fixed | |
| 08:55:47 | bauzas | let's pretend this whole conversation never existed | |
| 08:56:12 | bauzas | but my original point remains | |
| 08:56:37 | bauzas | we could let operators to isolate the allocations between different named groups | |
| 08:56:43 | bauzas | in their flavors | |
| 08:57:01 | bauzas | the point is, we will just swallow all of them but one | |
| 08:57:09 | gibi | ahh different falvor :) | |
| 08:58:04 | bauzas | I'm just about modifying the flavor to ask for one VGPU per group | |
| 08:58:18 | bauzas | and I'm pretty sure we'll end up with only one mdev | |
| 09:14:06 | ygk_12345 | hi all can anyone help me with this https://bugs.launchpad.net/nova/+bug/1978065 | |
| 09:18:09 | gibi | ygk_12345: please check the request-id in the conductor and scheduler logs as well | |
| 09:21:15 | ygk_12345 | gibi: All I can find are those messages from all nova logs. They are still stuck in scheduling and building state. Even now out of 10 vms, only 8 are created fine. rest two are in building state | |
| 09:22:15 | ygk_12345 | gibi: i have added those logs now to the case., pls check them | |
| 09:22:47 | gibi | I saw. that is awful small amount of log for an instance boot. do you see more logs for those VMs that booted successfully? | |
| 09:23:15 | ygk_12345 | gibi: let me check that | |
| 09:25:30 | opendevreview | Rajat Dhasmana proposed openstack/python-novaclient master: Add support to rebuild boot volume https://review.opendev.org/c/openstack/python-novaclient/+/827163 | |
| 09:28:58 | ygk_12345 | gibi: i have added the log | |
| 09:30:37 | gibi | I don't think you are actually having / finding all the logs. for a successfull boot you should see many log lines in the conductor / scheduler and compute service | |
| 09:32:18 | bauzas | gibi: sorry to interupt you but my a_c language is a bit rusty | |
| 09:32:30 | gibi | bauzas: no worries | |
| 09:32:38 | bauzas | gibi: /placement/allocation_candidates?group_policy=isolate&in_tree=adbdb144-d84b-4b80-b03b-a0bc520d91ba&limit=1000&resources=DISK_GB%3A20%2CMEMORY_MB%3A2048%2CVCPU%3A2&resources1=VGPU%3A1&resources2=VGPU%3A1&root_required=%21COMPUTE_STATUS_DISABLED gives me no valid candidates and I wonder why | |
| 09:33:06 | bauzas | the two RPs are supporting different types but I don't ask them | |
| 09:33:21 | bauzas | I just ask for one VGPU per RP | |
| 09:35:44 | gibi | do you have the PGPU RP in the tree of in_tree=adbdb144-d84b-4b80-b03b-a0bc520d91ba ? | |
| 09:36:29 | gibi | so you requests two vgpus from two different pGPUs (isolate) | |
| 09:36:42 | gibi | I mean one vgpu from each pgpu | |
| 09:37:53 | gibi | that should work if you have to pGPU RPs | |
| 09:39:04 | gibi | could you paste the output of https://github.com/gibizer/osc-placement-tree ? (or publish the test you are running so I can reproduce it?) | |
| 09:41:33 | bauzas | sorry was doing other thing | |
| 09:42:10 | bauzas | gibi: I can install it in the func venv | |
| 09:42:37 | bauzas | oh, you said it in the README :) | |
| 09:42:45 | bauzas | lemme pip it | |
| 09:48:11 | bauzas | gibi: I need to disappear for gym reasons but I'll work on it this afternoon | |
| 09:48:29 | bauzas | I just installed, I just need to add the placement api module in my test class | |
| 09:48:39 | gibi | bauzas: have a nice workout. feel free to ping me later | |
| 09:51:18 | bauzas | gibi: <3 | |
| 09:51:21 | bauzas | gibi: found my problem | |
| 09:51:30 | bauzas | I was only having one child RP | |
| 09:51:50 | gibi | ack, good to hear that | |
| 09:55:14 | gibi | bauzas, sean-k-mooney: an interesting performance bug https://bugs.launchpad.net/nova/+bug/1978372 | |
| 09:59:49 | sean-k-mooney | first tought is they dont know how to use the hw:numa_* extra specs | |
| 10:00:38 | sean-k-mooney | its very bad pratcit to use hw:numa_cpus or hw:numa_mem with symetric numa toplogies | |
| 10:00:46 | sean-k-mooney | its explictly an anti pattern | |
| 10:00:59 | sean-k-mooney | and should not be doen | |
| 10:01:38 | sean-k-mooney | it wont affect the perfrmace but the also mis understand the relation ship betweeh hw:cpu_threads and hw:cpu_max_threads | |
| 10:01:52 | sean-k-mooney | if hw:cpu_treads is specifed then max is ignored | |
| 10:02:03 | sean-k-mooney | sam for sockets | |
| 10:02:32 | sean-k-mooney | so that flavor defintion annoys me on a fundimental level :) | |
| 10:03:45 | sean-k-mooney | gibi: hoststly im not that suprised we know that this is a very slow algorthim | |
| 10:03:50 | gibi | then look at the minimal reproduction that has no falvor :) | |
| 10:04:17 | sean-k-mooney | i did some experiments a few year ago repimlementing it form scratch | |
| 10:04:39 | sean-k-mooney | but the reason i did not continue with that was we were going to implement numa in placement any day now | |
| 10:04:55 | gibi | also we are emitting 18G debug logs during that run which is pretty extreme | |
| 10:05:35 | sean-k-mooney | peopel complaied there was not enough logging in the hardware module :P | |
| 10:06:06 | sean-k-mooney | https://github.com/SeanMooney/cpu-pinning | |
| 10:06:13 | sean-k-mooney | this was my experimal approch | |
| 10:08:31 | gibi | as far as I understand this call does not call out to any extrnal system (no fs, db, rabbit calls) so it runs in pure python. So taking minutes indicate that we are doing something crazy | |
| 10:10:59 | sean-k-mooney | we are try to validate cpu (floating and pinned),ram ,pci, and pmem affinity based on the rules set in teh falvor with regards to thread affintiy, device affintiy and any semetic or asmetirc numa requirements | |
| 10:16:32 | sean-k-mooney | gibi: by the way we have know about this basically from the start that its qudratic or wrose as you scale the number of host/guest numa nodes | |
| 10:16:50 | sean-k-mooney | its why the numa toploty filter shoudl always be last in your filter list if you enable it | |
| 10:19:28 | sean-k-mooney | i am not sure if ic an repodcue that simpel example in my ohter implemntaion but im going to give it a try quickly | |
| 10:28:30 | sean-k-mooney | [11:27:50]❯ ./run.sh | |
| 10:28:32 | sean-k-mooney | Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning. | |
| 10:28:34 | sean-k-mooney | ||
| 10:28:36 | sean-k-mooney | 0:0,1:32,2:512,3:544 | |
| 10:28:38 | sean-k-mooney | 0:1,1:2,2:3,3:4 | |
| 10:28:40 | sean-k-mooney | 0:5,1:69,2:133,3:197 | |
| 10:28:42 | sean-k-mooney | 0:33,1:513,2:34,3:514,4:35,5:515,6:36,7:6 | |
| 10:28:44 | sean-k-mooney | 0:2047,1:1983,2:1919,3:1855,4:2015,5:1951,6:1887,7:1823 | |
| 10:28:46 | sean-k-mooney | bug repoducer | |
| 10:28:48 | sean-k-mooney | ---------------------------------------- | |
| 10:28:50 | sean-k-mooney | 0:0,1:3,2:6,3:9,4:12,5:15,6:18,7:21,8:48,9:51,10:54,11:57,12:60,13:63,14:66,15:69,16:1,17:4,18:7,19:10,20:13,21:16,22:19,23:22,24:49,25:52,26:55,27:58,28:61,29:64,30:67,31:70,32:2,33:5,34:8,35:11,36:14,37:17,38:20,39:23,40:50,41:53,42:56,43:59,44:62,45:65,46:68,47:71 | |
| 10:28:52 | sean-k-mooney | real 0m1.004s | |
| 10:28:54 | sean-k-mooney | user 0m0.893s | |