| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-19 | |||
| 14:04:10 | mgariepy | yes for a nic, that's true. | |
| 14:04:25 | mgariepy | that's my testbed , the other system i have uses graphic card :D | |
| 14:04:26 | mgariepy | haha | |
| 14:04:41 | sean-k-mooney | event for gpus we only the vendor id and produc id are recored | |
| 14:04:47 | sean-k-mooney | not the subvend id | |
| 14:05:32 | sean-k-mooney | so all GTX 1080s have the same vendor id and product id but an EVGA or asus one has different subvendor ids | |
| 14:05:48 | sean-k-mooney | not they should be identical but no all such product are. | |
| 14:06:09 | sean-k-mooney | well actully the clock speed/ram could chagne | |
| 14:06:28 | mgariepy | yeah but the drive would manage that for you. | |
| 14:07:32 | sean-k-mooney | it may but the point is if we deteact the device was removed we cannot trust that it si the same and cant reuse the entry | |
| 14:07:53 | sean-k-mooney | we do not recerate tehm on every reboot | |
| 14:08:18 | sean-k-mooney | we only do it if the agent does a pci scan and did not find them | |
| 14:08:49 | mgariepy | yeah, i never had issue with reboot before :D haha | |
| 14:09:22 | mgariepy | anyway at least now i know, and i'll be more careful next time. | |
| 14:10:30 | sean-k-mooney | so out of interst the device was allocated when it was removed | |
| 14:10:40 | sean-k-mooney | what was teh state of the vm | |
| 14:10:44 | stephenfin | mriedem: Yeah, I'm not actually sure how else we could resolve that besides a cold migrate/rebuild. It's a mismatch between two sources of truth: what the libvirt driver is finding on the host (based on the whitelist) and what the instance is saying it's using | |
| 14:11:25 | mgariepy | as long as you don't restart it's ok | |
| 14:11:29 | sean-k-mooney | mgariepy: i would have assumed you would have migrated the vm off the host before upgrade but if you didnt have you tried doing a openstack server reboot --hard to try and fix the issue | |
| 14:11:59 | mgariepy | when you restart, the libvirt config generated doesn't have the pci devices but boots anyway | |
| 14:12:27 | sean-k-mooney | right ok then the only way to fix that is likely to shelve and unshelve | |
| 14:13:00 | sean-k-mooney | e.g. to free up the node and its pci device and then recreate the vm with its data on the same node | |
| 14:13:05 | mgariepy | the reboot --hard doesn't work, since i guess the data is pulled from the DB and my ""new"" device is not allocated | |
| 14:13:09 | mgariepy | yeah. | |
| 14:13:35 | mgariepy | i'll have my client to rebuild his cluster. | |
| 14:14:13 | mgariepy | sean-k-mooney, i do inplace upgrades, it's not a big deal ;P | |
| 14:14:27 | sean-k-mooney | mgariepy: well you could also manually fix this in the db without too much hasel but i guess | |
| 14:14:53 | sean-k-mooney | on a larger cluster it may be more complicated however | |
| 14:15:10 | sean-k-mooney | mgariepy: i have done that but usally i create a test env first. | |
| 14:15:11 | mgariepy | i don't like updating the db. sometimes it comes back to bite me .. | |
| 14:15:58 | sean-k-mooney | mgariepy: yes it can it can be less painful then redeploying the cluster however | |
| 14:16:12 | sean-k-mooney | unless you jsut ment the applicateion in the cluster. | |
| 14:16:13 | mgariepy | it's a "special" contributed cluster part of another one | |
| 14:16:45 | sean-k-mooney | deleting the vm and recreating it would have the same effect but i assume this happend on all nodes so you would have to delete all the vms with passthough and recreate them | |
| 14:16:46 | mgariepy | he runs some kind of hpc cluster on kubernetes on the vms | |
| 14:17:56 | sean-k-mooney | so to be clear. you use openstack to spwan 1 vm per host that uses all the host resouces then they use kubernetes to run a distibuted hpc application on the vms | |
| 14:18:10 | mgariepy | anyway, not a big deal, i'll be more careful next time. i just messed up the nova.conf pci config on the upgrade | |
| 14:18:12 | mgariepy | yep. | |
| 14:18:15 | mgariepy | haha :D | |
| 14:18:32 | sean-k-mooney | that seam overly complicated but if ti work it works i guess :) | |
| 14:18:59 | mgariepy | it's shared, and this way the client doesn't really have to deal with the hardware... | |
| 14:19:17 | mgariepy | and have some other benefit like access to some storage and so on. | |
| 14:19:57 | sean-k-mooney | yep i totally get why you would do it its just you have at least 3 layers fo orchestration there | |
| 14:20:18 | mgariepy | yep, not all the same person do all 3. | |
| 14:20:33 | sean-k-mooney | openstack orchestrting the vms, kubernets orchestatin the hpc cluster and spark or whatever orchestatign the hpc jobs on the cluster | |
| 14:21:06 | mgariepy | probably slurm, but i'm not 100% sure. | |
| 14:21:33 | mgariepy | anyway, thanks a lot for your time and help. | |
| 14:21:42 | sean-k-mooney | if you deploy your openstack with kubernets you can make it nice an inceptione | |
| 14:21:55 | sean-k-mooney | no worres are you ok with me closing https://bugs.launchpad.net/nova/+bug/1809040 | |
| 14:21:56 | openstack | Launchpad bug 1809040 in OpenStack Compute (nova) "pci device lost when error in the configuration file " [Undecided,New] | |
| 14:22:18 | mgariepy | the question is ,will I be able to remove the physical server at some point. | |
| 14:22:20 | mgariepy | yep | |
| 14:25:07 | stephenfin | sean-k-mooney: Perhaps we could add a nova-compute start up check to see if there are unrecognized PCI devices attached to running instances and fail to start if so? | |
| 14:25:24 | stephenfin | sean-k-mooney: Thought I guess by then the old PCI devs in the manager would have been marked deleted and new ones created | |
| 14:25:48 | stephenfin | Unless we did it realllly early, but that would involve duplicating a lot of logic | |
| 14:26:01 | stephenfin | really early = before the PCI manager stuff kicks off | |
| 14:26:17 | sean-k-mooney | stephenfin: well i was going to suggest in the but if some wanted to retarget the bug to allow a hard reboot to fix it then it would be fine | |
| 14:26:26 | jaypipes | jackding: will try my best | |
| 14:26:42 | mgariepy | stephenfin, there are already something like that : https://paste.ubuntu.com/p/GVJQqMSTrM/ | |
| 14:28:12 | sean-k-mooney | stephenfin: e.g. if you had a vm with a passthough deivce in the flavor aliase we would revalidate taht we have claimed it and fix it on hard reboot | |
| 14:29:03 | sean-k-mooney | i was gong to triage it as wontfix and low priority | |
| 14:30:03 | mgariepy | anyone of you uses passthrough and seen: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1808412 | |
| 14:30:03 | openstack | Launchpad bug 1808412 in linux (Ubuntu) "4.15.0 memory allocation issue" [Undecided,Confirmed] | |
| 14:30:41 | mgariepy | when i start the vm with pci_passthrough ,it does pre-allocate the ram, and get stuck at some point. | |
| 14:30:42 | mriedem | stephenfin: rebuild won't fix it b/c we don't do a resource claim on rebuild | |
| 14:32:00 | mgariepy | shelve/unshelve works | |
| 14:35:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix a broken-link in nova doc https://review.openstack.org/626113 | |
| 14:38:11 | mriedem | yeah that would work, didn't think about that, but that will do a new instance resoure claim on unshelve | |
| 14:38:19 | mriedem | how are you pinning it back to the same host though? | |
| 14:38:31 | mriedem | or is that the only option for that instance? | |
| 14:38:38 | mriedem | s/instance/flavor/ | |
| 14:38:44 | mgariepy | i don't have another spot for it haha | |
| 14:39:04 | mgariepy | otherwise migrate should be better | |
| 14:39:20 | mgariepy | as shelve is not shelving ephemeral drive. | |
| 14:42:15 | mriedem | mgariepy: are you ok with https://bugs.launchpad.net/nova/+bug/1809040/comments/4 ? | |
| 14:42:15 | openstack | Launchpad bug 1809040 in OpenStack Compute (nova) "pci device lost when error in the configuration file " [Undecided,New] | |
| 14:42:49 | sean-k-mooney | sorry had to pop away for a bit. i have changed my mind i think i will set it to triaged and low instead of wontfix and low and state that the logic is working as desigend but that we shoudl be able to correct the issue with a hard reboot and we shoudl fix that | |
| 14:43:02 | sean-k-mooney | mriedem: mgariepy stephenfin does ^ sound resonable | |
| 14:43:44 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject migration requests when source is down https://review.openstack.org/623489 | |
| 14:44:05 | stephenfin | sean-k-mooney: Sounds fair | |
| 14:44:48 | mriedem | avolkov was working a fix at one point https://review.openstack.org/#/c/426243/ | |
| 14:44:52 | stephenfin | mriedem: Could you send this docs followup patch on its way? https://review.openstack.org/#/c/614322/ | |
| 14:45:47 | mriedem | done | |
| 14:46:28 | sean-k-mooney | mriedem: that occurred to me as well but i was not sure how involved that would be. e.g. updating the db from the deleted value. also not sure how safe it would be in some cases such as if the divece was chaged | |
| 14:46:40 | sean-k-mooney | ill link it in the bug for context too | |
| 14:46:54 | mriedem | there is also https://bugs.launchpad.net/nova/+bug/1633120 | |
| 14:46:54 | openstack | Launchpad bug 1633120 in OpenStack Compute (nova) "Nova scheduler tries to assign an already-in-use SRIOV QAT VF to a new instance" [Undecided,Confirmed] | |
| 14:46:58 | mriedem | all the same issue, | |
| 14:47:07 | mriedem | change pci whitelist and restart nova compute blows away allocated pci devices | |
| 14:48:03 | sean-k-mooney | you know when we start tracking these device in placement we are not going to be able to just blow away the resouce provider anymore if there are allocation against it | |
| 14:48:20 | mriedem | that was my reply on mgariepy's bug, | |
| 14:48:48 | mriedem | long-term when pci device inventory and allocation is in placement, this shouldn't be a problem, | |
| 14:49:05 | mriedem | because even if nova-compute tried to remove device inventory, if it's in-use by a consumer (instance) the inventory delete request will fail with a 409 | |
| 14:49:44 | mriedem | where is the code that removes / deletes devices on restart of nova-compute? shouldn't that be smarter and not delete those devices which are assigned to an instance? | |
| 14:50:17 | mriedem | as we can see in https://paste.ubuntu.com/p/Pn76QVmwqr/ pci device 8 is allocated to an instance but was deleted anyway | |
| 14:50:21 | sean-k-mooney | mriedem: well we will still need to resouce track to handel the pci device assingment aspect and we will need to be able to coralt it back to the rp | |
| 14:50:55 | mriedem | so in 4 years when pci devices are handled with placement, this will not suck as bad anymore yeah? | |
| 14:51:10 | mriedem | can't we just not delete allocated pci devices? | |
| 14:51:13 | sean-k-mooney | mriedem: actuly i think it will suck more | |
| 14:51:21 | mriedem | obviously that is some kind of referential constraint | |