| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-19 | |||
| 13:59:34 | sean-k-mooney | mgariepy: they are | |
| 14:00:09 | mgariepy | yep, but the issue is that if i remove the passthrough config from nova.conf, it get the deleted_at but if I re-add it it create new one. | |
| 14:00:17 | mgariepy | in the same host, same address. etc.. | |
| 14:00:36 | sean-k-mooney | yes | |
| 14:00:41 | sean-k-mooney | that is expected | |
| 14:01:29 | sean-k-mooney | the id filed is an auto incremting filed and the uuid is randomly generated if the device id not found in the database | |
| 14:01:34 | mgariepy | wouldn't be better to re-use the old entry if all the info matches? | |
| 14:02:06 | stephenfin | mgariepy: For what it's worth, that also confused me but it is expected | |
| 14:02:06 | sean-k-mooney | mgariepy: if we have already deleted it no | |
| 14:02:23 | mgariepy | the 2 first are the ""original"" one, then the 2 other are the new one created. | |
| 14:02:31 | mriedem | it seems that things break down on reboot here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5350 | |
| 14:02:42 | mriedem | pci_manager.get_instance_pci_devs(instance) must be returning [] | |
| 14:02:45 | sean-k-mooney | mgariepy: we do it this way because you could have pull the card and install a different one in the same slot | |
| 14:02:59 | mgariepy | yes. | |
| 14:03:24 | mgariepy | then the product id would have changed | |
| 14:03:26 | sean-k-mooney | that could break the guest if we jsut blindly reused it | |
| 14:03:29 | mriedem | and i think that's probably [] because i think instance.pci_devices is set during a resource claim, which doesn't happen on reboot | |
| 14:03:42 | sean-k-mooney | no the product ids could be the same but if they were nics | |
| 14:03:52 | sean-k-mooney | that were usidn for pf passthouhg the mac woudl have changed | |
| 14:03:58 | openstackgerrit | Merged openstack/nova master: Fix a broken-link in nova doc https://review.openstack.org/626110 | |
| 14:04:10 | mgariepy | yes for a nic, that's true. | |
| 14:04:25 | mgariepy | that's my testbed , the other system i have uses graphic card :D | |
| 14:04:26 | mgariepy | haha | |
| 14:04:41 | sean-k-mooney | event for gpus we only the vendor id and produc id are recored | |
| 14:04:47 | sean-k-mooney | not the subvend id | |
| 14:05:32 | sean-k-mooney | so all GTX 1080s have the same vendor id and product id but an EVGA or asus one has different subvendor ids | |
| 14:05:48 | sean-k-mooney | not they should be identical but no all such product are. | |
| 14:06:09 | sean-k-mooney | well actully the clock speed/ram could chagne | |
| 14:06:28 | mgariepy | yeah but the drive would manage that for you. | |
| 14:07:32 | sean-k-mooney | it may but the point is if we deteact the device was removed we cannot trust that it si the same and cant reuse the entry | |
| 14:07:53 | sean-k-mooney | we do not recerate tehm on every reboot | |
| 14:08:18 | sean-k-mooney | we only do it if the agent does a pci scan and did not find them | |
| 14:08:49 | mgariepy | yeah, i never had issue with reboot before :D haha | |
| 14:09:22 | mgariepy | anyway at least now i know, and i'll be more careful next time. | |
| 14:10:30 | sean-k-mooney | so out of interst the device was allocated when it was removed | |
| 14:10:40 | sean-k-mooney | what was teh state of the vm | |
| 14:10:44 | stephenfin | mriedem: Yeah, I'm not actually sure how else we could resolve that besides a cold migrate/rebuild. It's a mismatch between two sources of truth: what the libvirt driver is finding on the host (based on the whitelist) and what the instance is saying it's using | |
| 14:11:25 | mgariepy | as long as you don't restart it's ok | |
| 14:11:29 | sean-k-mooney | mgariepy: i would have assumed you would have migrated the vm off the host before upgrade but if you didnt have you tried doing a openstack server reboot --hard to try and fix the issue | |
| 14:11:59 | mgariepy | when you restart, the libvirt config generated doesn't have the pci devices but boots anyway | |
| 14:12:27 | sean-k-mooney | right ok then the only way to fix that is likely to shelve and unshelve | |
| 14:13:00 | sean-k-mooney | e.g. to free up the node and its pci device and then recreate the vm with its data on the same node | |
| 14:13:05 | mgariepy | the reboot --hard doesn't work, since i guess the data is pulled from the DB and my ""new"" device is not allocated | |
| 14:13:09 | mgariepy | yeah. | |
| 14:13:35 | mgariepy | i'll have my client to rebuild his cluster. | |
| 14:14:13 | mgariepy | sean-k-mooney, i do inplace upgrades, it's not a big deal ;P | |
| 14:14:27 | sean-k-mooney | mgariepy: well you could also manually fix this in the db without too much hasel but i guess | |
| 14:14:53 | sean-k-mooney | on a larger cluster it may be more complicated however | |
| 14:15:10 | sean-k-mooney | mgariepy: i have done that but usally i create a test env first. | |
| 14:15:11 | mgariepy | i don't like updating the db. sometimes it comes back to bite me .. | |
| 14:15:58 | sean-k-mooney | mgariepy: yes it can it can be less painful then redeploying the cluster however | |
| 14:16:12 | sean-k-mooney | unless you jsut ment the applicateion in the cluster. | |
| 14:16:13 | mgariepy | it's a "special" contributed cluster part of another one | |
| 14:16:45 | sean-k-mooney | deleting the vm and recreating it would have the same effect but i assume this happend on all nodes so you would have to delete all the vms with passthough and recreate them | |
| 14:16:46 | mgariepy | he runs some kind of hpc cluster on kubernetes on the vms | |
| 14:17:56 | sean-k-mooney | so to be clear. you use openstack to spwan 1 vm per host that uses all the host resouces then they use kubernetes to run a distibuted hpc application on the vms | |
| 14:18:10 | mgariepy | anyway, not a big deal, i'll be more careful next time. i just messed up the nova.conf pci config on the upgrade | |
| 14:18:12 | mgariepy | yep. | |
| 14:18:15 | mgariepy | haha :D | |
| 14:18:32 | sean-k-mooney | that seam overly complicated but if ti work it works i guess :) | |
| 14:18:59 | mgariepy | it's shared, and this way the client doesn't really have to deal with the hardware... | |
| 14:19:17 | mgariepy | and have some other benefit like access to some storage and so on. | |
| 14:19:57 | sean-k-mooney | yep i totally get why you would do it its just you have at least 3 layers fo orchestration there | |
| 14:20:18 | mgariepy | yep, not all the same person do all 3. | |
| 14:20:33 | sean-k-mooney | openstack orchestrting the vms, kubernets orchestatin the hpc cluster and spark or whatever orchestatign the hpc jobs on the cluster | |
| 14:21:06 | mgariepy | probably slurm, but i'm not 100% sure. | |
| 14:21:33 | mgariepy | anyway, thanks a lot for your time and help. | |
| 14:21:42 | sean-k-mooney | if you deploy your openstack with kubernets you can make it nice an inceptione | |
| 14:21:55 | sean-k-mooney | no worres are you ok with me closing https://bugs.launchpad.net/nova/+bug/1809040 | |
| 14:21:56 | openstack | Launchpad bug 1809040 in OpenStack Compute (nova) "pci device lost when error in the configuration file " [Undecided,New] | |
| 14:22:18 | mgariepy | the question is ,will I be able to remove the physical server at some point. | |
| 14:22:20 | mgariepy | yep | |
| 14:25:07 | stephenfin | sean-k-mooney: Perhaps we could add a nova-compute start up check to see if there are unrecognized PCI devices attached to running instances and fail to start if so? | |
| 14:25:24 | stephenfin | sean-k-mooney: Thought I guess by then the old PCI devs in the manager would have been marked deleted and new ones created | |
| 14:25:48 | stephenfin | Unless we did it realllly early, but that would involve duplicating a lot of logic | |
| 14:26:01 | stephenfin | really early = before the PCI manager stuff kicks off | |
| 14:26:17 | sean-k-mooney | stephenfin: well i was going to suggest in the but if some wanted to retarget the bug to allow a hard reboot to fix it then it would be fine | |
| 14:26:26 | jaypipes | jackding: will try my best | |
| 14:26:42 | mgariepy | stephenfin, there are already something like that : https://paste.ubuntu.com/p/GVJQqMSTrM/ | |
| 14:28:12 | sean-k-mooney | stephenfin: e.g. if you had a vm with a passthough deivce in the flavor aliase we would revalidate taht we have claimed it and fix it on hard reboot | |
| 14:29:03 | sean-k-mooney | i was gong to triage it as wontfix and low priority | |
| 14:30:03 | openstack | Launchpad bug 1808412 in linux (Ubuntu) "4.15.0 memory allocation issue" [Undecided,Confirmed] | |
| 14:30:03 | mgariepy | anyone of you uses passthrough and seen: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1808412 | |
| 14:30:41 | mgariepy | when i start the vm with pci_passthrough ,it does pre-allocate the ram, and get stuck at some point. | |
| 14:30:42 | mriedem | stephenfin: rebuild won't fix it b/c we don't do a resource claim on rebuild | |
| 14:32:00 | mgariepy | shelve/unshelve works | |
| 14:35:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix a broken-link in nova doc https://review.openstack.org/626113 | |
| 14:38:11 | mriedem | yeah that would work, didn't think about that, but that will do a new instance resoure claim on unshelve | |
| 14:38:19 | mriedem | how are you pinning it back to the same host though? | |
| 14:38:31 | mriedem | or is that the only option for that instance? | |
| 14:38:38 | mriedem | s/instance/flavor/ | |
| 14:38:44 | mgariepy | i don't have another spot for it haha | |
| 14:39:04 | mgariepy | otherwise migrate should be better | |
| 14:39:20 | mgariepy | as shelve is not shelving ephemeral drive. | |
| 14:42:15 | openstack | Launchpad bug 1809040 in OpenStack Compute (nova) "pci device lost when error in the configuration file " [Undecided,New] | |
| 14:42:15 | mriedem | mgariepy: are you ok with https://bugs.launchpad.net/nova/+bug/1809040/comments/4 ? | |
| 14:42:49 | sean-k-mooney | sorry had to pop away for a bit. i have changed my mind i think i will set it to triaged and low instead of wontfix and low and state that the logic is working as desigend but that we shoudl be able to correct the issue with a hard reboot and we shoudl fix that | |
| 14:43:02 | sean-k-mooney | mriedem: mgariepy stephenfin does ^ sound resonable | |
| 14:43:44 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject migration requests when source is down https://review.openstack.org/623489 | |
| 14:44:05 | stephenfin | sean-k-mooney: Sounds fair | |