| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-19 | |||
| 14:14:27 | sean-k-mooney | mgariepy: well you could also manually fix this in the db without too much hasel but i guess | |
| 14:14:53 | sean-k-mooney | on a larger cluster it may be more complicated however | |
| 14:15:10 | sean-k-mooney | mgariepy: i have done that but usally i create a test env first. | |
| 14:15:11 | mgariepy | i don't like updating the db. sometimes it comes back to bite me .. | |
| 14:15:58 | sean-k-mooney | mgariepy: yes it can it can be less painful then redeploying the cluster however | |
| 14:16:12 | sean-k-mooney | unless you jsut ment the applicateion in the cluster. | |
| 14:16:13 | mgariepy | it's a "special" contributed cluster part of another one | |
| 14:16:45 | sean-k-mooney | deleting the vm and recreating it would have the same effect but i assume this happend on all nodes so you would have to delete all the vms with passthough and recreate them | |
| 14:16:46 | mgariepy | he runs some kind of hpc cluster on kubernetes on the vms | |
| 14:17:56 | sean-k-mooney | so to be clear. you use openstack to spwan 1 vm per host that uses all the host resouces then they use kubernetes to run a distibuted hpc application on the vms | |
| 14:18:10 | mgariepy | anyway, not a big deal, i'll be more careful next time. i just messed up the nova.conf pci config on the upgrade | |
| 14:18:12 | mgariepy | yep. | |
| 14:18:15 | mgariepy | haha :D | |
| 14:18:32 | sean-k-mooney | that seam overly complicated but if ti work it works i guess :) | |
| 14:18:59 | mgariepy | it's shared, and this way the client doesn't really have to deal with the hardware... | |
| 14:19:17 | mgariepy | and have some other benefit like access to some storage and so on. | |
| 14:19:57 | sean-k-mooney | yep i totally get why you would do it its just you have at least 3 layers fo orchestration there | |
| 14:20:18 | mgariepy | yep, not all the same person do all 3. | |
| 14:20:33 | sean-k-mooney | openstack orchestrting the vms, kubernets orchestatin the hpc cluster and spark or whatever orchestatign the hpc jobs on the cluster | |
| 14:21:06 | mgariepy | probably slurm, but i'm not 100% sure. | |
| 14:21:33 | mgariepy | anyway, thanks a lot for your time and help. | |
| 14:21:42 | sean-k-mooney | if you deploy your openstack with kubernets you can make it nice an inceptione | |
| 14:21:55 | sean-k-mooney | no worres are you ok with me closing https://bugs.launchpad.net/nova/+bug/1809040 | |
| 14:21:56 | openstack | Launchpad bug 1809040 in OpenStack Compute (nova) "pci device lost when error in the configuration file " [Undecided,New] | |
| 14:22:18 | mgariepy | the question is ,will I be able to remove the physical server at some point. | |
| 14:22:20 | mgariepy | yep | |
| 14:25:07 | stephenfin | sean-k-mooney: Perhaps we could add a nova-compute start up check to see if there are unrecognized PCI devices attached to running instances and fail to start if so? | |
| 14:25:24 | stephenfin | sean-k-mooney: Thought I guess by then the old PCI devs in the manager would have been marked deleted and new ones created | |
| 14:25:48 | stephenfin | Unless we did it realllly early, but that would involve duplicating a lot of logic | |
| 14:26:01 | stephenfin | really early = before the PCI manager stuff kicks off | |
| 14:26:17 | sean-k-mooney | stephenfin: well i was going to suggest in the but if some wanted to retarget the bug to allow a hard reboot to fix it then it would be fine | |
| 14:26:26 | jaypipes | jackding: will try my best | |
| 14:26:42 | mgariepy | stephenfin, there are already something like that : https://paste.ubuntu.com/p/GVJQqMSTrM/ | |
| 14:28:12 | sean-k-mooney | stephenfin: e.g. if you had a vm with a passthough deivce in the flavor aliase we would revalidate taht we have claimed it and fix it on hard reboot | |
| 14:29:03 | sean-k-mooney | i was gong to triage it as wontfix and low priority | |
| 14:30:03 | openstack | Launchpad bug 1808412 in linux (Ubuntu) "4.15.0 memory allocation issue" [Undecided,Confirmed] | |
| 14:30:03 | mgariepy | anyone of you uses passthrough and seen: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1808412 | |
| 14:30:41 | mgariepy | when i start the vm with pci_passthrough ,it does pre-allocate the ram, and get stuck at some point. | |
| 14:30:42 | mriedem | stephenfin: rebuild won't fix it b/c we don't do a resource claim on rebuild | |
| 14:32:00 | mgariepy | shelve/unshelve works | |
| 14:35:37 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix a broken-link in nova doc https://review.openstack.org/626113 | |
| 14:38:11 | mriedem | yeah that would work, didn't think about that, but that will do a new instance resoure claim on unshelve | |
| 14:38:19 | mriedem | how are you pinning it back to the same host though? | |
| 14:38:31 | mriedem | or is that the only option for that instance? | |
| 14:38:38 | mriedem | s/instance/flavor/ | |
| 14:38:44 | mgariepy | i don't have another spot for it haha | |
| 14:39:04 | mgariepy | otherwise migrate should be better | |
| 14:39:20 | mgariepy | as shelve is not shelving ephemeral drive. | |
| 14:42:15 | openstack | Launchpad bug 1809040 in OpenStack Compute (nova) "pci device lost when error in the configuration file " [Undecided,New] | |
| 14:42:15 | mriedem | mgariepy: are you ok with https://bugs.launchpad.net/nova/+bug/1809040/comments/4 ? | |
| 14:42:49 | sean-k-mooney | sorry had to pop away for a bit. i have changed my mind i think i will set it to triaged and low instead of wontfix and low and state that the logic is working as desigend but that we shoudl be able to correct the issue with a hard reboot and we shoudl fix that | |
| 14:43:02 | sean-k-mooney | mriedem: mgariepy stephenfin does ^ sound resonable | |
| 14:43:44 | openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Reject migration requests when source is down https://review.openstack.org/623489 | |
| 14:44:05 | stephenfin | sean-k-mooney: Sounds fair | |
| 14:44:48 | mriedem | avolkov was working a fix at one point https://review.openstack.org/#/c/426243/ | |
| 14:44:52 | stephenfin | mriedem: Could you send this docs followup patch on its way? https://review.openstack.org/#/c/614322/ | |
| 14:45:47 | mriedem | done | |
| 14:46:28 | sean-k-mooney | mriedem: that occurred to me as well but i was not sure how involved that would be. e.g. updating the db from the deleted value. also not sure how safe it would be in some cases such as if the divece was chaged | |
| 14:46:40 | sean-k-mooney | ill link it in the bug for context too | |
| 14:46:54 | openstack | Launchpad bug 1633120 in OpenStack Compute (nova) "Nova scheduler tries to assign an already-in-use SRIOV QAT VF to a new instance" [Undecided,Confirmed] | |
| 14:46:54 | mriedem | there is also https://bugs.launchpad.net/nova/+bug/1633120 | |
| 14:46:58 | mriedem | all the same issue, | |
| 14:47:07 | mriedem | change pci whitelist and restart nova compute blows away allocated pci devices | |
| 14:48:03 | sean-k-mooney | you know when we start tracking these device in placement we are not going to be able to just blow away the resouce provider anymore if there are allocation against it | |
| 14:48:20 | mriedem | that was my reply on mgariepy's bug, | |
| 14:48:48 | mriedem | long-term when pci device inventory and allocation is in placement, this shouldn't be a problem, | |
| 14:49:05 | mriedem | because even if nova-compute tried to remove device inventory, if it's in-use by a consumer (instance) the inventory delete request will fail with a 409 | |
| 14:49:44 | mriedem | where is the code that removes / deletes devices on restart of nova-compute? shouldn't that be smarter and not delete those devices which are assigned to an instance? | |
| 14:50:17 | mriedem | as we can see in https://paste.ubuntu.com/p/Pn76QVmwqr/ pci device 8 is allocated to an instance but was deleted anyway | |
| 14:50:21 | sean-k-mooney | mriedem: well we will still need to resouce track to handel the pci device assingment aspect and we will need to be able to coralt it back to the rp | |
| 14:50:55 | mriedem | so in 4 years when pci devices are handled with placement, this will not suck as bad anymore yeah? | |
| 14:51:10 | mriedem | can't we just not delete allocated pci devices? | |
| 14:51:13 | sean-k-mooney | mriedem: actuly i think it will suck more | |
| 14:51:21 | mriedem | obviously that is some kind of referential constraint | |
| 14:52:12 | sean-k-mooney | we could delay deleting the entry until it is deallcoated | |
| 14:52:40 | sean-k-mooney | if the admin change the whitelist to prevent a device form being used we would still want to stop new instaces form using it | |
| 14:52:57 | sean-k-mooney | but it would be resoable to assume that an existing instance could continue to use it | |
| 14:53:35 | mriedem | i don't know how you delay that | |
| 14:54:02 | mriedem | without new logic / data modeling on the pci device record to say, "pending delete" or something once it's no longer allocated | |
| 14:54:09 | mriedem | but by then you'd have to see if it's back in the pci whitelist | |
| 14:54:29 | sean-k-mooney | this is where we figure out the available devices https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L119 | |
| 14:54:35 | mriedem | i would think a simple solution is fail nova-compute restart if you tried scrubbing an entry from the whitelist for an allocated devie | |
| 14:54:46 | mriedem | jaypipes: have you ever heard of this craziness? ^ | |
| 14:55:59 | mriedem | jaypipes: if you change the pci whitelist config to remove an already allocated device, and restart nova-compute, compute deletes the pci device record that is still allocated to an instance | |
| 14:56:14 | mriedem | if you then add it back to the whitelist and restart, nova-compute creates a new 'available' pci device record and the scheduler can try to use it | |
| 14:56:15 | sean-k-mooney | so we could just change the if form self.dev_filter.device_assignable(dev) to self.dev_filter.device_assignable(dev) or is_allocated(deve) where is allocated is a new fuction | |
| 14:56:22 | mriedem | which the hypervisor will reject | |
| 14:56:30 | mriedem | you can also lose the old device assigned to the old instance if you reboot the instance | |
| 14:57:26 | mriedem | sean-k-mooney: maybe? i don't know what devices_json is | |
| 14:57:32 | mriedem | is that from the db or the config? | |
| 14:58:41 | mriedem | god even finding the object code to see where pci device records are deleted is hard | |
| 14:59:04 | mriedem | oh it's in save() i should have known! | |
| 14:59:45 | mriedem | https://github.com/openstack/nova/blob/master/nova/objects/pci_device.py#L244 | |
| 14:59:56 | sean-k-mooney | so this is suspicious https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L166-L181 | |
| 15:01:08 | mriedem | idk, seems like a very easy solution to avoid screwing up the db state would just be raise an exception here https://github.com/openstack/nova/blob/master/nova/objects/pci_device.py#L244 if self.instance_uuid is not None | |
| 15:01:17 | mriedem | i.e. you can't remove/delete an allocated pci device | |
| 15:02:31 | sean-k-mooney | yes but it looks like https://github.com/openstack/nova/blob/master/nova/pci/manager.py#L166-L181 was maybe working around that | |
| 15:04:02 | mriedem | that was added 6 years ago https://github.com/openstack/nova/commit/4855239497050c9ee03fed627c5f41d6b59eddc6 | |
| 15:04:05 | mriedem | and clearly it sucks | |
| 15:04:26 | gibi | mriedem: FYI my wall of text as a review guide for the bandwidth series http://lists.openstack.org/pipermail/openstack-discuss/2018-December/001129.html | |