| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-29 | |||
| 16:48:39 | jmlowe | yes | |
| 16:48:43 | jaypipes | jmlowe: because I don't believe shelving (without offloading) affects allocations. could be wrong though. | |
| 16:49:01 | jaypipes | shelve is possibly one of the worst designed things in Nova... </grumbling> | |
| 16:49:05 | mriedem | when we shelve offload, the allocations for the node on which the instance is running will be deleted | |
| 16:49:42 | mriedem | https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L4496 | |
| 16:50:07 | dansmith | he said "without offloading" | |
| 16:50:11 | mriedem | when you unshelve, you go through the scheduler which will create allocations against the selected destination host | |
| 16:50:44 | mriedem | where did he say 'without offloading'? | |
| 16:51:00 | jmlowe | Really need it for our workflows, scientist does data analysis and runs for several weeks, shelves and goes off to write up results, submits paper and waits a couple of months to hear back from reviewers, unshelves and reruns based on reviewers comments | |
| 16:51:01 | dansmith | <jaypipes>jmlowe: because I don't believe shelving (without offloading) affects allocations. could be wrong though. | |
| 16:51:02 | mriedem | jay said that | |
| 16:51:11 | mriedem | by default shelve offloads immediately | |
| 16:51:15 | mriedem | and removes allocatoins | |
| 16:51:18 | dansmith | yeah, I thought you were contradicting jaypipes' comment | |
| 16:51:33 | mriedem | no, i thought jay was asking for clarification | |
| 16:51:34 | dansmith | jaypipes: I learned something about the importance of shelve this summit by the way | |
| 16:51:41 | mriedem | to which jmlowe said "yes" to shelve_offload | |
| 16:51:46 | jmlowe | that default behavior is why I tend to drop the offloading when referring to it | |
| 16:51:51 | dansmith | jaypipes: related to my apparent misunderstanding of how AWS works | |
| 16:51:58 | mriedem | jmlowe: same here | |
| 16:52:15 | mriedem | jmlowe: so shelve / unshelve should be ok wrt allocations | |
| 16:52:44 | mriedem | https://review.openstack.org/#/c/494632/ | |
| 16:52:57 | mriedem | that was in the pike GA also | |
| 16:53:01 | jmlowe | I'll start spot checking history once I get a list of all the errant allocations | |
| 16:53:52 | cfriesen | dansmith: I'll bite...what did you learn about shelve? | |
| 16:54:14 | dansmith | cfriesen: weren't you in that cyborg session? | |
| 16:54:51 | cfriesen | dansmith: if I was I missed the important bit. :) | |
| 16:55:34 | dansmith | cfriesen: that the AWS equivalent of our stop/start involves a reschedule and is actually a little more equivalent to our shelve(-offload) and unshelve in terms of freeing resources (and the associated billing) | |
| 16:56:19 | cfriesen | dansmith: ah, okay. makes sense. I never could figure out the logic of why we have stop/suspend/pause when they all still consume resources. | |
| 16:57:03 | dansmith | well, just lifecycle of a pet, but I had assumed we imported that from AWS for parity | |
| 16:57:58 | jmlowe | I'd kill for the ability to offline migrate a suspended instance btw | |
| 16:58:27 | cfriesen | jmlowe: should be technically doable. | |
| 16:58:55 | jmlowe | cfriesen: right, just move the definition and the file where the state is written? | |
| 16:59:34 | cfriesen | jmlowe: essentially. the resource tracking and wiring up the networking and storage and whatnot are the tricky bits | |
| 16:59:57 | dansmith | yeah, a little more complicated than that | |
| 16:59:58 | dansmith | I thought that cold migrate could do it though | |
| 17:00:31 | cfriesen | dansmith: if not, I bet it wouldn't be horrendous to add it | |
| 17:00:38 | jmlowe | haven't tried for a while | |
| 17:01:01 | dansmith | or do you mean actually suspended (i.e. memory state saved) ? | |
| 17:01:44 | cfriesen | dansmith: looks like we only cold-migrate "active" or "stopped" instances | |
| 17:02:00 | dansmith | yeah, I assumed he meant suspended==stopped | |
| 17:02:15 | dansmith | I'm not sure I understand the rationale for suspend in the cloud either | |
| 17:02:30 | dansmith | maybe faster boot for slow-to-start applications or something, but.. | |
| 17:02:39 | yumapath | hi can i get some help on putting pci devices in passthrough mode | |
| 17:02:48 | yumapath | https://docs.openstack.org/nova/pike/admin/pci-passthrough.html | |
| 17:02:52 | cfriesen | dansmith: in order to manage disk space on compute nodes we ended up making "suspend" behave like "pause" | |
| 17:03:04 | yumapath | i followed this link , but not able to get things working | |
| 17:03:26 | yumapath | if anyone out there can help me that would be great | |
| 17:05:14 | dansmith | cfriesen: for what gain? | |
| 17:05:21 | dansmith | oh, | |
| 17:05:23 | dansmith | suspend like pause | |
| 17:05:29 | dansmith | so they don't take up extra space with the memory image? | |
| 17:12:31 | jmlowe | hmm, so it would seem out of 946 instances two were in the wrong place, about two dozen don't have allocations | |
| 17:14:14 | jmlowe | last instance action for the one in the wrong place was an errored live migration | |
| 17:15:06 | openstackgerrit | Merged openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640 | |
| 17:20:34 | cfriesen | dansmith: was away from keyboard. yeah, otherwise we'd need to reserve as much disk space as we have guest RAM, which might be an issue on small-disk compute nodes. | |
| 17:21:03 | dansmith | yeah, makes sense. unfortunately it changes the behavior the api user expects | |
| 17:23:44 | cfriesen | dansmith: agreed. though the practical difference between the two is minimal | |
| 17:24:28 | jaypipes | jmlowe: sorry, had a phone call... ok, sorry, hope you are ready for a stream of opinions coming from me... here goes... | |
| 17:24:49 | dansmith | cfriesen: well, if you're using pause for fast HA, the startup time may differ significantly, but also suspend should survive host reboots, power outages, etc | |
| 17:24:53 | dansmith | cfriesen: but yes, similar | |
| 17:25:09 | jaypipes | jmlowe: a) application state should *never* be saved in a VM disk image. it should always be saved only to a persistent volume that can be attached/detached as needed. | |
| 17:25:58 | jmlowe | great in theory, now fix 40 years of domain specific scientific code | |
| 17:26:13 | jaypipes | jmlowe: b) if you need persistent IP addresses, then those IP addresses should be managed outside of the scope of Nova (i.e. created in Neutron and attached to a Nova VM on boot). Then detached from the VM when no longer needed. | |
| 17:26:25 | jaypipes | jmlowe: like I said, my opinion... :) | |
| 17:27:44 | jmlowe | case in point, most domestic weather forecasting is done with WRF, there is exactly one guy in the world who still knows how it works, he was brought out of retirement at UCAR, there is a new model in the works but I don't think it's ready for primetime yet | |
| 17:27:52 | jaypipes | jmlowe: c) you will end up spending more money and time working around coupled app state and configuration instead of investing the time in decoupling the app configuration and state and treating all your VMs as cattle instead of pets. | |
| 17:29:43 | jaypipes | jmlowe: ok, I'm done with my opinion stream. :) I recognize there are real and unworkaroundble constraints that you and others suffer with. I'm just pointing out my experience and long-term strategy, nothing more. | |
| 17:29:55 | openstackgerrit | Merged openstack/nova stable/pike: Stringify instance UUID https://review.openstack.org/569421 | |
| 17:30:06 | jmlowe | I have thousands of pets, they technically aren't even mine, I'd guess of my ~2400 users 100 have the chops to rework their applications | |
| 17:30:27 | jaypipes | jmlowe: understood. | |
| 17:30:59 | jmlowe | I tackle them one at a time, in a few centuries I will have made major progress | |
| 17:31:07 | jaypipes | heh :) touche. | |
| 17:32:00 | jaypipes | yumapath: you will need to be more specific that "but not able to get things working"./ | |
| 17:32:08 | jaypipes | s/that/than | |
| 17:37:31 | openstackgerrit | Merged openstack/nova stable/queens: Fix shelving a paused instance https://review.openstack.org/569650 | |
| 17:39:13 | mriedem | cfriesen: are you aware of anything in the pci whitelist that must be case sensitive? https://review.openstack.org/#/c/565808/4/nova/pci/utils.py | |
| 17:47:35 | cfriesen | mriedem: not my area of expertise | |
| 17:48:01 | cfriesen | mriedem: our pci guy seems to be away at the moment. | |
| 18:03:21 | mriedem | jmlowe: ok so it sounds like what i expected, the failed live migration was probably before we patched pike to cleanup allocations properly in that case | |
| 18:04:02 | mriedem | jmlowe: any ideas on the ones that don't have allocations at all? | |
| 18:05:04 | jmlowe | I haven't really found a pattern, other than they existed during 16.0 vs our current 16.1 | |
| 18:05:16 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (3) https://review.openstack.org/569320 | |
| 18:05:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (4) https://review.openstack.org/570750 | |
| 18:05:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (5) https://review.openstack.org/570842 | |
| 18:08:53 | mriedem | jmlowe: did you look at the instance action records for the ones that don't have allocations? | |
| 18:09:25 | jmlowe | http://paste.openstack.org/show/722279/ | |
| 18:10:03 | mriedem | did req-e0d0edd7-74ee-40d0-b538-9f15f0242183 fail? | |
| 18:10:22 | jmlowe | NAFAICT | |
| 18:10:51 | jmlowe | failed ones tended to get double allocations | |
| 18:11:16 | jmlowe | or allocated on the wrong (probably target) host | |
| 18:11:53 | mriedem | can you do a "nova instance-action <server> req-e0d0edd7-74ee-40d0-b538-9f15f0242183" on that same instance from the paste? | |
| 18:12:56 | jmlowe | http://paste.openstack.org/show/722283/ | |
| 18:13:55 | mriedem | do you have any ocata compute services running? | |
| 18:14:02 | jmlowe | no | |
| 18:14:38 | jmlowe | I get the warnings about only have pike won't update allocation in the nova compute logs | |
| 18:14:54 | mriedem | ok yeah that's what i was thinking of | |
| 18:17:04 | efried | jaypipes: Finally finished https://review.openstack.org/#/c/553597/ -- thanks for your patience | |
| 18:19:01 | mriedem | wonder if we have a bug here https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L5823 | |
| 18:19:47 | mriedem | that should only remove the instance's allocations against the source node after a successful live migration, | |