| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-29 | |||
| 16:52:57 | mriedem | that was in the pike GA also | |
| 16:53:01 | jmlowe | I'll start spot checking history once I get a list of all the errant allocations | |
| 16:53:52 | cfriesen | dansmith: I'll bite...what did you learn about shelve? | |
| 16:54:14 | dansmith | cfriesen: weren't you in that cyborg session? | |
| 16:54:51 | cfriesen | dansmith: if I was I missed the important bit. :) | |
| 16:55:34 | dansmith | cfriesen: that the AWS equivalent of our stop/start involves a reschedule and is actually a little more equivalent to our shelve(-offload) and unshelve in terms of freeing resources (and the associated billing) | |
| 16:56:19 | cfriesen | dansmith: ah, okay. makes sense. I never could figure out the logic of why we have stop/suspend/pause when they all still consume resources. | |
| 16:57:03 | dansmith | well, just lifecycle of a pet, but I had assumed we imported that from AWS for parity | |
| 16:57:58 | jmlowe | I'd kill for the ability to offline migrate a suspended instance btw | |
| 16:58:27 | cfriesen | jmlowe: should be technically doable. | |
| 16:58:55 | jmlowe | cfriesen: right, just move the definition and the file where the state is written? | |
| 16:59:34 | cfriesen | jmlowe: essentially. the resource tracking and wiring up the networking and storage and whatnot are the tricky bits | |
| 16:59:57 | dansmith | yeah, a little more complicated than that | |
| 16:59:58 | dansmith | I thought that cold migrate could do it though | |
| 17:00:31 | cfriesen | dansmith: if not, I bet it wouldn't be horrendous to add it | |
| 17:00:38 | jmlowe | haven't tried for a while | |
| 17:01:01 | dansmith | or do you mean actually suspended (i.e. memory state saved) ? | |
| 17:01:44 | cfriesen | dansmith: looks like we only cold-migrate "active" or "stopped" instances | |
| 17:02:00 | dansmith | yeah, I assumed he meant suspended==stopped | |
| 17:02:15 | dansmith | I'm not sure I understand the rationale for suspend in the cloud either | |
| 17:02:30 | dansmith | maybe faster boot for slow-to-start applications or something, but.. | |
| 17:02:39 | yumapath | hi can i get some help on putting pci devices in passthrough mode | |
| 17:02:48 | yumapath | https://docs.openstack.org/nova/pike/admin/pci-passthrough.html | |
| 17:02:52 | cfriesen | dansmith: in order to manage disk space on compute nodes we ended up making "suspend" behave like "pause" | |
| 17:03:04 | yumapath | i followed this link , but not able to get things working | |
| 17:03:26 | yumapath | if anyone out there can help me that would be great | |
| 17:05:14 | dansmith | cfriesen: for what gain? | |
| 17:05:21 | dansmith | oh, | |
| 17:05:23 | dansmith | suspend like pause | |
| 17:05:29 | dansmith | so they don't take up extra space with the memory image? | |
| 17:12:31 | jmlowe | hmm, so it would seem out of 946 instances two were in the wrong place, about two dozen don't have allocations | |
| 17:14:14 | jmlowe | last instance action for the one in the wrong place was an errored live migration | |
| 17:15:06 | openstackgerrit | Merged openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640 | |
| 17:20:34 | cfriesen | dansmith: was away from keyboard. yeah, otherwise we'd need to reserve as much disk space as we have guest RAM, which might be an issue on small-disk compute nodes. | |
| 17:21:03 | dansmith | yeah, makes sense. unfortunately it changes the behavior the api user expects | |
| 17:23:44 | cfriesen | dansmith: agreed. though the practical difference between the two is minimal | |
| 17:24:28 | jaypipes | jmlowe: sorry, had a phone call... ok, sorry, hope you are ready for a stream of opinions coming from me... here goes... | |
| 17:24:49 | dansmith | cfriesen: well, if you're using pause for fast HA, the startup time may differ significantly, but also suspend should survive host reboots, power outages, etc | |
| 17:24:53 | dansmith | cfriesen: but yes, similar | |
| 17:25:09 | jaypipes | jmlowe: a) application state should *never* be saved in a VM disk image. it should always be saved only to a persistent volume that can be attached/detached as needed. | |
| 17:25:58 | jmlowe | great in theory, now fix 40 years of domain specific scientific code | |
| 17:26:13 | jaypipes | jmlowe: b) if you need persistent IP addresses, then those IP addresses should be managed outside of the scope of Nova (i.e. created in Neutron and attached to a Nova VM on boot). Then detached from the VM when no longer needed. | |
| 17:26:25 | jaypipes | jmlowe: like I said, my opinion... :) | |
| 17:27:44 | jmlowe | case in point, most domestic weather forecasting is done with WRF, there is exactly one guy in the world who still knows how it works, he was brought out of retirement at UCAR, there is a new model in the works but I don't think it's ready for primetime yet | |
| 17:27:52 | jaypipes | jmlowe: c) you will end up spending more money and time working around coupled app state and configuration instead of investing the time in decoupling the app configuration and state and treating all your VMs as cattle instead of pets. | |
| 17:29:43 | jaypipes | jmlowe: ok, I'm done with my opinion stream. :) I recognize there are real and unworkaroundble constraints that you and others suffer with. I'm just pointing out my experience and long-term strategy, nothing more. | |
| 17:29:55 | openstackgerrit | Merged openstack/nova stable/pike: Stringify instance UUID https://review.openstack.org/569421 | |
| 17:30:06 | jmlowe | I have thousands of pets, they technically aren't even mine, I'd guess of my ~2400 users 100 have the chops to rework their applications | |
| 17:30:27 | jaypipes | jmlowe: understood. | |
| 17:30:59 | jmlowe | I tackle them one at a time, in a few centuries I will have made major progress | |
| 17:31:07 | jaypipes | heh :) touche. | |
| 17:32:00 | jaypipes | yumapath: you will need to be more specific that "but not able to get things working"./ | |
| 17:32:08 | jaypipes | s/that/than | |
| 17:37:31 | openstackgerrit | Merged openstack/nova stable/queens: Fix shelving a paused instance https://review.openstack.org/569650 | |
| 17:39:13 | mriedem | cfriesen: are you aware of anything in the pci whitelist that must be case sensitive? https://review.openstack.org/#/c/565808/4/nova/pci/utils.py | |
| 17:47:35 | cfriesen | mriedem: not my area of expertise | |
| 17:48:01 | cfriesen | mriedem: our pci guy seems to be away at the moment. | |
| 18:03:21 | mriedem | jmlowe: ok so it sounds like what i expected, the failed live migration was probably before we patched pike to cleanup allocations properly in that case | |
| 18:04:02 | mriedem | jmlowe: any ideas on the ones that don't have allocations at all? | |
| 18:05:04 | jmlowe | I haven't really found a pattern, other than they existed during 16.0 vs our current 16.1 | |
| 18:05:16 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (3) https://review.openstack.org/569320 | |
| 18:05:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (4) https://review.openstack.org/570750 | |
| 18:05:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (5) https://review.openstack.org/570842 | |
| 18:08:53 | mriedem | jmlowe: did you look at the instance action records for the ones that don't have allocations? | |
| 18:09:25 | jmlowe | http://paste.openstack.org/show/722279/ | |
| 18:10:03 | mriedem | did req-e0d0edd7-74ee-40d0-b538-9f15f0242183 fail? | |
| 18:10:22 | jmlowe | NAFAICT | |
| 18:10:51 | jmlowe | failed ones tended to get double allocations | |
| 18:11:16 | jmlowe | or allocated on the wrong (probably target) host | |
| 18:11:53 | mriedem | can you do a "nova instance-action <server> req-e0d0edd7-74ee-40d0-b538-9f15f0242183" on that same instance from the paste? | |
| 18:12:56 | jmlowe | http://paste.openstack.org/show/722283/ | |
| 18:13:55 | mriedem | do you have any ocata compute services running? | |
| 18:14:02 | jmlowe | no | |
| 18:14:38 | jmlowe | I get the warnings about only have pike won't update allocation in the nova compute logs | |
| 18:14:54 | mriedem | ok yeah that's what i was thinking of | |
| 18:17:04 | efried | jaypipes: Finally finished https://review.openstack.org/#/c/553597/ -- thanks for your patience | |
| 18:19:01 | mriedem | wonder if we have a bug here https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L5823 | |
| 18:19:47 | mriedem | that should only remove the instance's allocations against the source node after a successful live migration, | |
| 18:19:53 | mriedem | the dest host allocations are created during scheduling | |
| 18:59:37 | jmlowe | mriedem: not exactly the same thing, but along those lines http://paste.openstack.org/show/722290/ | |
| 19:00:35 | jmlowe | something doesn't add up | |
| 19:04:03 | mriedem | is instance 6c4e5035-8a42-4e37-8e90-3ba142ecd5aa running on r06c2b03? | |
| 19:06:05 | openstackgerrit | Merged openstack/nova master: Remove unused function https://review.openstack.org/570865 | |
| 19:07:17 | mriedem | jmlowe: what does "openstack resource provider inventory list b0f5d569-687f-43a2-9bdb-309a29597125" show? | |
| 19:08:29 | mriedem | looking at that allocation output, you have 3 instances running on that host with a total of 126976 memory mb consumed | |
| 19:08:52 | jmlowe | http://paste.openstack.org/show/722293/ | |
| 19:09:46 | mriedem | ok and that says there is 130850 mb total of memory | |
| 19:11:19 | mriedem | jaypipes: efried: should max_unit ever be > total - reserved? | |
| 19:11:41 | dansmith | mriedem: I would think it could be | |
| 19:11:42 | efried | mriedem: I think it should be allowed. | |
| 19:11:59 | dansmith | mriedem: consider the case of ironic | |
| 19:12:03 | dansmith | well, that'd be equal I guess | |
| 19:12:07 | efried | Because you should be able to change total and/or reserved without having to worry about also "fixing" max_unit at the same time. | |
| 19:12:12 | dansmith | yeah | |
| 19:12:18 | mriedem | fine fine | |
| 19:12:35 | dansmith | well, yeah, ironic would be total=1, reserved=1, max_unit=1, total-reserved=0 | |
| 19:12:48 | dansmith | for the cleaning state | |
| 19:14:22 | jmlowe | there are 3 instances running on that host, 158726ec-f09d-443c-a700-cce79cd065e0, 5053ebcc-9145-4297-b60f-f2c74ce95aef, 950be30e-a275-4533-9d42-53df6145545b | |
| 19:14:32 | mriedem | jmlowe: how many servers show up on that host if you do "nova hypervisor-servers r06c2b03"? | |
| 19:14:33 | mriedem | same 3? | |