| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-29 | |||
| 17:03:04 | yumapath | i followed this link , but not able to get things working | |
| 17:03:26 | yumapath | if anyone out there can help me that would be great | |
| 17:05:14 | dansmith | cfriesen: for what gain? | |
| 17:05:21 | dansmith | oh, | |
| 17:05:23 | dansmith | suspend like pause | |
| 17:05:29 | dansmith | so they don't take up extra space with the memory image? | |
| 17:12:31 | jmlowe | hmm, so it would seem out of 946 instances two were in the wrong place, about two dozen don't have allocations | |
| 17:14:14 | jmlowe | last instance action for the one in the wrong place was an errored live migration | |
| 17:15:06 | openstackgerrit | Merged openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640 | |
| 17:20:34 | cfriesen | dansmith: was away from keyboard. yeah, otherwise we'd need to reserve as much disk space as we have guest RAM, which might be an issue on small-disk compute nodes. | |
| 17:21:03 | dansmith | yeah, makes sense. unfortunately it changes the behavior the api user expects | |
| 17:23:44 | cfriesen | dansmith: agreed. though the practical difference between the two is minimal | |
| 17:24:28 | jaypipes | jmlowe: sorry, had a phone call... ok, sorry, hope you are ready for a stream of opinions coming from me... here goes... | |
| 17:24:49 | dansmith | cfriesen: well, if you're using pause for fast HA, the startup time may differ significantly, but also suspend should survive host reboots, power outages, etc | |
| 17:24:53 | dansmith | cfriesen: but yes, similar | |
| 17:25:09 | jaypipes | jmlowe: a) application state should *never* be saved in a VM disk image. it should always be saved only to a persistent volume that can be attached/detached as needed. | |
| 17:25:58 | jmlowe | great in theory, now fix 40 years of domain specific scientific code | |
| 17:26:13 | jaypipes | jmlowe: b) if you need persistent IP addresses, then those IP addresses should be managed outside of the scope of Nova (i.e. created in Neutron and attached to a Nova VM on boot). Then detached from the VM when no longer needed. | |
| 17:26:25 | jaypipes | jmlowe: like I said, my opinion... :) | |
| 17:27:44 | jmlowe | case in point, most domestic weather forecasting is done with WRF, there is exactly one guy in the world who still knows how it works, he was brought out of retirement at UCAR, there is a new model in the works but I don't think it's ready for primetime yet | |
| 17:27:52 | jaypipes | jmlowe: c) you will end up spending more money and time working around coupled app state and configuration instead of investing the time in decoupling the app configuration and state and treating all your VMs as cattle instead of pets. | |
| 17:29:43 | jaypipes | jmlowe: ok, I'm done with my opinion stream. :) I recognize there are real and unworkaroundble constraints that you and others suffer with. I'm just pointing out my experience and long-term strategy, nothing more. | |
| 17:29:55 | openstackgerrit | Merged openstack/nova stable/pike: Stringify instance UUID https://review.openstack.org/569421 | |
| 17:30:06 | jmlowe | I have thousands of pets, they technically aren't even mine, I'd guess of my ~2400 users 100 have the chops to rework their applications | |
| 17:30:27 | jaypipes | jmlowe: understood. | |
| 17:30:59 | jmlowe | I tackle them one at a time, in a few centuries I will have made major progress | |
| 17:31:07 | jaypipes | heh :) touche. | |
| 17:32:00 | jaypipes | yumapath: you will need to be more specific that "but not able to get things working"./ | |
| 17:32:08 | jaypipes | s/that/than | |
| 17:37:31 | openstackgerrit | Merged openstack/nova stable/queens: Fix shelving a paused instance https://review.openstack.org/569650 | |
| 17:39:13 | mriedem | cfriesen: are you aware of anything in the pci whitelist that must be case sensitive? https://review.openstack.org/#/c/565808/4/nova/pci/utils.py | |
| 17:47:35 | cfriesen | mriedem: not my area of expertise | |
| 17:48:01 | cfriesen | mriedem: our pci guy seems to be away at the moment. | |
| 18:03:21 | mriedem | jmlowe: ok so it sounds like what i expected, the failed live migration was probably before we patched pike to cleanup allocations properly in that case | |
| 18:04:02 | mriedem | jmlowe: any ideas on the ones that don't have allocations at all? | |
| 18:05:04 | jmlowe | I haven't really found a pattern, other than they existed during 16.0 vs our current 16.1 | |
| 18:05:16 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (3) https://review.openstack.org/569320 | |
| 18:05:31 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (4) https://review.openstack.org/570750 | |
| 18:05:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in libvirt/test_driver.py (5) https://review.openstack.org/570842 | |
| 18:08:53 | mriedem | jmlowe: did you look at the instance action records for the ones that don't have allocations? | |
| 18:09:25 | jmlowe | http://paste.openstack.org/show/722279/ | |
| 18:10:03 | mriedem | did req-e0d0edd7-74ee-40d0-b538-9f15f0242183 fail? | |
| 18:10:22 | jmlowe | NAFAICT | |
| 18:10:51 | jmlowe | failed ones tended to get double allocations | |
| 18:11:16 | jmlowe | or allocated on the wrong (probably target) host | |
| 18:11:53 | mriedem | can you do a "nova instance-action <server> req-e0d0edd7-74ee-40d0-b538-9f15f0242183" on that same instance from the paste? | |
| 18:12:56 | jmlowe | http://paste.openstack.org/show/722283/ | |
| 18:13:55 | mriedem | do you have any ocata compute services running? | |
| 18:14:02 | jmlowe | no | |
| 18:14:38 | jmlowe | I get the warnings about only have pike won't update allocation in the nova compute logs | |
| 18:14:54 | mriedem | ok yeah that's what i was thinking of | |
| 18:17:04 | efried | jaypipes: Finally finished https://review.openstack.org/#/c/553597/ -- thanks for your patience | |
| 18:19:01 | mriedem | wonder if we have a bug here https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L5823 | |
| 18:19:47 | mriedem | that should only remove the instance's allocations against the source node after a successful live migration, | |
| 18:19:53 | mriedem | the dest host allocations are created during scheduling | |
| 18:59:37 | jmlowe | mriedem: not exactly the same thing, but along those lines http://paste.openstack.org/show/722290/ | |
| 19:00:35 | jmlowe | something doesn't add up | |
| 19:04:03 | mriedem | is instance 6c4e5035-8a42-4e37-8e90-3ba142ecd5aa running on r06c2b03? | |
| 19:06:05 | openstackgerrit | Merged openstack/nova master: Remove unused function https://review.openstack.org/570865 | |
| 19:07:17 | mriedem | jmlowe: what does "openstack resource provider inventory list b0f5d569-687f-43a2-9bdb-309a29597125" show? | |
| 19:08:29 | mriedem | looking at that allocation output, you have 3 instances running on that host with a total of 126976 memory mb consumed | |
| 19:08:52 | jmlowe | http://paste.openstack.org/show/722293/ | |
| 19:09:46 | mriedem | ok and that says there is 130850 mb total of memory | |
| 19:11:19 | mriedem | jaypipes: efried: should max_unit ever be > total - reserved? | |
| 19:11:41 | dansmith | mriedem: I would think it could be | |
| 19:11:42 | efried | mriedem: I think it should be allowed. | |
| 19:11:59 | dansmith | mriedem: consider the case of ironic | |
| 19:12:03 | dansmith | well, that'd be equal I guess | |
| 19:12:07 | efried | Because you should be able to change total and/or reserved without having to worry about also "fixing" max_unit at the same time. | |
| 19:12:12 | dansmith | yeah | |
| 19:12:18 | mriedem | fine fine | |
| 19:12:35 | dansmith | well, yeah, ironic would be total=1, reserved=1, max_unit=1, total-reserved=0 | |
| 19:12:48 | dansmith | for the cleaning state | |
| 19:14:22 | jmlowe | there are 3 instances running on that host, 158726ec-f09d-443c-a700-cce79cd065e0, 5053ebcc-9145-4297-b60f-f2c74ce95aef, 950be30e-a275-4533-9d42-53df6145545b | |
| 19:14:32 | mriedem | jmlowe: how many servers show up on that host if you do "nova hypervisor-servers r06c2b03"? | |
| 19:14:33 | mriedem | same 3? | |
| 19:14:50 | jmlowe | yep, check nova and virsh | |
| 19:15:13 | jaypipes | efried: np | |
| 19:15:13 | mriedem | ok, i'm not sure why free_ram_mb shows up as -122863 | |
| 19:16:18 | jmlowe | this host did once have a allocation for an instance that wasn't present | |
| 19:16:30 | mriedem | based on my calculations you should have 17mb of free ram on that host | |
| 19:16:50 | mriedem | 130850 total - 3857 reserved - 126976 (sum memory mb of allocations of the 3 instances on that host) | |
| 19:17:24 | jmlowe | Is there some cache somewhere that needs updated? | |
| 19:18:26 | mriedem | don't think so, i'm probably missing something in the free ram calculation in the resource tracker, like what the virt driver computes for overhead | |
| 19:19:03 | mriedem | although the libvirt driver doesn't do anything wrt memory_mb overhead | |
| 19:19:21 | esberglu | melwitt: Not sure if you saw my message last week with the summit | |
| 19:19:32 | esberglu | PowerVM vSCSI CI is enabled | |
| 19:19:46 | esberglu | So you should be able to remove the procedural -2 from https://review.openstack.org/#/c/526094/ | |
| 19:21:59 | mriedem | jmlowe: i see this comment but i'm not sure what it means about policy https://github.com/openstack/nova/blob/stable/pike/nova/compute/resource_tracker.py#L913 | |
| 19:22:16 | mriedem | i assume oversubscription? | |
| 19:22:22 | cfriesen | mriedem: jmlowe: is anything else on that compute node consuming ram outside of openstack/qemu/etc? | |
| 19:22:55 | jmlowe | free shows 82G free | |
| 19:28:36 | cfriesen | jmlowe: I think that's largely irrelevent for 4KB pages since they're only allocated on demand. | |
| 19:29:07 | dansmith | cfriesen: other stuff on the box consuming memory shouldn't be represented in placement in any way | |
| 19:29:13 | dansmith | other than reserved if configured in nova | |
| 19:32:05 | melwitt | esberglu: yep, just need to take a look through and will remove the -2 | |
| 19:32:24 | esberglu | Sounds good thanks | |
| 19:33:00 | cfriesen | libvirt.host.Host.get_memory_mb_used() looks at actual memory usage on the host | |
| 19:33:06 | cfriesen | dansmith: ^ | |
| 19:34:03 | dansmith | cfriesen: right but that shouldn't be what we're reporting as reserved | |