| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-22 | |||
| 11:57:40 | openstackgerrit | Alex Xu proposed openstack/nova master: Populates the existing mediated devices in the libvirt device manager https://review.opendev.org/670787 | |
| 11:57:40 | openstackgerrit | Alex Xu proposed openstack/nova master: Add DeviceManager to the libvirt virt driver https://review.opendev.org/671388 | |
| 11:57:41 | openstackgerrit | Alex Xu proposed openstack/nova master: Adds functional test for creating the instance with vgpus https://review.opendev.org/671398 | |
| 11:57:41 | openstackgerrit | Alex Xu proposed openstack/nova master: Using the claim/unclaim_for_instance for mdevs https://review.opendev.org/671222 | |
| 12:01:44 | ygk_12345 | hi all | |
| 12:02:53 | ygk_12345 | some of the vms on the compute node are eating too much of memory. Can someone point me to any links for finding out what is happening in those vms from the hypervisro prespective ? | |
| 12:18:01 | TheJulia | sean-k-mooney: I never get enough sleep and if I don't take meds It would be even worse... so I always needssleep :) | |
| 12:18:33 | mdbooth | ygk_12345: You're unlikely to find the right expertise here for that. Assuming you're using libvirt/kvm you'll want to talk to a libvirt/kvm forum. Or your vendor... | |
| 12:19:08 | ygk_12345 | mdbooth is there any irc channel for kvm in general ? | |
| 12:23:15 | openstackgerrit | sean mooney proposed openstack/nova master: Libvirt: add support for vPMU configuration. https://review.opendev.org/671338 | |
| 12:23:27 | mdbooth | ygk_12345: Try #virt on irc.oftc.net | |
| 12:23:40 | ygk_12345 | mdbooth ok thanks | |
| 12:24:32 | sean-k-mooney | ygk_12345: how much memory are they consuming over what you execpt | |
| 12:25:36 | ygk_12345 | sean-k-mooney what I observer is in general the cpu usage against those qemu-kvm processes is shooting above 150 sometimes and so I am suspecting them. If I shut them down it is normal again | |
| 12:26:20 | sean-k-mooney | ygk_12345: are you seeing high cpu usage, or high memory usage or both | |
| 12:26:26 | ygk_12345 | both | |
| 12:26:30 | ygk_12345 | sometimes | |
| 12:26:49 | ygk_12345 | how to find out whats exactly happening at those times ? | |
| 12:28:30 | sean-k-mooney | you would likely need to look at the guest journal/logs to be honest but you could check the host for kernel errors | |
| 12:28:41 | ygk_12345 | hmm | |
| 12:28:48 | sean-k-mooney | e.g. check if the kernel is soft locking a core | |
| 12:29:12 | sean-k-mooney | that could be caused by the guest soft locking up | |
| 12:29:21 | ygk_12345 | ok | |
| 12:29:45 | sean-k-mooney | are you currently using q35 machine type or are your still using the pc i440 machine type | |
| 12:30:04 | ygk_12345 | we use dell with latest intel | |
| 12:30:07 | sean-k-mooney | we have seen that q35 while more modern tends to use more memory | |
| 12:30:17 | ygk_12345 | oh ok | |
| 12:49:24 | boxiang_ | https://review.opendev.org/#/c/649963/ and https://review.opendev.org/#/c/651969/ Can someone help review these two patches? thanks :) | |
| 13:00:06 | efried | alex_xu: I don't understand. Why would we not use the migration consumer to migrate? | |
| 13:08:57 | sean-k-mooney | gibi_off: not sure if your are on vaction/PTO this week but just an fyi that i should have adressed your feedback in https://review.opendev.org/671338 | |
| 13:14:11 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Restore RT.old_resources if ComputeNode.save() fails https://review.opendev.org/672038 | |
| 13:14:59 | sean-k-mooney | efried: i am also confused. isnt a migration consumer proposed for use when doing cold and live migrations to hold the allocation on the second node | |
| 13:15:23 | efried | sean-k-mooney: He's off for two weeks. It looks like the changes are minor; if so another core will probably proxy his +2. | |
| 13:15:30 | efried | sean-k-mooney: that's my confusion as well. | |
| 13:15:42 | efried | perhaps we only use the migration consumer for live migration? | |
| 13:15:44 | efried | I don't know. | |
| 13:15:52 | sean-k-mooney | efried: ah ok good to know. | |
| 13:15:54 | mriedem | they're used for both cold and live | |
| 13:16:13 | mriedem | just not evacuate | |
| 13:16:19 | sean-k-mooney | mriedem: o/ welcome back, have a good vaction? | |
| 13:16:25 | mriedem | thanks, yeah | |
| 13:16:47 | sean-k-mooney | we dont use them for evac because we dont use them in rebuild | |
| 13:17:17 | mriedem | um | |
| 13:17:23 | mriedem | that's not really the reason | |
| 13:17:48 | mriedem | it's because for evac the source compute service is down, so it's not really the same flow for how the migration-based allocations are handled at the end | |
| 13:18:06 | mriedem | we have migration records for evac | |
| 13:18:25 | alex_xu | efried: if we use migration as consumer for the dst, when we confirm the resize, there isn't a call to dest host to change migration consumer back to instance consumer | |
| 13:18:35 | sean-k-mooney | right however event with a down host we could still free the allocation on the source node in placment | |
| 13:18:52 | mriedem | sean-k-mooney: we do that when the source compute service comes back up | |
| 13:19:12 | alex_xu | efried: if we use migration as consumer for the src, we will call the the dest host to claim the device, so we have no way to change the src host's claim consumer to migration. | |
| 13:19:12 | efried | alex_xu: oh, you're talking about swapping which side gets which consumer? | |
| 13:19:14 | sean-k-mooney | yes that is true. | |
| 13:19:18 | alex_xu | efried: yes | |
| 13:19:34 | mriedem | we *could* use migration-based consumers for evac, i just don't think it was really a priority when dansmith implemented that blueprint | |
| 13:20:04 | mriedem | i'm obviously walking into something late here though | |
| 13:20:09 | mriedem | who is out for 2 weeks? | |
| 13:20:13 | efried | gibi_off: | |
| 13:20:31 | efried | mriedem: but that's a separate topic | |
| 13:20:44 | mriedem | ok, which series are you talking about? | |
| 13:21:06 | efried | mriedem: we're talking about alex_xu's work to make hypervisor-specific claiming (atm for VGPUs and VPMEMs) happen in the virt driver. | |
| 13:21:23 | alex_xu | mriedem: https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:claim_for_instance | |
| 13:22:09 | mriedem | ok and this is prep work for vpmem i guess | |
| 13:22:23 | mriedem | claims do happen in the rt today | |
| 13:22:31 | mriedem | the only hypervisor-specific thing though is the overhead stuff | |
| 13:22:57 | alex_xu | mriedem: vgpu allocation has race problem also, in libvirt, we allocate vgpu out of rt, that means no lock | |
| 13:23:45 | efried | also, it just makes sense for hypervisor-specific claiming to happen in the virt driver. | |
| 13:24:31 | mriedem | alex_xu: it happens during driver.spawn correct? | |
| 13:24:41 | efried | So tldr we're adding a hook from the various *_claimZ to call into a new ComputeDriver interface which interprets allocations etc to earmark the real devices corresponding to the allocations | |
| 13:24:42 | openstack | Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj) | |
| 13:24:42 | alex_xu | mriedem: yes, https://bugs.launchpad.net/nova/+bug/1836204 | |
| 13:25:43 | mriedem | i guess we're not modeling each specific device in placement right? just the total number of available in inventory, and then it's up to the virt driver to pick one, which could conflict. | |
| 13:26:04 | efried | exactly | |
| 13:26:21 | mriedem | weee, so much for ever getting rid of the rt claims code... | |
| 13:26:59 | efried | The RT claims *framework* will need to stay, but IMO eventually "all" of the guts should wind up moving into this ComputeDriver.claim_for_instance | |
| 13:27:11 | efried | especially for things like PCI devices | |
| 13:27:18 | efried | and NUMA | |
| 13:27:44 | efried | the line where it makes sense to cut things over is when we move to tracking a resource in placement. | |
| 13:28:02 | efried | the messy ones obviously being VCPU, MEMORY_MB, DISK_GB | |
| 13:28:53 | sean-k-mooney | efried: well placement just keeps a tally count. if we need to do device assignment we need to continue to do tracking the the RT | |
| 13:29:03 | efried | yup | |
| 13:29:25 | efried | where "device" will eventually encompass NUMA-split proc/mem | |
| 13:29:50 | sean-k-mooney | ram and cpus are just devices in disgiuse | |
| 13:30:32 | alex_xu | I have too much fun with same host resize today | |
| 13:30:58 | efried | bbiab | |
| 13:36:47 | stephenfin | mriedem, efried: I'm hitting the boundaries of my resource tracking know how and could do with some help | |
| 13:37:03 | stephenfin | I've got this patch to start tracking pcpus as a ComputeNode/HostState field https://review.opendev.org/#/c/671794/ | |
| 13:37:47 | stephenfin | It's not complete, and it's also breaking some functional/tempest tests because one of those objects is being spat out for the os-hypervisors API | |
| 13:38:42 | stephenfin | However, does any of the resource tracking claims stuff matter for vcpus, ram and disk anymore, given we're actually requesting things from placement? | |
| 13:39:25 | openstackgerrit | sahid proposed openstack/nova master: cellv2: make update_cell to support cell0 https://review.opendev.org/672045 | |
| 13:39:28 | stephenfin | Referring specifically to this https://github.com/openstack/nova/blob/master/nova/compute/claims.py#L97-L109 | |
| 13:40:01 | mriedem | they used to matter for scheduler drivers that didn't use placement, like the caching scheduler, but that's gone now, | |
| 13:40:09 | mriedem | the overhead stuff comes from the driver and that's still part of the claim | |
| 13:40:17 | mriedem | i know the libvirt driver claims overhead in certain case | |
| 13:40:18 | mriedem | *cases | |
| 13:40:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L884 | |
| 13:40:47 | stephenfin | Yup, for emulator threads, though I do have a chance to stop doing that now with the PCPU stuff | |
| 13:41:08 | mriedem | there are other drivers that implement that method as well | |
| 13:41:10 | mriedem | hyperv i think is one | |
| 13:41:46 | mriedem | https://github.com/openstack/nova/blob/master/nova/virt/hyperv/vmops.py#L119 | |
| 13:42:43 | mriedem | besides the overhead stuff i'm not sure anything cares about vcpus/ram/disk claims in the rt anymore since that should be handled by the scheduler + placement now | |