| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-21 | |||
| 13:52:27 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start reporting PCPU inventory to placement https://review.opendev.org/671793 | |
| 13:52:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove unused function parameter https://review.opendev.org/671796 | |
| 13:52:28 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Rename exception argument https://review.opendev.org/671795 | |
| 13:52:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.host_topology_and_format_from_host' https://review.opendev.org/671798 | |
| 13:52:29 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.get_host_numa_usage_from_instance' https://review.opendev.org/671797 | |
| 13:52:30 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: hardware: Differentiate between shared and dedicated CPUs https://review.opendev.org/671800 | |
| 13:52:30 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'hardware.instance_topology_from_instance' https://review.opendev.org/671799 | |
| 13:52:31 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support translating CPU policy extra specs, image meta https://review.opendev.org/671801 | |
| #openstack-nova - 2019-07-22 | |||
| 01:31:50 | brinzhang | lyarwood: After you review this spec[1] 5/20, now this spec wait for your update, when do you have time to do it? [1]https://review.opendev.org/#/c/580336/ | |
| 05:47:52 | alex_xu | efried: bauzas Luyao found a problem https://review.opendev.org/#/c/671222/4/nova/virt/libvirt/device.py@119, I guess we should use the migration as the consumer when migrate, just like we did for the placement | |
| 07:32:57 | openstackgerrit | wangwei1 proposed openstack/nova master: fix spelling error in nova/api/validation/__init__.py https://review.opendev.org/669244 | |
| 08:03:05 | bauzas | alex_xu: ack, I'll need to look | |
| 09:00:57 | alex_xu | bauzas: thanks, after take a look the detail, I don't think the migration uuid works. in the end, I'm thinking passing the flavor id to the claim/unclaim_for_instance. Then claim/unclaim based on the (instance_uuid, flavor_id) | |
| 09:26:00 | openstackgerrit | **** proposed openstack/nova master: Nova: node should be deleted when last service is deleted https://review.opendev.org/671731 | |
| 11:29:23 | sean-k-mooney | stephenfin: when you get back form lunch we have killed the cacheing schduler so maybe we should delete the core, ram and disk filters then nuke the filed for the resouce tracker? | |
| 11:30:20 | sean-k-mooney | it would be a change to the hyperviors api but you could always proxy the data from placement. | |
| 11:37:25 | openstackgerrit | **** proposed openstack/nova master: Nova: node should be deleted when last service is deleted https://review.opendev.org/671731 | |
| 11:41:04 | sean-k-mooney | johnthetubaguy: lyarwood can ye take a look at this stable backport https://review.opendev.org/#/c/671532/ if ye have time | |
| 11:57:38 | openstackgerrit | Alex Xu proposed openstack/nova master: Add the virt driver interface for claim and unclaim the devices https://review.opendev.org/670782 | |
| 11:57:39 | openstackgerrit | Alex Xu proposed openstack/nova master: Calling the virt driver's claim/unclaim_for_instance in resource tracker https://review.opendev.org/670784 | |
| 11:57:39 | openstackgerrit | Alex Xu proposed openstack/nova master: Moves the allocation retrieving early https://review.opendev.org/670783 | |
| 11:57:40 | openstackgerrit | Alex Xu proposed openstack/nova master: Populates the existing mediated devices in the libvirt device manager https://review.opendev.org/670787 | |
| 11:57:40 | openstackgerrit | Alex Xu proposed openstack/nova master: Add DeviceManager to the libvirt virt driver https://review.opendev.org/671388 | |
| 11:57:41 | openstackgerrit | Alex Xu proposed openstack/nova master: Adds functional test for creating the instance with vgpus https://review.opendev.org/671398 | |
| 11:57:41 | openstackgerrit | Alex Xu proposed openstack/nova master: Using the claim/unclaim_for_instance for mdevs https://review.opendev.org/671222 | |
| 12:01:44 | ygk_12345 | hi all | |
| 12:02:53 | ygk_12345 | some of the vms on the compute node are eating too much of memory. Can someone point me to any links for finding out what is happening in those vms from the hypervisro prespective ? | |
| 12:18:01 | TheJulia | sean-k-mooney: I never get enough sleep and if I don't take meds It would be even worse... so I always needssleep :) | |
| 12:18:33 | mdbooth | ygk_12345: You're unlikely to find the right expertise here for that. Assuming you're using libvirt/kvm you'll want to talk to a libvirt/kvm forum. Or your vendor... | |
| 12:19:08 | ygk_12345 | mdbooth is there any irc channel for kvm in general ? | |
| 12:23:15 | openstackgerrit | sean mooney proposed openstack/nova master: Libvirt: add support for vPMU configuration. https://review.opendev.org/671338 | |
| 12:23:27 | mdbooth | ygk_12345: Try #virt on irc.oftc.net | |
| 12:23:40 | ygk_12345 | mdbooth ok thanks | |
| 12:24:32 | sean-k-mooney | ygk_12345: how much memory are they consuming over what you execpt | |
| 12:25:36 | ygk_12345 | sean-k-mooney what I observer is in general the cpu usage against those qemu-kvm processes is shooting above 150 sometimes and so I am suspecting them. If I shut them down it is normal again | |
| 12:26:20 | sean-k-mooney | ygk_12345: are you seeing high cpu usage, or high memory usage or both | |
| 12:26:26 | ygk_12345 | both | |
| 12:26:30 | ygk_12345 | sometimes | |
| 12:26:49 | ygk_12345 | how to find out whats exactly happening at those times ? | |
| 12:28:30 | sean-k-mooney | you would likely need to look at the guest journal/logs to be honest but you could check the host for kernel errors | |
| 12:28:41 | ygk_12345 | hmm | |
| 12:28:48 | sean-k-mooney | e.g. check if the kernel is soft locking a core | |
| 12:29:12 | sean-k-mooney | that could be caused by the guest soft locking up | |
| 12:29:21 | ygk_12345 | ok | |
| 12:29:45 | sean-k-mooney | are you currently using q35 machine type or are your still using the pc i440 machine type | |
| 12:30:04 | ygk_12345 | we use dell with latest intel | |
| 12:30:07 | sean-k-mooney | we have seen that q35 while more modern tends to use more memory | |
| 12:30:17 | ygk_12345 | oh ok | |
| 12:49:24 | boxiang_ | https://review.opendev.org/#/c/649963/ and https://review.opendev.org/#/c/651969/ Can someone help review these two patches? thanks :) | |
| 13:00:06 | efried | alex_xu: I don't understand. Why would we not use the migration consumer to migrate? | |
| 13:08:57 | sean-k-mooney | gibi_off: not sure if your are on vaction/PTO this week but just an fyi that i should have adressed your feedback in https://review.opendev.org/671338 | |
| 13:14:11 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Restore RT.old_resources if ComputeNode.save() fails https://review.opendev.org/672038 | |
| 13:14:59 | sean-k-mooney | efried: i am also confused. isnt a migration consumer proposed for use when doing cold and live migrations to hold the allocation on the second node | |
| 13:15:23 | efried | sean-k-mooney: He's off for two weeks. It looks like the changes are minor; if so another core will probably proxy his +2. | |
| 13:15:30 | efried | sean-k-mooney: that's my confusion as well. | |
| 13:15:42 | efried | perhaps we only use the migration consumer for live migration? | |
| 13:15:44 | efried | I don't know. | |
| 13:15:52 | sean-k-mooney | efried: ah ok good to know. | |
| 13:15:54 | mriedem | they're used for both cold and live | |
| 13:16:13 | mriedem | just not evacuate | |
| 13:16:19 | sean-k-mooney | mriedem: o/ welcome back, have a good vaction? | |
| 13:16:25 | mriedem | thanks, yeah | |
| 13:16:47 | sean-k-mooney | we dont use them for evac because we dont use them in rebuild | |
| 13:17:17 | mriedem | um | |
| 13:17:23 | mriedem | that's not really the reason | |
| 13:17:48 | mriedem | it's because for evac the source compute service is down, so it's not really the same flow for how the migration-based allocations are handled at the end | |
| 13:18:06 | mriedem | we have migration records for evac | |
| 13:18:25 | alex_xu | efried: if we use migration as consumer for the dst, when we confirm the resize, there isn't a call to dest host to change migration consumer back to instance consumer | |
| 13:18:35 | sean-k-mooney | right however event with a down host we could still free the allocation on the source node in placment | |
| 13:18:52 | mriedem | sean-k-mooney: we do that when the source compute service comes back up | |
| 13:19:12 | alex_xu | efried: if we use migration as consumer for the src, we will call the the dest host to claim the device, so we have no way to change the src host's claim consumer to migration. | |
| 13:19:12 | efried | alex_xu: oh, you're talking about swapping which side gets which consumer? | |
| 13:19:14 | sean-k-mooney | yes that is true. | |
| 13:19:18 | alex_xu | efried: yes | |
| 13:19:34 | mriedem | we *could* use migration-based consumers for evac, i just don't think it was really a priority when dansmith implemented that blueprint | |
| 13:20:04 | mriedem | i'm obviously walking into something late here though | |
| 13:20:09 | mriedem | who is out for 2 weeks? | |
| 13:20:13 | efried | gibi_off: | |
| 13:20:31 | efried | mriedem: but that's a separate topic | |
| 13:20:44 | mriedem | ok, which series are you talking about? | |
| 13:21:06 | efried | mriedem: we're talking about alex_xu's work to make hypervisor-specific claiming (atm for VGPUs and VPMEMs) happen in the virt driver. | |
| 13:21:23 | alex_xu | mriedem: https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:claim_for_instance | |
| 13:22:09 | mriedem | ok and this is prep work for vpmem i guess | |
| 13:22:23 | mriedem | claims do happen in the rt today | |
| 13:22:31 | mriedem | the only hypervisor-specific thing though is the overhead stuff | |
| 13:22:57 | alex_xu | mriedem: vgpu allocation has race problem also, in libvirt, we allocate vgpu out of rt, that means no lock | |
| 13:23:45 | efried | also, it just makes sense for hypervisor-specific claiming to happen in the virt driver. | |
| 13:24:31 | mriedem | alex_xu: it happens during driver.spawn correct? | |
| 13:24:41 | efried | So tldr we're adding a hook from the various *_claimZ to call into a new ComputeDriver interface which interprets allocations etc to earmark the real devices corresponding to the allocations | |
| 13:24:42 | openstack | Launchpad bug 1836204 in OpenStack Compute (nova) "The allocation of VGPU has race problem" [High,Triaged] - Assigned to Alex Xu (xuhj) | |
| 13:24:42 | alex_xu | mriedem: yes, https://bugs.launchpad.net/nova/+bug/1836204 | |
| 13:25:43 | mriedem | i guess we're not modeling each specific device in placement right? just the total number of available in inventory, and then it's up to the virt driver to pick one, which could conflict. | |
| 13:26:04 | efried | exactly | |
| 13:26:21 | mriedem | weee, so much for ever getting rid of the rt claims code... | |
| 13:26:59 | efried | The RT claims *framework* will need to stay, but IMO eventually "all" of the guts should wind up moving into this ComputeDriver.claim_for_instance | |
| 13:27:11 | efried | especially for things like PCI devices | |
| 13:27:18 | efried | and NUMA | |
| 13:27:44 | efried | the line where it makes sense to cut things over is when we move to tracking a resource in placement. | |
| 13:28:02 | efried | the messy ones obviously being VCPU, MEMORY_MB, DISK_GB | |
| 13:28:53 | sean-k-mooney | efried: well placement just keeps a tally count. if we need to do device assignment we need to continue to do tracking the the RT | |