| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-05 | |||
| 23:55:30 | eandersson | cfriesen_, > During the sync_power process the instance has moved from host X to host Y | |
| 23:55:36 | eandersson | Is all I see in the normal logs | |
| 23:55:55 | cfriesen_ | eandersson: mitaka, right? | |
| 23:55:58 | eandersson | yea | |
| 23:56:04 | eandersson | > CPUPinningInvalid: Cannot pin/unpin cpus [1, 21] from the following pinned set [0, 1, 20, 5, 25, 21] | |
| 23:57:15 | eandersson | It's complaning about that after it was moved | |
| #openstack-nova - 2018-04-06 | |||
| 00:00:33 | cfriesen_ | eandersson: you might try enabling debug logs and redoing the test. I'd expect that on the dest nova-compute rebuild_instance() would call ResourceTracker.rebuild_claim() | |
| 00:00:48 | cfriesen_ | eandersson: that in turn should emit logs like "Attempting claim: ....." | |
| 00:02:10 | cfriesen_ | Spaz-Home: no, it should claim resources for scheduler-chosen destinations as well since even now placement doesn't do cpu pinning, hugepages, etc. | |
| 00:02:52 | cfriesen_ | eandersson: just to be clear, the "Attempting claim" log should come out at "info" level | |
| 00:03:45 | eandersson | Is that in the compute logs? | |
| 00:03:54 | cfriesen_ | eandersson: yes, on the dest node | |
| 00:04:42 | eandersson | I only see three of those, and none from the evacuated nodes | |
| 00:05:08 | cfriesen_ | eandersson: that claim is what calls _test_numa_topology(), which is what handles setting up the CPU pinning for the instance that is being evacuated. | |
| 00:06:11 | openstackgerrit | Michael Still proposed openstack/nova master: Rename network.utils to network.linux_utils. https://review.openstack.org/559197 | |
| 00:07:33 | cfriesen_ | eandersson: I'd expect to see the "Attempting claim" log on instance boot as well. | |
| 00:09:55 | eandersson | > Creating Image | |
| 00:09:56 | eandersson | > Instance spawned successfully. | |
| 00:10:12 | eandersson | That is it for INFO logging | |
| 00:11:13 | eandersson | I'll enable debug logging and see in a bit | |
| 00:11:14 | cfriesen_ | do you see a "Rebuilding instance" log on the dest compute node when doing the evacuate? | |
| 00:16:14 | eandersson | Yea - missed that line > [instance: xxxx] Rebuilding instance | |
| 00:16:35 | eandersson | > disk not on shared storage, rebuilding from: 'yyyy' | |
| 00:16:53 | cfriesen_ | okay, so pretty much right after that I'd expect to see an "Attempting claim" log, followed by a "Claim successful" log | |
| 00:19:38 | cfriesen_ | if you want you could add extra logging in nova.compute.manager.ComputeManager.rebuild_instance() around where it sets "rebuild_claim" and then calls it. | |
| 00:21:08 | cfriesen_ | eandersson: crazy question maybe, but did you live-migrate any of these pinned instances? | |
| 00:21:26 | eandersson | nope | |
| 00:21:34 | eandersson | I created a lab and 5 new VMs. | |
| 00:21:41 | eandersson | with two computes | |
| 00:22:07 | eandersson | evacuated the two vms on compute2 | |
| 00:24:37 | eandersson | > Lock "compute_resources" acquired by "nova.compute.resource_tracker.rebuild_claim" | |
| 00:28:26 | cfriesen_ | eandersson: that's promising, it corresponds to the locking decorator on ResourceTracker.rebuild_claim | |
| 00:31:10 | cfriesen_ | eandersson: just had a thought...what virt driver are you using? | |
| 00:32:16 | cfriesen_ | eandersson: do you see a log like "Virt driver does not support 'get_available_resource'. Compute tracking is disabled." ? | |
| 00:34:07 | eandersson | nope | |
| 00:35:54 | eandersson | cfriesen_, default kvm | |
| 00:47:59 | eandersson | libvirt, qemu-kvm | |
| 01:31:37 | melwitt | logan-: here's the link http://logs.openstack.org/84/534384/24/check/tempest-full/ae551f8/ | |
| 01:58:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove non-libguestfs file injection for libvirt https://review.openstack.org/324720 | |
| 02:34:26 | gmann | alex_xu_: ping | |
| 02:45:22 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (3) https://review.openstack.org/559258 | |
| 04:36:59 | homeski | is osprofiler working in Nova --Newton? | |
| 04:37:09 | homeski | can't find any docs on it .. | |
| 04:53:30 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_user_data.py https://review.openstack.org/559264 | |
| 05:23:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/api/openstack/compute/test_hosts.py https://review.openstack.org/559266 | |
| 05:45:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/api/*/test_instance_actions.py https://review.openstack.org/559269 | |
| 06:53:42 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772 | |
| 07:11:34 | kashyap | zigo: Thanks. Will look in a few. | |
| 07:19:18 | openstackgerrit | Roman Dobosz proposed openstack/nova master: Pass hosts or nodes to HostStates update method depending on a driver. https://review.openstack.org/556761 | |
| 07:19:18 | openstackgerrit | Roman Dobosz proposed openstack/nova master: Added weight for aggregate soft (anti) affinity. https://review.openstack.org/556762 | |
| 07:19:19 | openstackgerrit | Roman Dobosz proposed openstack/nova master: Added node field for InstanceGroup objects https://review.openstack.org/559284 | |
| 08:32:00 | kaisers1 | mikal_: ping | |
| 08:46:27 | kashyap | It's often best to just ask the question -- when the person wakes up, they could respond directly to the question. | |
| 09:27:45 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Cleanup patch for the cell-disable series https://review.openstack.org/559299 | |
| 09:34:25 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add missing encryption_secret_uuid tests https://review.openstack.org/540679 | |
| 09:44:38 | kashyap | zigo: Hi there | |
| 09:45:18 | kashyap | zigo: I'm just about to respond to your email, although it's easy to use the versions in Stretch | |
| 09:45:33 | zigo | kashyap: Great ! :) | |
| 09:45:38 | kashyap | zigo: Thanks for the compilation trying; it looks like a fairly small one. | |
| 09:46:04 | kashyap | zigo: But -- it would be nice to use 3.2.0, as I just spent 30 minutes comparing the release notes of 3.0.0 and 3.2.0 | |
| 09:46:06 | zigo | As I wrote, right now, not much of a problem. In a year from now, I don't know ! | |
| 09:46:22 | zigo | kashyap: Oh, interesting. What's the added features? | |
| 09:46:23 | kashyap | That's why instead of such new versions from Buster / Sid | |
| 09:46:57 | kashyap | zigo: So, one of the core things we'd get with 3.2.0 is this: | |
| 09:47:21 | kashyap | zigo: https://www.redhat.com/archives/libvir-list/2017-February/msg01295.html -- "Detect host CPU model by askingQEMU on x86_64" | |
| 09:47:36 | kashyap | zigo: It fixes a ton of bugs in that area of CPU model comparision | |
| 09:47:47 | zigo | Oh ... | |
| 09:47:51 | zigo | I see, interesting indeed. | |
| 09:48:35 | zigo | Though that's bugs in libvirt itself, not really a new API feature, right? | |
| 09:49:01 | kashyap | zigo: Yes, indeed | |
| 09:49:09 | kashyap | zigo: Bug hang on, let me look at the details | |
| 09:49:55 | zigo | What I was refering was missing feature on the way of the enhancement of Nova. | |
| 09:49:57 | kashyap | zigo: Some new APIs too | |
| 09:50:44 | zigo | Is there any incompatibilities? | |
| 09:51:32 | zigo | If there are bugfixes, then I can try to do my best to get the latest version from Buster backported. But that's a whole of a different story to have this backport being mandatory because of API breaks. | |
| 09:51:33 | kashyap | zigo: That's a broad question, you mean API incompat? | |
| 09:51:36 | kashyap | zigo: No, not really | |
| 09:51:38 | zigo | Yeah... | |
| 09:52:03 | kashyap | zigo: Isn't it too much to backport the 'Buster' versions? Rather than introducing the bit more conservative 3.2.0 and 2.9.0 (QEMU)? | |
| 09:52:15 | zigo | If 3.2.0 is still backward compatible with 3.0.0, then just require 3.0.0 and I'll do what I can to get the backport done. It's just nicer to not have it mandatory. | |
| 09:52:30 | kashyap | zigo: libvirt guarantees API compatibility "forever" | |
| 09:52:35 | zigo | Oh ! | |
| 09:52:42 | zigo | So why are you willing to even require anything? | |
| 09:53:13 | kashyap | Wait, let me retract that :-) | |
| 09:53:26 | kashyap | Read here about API guarantees: https://libvirt.org/support.html | |
| 09:54:12 | zigo | Another thing: isn't Nova only interested in the python-libvirt API, rather than the engine behind? | |
| 09:54:12 | kashyap | zigo: I meant, the "forever" part applies to ABI compatibility from the primary public API | |
| 09:55:05 | zigo | " IOW they will be ABI compatible forever." <--- Read this, indeed ! | |
| 09:55:34 | zigo | That's the Linux kernel userland API policy (though here, ABI, which is even nicer). | |
| 09:55:38 | kashyap | zigo: Each Compute node runs libvirt daemon | |
| 09:56:19 | zigo | Linuz told Debian folks (at Debconf in portland) that he didn't understand why it wasn't a common practice on all the Linux echosystem, and I very much agree ! :) | |
| 09:58:00 | zigo | Ok, so libvirt C lib is super-stable, but is there anything new in 3.2.0 that you would need that isn't in 3.0.0 ? | |
| 09:59:05 | kashyap | zigo: Yeah, like I said 3.2.0 libvirt (in conjunction with QEMU 2.9.0) by default saves a ton of headache in the CPU modelling area | |
| 10:02:03 | kashyap | zigo: But if it's too much time and effort, it's alright, can settle on 3.0.0 & 2.8.0 | |
| 10:04:10 | zigo | Great, thanks. Just do that, and advise for 3.0.0, then I'll keep in mind that backporting is prefered. | |
| 10:04:38 | zigo | The real issue isn't "too much effort", right now, it's not. The issue is: I don't know if it's going to be hard or even impossible in one year from now. | |
| 10:04:54 | zigo | Anyway, thanks so much for your care. | |
| 10:07:23 | kashyap | zigo: Email sent. | |
| 10:26:55 | openstackgerrit | Chris Dent proposed openstack/nova master: Move test_report_client out of placement namespace https://review.openstack.org/558911 | |
| 11:58:43 | kashyap | Anyone also seeing these timeout issues: | |