| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-05 | |||
| 21:00:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: libvirt: use dest host vif migrate details for live migration https://review.openstack.org/551370 | |
| 21:00:04 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: compute: use port binding extended API during live migration https://review.openstack.org/551371 | |
| 21:00:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Port binding based on events during live migration https://review.openstack.org/434870 | |
| 21:00:05 | openstackgerrit | Matt Riedemann proposed openstack/nova master: conductor: use port binding extended API in during live migrate https://review.openstack.org/522537 | |
| 21:03:16 | melwitt | owalsh: thanks. I misunderstood you earlier, I thought there was a second LP bug you had opened about the other issue | |
| 21:03:36 | openstackgerrit | Merged openstack/nova master: Make generation optional in ProviderTree https://review.openstack.org/539324 | |
| 21:04:12 | openstackgerrit | Tyler Blakeslee proposed openstack/nova master: Use six.text_type instead of str in NovaException __repr__ https://review.openstack.org/559169 | |
| 21:04:25 | owalsh | melwitt: ah, yea, not had time to investigate it yet... | |
| 21:04:55 | melwitt | yeah, no worries. just explaining why I said, "link me the LP" because I thought there was a second LP :) | |
| 21:05:00 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: test live_migration_wait_for_vif_plug=True https://review.openstack.org/558006 | |
| 21:05:39 | melwitt | you were probably like, "what ... link the same bug you told me about?" | |
| 21:06:04 | melwitt | haha | |
| 21:17:35 | arvindn05 | i am having issues running tox -epep8. Was working fine unti it wasnt... | |
| 21:17:54 | arvindn05 | gettting the error netifaces.c:1:20: fatal error: Python.h: No such file or directory | |
| 21:18:04 | arvindn05 | anyone saw this error before? | |
| 21:20:28 | melwitt | we recently switched it to run with python3 by default | |
| 21:20:49 | melwitt | so maybe you need the python3-dev package, or the like | |
| 21:21:47 | openstackgerrit | Arvind Nadendla proposed openstack/nova master: Update ImageMetaProp object to expose traits https://review.openstack.org/557795 | |
| 21:24:15 | arvindn05 | melwitt: thanks...i tried that based on similar recommendation | |
| 21:24:18 | arvindn05 | The following packages have unmet dependencies: | |
| 21:24:18 | arvindn05 | python3-dev : Depends: libpython3-dev (= 3.5.1-3) but it is not going to be installed | |
| 21:24:19 | arvindn05 | Depends: python3.5-dev (>= 3.5.1-2~) but it is not going to be installed | |
| 21:24:19 | arvindn05 | E: Unable to correct problems, you have held broken packages. | |
| 21:24:55 | arvindn05 | unable to install them because it says i have unmet dependencies...i looked for broken packages...but no luck | |
| 21:29:01 | arvindn05 | melwitt: nevermind got it to work. Thanks for the hint | |
| 21:29:20 | arvindn05 | had to use a different package manager aptitude to resolve the dependencies | |
| 21:29:45 | melwitt | k, cool | |
| 21:32:56 | openstackgerrit | Merged openstack/nova master: [placement] Fix incorrect exception import https://review.openstack.org/558916 | |
| 21:47:43 | melwitt | looks like limestone hosts still timing out | |
| 22:27:53 | openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi disk resizing to privsep. https://review.openstack.org/552242 | |
| 22:27:54 | openstackgerrit | Michael Still proposed openstack/nova master: Sync xenapi and libvirt on what flags to pass e2fsck. https://review.openstack.org/554078 | |
| 22:27:54 | openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi partition copies to privsep. https://review.openstack.org/553605 | |
| 22:27:55 | openstackgerrit | Michael Still proposed openstack/nova master: Move image conversion to privsep. https://review.openstack.org/554437 | |
| 22:27:55 | openstackgerrit | Michael Still proposed openstack/nova master: We don't need utils.trycmd any more. https://review.openstack.org/554439 | |
| 22:27:56 | openstackgerrit | Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | |
| 22:31:40 | logan- | melwitt: would you mind linking me to the job log? im not seeing the fail in logstash. we (limestone) are actively monitoring and fixing the timeouts noticed yesterday and the failure rate has been much lower today. | |
| 22:33:44 | openstackgerrit | Michael Still proposed openstack/nova master: Use a pythonic delete, with a retry. https://review.openstack.org/554793 | |
| 22:51:55 | cfriesen_ | eandersson: I was off working other stuff today, didn't see your messages. If you have questions abut numa stuff, feel free to ping me. Just wondering, do you have the NUMATopologyFilter enabled in the "scheduler_default_filters" config option in nova.conf? | |
| 23:01:42 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: placement: Granular GET /allocation_candidates https://review.openstack.org/517757 | |
| 23:04:13 | mriedem | cfriesen_: i think the net is https://bugs.launchpad.net/nova/+bug/1417667/comments/45 | |
| 23:04:13 | openstack | Launchpad bug 1417667 in OpenStack Compute (nova) "migration/evacuation/resize/unshelve of instance with NUMA topology needs to recalculate NUMA topology" [Medium,In progress] - Assigned to sahid (sahid-ferdjaoui) | |
| 23:04:38 | mriedem | cfriesen_: ah you replied | |
| 23:04:48 | mriedem | ok, so i think we should probably close that bug | |
| 23:08:03 | mriedem | mikal: fyi, i'm cleaning up https://review.openstack.org/#/c/324720/ | |
| 23:11:43 | eandersson | cfriesen_, yea it's enabled | |
| 23:15:15 | openstackgerrit | Arvind Nadendla proposed openstack/nova master: Update ImageMetaProp object to expose traits https://review.openstack.org/557795 | |
| 23:16:14 | openstackgerrit | Merged openstack/nova master: Add --enable and --disable options to nova-manage update_cell https://review.openstack.org/555416 | |
| 23:16:25 | openstackgerrit | Merged openstack/nova master: Update the cells FAQs and scheduler maintenance docs. https://review.openstack.org/556459 | |
| 23:16:53 | arvindn05 | mriedem: added the check to unit test for the image traits review | |
| 23:20:37 | Spaz-Home | Morning folks | |
| 23:27:55 | openstackgerrit | Merged openstack/nova master: update_provider_tree devref and docstring updates https://review.openstack.org/553476 | |
| 23:30:18 | cfriesen_ | mriedem: I guess we've got https://bugs.launchpad.net/nova/+bug/1289064 to track the live migration issues, so yeah I'm okay with closing that one. | |
| 23:30:18 | openstack | Launchpad bug 1289064 in OpenStack Compute (nova) "live migration of instance should claim resources on target compute node" [Medium,In progress] - Assigned to sahid (sahid-ferdjaoui) | |
| 23:33:42 | cfriesen_ | eandersson: in that case I'm a bit surprised. Do you see resource claims being made on the destination? | |
| 23:52:40 | Spaz-Home | Interesting.. does it only do those claims if a host is specified? | |
| 23:55:30 | eandersson | cfriesen_, > During the sync_power process the instance has moved from host X to host Y | |
| 23:55:36 | eandersson | Is all I see in the normal logs | |
| 23:55:55 | cfriesen_ | eandersson: mitaka, right? | |
| 23:55:58 | eandersson | yea | |
| 23:56:04 | eandersson | > CPUPinningInvalid: Cannot pin/unpin cpus [1, 21] from the following pinned set [0, 1, 20, 5, 25, 21] | |
| 23:57:15 | eandersson | It's complaning about that after it was moved | |
| #openstack-nova - 2018-04-06 | |||
| 00:00:33 | cfriesen_ | eandersson: you might try enabling debug logs and redoing the test. I'd expect that on the dest nova-compute rebuild_instance() would call ResourceTracker.rebuild_claim() | |
| 00:00:48 | cfriesen_ | eandersson: that in turn should emit logs like "Attempting claim: ....." | |
| 00:02:10 | cfriesen_ | Spaz-Home: no, it should claim resources for scheduler-chosen destinations as well since even now placement doesn't do cpu pinning, hugepages, etc. | |
| 00:02:52 | cfriesen_ | eandersson: just to be clear, the "Attempting claim" log should come out at "info" level | |
| 00:03:45 | eandersson | Is that in the compute logs? | |
| 00:03:54 | cfriesen_ | eandersson: yes, on the dest node | |
| 00:04:42 | eandersson | I only see three of those, and none from the evacuated nodes | |
| 00:05:08 | cfriesen_ | eandersson: that claim is what calls _test_numa_topology(), which is what handles setting up the CPU pinning for the instance that is being evacuated. | |
| 00:06:11 | openstackgerrit | Michael Still proposed openstack/nova master: Rename network.utils to network.linux_utils. https://review.openstack.org/559197 | |
| 00:07:33 | cfriesen_ | eandersson: I'd expect to see the "Attempting claim" log on instance boot as well. | |
| 00:09:55 | eandersson | > Creating Image | |
| 00:09:56 | eandersson | > Instance spawned successfully. | |
| 00:10:12 | eandersson | That is it for INFO logging | |
| 00:11:13 | eandersson | I'll enable debug logging and see in a bit | |
| 00:11:14 | cfriesen_ | do you see a "Rebuilding instance" log on the dest compute node when doing the evacuate? | |
| 00:16:14 | eandersson | Yea - missed that line > [instance: xxxx] Rebuilding instance | |
| 00:16:35 | eandersson | > disk not on shared storage, rebuilding from: 'yyyy' | |
| 00:16:53 | cfriesen_ | okay, so pretty much right after that I'd expect to see an "Attempting claim" log, followed by a "Claim successful" log | |
| 00:19:38 | cfriesen_ | if you want you could add extra logging in nova.compute.manager.ComputeManager.rebuild_instance() around where it sets "rebuild_claim" and then calls it. | |
| 00:21:08 | cfriesen_ | eandersson: crazy question maybe, but did you live-migrate any of these pinned instances? | |
| 00:21:26 | eandersson | nope | |
| 00:21:34 | eandersson | I created a lab and 5 new VMs. | |
| 00:21:41 | eandersson | with two computes | |
| 00:22:07 | eandersson | evacuated the two vms on compute2 | |
| 00:24:37 | eandersson | > Lock "compute_resources" acquired by "nova.compute.resource_tracker.rebuild_claim" | |
| 00:28:26 | cfriesen_ | eandersson: that's promising, it corresponds to the locking decorator on ResourceTracker.rebuild_claim | |
| 00:31:10 | cfriesen_ | eandersson: just had a thought...what virt driver are you using? | |
| 00:32:16 | cfriesen_ | eandersson: do you see a log like "Virt driver does not support 'get_available_resource'. Compute tracking is disabled." ? | |
| 00:34:07 | eandersson | nope | |
| 00:35:54 | eandersson | cfriesen_, default kvm | |
| 00:47:59 | eandersson | libvirt, qemu-kvm | |
| 01:31:37 | melwitt | logan-: here's the link http://logs.openstack.org/84/534384/24/check/tempest-full/ae551f8/ | |
| 01:58:28 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove non-libguestfs file injection for libvirt https://review.openstack.org/324720 | |
| 02:34:26 | gmann | alex_xu_: ping | |
| 02:45:22 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vm_utils.py (3) https://review.openstack.org/559258 | |
| 04:36:59 | homeski | is osprofiler working in Nova --Newton? | |
| 04:37:09 | homeski | can't find any docs on it .. | |
| 04:53:30 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in test_user_data.py https://review.openstack.org/559264 | |
| 05:23:50 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/api/openstack/compute/test_hosts.py https://review.openstack.org/559266 | |