| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-15 | |||
| 17:54:47 | efried | sdague Can we get a quick merge on this 'un: https://review.openstack.org/#/c/473975/ ? | |
| 17:54:54 | sdague | efried: looking | |
| 17:55:34 | sdague | efried: +2 | |
| 17:55:38 | efried | Thanks. | |
| 18:07:02 | mriedem | efried: powervm ci failed on that change http://184.172.12.213/75/473975/1/check/nova-in-tree-pvm/973064a/console.html | |
| 18:07:18 | mriedem | /opt/stack/devstack/lib/neutron: line 301: neutron_plugin_create_nova_conf: command not found | |
| 18:10:37 | efried | mriedem Yeah, it's the CI, not the change; we're working on getting it back up, allllmost there. | |
| 18:11:15 | eandersson | melwitt I think I added the dependency on the wrong patch :D 474404 depends on 390984 | |
| 18:11:18 | eandersson | Should be the other way around | |
| 18:13:25 | melwitt | jaypipes: heh. it's not wrong bc Melania is the same name, just in a different language | |
| 18:14:57 | melwitt | eandersson: yeah. looks like you're still seeing failures with it though? looking ... | |
| 18:15:22 | eandersson | Yea - it's odd | |
| 18:15:45 | melwitt | it keeps saying the "received update from unknown host" | |
| 18:16:09 | eandersson | I almost believe that the successful runs are actually the bad ones | |
| 18:16:53 | eandersson | And the fails are legit | |
| 18:19:57 | eandersson | It basically looks like the resource usage isn't registered at all, and because there are two nodes, when the host gets re-registered new vms can land on either host1 or host2 (not based on resource usage). | |
| 18:21:13 | eandersson | Maybe an issue with the new variable memory_mb_used ? https://review.openstack.org/#/c/390984/13/nova/scheduler/host_manager.py | |
| 18:21:30 | eandersson | This for sure worked when I tested it, but maybe something has changed since I made the patch... last year | |
| 18:22:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: fix hypervisor_hostname description for Ironic https://review.openstack.org/474723 | |
| 18:23:54 | eandersson | Yea - I don't think that variable is updated | |
| 18:25:23 | openstackgerrit | melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634 | |
| 18:25:24 | openstackgerrit | melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635 | |
| 18:29:52 | eandersson | Those values aren't updated on master either, so I guess that is wai | |
| 18:32:46 | melwitt | eandersson: I'm going to pull down your patch and run the problematic test with debug logging enabled (you can do by OS_DEBUG=1 tox -efunctional ...) and see where the instances are landing. the test is trying to boot 3 servers and the third is supposed to fail to find a host with capacity, because of the affinity policy they're supposed to all land on the same host, I think | |
| 18:33:22 | eandersson | So I am pretty sure the issue is with the above variable | |
| 18:33:44 | eandersson | Unless having spent too many hours on this is making me delirious | |
| 18:33:50 | eandersson | It looks like the value is never updated | |
| 18:33:56 | melwitt | eandersson: oh, okay. sorry I didn't understand that part :) | |
| 18:36:25 | eandersson | No worries - I understand very little of this in general hah | |
| 18:38:15 | eandersson | and thaks for the help melwitt | |
| 18:40:56 | melwitt | np. I'm concerned if that test is busted | |
| 18:41:33 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: [WIP] Deactivate powervm compute until ready https://review.openstack.org/471773 | |
| 18:47:32 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: [WIP] Implement virt driver set_host_enabled https://review.openstack.org/472794 | |
| 18:49:45 | melwitt | eandersson: from what I see in the debug logging, you're right that memory_mb_used stays stuck at 512. it never increases after each instance is booted | |
| 18:51:31 | eandersson | Maybe that calculation has moved since I created that patch. | |
| 18:52:32 | eandersson | Or maybe I messed up somewhere! :D | |
| 18:52:52 | eandersson | Then again not sure how all those unit-tests could fail if that was so | |
| 18:53:12 | eandersson | *could not | |
| 18:53:19 | melwitt | hang on, I'm also seeing free_ram_mb not changing. | |
| 19:06:25 | mriedem | sdague: huzzah! | |
| 19:06:58 | mriedem | always have to remember my old thunderbird bug trick - after an update, threading is broken, and you have to select "threading with unread" to fix threading, and then do "threading > all" to get threading on read and unread | |
| 19:07:01 | mriedem | yay for bugs | |
| 19:07:51 | melwitt | my thunderbird trick is not using threading (I didn't know it could thread) | |
| 19:08:07 | mriedem | i need threading | |
| 19:08:27 | melwitt | I might need it but I don't know it yet | |
| 19:09:54 | mriedem | we already had like 10lb of brown sugar in the house but since laura couldn't find it immediately she decided that we needed another 5lbs | |
| 19:10:02 | mriedem | so now i'm swimming in gd brown sugar | |
| 19:10:16 | openstackgerrit | Merged openstack/nova master: Stop caching compute nodes in the request https://review.openstack.org/474382 | |
| 19:11:42 | melwitt | heh | |
| 19:12:12 | melwitt | you have a lot of baked goods in your future | |
| 19:13:03 | mriedem | i do | |
| 19:13:14 | mriedem | i asked for homemade oatmeal raisin cookies for fathers day | |
| 19:13:19 | mriedem | sans chocolate chips | |
| 19:13:29 | mriedem | b/c if i wanted chocolate chip cookies, i'd have asked for those | |
| 19:14:16 | melwitt | haha | |
| 19:14:24 | clarkb | mriedem: as a cookie eater I'm happy that people like you don't eat the chocolate chip cookies so that there are more for me | |
| 19:14:28 | clarkb | :P | |
| 19:14:48 | mriedem | oatmeal raisin is a cookie too clark | |
| 19:14:52 | eandersson | melwitt, I think self.memory_mb_used may be missing here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L276 | |
| 19:16:22 | melwitt | eandersson: hm, yeah looks like that could be a problem | |
| 19:17:35 | melwitt | it's weird that I also see free_ram_mb not changing with the patches. I don't know how that could be. without the patches, I see both free_ram_mb and memory_mb_used changing | |
| 19:17:56 | melwitt | I'm gonna try again with those set_nodes() on and see what that does | |
| 19:19:54 | eandersson | I added the following self.memory_mb_used += ram_mb and everything looks good | |
| 19:20:18 | eandersson | without any modifications to the test_server_group | |
| 19:20:32 | melwitt | though I will say the test isn't doing what it thinks it's doing. it's supposed to be booting two instances on the same host and then expecting the third one to fail. but it's really booting one server on each host and then failing the third | |
| 19:23:15 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Re-worked ram filter and weights to support overcommit https://review.openstack.org/390984 | |
| 19:24:23 | eandersson | Will need additional review on that component, because I thought I understood how that part worked. ^ | |
| 19:26:08 | eandersson | Actually the above is very confusing in general, because it looks like free_mb can go negative, and the assumption was that it shouldn't be able to go negative | |
| 19:28:01 | melwitt | eandersson: I noticed the old code comment said it can go negative, but it didn't explain why. and I can't remember why it can or was supposed to be able to | |
| 19:28:21 | eandersson | The reason overcommit broke was because they changed it so that it couldn't go negative | |
| 19:29:25 | melwitt | I see | |
| 19:29:48 | eandersson | but maybe in some circumstances it's still possible | |
| 19:30:47 | eandersson | https://github.com/openstack/nova/commit/016b810f675b20e8ce78f4c82dc9c679c0162b7a | |
| 19:46:49 | openstackgerrit | Merged openstack/nova master: Use CONF.host for powervm nodename https://review.openstack.org/473975 | |
| 20:07:45 | rickd | Hi, Anyone aware of a bug within libvirtd v2.5.0 and nested virtulization. Having issues running Ocata on a VMware guest : ( | |
| 20:08:41 | rickd | Booting from hardisk ... GRUB... | |
| 20:09:52 | clarkb | rickd: openstack infra's experience is that it is quite buggy. We disable it by default as a result | |
| 20:09:55 | clarkb | it being nested virt | |
| 20:11:17 | rickd | clarkb I didnt release the infras had issues as well (?) | |
| 20:11:26 | rickd | Thought it would just be the computes... | |
| 20:11:43 | clarkb | rickd: sorry the infra team, eg the group that runs all the CI testing systems | |
| 20:11:49 | rickd | When you say you disabled it. Do you mean within Libvirt | |
| 20:11:58 | clarkb | rickd: we test with nova and qemu not nova and kvm | |
| 20:12:39 | rickd | ie you test nova and qemu for nested? | |
| 20:12:50 | clarkb | also when I dug into it I found the linux kernel does not enable nested virt by default on Intel CPUs because it is considered too buggy. It is enabled by default with amd though | |
| 20:13:16 | clarkb | rickd: yes, beacuse the testing itself runs on virtual machines we have the nested nova compute use qemu not kvm otherwise things fail in weird ways | |
| 20:14:04 | rickd | ok but for prod, its Nova + KVM? | |
| 20:14:25 | clarkb | yes the cloud we run tests on use kvm. I'm just saying we don't do nested virtualization | |
| 20:14:33 | clarkb | (cloud we run runs on baremetal so no nesting) | |
| 20:15:07 | rickd | ah ok, just trying to understand if there is a way around this. | |
| 20:15:31 | clarkb | rickd: according to the linux kernel use amd cpus... (I haven't been able to test reliability on amd though) | |
| 20:15:31 | rickd | But it looks like libvirt has pulled support for nested anyway | |
| 20:16:05 | clarkb | I don't think its a libvirt issue, it is a kvm thing | |
| 20:17:04 | rickd | from what I can see its was buggy before libvirt 2.5.0. Then nested support was fully pulled. | |
| 20:17:11 | rickd | I could be wrong though.. | |
| 20:18:03 | clarkb | where do you see that? my understanding is its purely a kvm option and libvirt will just do whatever with kvm being configured the way it is | |
| 20:19:26 | rickd | ah yer see your point - https://bugs.launchpad.net/devstack/+bug/1689370 | |
| 20:19:26 | openstack | Launchpad bug 1689370 in devstack "using ubuntu cloud archive gets a version of libvirtd which doesn't do nested virtualization" [Undecided,Fix released] - Assigned to Amrith Kumar (amrith) | |
| 20:22:30 | rickd | I wonder if I will be ok with a physical compute and virtual controller nodes. Think I would be fine. | |
| 20:41:41 | eandersson | melwitt, so yea looks like it was an issue with the code | |