Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-06-15
17:52:36 cfriesen bauzas: when running "nova-manage db online_data_migrations" in newton is it expected that the pci_requests field in the "instance_extra" table would be nulled out as the data is copied to the "request_specs" table in the nova_api db? This seems like it would prevent the instance from getting at instance.pci_requests.
17:53:45 cfriesen bauzas: ah, nevermind. looks like it was a different failure that was doing that, not the actual online data migration.
17:54:47 efried sdague Can we get a quick merge on this 'un: https://review.openstack.org/#/c/473975/ ?
17:54:54 sdague efried: looking
17:55:34 sdague efried: +2
17:55:38 efried Thanks.
18:07:02 mriedem efried: powervm ci failed on that change http://184.172.12.213/75/473975/1/check/nova-in-tree-pvm/973064a/console.html
18:07:18 mriedem /opt/stack/devstack/lib/neutron: line 301: neutron_plugin_create_nova_conf: command not found
18:10:37 efried mriedem Yeah, it's the CI, not the change; we're working on getting it back up, allllmost there.
18:11:15 eandersson melwitt I think I added the dependency on the wrong patch :D 474404 depends on 390984
18:11:18 eandersson Should be the other way around
18:13:25 melwitt jaypipes: heh. it's not wrong bc Melania is the same name, just in a different language
18:14:57 melwitt eandersson: yeah. looks like you're still seeing failures with it though? looking ...
18:15:22 eandersson Yea - it's odd
18:15:45 melwitt it keeps saying the "received update from unknown host"
18:16:09 eandersson I almost believe that the successful runs are actually the bad ones
18:16:53 eandersson And the fails are legit
18:19:57 eandersson It basically looks like the resource usage isn't registered at all, and because there are two nodes, when the host gets re-registered new vms can land on either host1 or host2 (not based on resource usage).
18:21:13 eandersson Maybe an issue with the new variable memory_mb_used ? https://review.openstack.org/#/c/390984/13/nova/scheduler/host_manager.py
18:21:30 eandersson This for sure worked when I tested it, but maybe something has changed since I made the patch... last year
18:22:55 openstackgerrit Matt Riedemann proposed openstack/nova master: api-ref: fix hypervisor_hostname description for Ironic https://review.openstack.org/474723
18:23:54 eandersson Yea - I don't think that variable is updated
18:25:23 openstackgerrit melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634
18:25:24 openstackgerrit melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635
18:29:52 eandersson Those values aren't updated on master either, so I guess that is wai
18:32:46 melwitt eandersson: I'm going to pull down your patch and run the problematic test with debug logging enabled (you can do by OS_DEBUG=1 tox -efunctional ...) and see where the instances are landing. the test is trying to boot 3 servers and the third is supposed to fail to find a host with capacity, because of the affinity policy they're supposed to all land on the same host, I think
18:33:22 eandersson So I am pretty sure the issue is with the above variable
18:33:44 eandersson Unless having spent too many hours on this is making me delirious
18:33:50 eandersson It looks like the value is never updated
18:33:56 melwitt eandersson: oh, okay. sorry I didn't understand that part :)
18:36:25 eandersson No worries - I understand very little of this in general hah
18:38:15 eandersson and thaks for the help melwitt
18:40:56 melwitt np. I'm concerned if that test is busted
18:41:33 openstackgerrit Matthew Edmonds proposed openstack/nova master: [WIP] Deactivate powervm compute until ready https://review.openstack.org/471773
18:47:32 openstackgerrit Matthew Edmonds proposed openstack/nova master: [WIP] Implement virt driver set_host_enabled https://review.openstack.org/472794
18:49:45 melwitt eandersson: from what I see in the debug logging, you're right that memory_mb_used stays stuck at 512. it never increases after each instance is booted
18:51:31 eandersson Maybe that calculation has moved since I created that patch.
18:52:32 eandersson Or maybe I messed up somewhere! :D
18:52:52 eandersson Then again not sure how all those unit-tests could fail if that was so
18:53:12 eandersson *could not
18:53:19 melwitt hang on, I'm also seeing free_ram_mb not changing.
19:06:25 mriedem sdague: huzzah!
19:06:58 mriedem always have to remember my old thunderbird bug trick - after an update, threading is broken, and you have to select "threading with unread" to fix threading, and then do "threading > all" to get threading on read and unread
19:07:01 mriedem yay for bugs
19:07:51 melwitt my thunderbird trick is not using threading (I didn't know it could thread)
19:08:07 mriedem i need threading
19:08:27 melwitt I might need it but I don't know it yet
19:09:54 mriedem we already had like 10lb of brown sugar in the house but since laura couldn't find it immediately she decided that we needed another 5lbs
19:10:02 mriedem so now i'm swimming in gd brown sugar
19:10:16 openstackgerrit Merged openstack/nova master: Stop caching compute nodes in the request https://review.openstack.org/474382
19:11:42 melwitt heh
19:12:12 melwitt you have a lot of baked goods in your future
19:13:03 mriedem i do
19:13:14 mriedem i asked for homemade oatmeal raisin cookies for fathers day
19:13:19 mriedem sans chocolate chips
19:13:29 mriedem b/c if i wanted chocolate chip cookies, i'd have asked for those
19:14:16 melwitt haha
19:14:24 clarkb mriedem: as a cookie eater I'm happy that people like you don't eat the chocolate chip cookies so that there are more for me
19:14:28 clarkb :P
19:14:48 mriedem oatmeal raisin is a cookie too clark
19:14:52 eandersson melwitt, I think self.memory_mb_used may be missing here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L276
19:16:22 melwitt eandersson: hm, yeah looks like that could be a problem
19:17:35 melwitt it's weird that I also see free_ram_mb not changing with the patches. I don't know how that could be. without the patches, I see both free_ram_mb and memory_mb_used changing
19:17:56 melwitt I'm gonna try again with those set_nodes() on and see what that does
19:19:54 eandersson I added the following self.memory_mb_used += ram_mb and everything looks good
19:20:18 eandersson without any modifications to the test_server_group
19:20:32 melwitt though I will say the test isn't doing what it thinks it's doing. it's supposed to be booting two instances on the same host and then expecting the third one to fail. but it's really booting one server on each host and then failing the third
19:23:15 openstackgerrit Erik Olof Gunnar Andersson proposed openstack/nova master: Re-worked ram filter and weights to support overcommit https://review.openstack.org/390984
19:24:23 eandersson Will need additional review on that component, because I thought I understood how that part worked. ^
19:26:08 eandersson Actually the above is very confusing in general, because it looks like free_mb can go negative, and the assumption was that it shouldn't be able to go negative
19:28:01 melwitt eandersson: I noticed the old code comment said it can go negative, but it didn't explain why. and I can't remember why it can or was supposed to be able to
19:28:21 eandersson The reason overcommit broke was because they changed it so that it couldn't go negative
19:29:25 melwitt I see
19:29:48 eandersson but maybe in some circumstances it's still possible
19:30:47 eandersson https://github.com/openstack/nova/commit/016b810f675b20e8ce78f4c82dc9c679c0162b7a
19:46:49 openstackgerrit Merged openstack/nova master: Use CONF.host for powervm nodename https://review.openstack.org/473975
20:07:45 rickd Hi, Anyone aware of a bug within libvirtd v2.5.0 and nested virtulization. Having issues running Ocata on a VMware guest : (
20:08:41 rickd Booting from hardisk ... GRUB...
20:09:52 clarkb rickd: openstack infra's experience is that it is quite buggy. We disable it by default as a result
20:09:55 clarkb it being nested virt
20:11:17 rickd clarkb I didnt release the infras had issues as well (?)
20:11:26 rickd Thought it would just be the computes...
20:11:43 clarkb rickd: sorry the infra team, eg the group that runs all the CI testing systems
20:11:49 rickd When you say you disabled it. Do you mean within Libvirt
20:11:58 clarkb rickd: we test with nova and qemu not nova and kvm
20:12:39 rickd ie you test nova and qemu for nested?
20:12:50 clarkb also when I dug into it I found the linux kernel does not enable nested virt by default on Intel CPUs because it is considered too buggy. It is enabled by default with amd though
20:13:16 clarkb rickd: yes, beacuse the testing itself runs on virtual machines we have the nested nova compute use qemu not kvm otherwise things fail in weird ways
20:14:04 rickd ok but for prod, its Nova + KVM?
20:14:25 clarkb yes the cloud we run tests on use kvm. I'm just saying we don't do nested virtualization
20:14:33 clarkb (cloud we run runs on baremetal so no nesting)
20:15:07 rickd ah ok, just trying to understand if there is a way around this.
20:15:31 clarkb rickd: according to the linux kernel use amd cpus... (I haven't been able to test reliability on amd though)
20:15:31 rickd But it looks like libvirt has pulled support for nested anyway
20:16:05 clarkb I don't think its a libvirt issue, it is a kvm thing
20:17:04 rickd from what I can see its was buggy before libvirt 2.5.0. Then nested support was fully pulled.
20:17:11 rickd I could be wrong though..
20:18:03 clarkb where do you see that? my understanding is its purely a kvm option and libvirt will just do whatever with kvm being configured the way it is
20:19:26 rickd ah yer see your point - https://bugs.launchpad.net/devstack/+bug/1689370
20:19:26 openstack Launchpad bug 1689370 in devstack "using ubuntu cloud archive gets a version of libvirtd which doesn't do nested virtualization" [Undecided,Fix released] - Assigned to Amrith Kumar (amrith)

Earlier   Later