| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-06-15 | |||
| 16:29:35 | bauzas | mriedem: from compute.monitors | |
| 16:29:51 | bauzas | are you really sure you want to enter that rathole ? | |
| 16:30:01 | openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/474672 | |
| 16:30:50 | mriedem | i'm not making the connection between the cpu metric monitor and how that sets a "stats" key in the resources dict that the RT is using | |
| 16:32:28 | openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif master: Updated from global requirements https://review.openstack.org/467173 | |
| 16:32:48 | mriedem | i don't think anything sets this https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L554 | |
| 16:33:32 | bauzas | I have a fun KB | |
| 16:33:33 | bauzas | https://www.ibm.com/support/knowledgecenter/SS8MU9_2.3.0/Admin/tasks/integratingplugin.html | |
| 16:34:23 | mriedem | well it used to be stevedore plugin extensions i think | |
| 16:34:30 | mriedem | i think we might have killed that in newton or so | |
| 16:35:21 | mriedem | anyway, i don't even remember what i was doing 40 minutes ago when i went down this mess, and now it's lunch time | |
| 16:37:36 | bauzas | gosh, all that spaghetti code is confusing me too | |
| 16:39:25 | bauzas | hmpf, looks like I was confusing stats and metrics | |
| 16:39:54 | mriedem | i don't see how that could be possibly confusing | |
| 16:40:19 | bauzas | but anyway, we have *something* that is amending the HostState so that ComputeCapabilitiesFilter can use it thru the extra specs | |
| 16:44:01 | bauzas | oh, and I remember other thing | |
| 16:44:15 | bauzas | mriedem: virt drivers are reporting the instances they can support | |
| 16:44:25 | bauzas | mriedem: thru a supported_instances field | |
| 16:45:07 | bauzas | anyway, I need to bail out | |
| 16:45:41 | bauzas | not sure we can just summarize all of that but saying it's just hard | |
| 16:46:16 | NostawRm | lol, I thought I'd try to do what I was saying with the accessIPv4, but its not as straight forward as I'd thought just because i was confused thinking that 'instance' was an arg for get_injected_network_template() | |
| 17:01:16 | openstackgerrit | Merged openstack/python-novaclient master: Revert "client.logger.warning wrongly used in migrations" https://review.openstack.org/474600 | |
| 17:25:52 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.resize_prep notification https://review.openstack.org/465081 | |
| 17:44:34 | eandersson | How do I output python logging even on successful tests? | |
| 17:45:24 | eandersson | Want to compare the output from successful and failed runs | |
| 17:45:36 | clarkb | eandersson: the logs are collected and are in the subunit, you should be able to run the subunit output through a filter or just look at the subunit log file after hte tests run (I tend to just read the data afterwards .testrepository/$testrunid | |
| 17:52:36 | cfriesen | bauzas: when running "nova-manage db online_data_migrations" in newton is it expected that the pci_requests field in the "instance_extra" table would be nulled out as the data is copied to the "request_specs" table in the nova_api db? This seems like it would prevent the instance from getting at instance.pci_requests. | |
| 17:53:45 | cfriesen | bauzas: ah, nevermind. looks like it was a different failure that was doing that, not the actual online data migration. | |
| 17:54:47 | efried | sdague Can we get a quick merge on this 'un: https://review.openstack.org/#/c/473975/ ? | |
| 17:54:54 | sdague | efried: looking | |
| 17:55:34 | sdague | efried: +2 | |
| 17:55:38 | efried | Thanks. | |
| 18:07:02 | mriedem | efried: powervm ci failed on that change http://184.172.12.213/75/473975/1/check/nova-in-tree-pvm/973064a/console.html | |
| 18:07:18 | mriedem | /opt/stack/devstack/lib/neutron: line 301: neutron_plugin_create_nova_conf: command not found | |
| 18:10:37 | efried | mriedem Yeah, it's the CI, not the change; we're working on getting it back up, allllmost there. | |
| 18:11:15 | eandersson | melwitt I think I added the dependency on the wrong patch :D 474404 depends on 390984 | |
| 18:11:18 | eandersson | Should be the other way around | |
| 18:13:25 | melwitt | jaypipes: heh. it's not wrong bc Melania is the same name, just in a different language | |
| 18:14:57 | melwitt | eandersson: yeah. looks like you're still seeing failures with it though? looking ... | |
| 18:15:22 | eandersson | Yea - it's odd | |
| 18:15:45 | melwitt | it keeps saying the "received update from unknown host" | |
| 18:16:09 | eandersson | I almost believe that the successful runs are actually the bad ones | |
| 18:16:53 | eandersson | And the fails are legit | |
| 18:19:57 | eandersson | It basically looks like the resource usage isn't registered at all, and because there are two nodes, when the host gets re-registered new vms can land on either host1 or host2 (not based on resource usage). | |
| 18:21:13 | eandersson | Maybe an issue with the new variable memory_mb_used ? https://review.openstack.org/#/c/390984/13/nova/scheduler/host_manager.py | |
| 18:21:30 | eandersson | This for sure worked when I tested it, but maybe something has changed since I made the patch... last year | |
| 18:22:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: fix hypervisor_hostname description for Ironic https://review.openstack.org/474723 | |
| 18:23:54 | eandersson | Yea - I don't think that variable is updated | |
| 18:25:23 | openstackgerrit | melanie witt proposed openstack/nova master: placement project_id, user_id in PUT /allocations https://review.openstack.org/469634 | |
| 18:25:24 | openstackgerrit | melanie witt proposed openstack/nova master: placement: Add GET /usages to placement API https://review.openstack.org/469635 | |
| 18:29:52 | eandersson | Those values aren't updated on master either, so I guess that is wai | |
| 18:32:46 | melwitt | eandersson: I'm going to pull down your patch and run the problematic test with debug logging enabled (you can do by OS_DEBUG=1 tox -efunctional ...) and see where the instances are landing. the test is trying to boot 3 servers and the third is supposed to fail to find a host with capacity, because of the affinity policy they're supposed to all land on the same host, I think | |
| 18:33:22 | eandersson | So I am pretty sure the issue is with the above variable | |
| 18:33:44 | eandersson | Unless having spent too many hours on this is making me delirious | |
| 18:33:50 | eandersson | It looks like the value is never updated | |
| 18:33:56 | melwitt | eandersson: oh, okay. sorry I didn't understand that part :) | |
| 18:36:25 | eandersson | No worries - I understand very little of this in general hah | |
| 18:38:15 | eandersson | and thaks for the help melwitt | |
| 18:40:56 | melwitt | np. I'm concerned if that test is busted | |
| 18:41:33 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: [WIP] Deactivate powervm compute until ready https://review.openstack.org/471773 | |
| 18:47:32 | openstackgerrit | Matthew Edmonds proposed openstack/nova master: [WIP] Implement virt driver set_host_enabled https://review.openstack.org/472794 | |
| 18:49:45 | melwitt | eandersson: from what I see in the debug logging, you're right that memory_mb_used stays stuck at 512. it never increases after each instance is booted | |
| 18:51:31 | eandersson | Maybe that calculation has moved since I created that patch. | |
| 18:52:32 | eandersson | Or maybe I messed up somewhere! :D | |
| 18:52:52 | eandersson | Then again not sure how all those unit-tests could fail if that was so | |
| 18:53:12 | eandersson | *could not | |
| 18:53:19 | melwitt | hang on, I'm also seeing free_ram_mb not changing. | |
| 19:06:25 | mriedem | sdague: huzzah! | |
| 19:06:58 | mriedem | always have to remember my old thunderbird bug trick - after an update, threading is broken, and you have to select "threading with unread" to fix threading, and then do "threading > all" to get threading on read and unread | |
| 19:07:01 | mriedem | yay for bugs | |
| 19:07:51 | melwitt | my thunderbird trick is not using threading (I didn't know it could thread) | |
| 19:08:07 | mriedem | i need threading | |
| 19:08:27 | melwitt | I might need it but I don't know it yet | |
| 19:09:54 | mriedem | we already had like 10lb of brown sugar in the house but since laura couldn't find it immediately she decided that we needed another 5lbs | |
| 19:10:02 | mriedem | so now i'm swimming in gd brown sugar | |
| 19:10:16 | openstackgerrit | Merged openstack/nova master: Stop caching compute nodes in the request https://review.openstack.org/474382 | |
| 19:11:42 | melwitt | heh | |
| 19:12:12 | melwitt | you have a lot of baked goods in your future | |
| 19:13:03 | mriedem | i do | |
| 19:13:14 | mriedem | i asked for homemade oatmeal raisin cookies for fathers day | |
| 19:13:19 | mriedem | sans chocolate chips | |
| 19:13:29 | mriedem | b/c if i wanted chocolate chip cookies, i'd have asked for those | |
| 19:14:16 | melwitt | haha | |
| 19:14:24 | clarkb | mriedem: as a cookie eater I'm happy that people like you don't eat the chocolate chip cookies so that there are more for me | |
| 19:14:28 | clarkb | :P | |
| 19:14:48 | mriedem | oatmeal raisin is a cookie too clark | |
| 19:14:52 | eandersson | melwitt, I think self.memory_mb_used may be missing here https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L276 | |
| 19:16:22 | melwitt | eandersson: hm, yeah looks like that could be a problem | |
| 19:17:35 | melwitt | it's weird that I also see free_ram_mb not changing with the patches. I don't know how that could be. without the patches, I see both free_ram_mb and memory_mb_used changing | |
| 19:17:56 | melwitt | I'm gonna try again with those set_nodes() on and see what that does | |
| 19:19:54 | eandersson | I added the following self.memory_mb_used += ram_mb and everything looks good | |
| 19:20:18 | eandersson | without any modifications to the test_server_group | |
| 19:20:32 | melwitt | though I will say the test isn't doing what it thinks it's doing. it's supposed to be booting two instances on the same host and then expecting the third one to fail. but it's really booting one server on each host and then failing the third | |
| 19:23:15 | openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/nova master: Re-worked ram filter and weights to support overcommit https://review.openstack.org/390984 | |
| 19:24:23 | eandersson | Will need additional review on that component, because I thought I understood how that part worked. ^ | |
| 19:26:08 | eandersson | Actually the above is very confusing in general, because it looks like free_mb can go negative, and the assumption was that it shouldn't be able to go negative | |
| 19:28:01 | melwitt | eandersson: I noticed the old code comment said it can go negative, but it didn't explain why. and I can't remember why it can or was supposed to be able to | |
| 19:28:21 | eandersson | The reason overcommit broke was because they changed it so that it couldn't go negative | |
| 19:29:25 | melwitt | I see | |
| 19:29:48 | eandersson | but maybe in some circumstances it's still possible | |