| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-04-19 | |||
| 14:27:54 | dansmith | the fact that we have larger allocations than the flavor is not ideal, but at the moment we have no choice | |
| 14:28:15 | bauzas | but you would not just compare with the flavor for the instance, but flavor+cached_overhead ? | |
| 14:28:25 | bauzas | well, huge thing | |
| 14:28:40 | mriedem | the allocations are made up of the flavor today | |
| 14:28:42 | bauzas | dansmith: mriedem: okay, I think why you'd like to cache that | |
| 14:29:46 | mriedem | should i go back and update the spec with a summary? | |
| 14:30:16 | mriedem | bauzas: unfortunately for you, this means redoing a large part of the spec where you changed it from posting allocations in conductor to scheduler, now back to conductor :( | |
| 14:31:07 | cdent | mriedem: what does that mean with regard to the reasons it moved back to the scheduler in the first place? | |
| 14:31:53 | cdent | conductor ->(reasons)->sched->(new reasons)->conductor | |
| 14:32:53 | bauzas | mriedem: this is fine https://www.nytimes.com/2016/08/06/arts/this-is-fine-meme-dog-fire.html | |
| 14:33:13 | mriedem | cdent: we moved it to scheduler because that's where the filtered host list is, so retries there might be theoretically faster, | |
| 14:33:33 | cdent | so it's just a performance thing? I guess that's okay then | |
| 14:33:35 | mriedem | cdent: we're moving to conductor now because conductor is what's going to need to do the rpc call into the compute to get the overhead values, | |
| 14:33:49 | bauzas | mriedem: dansmith: wait, we expose flavors to scheduler, right? | |
| 14:34:04 | bauzas | oh crap, nvm | |
| 14:34:28 | mriedem | so conductor -> scheduler (select_destinations) -> conductor (rpc call to compute to get overhead for the selected host) -> post allocations (flavor + overhead) to placement -> (1) success, build on compute, or (2) fail, retry through scheduler for a new host | |
| 14:34:35 | mriedem | cdent: ^ make sense? | |
| 14:34:41 | bauzas | I'm trying to consider how we could inform placement of the overhead for the specific instance | |
| 14:34:51 | mriedem | bauzas: via the allocations that we post | |
| 14:34:53 | mriedem | see ^ | |
| 14:35:12 | cdent | mriedem: yeah, makes sense, but is frustrating | |
| 14:35:20 | mriedem | cdent: agree, | |
| 14:35:36 | bauzas | mriedem: I'd love to have posted when we do GET /RPs=memory=X | |
| 14:35:37 | mriedem | especially since i for one didn't even know this overhead thing existed until yesterday | |
| 14:35:47 | mriedem | bauzas: we don't know the host then | |
| 14:35:50 | mriedem | it's a chicken and egg | |
| 14:35:52 | bauzas | mriedem: where X would be flavor+overhead | |
| 14:36:01 | bauzas | fuuuuuuuuuu | |
| 14:36:03 | mriedem | bauzas: we have to have the host to get the overhead | |
| 14:36:09 | mriedem | because the virt driver gives the overhead values | |
| 14:36:47 | bauzas | right, so the only problem is with the new POST allocation | |
| 14:36:52 | bauzas | sorry, slow brain here | |
| 14:37:12 | bauzas | my brain is fried by that shit ton of political discussions I'm eating | |
| 14:37:59 | mriedem | i don't read the news anymore | |
| 14:38:08 | mriedem | i heard we bombed afghanistan with a large bomb | |
| 14:38:27 | mriedem | to show how cool we are | |
| 14:38:39 | edleafe | Is the overhead specific to each hypervisor, but the same for all instances on that hypervisor, regardless of instance size? | |
| 14:39:07 | mriedem | edleafe: no | |
| 14:39:15 | edleafe | ugh | |
| 14:39:24 | bauzas | mriedem: I do recommend https://www.youtube.com/watch?v=hkZir1L7fSY for your knowledge | |
| 14:40:13 | mriedem | bauzas: ok, saved for later | |
| 14:40:21 | bauzas | mriedem: NSFW | |
| 14:40:29 | mriedem | i'm at home | |
| 14:40:31 | mriedem | so everything is safe | |
| 14:40:45 | mriedem | between the hours of 9am and 4pm that is | |
| 14:40:45 | bauzas | s/NSFW/NSFWFH | |
| 14:40:56 | bauzas | anyway | |
| 14:41:04 | mriedem | edleafe: the hyperv overhead calculation is pretty simple | |
| 14:41:08 | bauzas | mriedem: posting a summary in the spec ? | |
| 14:41:10 | mriedem | 'disk_gb': (instance_info['memory_mb'] + 512) // units.Ki | |
| 14:41:20 | mriedem | bauzas: yeah, in a minute | |
| 14:41:26 | mriedem | looking at this py35 weirdness | |
| 14:41:36 | bauzas | mriedem: okay, because I was about writing a new rev -ish | |
| 14:42:36 | edleafe | mriedem: oh, that's much clearer | |
| 14:43:15 | mriedem | edleafe: the xen and libvirt ones are more confusing | |
| 14:44:07 | mriedem | gcb: i'm having a hard time knowing which test is doing the stack overflow | |
| 14:44:18 | mriedem | but there appears to be an infinite recursion in oslo.config | |
| 14:44:35 | mriedem | gcb: i also see at the end: nova.tests.unit.test_rpc.TestRPC.test_add_extra_exmods [] ... inprogress | |
| 14:44:37 | bauzas | mriedem: the libvirt one is scarying me, because I wonder if we could even cache that | |
| 14:44:38 | mriedem | so maybe it's that one | |
| 14:44:47 | mriedem | bauzas: because it's config driven? | |
| 14:44:51 | mriedem | and per-image | |
| 14:45:15 | mriedem | bauzas: yeah that's a good point | |
| 14:45:17 | mriedem | dansmith: ^ | |
| 14:45:24 | mriedem | the libvirt overhead one kind of screws us | |
| 14:45:24 | melwitt | mriedem: is anyone working on the intermittent ironic test failure? I want to take a stab at it if not | |
| 14:45:28 | mriedem | melwitt: no | |
| 14:45:30 | bauzas | mriedem: exactly this | |
| 14:45:41 | mriedem | melwitt: i was considering just skipping it for now | |
| 14:45:41 | dansmith | hmm, I don't remember the config bit | |
| 14:45:50 | mriedem | dansmith: might not be config, but it's based on image meta | |
| 14:46:00 | gcb | mriedem, Do you get error from https://review.openstack.org/#/c/456397 ? It seems oslo.config have recursion calls | |
| 14:46:02 | johnthetubaguy | melwitt: I saw talk of something in the neutron channel about ironic, but seemed more like a hard failure | |
| 14:46:09 | dansmith | mriedem: yeah image meta | |
| 14:46:24 | mriedem | gcb: it's multiple changes failing on this | |
| 14:46:27 | mriedem | according to logstash | |
| 14:46:45 | melwitt | mriedem: okay. I think the right thing to do is just raise the Timeout exception. we shouldn't be doing a real timeout | |
| 14:46:55 | melwitt | (as discussed on the bug) | |
| 14:47:11 | mriedem | melwitt: so remove all of the fake stub methods that are doing logic to trigger the timeout? | |
| 14:47:15 | mriedem | melwitt: that's the easy fix | |
| 14:47:34 | mriedem | i think i get the point of why the test was doing that though | |
| 14:47:39 | mriedem | to stub and mimic what the driver is doing | |
| 14:47:43 | mriedem | but it's pretty fragile | |
| 14:48:07 | melwitt | mriedem: yeah, stub the looping call to raise the timeout. I thought the test wants to check that it does the right thing when a timeout occurs, that it raises the ConsoleNotAvailable | |
| 14:49:21 | melwitt | that is, I think we can assume that oslo_service looping calls work correctly, else we're also testing oslo functionality | |
| 14:49:30 | mriedem | melwitt: yeah it's a fine line | |
| 14:49:49 | bauzas | mriedem: dansmith: we also lookup the instance if we don't have the numa topology set in the instance object for knowing the overhead | |
| 14:49:53 | mriedem | melwitt: i'm fine with doing that | |
| 14:49:56 | dansmith | mriedem: so, a couple things | |
| 14:50:00 | mriedem | melwitt: because my solution was skipTest :) | |
| 14:50:18 | gcb | mriedem, it's really strange, nova.tests.unit.test_rpc.TestRPC.test_add_extra_exmods is simple test. what I found just deprecated warning from oslo.messaging | |
| 14:50:41 | dansmith | mriedem: (1) presumably we'll do this without the cache first, (2) we can cache on flavorid/hv/ver first, (3) we might just make a hash key function that generates a key based on all the things currently known to matter | |
| 14:51:09 | mriedem | dansmith: where (3) could take into account the image id too? | |
| 14:51:12 | dansmith | mriedem: and (4) the penalty for missing means we *might* fail to build because of overhead on the compute, which sucks, but.. builds can fail | |
| 14:51:17 | dansmith | mriedem: yeah | |
| 14:51:26 | mriedem | dansmith: and the image meta could change w/o changing the image id | |
| 14:51:37 | dansmith | mriedem: or just hash in the actual fields we know are ever looked at | |
| 14:51:45 | dansmith | mriedem: right well, that's why I was thinking ^ instead of id, but yeah | |
| 14:51:55 | bauzas | mriedem: dansmith: so just to make it clear, we introspect the instance every time we claim the resource in the RT if libvirt | |
| 14:51:57 | melwitt | mriedem: heh, okay. I'll upload it and we'll see | |