| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-09-27 | |||
| 12:42:36 | gmann | alex_xu: take rest and get well soon | |
| 12:53:16 | tssurya | Hi, I was just playing around with cell_v2 and came across the nova-manage cell_v2 map_instances command; so I went through the source code (manage.py) but was not able to find a connection as to which sql_connection string is used to query the 'instances' table in the nova DB. It should be taking this from the conf file ? But could someone point me to the section of code that does this ? | |
| 13:00:36 | sdague | tssurya: it's coming from the same nova.conf that your API server is expected to be using | |
| 13:01:10 | sdague | tssurya: all nova-manage commands are assumed to be run from the shell on the API server (and have the same db access) | |
| 13:03:36 | openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep. https://review.openstack.org/507848 | |
| 13:05:12 | tssurya | sdague : thanks for the reply, so here is my situation : I have some instances lying in my nova_cell1 DB which need to be mapped to my new cell (cell1). however the db access URL for this is inside nova_cell1.conf ? so when I run the map instances command, they do not get mapped. | |
| 13:06:42 | tssurya | sdague: I think its because the queries are sent to the nova_cell0 DB since nova.conf has that access url. | |
| 13:07:29 | efried | cdent Yo | |
| 13:07:52 | openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | |
| 13:08:13 | cdent | efried: yo | |
| 13:09:27 | cdent | efried: my question is basically: in powervm does the thing which is the actual hypervisor ever host workloads that are not managed by nova. The comments on this bug for context: https://bugs.launchpad.net/nova/+bug/1718212 | |
| 13:09:28 | openstack | Launchpad bug 1718212 in OpenStack Compute (nova) "Compute resource tracker does not report correct information for drivers such as vSphere" [Medium,In progress] - Assigned to Radoslav Gerganov (rgerganov) | |
| 13:10:27 | cdent | since placement is authoritative for dynamic workloads if nova is not doing all the managing, there may be need for additional things talking to placement | |
| 13:10:46 | efried | cdent That's a multi-pronged question. | |
| 13:10:56 | cdent | It’s a full on dinner fork | |
| 13:11:03 | openstackgerrit | Sean Dague proposed openstack/nova master: Support qemu >= 2.10 https://review.openstack.org/505673 | |
| 13:11:14 | efried | First, nothing is stopping the user from creating VMs outside the auspices of OpenStack. | |
| 13:12:43 | efried | Second, there's a mode (the most common one, historically) where the I/O virtualization is done by one (or two, cause redundancy/HA) separate partitions, in which case those guys are consuming resources outside of Nova's purview. | |
| 13:13:02 | efried | cdent Now I'll look at the bug... | |
| 13:13:32 | bauzas | efried: creating VMs by the hypervisor directly is not supported by Nova | |
| 13:14:02 | efried | bauzas Oh, Nova won't pick them up, for sure. But there's nothing stopping the user from doing it. | |
| 13:14:05 | cdent | efried: So, short answer is “yes” so second question is “Do you yet have a plan on how to manage it” | |
| 13:14:26 | bauzas | efried: if the operator does that, then any bug related to that should be Wontfix | |
| 13:14:47 | efried | bauzas Well... | |
| 13:14:53 | efried | I agree in principle. | |
| 13:14:56 | bauzas | efried: lemme find where we say that we don't support direct hypervisor calls | |
| 13:15:11 | efried | But that doesn't mean we won't try to accomodate out-of-band partitions when reporting inventory/usage. | |
| 13:15:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Log consumer uuid when retrying claims in the scheduler https://review.openstack.org/507705 | |
| 13:15:33 | efried | bauzas Sure, but I totally believe you. | |
| 13:16:18 | openstackgerrit | Andrey Volkov proposed openstack/nova master: [WIP] List instances performace optimization https://review.openstack.org/507854 | |
| 13:17:28 | bauzas | efried: I didn't found any explanations either in https://docs.openstack.org/nova/latest/contributor/policies.html or https://docs.openstack.org/nova/latest/contributor/project-scope.html | |
| 13:17:41 | bauzas | maybe we should say that | |
| 13:17:46 | efried | cdent Okay, in general, we have very similar issues as described in comment #6 in that bug. | |
| 13:18:18 | efried | When I started looking into converting over to get_inventory, it was going to be a matter of reporting "reserved" amounts based on whatever's going on OOB. | |
| 13:18:56 | johnthetubaguy | I thought we were heading down the not doing live updates of resource usage? | |
| 13:18:56 | cdent | efried: do you have a convenient way of distinguishing between nova managed and not-nova managed stuff? | |
| 13:19:04 | efried | ...and it's not trivial to figure out what's OOB. | |
| 13:19:09 | efried | (I was just about to say :) | |
| 13:19:10 | cdent | jinx! | |
| 13:19:35 | efried | I can know off the bat to account for my Novalink partition (the node on which the compute service runs) and my Virtual I/O Servers. | |
| 13:19:35 | cdent | johnthetubaguy: that seems to work okay for libvirt, but not so great otherwise | |
| 13:20:03 | efried | But any plain ol' worker bees I have to figure out whether they came from Nova. | |
| 13:20:17 | johnthetubaguy | cdent: I guess I am not seeing why, is that vmware bug the example? | |
| 13:20:19 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Migrate from 'ip' commands to 'pyroute2' https://review.openstack.org/484386 | |
| 13:20:26 | cdent | johnthetubaguy: at this stage efried, rgerganov and I are just sort of having a chat, not making any decisions | |
| 13:20:58 | bhagyashris | johnthetubaguy, mriedem: Hi, | |
| 13:21:01 | efried | Simplest would be, every time get_inventory/get_available_resource is called, I ask Nova for the full list of instances it knows about, ask my hypervisor for the instances IT knows about, and do a set-diff. | |
| 13:21:01 | johnthetubaguy | I am just curious where the problem isn't nova driver expected resource usage | |
| 13:21:20 | cdent | johnthetubaguy: that’s a specific case of a general issue of “things other than the nova-compute node using the stuff that nova-compute is also using” | |
| 13:21:34 | efried | just so. | |
| 13:21:55 | johnthetubaguy | I think I need a more concrete example | |
| 13:22:01 | mriedem | cdent: i think the vcenter vms have some nova metadata associated with them, so you can tell which ones are nova-managed and which were created oob | |
| 13:22:20 | efried | johnthetubaguy You try to spawn an instance and ask for 3 VCPU. Resource tracker reports you've got 3 VCPU, so the claim passes. | |
| 13:22:34 | efried | johnthetubaguy But in fact, one VCPU is being consumed by a VM that was spawned out of band | |
| 13:22:36 | rgerganov | mriedem, that may work for vcpu and memory but it won't work well for storage | |
| 13:22:45 | johnthetubaguy | its the out of band VMs, I thought we explicitly didn't support that | |
| 13:23:03 | efried | johnthetubaguy Yeah, bauzas said the same, but didn't find where that's documented. | |
| 13:23:11 | efried | Not saying that means it's supported, or that it should be :) | |
| 13:23:40 | bauzas | so, Nova isn't a proxy layer for hypervisors | |
| 13:23:49 | johnthetubaguy | yeah, I could have swarn it was in here, but I don't see it: https://docs.openstack.org/nova/latest/contributor/project-scope.html | |
| 13:24:08 | bauzas | johnthetubaguy: yeah, I verified that | |
| 13:24:14 | bauzas | lemme provide a change for it | |
| 13:24:25 | johnthetubaguy | well, its a bit late I guess | |
| 13:25:13 | cdent | So, even if the statement is “we don’t do that” the problem still holds for shared storage | |
| 13:25:21 | efried | right | |
| 13:25:22 | mriedem | nova doesn't import existing vms on the hypervisor, | |
| 13:25:37 | mriedem | but that doesn't mean we don't try to adjust inventory based on things running on the hypervisor host | |
| 13:25:37 | cdent | where “the problem” is the generic notion of mixed accounting | |
| 13:25:39 | efried | mriedem Which actually makes it more problematic. | |
| 13:25:40 | efried | right. | |
| 13:25:43 | johnthetubaguy | cdent: yeah, that sounds like a real thing we have to support | |
| 13:25:43 | mriedem | that's why we have reserved | |
| 13:25:48 | efried | right | |
| 13:26:08 | mriedem | even libvirt hosts have to account for things like ovs running on the same host | |
| 13:26:09 | cdent | Right, so one of the underlying questions is: | |
| 13:26:21 | cdent | Do we intend/expect that reserved will by dynamically adjuted, frequently | |
| 13:26:43 | cdent | Or in the cases where we want it to be dynamically adjusted we should instead make allocations, via some third party? | |
| 13:26:45 | efried | Right, back to what johnthetubaguy mentioned earlier: "not doing live updates of resource usage" | |
| 13:27:06 | efried | That is, will get_inventory() eventually be a thing that's run only once, rather than on a periodic? | |
| 13:27:15 | cdent | exactly | |
| 13:27:17 | bhagyashris | johnthetubaguy, mriedem: Could you please review patch: https://review.openstack.org/#/c/409644/ ? Addressed all review comments. Thank you :) | |
| 13:27:23 | mriedem | i don't think it will no | |
| 13:27:27 | bauzas | I'm not seeing the reserved bit to be that dynamicv | |
| 13:27:31 | mriedem | the operator can adjust reserved space dynamically | |
| 13:27:35 | johnthetubaguy | cdent: I separate allocations are easier to update than a single reserved value | |
| 13:27:42 | efried | I've heard tell (possibly from jay) that it's a no-no to update the *total* amounts. | |
| 13:27:44 | mriedem | we talked about this in boston | |
| 13:27:56 | bauzas | but yeah, the operator can just set or raise the reserved bit when they want | |
| 13:27:58 | mriedem | about how we don't account for overhead, and operators would have to handle that by toggling reserved values | |
| 13:28:23 | cdent | mriedem: yes, but the new ingredient here is the dynamism | |
| 13:28:25 | cdent | maybe | |
| 13:28:27 | bauzas | mriedem: do you think we should clearly state that in https://docs.openstack.org/nova/latest/contributor/project-scope.html ? | |
| 13:28:40 | efried | ...unless, like, they hot-plug a whole new disk, or jack up the disk size on the SAN or whatever. In that case, we should be able to bump the totals, yes? | |
| 13:28:43 | bauzas | the fact that Nova doesn't support direct hypervisor VMs, but can leave room for them | |
| 13:28:49 | cdent | efried: yes | |
| 13:28:56 | efried | Same actually applies to CPUs | |
| 13:29:30 | efried | Not sure if more primitive hypervisors have this, but there's a dynamic entitlement thing where you can unlock previously-unavailable CPUs on the fly. | |
| 13:29:35 | johnthetubaguy | bhagyashris: there was a -1 review on that patch with no answer when I last looked, so I skipped looking any deeper, did you respond to their questions yet? | |
| 13:30:28 | johnthetubaguy | mriedem: it totally feels like the toggling reserved values ticks the 80% case, with very few surpizes | |
| 13:31:02 | cdent | efried, mriedem, bauzas, johnthetubaguy, rgerganov: it feels like we probably have the tooling to make something workable for this, but as we figure it out it would be good to document some kind of (hate this phrase) best practice, | |