| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-05 | |||
| 18:34:25 | jaypipes | efried: could you please explain to me your comment on https://review.openstack.org/#/c/534339/ then? If I have a single compute node with 2 PFs with 2 SRIOV_NET_VFs, and I request 4 SRIOV_NET_VF, how exactly are you proposing that we determine that that compute node is a good match for the request? | |
| 18:34:31 | odyssey4me | so I shouldn't be so nice and just wipe them :) | |
| 18:34:51 | efried | jaypipes: stand by | |
| 18:35:30 | odyssey4me | ok, many thanks - it makes much more sense now.. effectively it's no longer a thing, so don't do it | |
| 18:35:33 | mriedem | odyssey4me: likely a relic of your upgrade tooling | |
| 18:35:45 | mriedem | worked going from pre-cellsv2 to cellsv2, but not cellsv2+ | |
| 18:36:03 | mriedem | and yeah, was never a use in disabling / enabling non-compute services | |
| 18:36:40 | efried | jaypipes: I'm not sure which comment you're referring to, but to answer this: "If I have a single compute node with 2 PFs with 2 SRIOV_NET_VFs, and I request 4 SRIOV_NET_VF, how exactly are you proposing that we determine that that compute node is a good match for the request?" | |
| 18:38:08 | efried | jaypipes: Without knowing beforehand whether the compute node has one PF or two or four, or how many VFs are available on any of them, the most flexible way to issue that request and have the highest chance of landing *somewhere* would be: resources1=VF:1&resources2=VF:1&resources3=VF:1&resources4=VF:1 | |
| 18:38:55 | efried | that's of course assuming there are no traits in play (which in the case of PFs there likely would be, to represent nets or whatever) | |
| 18:39:34 | efried | One can envision a HAWeigher that would prefer results that were "spread". Etc. etc. | |
| 18:42:04 | jaypipes | efried: wow... | |
| 18:42:54 | jaypipes | efried: and what about DISK_GB? if we don't get a single provider that has 1024 GB of disk space from a single disk, do we break the request into two requests for 512 MB from two different resource providers? | |
| 18:43:12 | efried | not unless you want two disks. | |
| 18:43:30 | jaypipes | efried: the resource is not a disk. The resource is DISK_GB. i.e. an amount of GB of disk space. | |
| 18:43:55 | efried | Right. Point is that the *caller* understands the semantics of that, though. Not placement. | |
| 18:44:41 | efried | So if you want two 512GB disks, you say resources1=DISK_GB:512&resources2=DISK_GB:512. They may come from the same provider (which will respond with DISK_GB:1024 and you'll have to go back to your request to figure out that you wanted to split it up) or they may come from separate providers. | |
| 18:45:45 | efried | It would be nice if the user didn't have to know that the systems in the cloud have one disk or ten, on PF or eight, etc. He just wants his instance to land. | |
| 18:46:06 | efried | Or maybe that's wrong. Maybe there needs to be tight coupling between the flavors and the exact topology of the cloud. | |
| 18:46:18 | efried | but that doesn't seem very... cloudy to me. | |
| 18:46:50 | jaypipes | efried: I just want to make sure we're tackling real-world problems. | |
| 18:47:12 | efried | okay: what's the real-world problem that demands separation? NUMA? | |
| 18:47:28 | jaypipes | efried: I don't believe that there is a viable use case for a request for 4 SRIOV VFs and the requester doesn't care whether the VFs are provided by a single PF or two PFs. | |
| 18:47:56 | efried | FWIW, we implemented exactly that use case in powervm for SR-IOV. | |
| 18:48:08 | efried | For HA. | |
| 18:48:37 | jaypipes | efried: when does a requester of SRIOV VFs *not* want HA? | |
| 18:48:52 | openstackgerrit | Merged openstack/nova master: Avoid showing password in log https://review.openstack.org/558694 | |
| 18:49:02 | openstackgerrit | Merged openstack/nova master: Add __repr__ for NovaException https://review.openstack.org/555812 | |
| 18:49:11 | efried | When there's only one PF available. | |
| 18:50:02 | jaypipes | efried: if there's only one PF available, and the image/flavor needs >1 SRIOV VF, then IMHO, that compute node with only a single PF shouldn't be a legit destination node. | |
| 18:50:31 | efried | jaypipes: If I don't care about HA, and I said resources1=SRIOV_NET_VF:1&required1=CUSTOM_PHYSNET_A&resources2=SRIOV_NET_VF:1&required2=CUSTOM_PHYSNET_B (because that's the only way I can get two VFs on separate physnets), and I was *forced* to use two separate PFs to make that happen, I would not be able to land an instance on a node with just one PF. | |
| 18:50:42 | jaypipes | efried: the whole purpose of having multiple VFs is to allow active/failover for links... | |
| 18:50:50 | efried | When HA is the issue, yes. | |
| 18:54:25 | jaypipes | welcome back | |
| 18:56:23 | efried | Was that my glitch or a server thing? | |
| 18:56:30 | efried | Did I miss stuff? | |
| 18:56:37 | efried | Here's what I was about to say: | |
| 18:58:03 | efried | Look, I'm (still) not arguing that there are cases where it will be useful to force split. I'm (still) asserting that we're going to need to be able to handle both. One behavior will be the default, and the other will require some extra syntax to make it happen. I'm not convinced we truly need *either* immediately. So I've been advocating for the default being the one that's in the spec, because that's the one that make | |
| 18:58:03 | efried | head since Denver. But as I said here https://review.openstack.org/#/c/555081/4/specs/rocky/approved/cpu-resources.rst@412 I get that I'm being outvoted. So let's stop trying to convince each other that ours is the one true vision and just get on with making things happen. I'm implementing the granular algorithm right now. If you want to propose an amendment to the spec, let's git r done. | |
| 18:59:16 | efried | (But on that note, I don't have a good way to tie separate request groups together unless allocation requests include the anchor provider.) | |
| 19:01:04 | jaypipes | efried: ack | |
| 19:18:56 | eandersson | What is the status of NUMA migration / evacuation etc? | |
| 19:19:40 | mriedem | eandersson: there is a spec for supporting live migration of numa instances https://review.openstack.org/#/c/552722/ | |
| 19:19:43 | mriedem | artom owns that | |
| 19:19:49 | mriedem | evac.... | |
| 19:19:58 | eandersson | Do you know when resize etc was fixed? | |
| 19:20:12 | mriedem | evac should work, not sure when cold migrate was fixed | |
| 19:20:18 | mriedem | cfriesen_ might know | |
| 19:29:10 | melwitt | what sort of testing do we generally do for the metadata API service? mostly only unit tests? | |
| 19:31:22 | melwitt | nvm, I found some func tests too | |
| 19:34:00 | eandersson | mriedem, cfriesen_ thanks - yea was hoping to figure out when (resize, evacuate etc) was actually fixed | |
| 19:34:35 | mriedem | eandersson: i'm not sure what about evac would be broken for instances with numa | |
| 19:34:56 | mriedem | evac goes through the scheduler to find a new host and has to do a claim on that new host, just like normal server create | |
| 19:34:58 | eandersson | Yea - it's possible we were mislead by this bug report https://bugs.launchpad.net/nova/+bug/1417667 | |
| 19:34:58 | openstack | Launchpad bug 1417667 in OpenStack Compute (nova) "migration/evacuation/rebuild/resize/unshelve of instance with NUMA topology needs to recalculate NUMA topology" [Medium,In progress] - Assigned to sahid (sahid-ferdjaoui) | |
| 19:35:50 | mriedem | that does look misleading, | |
| 19:36:07 | mriedem | cfriesen_ reported that for live migration, and then it looks like a blanket statement was made about other operations, including rebuild, which isn't a move operation | |
| 19:37:01 | eandersson | We did do a lot of testing on this, but honestly don't remember what was actually broken (besides live migration) for us | |
| 19:37:05 | mriedem | eandersson: so https://review.openstack.org/#/q/topic:bug/1417667+(status:open+OR+status:merged) | |
| 19:37:14 | mriedem | should handle all of the cold migrate/evac/resize cases i think | |
| 19:38:08 | mriedem | https://review.openstack.org/#/c/226411/ goes back to liberty | |
| 19:38:40 | mriedem | https://review.openstack.org/#/c/218938/ looks like that is the fix for cold migrate / resize | |
| 19:40:37 | eandersson | btw unrelated by is there a reason why filters and weights weren't made into plugins? | |
| 19:40:44 | mriedem | so i think we can probably mark this as fixed for evac and cold migrate, and open a new bug for tracking the live migration issue, which is being resolved via https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-live-migration.html in rocky | |
| 19:41:00 | mriedem | you can plug those in | |
| 19:41:04 | eandersson | oh | |
| 19:41:29 | eandersson | we have been adding them to the nova/schedulers/filters etc :D | |
| 19:41:35 | mriedem | cfriesen_: if you're around and https://bugs.launchpad.net/nova/+bug/1417667/comments/45 is accurate, we should work on closing out that bug to reflect reality since Liberty | |
| 19:41:35 | openstack | Launchpad bug 1417667 in OpenStack Compute (nova) "migration/evacuation/rebuild/resize/unshelve of instance with NUMA topology needs to recalculate NUMA topology" [Medium,In progress] - Assigned to sahid (sahid-ferdjaoui) | |
| 19:41:49 | mriedem | eandersson: we == ? | |
| 19:41:50 | mriedem | suse? | |
| 19:42:02 | eandersson | *I have :D for my deployment | |
| 19:42:23 | melwitt | you mean adding custom out-of-tree ones right? if you have generally useful weighers/filters you can upstream them | |
| 19:42:57 | eandersson | Yea for sure :D | |
| 19:42:59 | melwitt | do we allow custom out of tree ones? I thought maybe not | |
| 19:43:02 | mriedem | eandersson: https://docs.openstack.org/nova/latest/user/filter-scheduler.html#writing-your-own-filter | |
| 19:43:08 | mriedem | melwitt: yes we do | |
| 19:43:15 | melwitt | I can't remember if we restricted it to in-tree only | |
| 19:43:18 | melwitt | okay, nevermind then | |
| 19:43:18 | mriedem | that's also why scheduler hints are wild west | |
| 19:43:40 | mriedem | hell you can even load your own scheduler driver | |
| 19:44:01 | eandersson | I think we may just have backported some of the newer filter fixes | |
| 19:44:16 | mriedem | eandersson: which release are you on? | |
| 19:44:23 | eandersson | mitaka atm | |
| 19:44:33 | openstackgerrit | Tyler Blakeslee proposed openstack/nova stable/queens: Add __repr__ for NovaException https://review.openstack.org/559158 | |
| 19:47:53 | mriedem | ok so i think i was wrong about the plugin, | |
| 19:47:59 | mriedem | i think that might just be the scheduler driver entrypoint | |
| 19:48:14 | mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/__init__.py#L64 | |
| 19:48:21 | mriedem | loads up the filters from nova/scheduler/filters as you said | |
| 19:48:46 | mriedem | https://docs.openstack.org/nova/latest/user/filter-scheduler.html#writing-your-own-filter doesn't mention that you have to drop them into the code tree | |
| 19:49:09 | eandersson | It would be a nice feature, because someone might write a scheduler to automatically upgrade their computes | |
| 19:49:17 | eandersson | you know sounds like a cool thing | |
| 19:49:27 | melwitt | it used to be free-form pluggable but it locked down more recently, IIRC. and the doc not updated | |
| 19:49:36 | eandersson | I see | |
| 19:49:39 | mriedem | eandersson: typically nova is anti-plugin | |
| 19:49:55 | mriedem | and have slowly evolved over time to cut out plugin points, as they aren't tested and we break them as such | |
| 19:50:07 | mriedem | plus, interop is a concern with too many plug points | |
| 19:50:30 | mriedem | for a private cloud that just wants to customize everything, interop isn't a concern, but it's a concern for public cloud | |
| 19:50:52 | mriedem | well, could be a concern for private cloud if it ever wants to use the same APIs on a public cloud | |
| 19:51:18 | melwitt | yeah, I've seen it bite private cloud when people used other open source tools that talk to openstack | |