| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-05 | |||
| 18:28:11 | efried | :param use_same_provider: | |
| 18:28:11 | efried | If True, (the default) this RequestGroup represents requests for | |
| 18:28:11 | efried | resources and traits which must be satisfied by a single resource | |
| 18:28:11 | efried | provider. If False, represents a request for resources and traits | |
| 18:28:11 | efried | in any resource provider in the same tree, or a sharing provider. | |
| 18:28:42 | jaypipes | odyssey4me: yeah, sec, grabbing one | |
| 18:28:44 | efried | To put it in context, though, a RequestGroup is *one* request group. This says nothing of how this RequestGroup interacts with other RequestGroups. | |
| 18:29:05 | jaypipes | odyssey4me: https://docs.openstack.org/nova/latest/cli/nova-manage.html#nova-cells-v2 | |
| 18:29:05 | odyssey4me | tyvm, appreciate your time jaypipes | |
| 18:29:33 | jaypipes | odyssey4me: also try this: | |
| 18:29:34 | jaypipes | https://docs.openstack.org/nova/latest/cli/nova-status.html#upgrade | |
| 18:29:39 | jaypipes | nova-manage status check | |
| 18:30:31 | odyssey4me | ooh, a new command :) | |
| 18:30:53 | jaypipes | odyssey4me: sorry, that should have been nova-status upgrade check | |
| 18:31:13 | odyssey4me | with return codes and everything, fancy :) | |
| 18:31:22 | odyssey4me | yeah, got that - it says all is well | |
| 18:31:23 | jaypipes | odyssey4me: you can thank mriedem for it. | |
| 18:32:00 | mriedem | i think that's microversion 2.53 you're hitting | |
| 18:32:05 | odyssey4me | many thanks to mriedem then - we shall make use of this in ways similar to keystone doctor | |
| 18:32:20 | mriedem | odyssey4me: the services API only allows enabling/disabling 'compute' services once you're on cells v2 | |
| 18:32:43 | mriedem | b/c those compute service are mapped to a cell in the api db, but other non-compute services aren't | |
| 18:32:45 | mriedem | plus, | |
| 18:32:56 | mriedem | it doesn't make sense to disable non-compute services, as that doesn't mean/do anything | |
| 18:33:07 | odyssey4me | ok, then something's probably wrong in our config or something - because this started life as pike, so cellsv2 should have just been there from the start | |
| 18:33:27 | mriedem | odyssey4me: https://docs.openstack.org/releasenotes/nova/pike.html#id23 | |
| 18:33:32 | mriedem | The PUT /os-services/disable, PUT /os-services/enable and PUT /os-services/force-down APIs to enable, disable, or force-down a service will now only work with nova-compute services. If you are using those APIs to try and disable a non-compute service, like nova-scheduler or nova-conductor, those APIs will result in a 404 response. | |
| 18:33:35 | odyssey4me | oh doesn't it? I'm trying to do a nice process of disable, then shut down, then wipe the containers | |
| 18:34:09 | mriedem | check that release note | |
| 18:34:21 | odyssey4me | aha, I see | |
| 18:34:25 | jaypipes | efried: could you please explain to me your comment on https://review.openstack.org/#/c/534339/ then? If I have a single compute node with 2 PFs with 2 SRIOV_NET_VFs, and I request 4 SRIOV_NET_VF, how exactly are you proposing that we determine that that compute node is a good match for the request? | |
| 18:34:31 | odyssey4me | so I shouldn't be so nice and just wipe them :) | |
| 18:34:51 | efried | jaypipes: stand by | |
| 18:35:30 | odyssey4me | ok, many thanks - it makes much more sense now.. effectively it's no longer a thing, so don't do it | |
| 18:35:33 | mriedem | odyssey4me: likely a relic of your upgrade tooling | |
| 18:35:45 | mriedem | worked going from pre-cellsv2 to cellsv2, but not cellsv2+ | |
| 18:36:03 | mriedem | and yeah, was never a use in disabling / enabling non-compute services | |
| 18:36:40 | efried | jaypipes: I'm not sure which comment you're referring to, but to answer this: "If I have a single compute node with 2 PFs with 2 SRIOV_NET_VFs, and I request 4 SRIOV_NET_VF, how exactly are you proposing that we determine that that compute node is a good match for the request?" | |
| 18:38:08 | efried | jaypipes: Without knowing beforehand whether the compute node has one PF or two or four, or how many VFs are available on any of them, the most flexible way to issue that request and have the highest chance of landing *somewhere* would be: resources1=VF:1&resources2=VF:1&resources3=VF:1&resources4=VF:1 | |
| 18:38:55 | efried | that's of course assuming there are no traits in play (which in the case of PFs there likely would be, to represent nets or whatever) | |
| 18:39:34 | efried | One can envision a HAWeigher that would prefer results that were "spread". Etc. etc. | |
| 18:42:04 | jaypipes | efried: wow... | |
| 18:42:54 | jaypipes | efried: and what about DISK_GB? if we don't get a single provider that has 1024 GB of disk space from a single disk, do we break the request into two requests for 512 MB from two different resource providers? | |
| 18:43:12 | efried | not unless you want two disks. | |
| 18:43:30 | jaypipes | efried: the resource is not a disk. The resource is DISK_GB. i.e. an amount of GB of disk space. | |
| 18:43:55 | efried | Right. Point is that the *caller* understands the semantics of that, though. Not placement. | |
| 18:44:41 | efried | So if you want two 512GB disks, you say resources1=DISK_GB:512&resources2=DISK_GB:512. They may come from the same provider (which will respond with DISK_GB:1024 and you'll have to go back to your request to figure out that you wanted to split it up) or they may come from separate providers. | |
| 18:45:45 | efried | It would be nice if the user didn't have to know that the systems in the cloud have one disk or ten, on PF or eight, etc. He just wants his instance to land. | |
| 18:46:06 | efried | Or maybe that's wrong. Maybe there needs to be tight coupling between the flavors and the exact topology of the cloud. | |
| 18:46:18 | efried | but that doesn't seem very... cloudy to me. | |
| 18:46:50 | jaypipes | efried: I just want to make sure we're tackling real-world problems. | |
| 18:47:12 | efried | okay: what's the real-world problem that demands separation? NUMA? | |
| 18:47:28 | jaypipes | efried: I don't believe that there is a viable use case for a request for 4 SRIOV VFs and the requester doesn't care whether the VFs are provided by a single PF or two PFs. | |
| 18:47:56 | efried | FWIW, we implemented exactly that use case in powervm for SR-IOV. | |
| 18:48:08 | efried | For HA. | |
| 18:48:37 | jaypipes | efried: when does a requester of SRIOV VFs *not* want HA? | |
| 18:48:52 | openstackgerrit | Merged openstack/nova master: Avoid showing password in log https://review.openstack.org/558694 | |
| 18:49:02 | openstackgerrit | Merged openstack/nova master: Add __repr__ for NovaException https://review.openstack.org/555812 | |
| 18:49:11 | efried | When there's only one PF available. | |
| 18:50:02 | jaypipes | efried: if there's only one PF available, and the image/flavor needs >1 SRIOV VF, then IMHO, that compute node with only a single PF shouldn't be a legit destination node. | |
| 18:50:31 | efried | jaypipes: If I don't care about HA, and I said resources1=SRIOV_NET_VF:1&required1=CUSTOM_PHYSNET_A&resources2=SRIOV_NET_VF:1&required2=CUSTOM_PHYSNET_B (because that's the only way I can get two VFs on separate physnets), and I was *forced* to use two separate PFs to make that happen, I would not be able to land an instance on a node with just one PF. | |
| 18:50:42 | jaypipes | efried: the whole purpose of having multiple VFs is to allow active/failover for links... | |
| 18:50:50 | efried | When HA is the issue, yes. | |
| 18:54:25 | jaypipes | welcome back | |
| 18:56:23 | efried | Was that my glitch or a server thing? | |
| 18:56:30 | efried | Did I miss stuff? | |
| 18:56:37 | efried | Here's what I was about to say: | |
| 18:58:03 | efried | Look, I'm (still) not arguing that there are cases where it will be useful to force split. I'm (still) asserting that we're going to need to be able to handle both. One behavior will be the default, and the other will require some extra syntax to make it happen. I'm not convinced we truly need *either* immediately. So I've been advocating for the default being the one that's in the spec, because that's the one that make | |
| 18:58:03 | efried | head since Denver. But as I said here https://review.openstack.org/#/c/555081/4/specs/rocky/approved/cpu-resources.rst@412 I get that I'm being outvoted. So let's stop trying to convince each other that ours is the one true vision and just get on with making things happen. I'm implementing the granular algorithm right now. If you want to propose an amendment to the spec, let's git r done. | |
| 18:59:16 | efried | (But on that note, I don't have a good way to tie separate request groups together unless allocation requests include the anchor provider.) | |
| 19:01:04 | jaypipes | efried: ack | |
| 19:18:56 | eandersson | What is the status of NUMA migration / evacuation etc? | |
| 19:19:40 | mriedem | eandersson: there is a spec for supporting live migration of numa instances https://review.openstack.org/#/c/552722/ | |
| 19:19:43 | mriedem | artom owns that | |
| 19:19:49 | mriedem | evac.... | |
| 19:19:58 | eandersson | Do you know when resize etc was fixed? | |
| 19:20:12 | mriedem | evac should work, not sure when cold migrate was fixed | |
| 19:20:18 | mriedem | cfriesen_ might know | |
| 19:29:10 | melwitt | what sort of testing do we generally do for the metadata API service? mostly only unit tests? | |
| 19:31:22 | melwitt | nvm, I found some func tests too | |
| 19:34:00 | eandersson | mriedem, cfriesen_ thanks - yea was hoping to figure out when (resize, evacuate etc) was actually fixed | |
| 19:34:35 | mriedem | eandersson: i'm not sure what about evac would be broken for instances with numa | |
| 19:34:56 | mriedem | evac goes through the scheduler to find a new host and has to do a claim on that new host, just like normal server create | |
| 19:34:58 | eandersson | Yea - it's possible we were mislead by this bug report https://bugs.launchpad.net/nova/+bug/1417667 | |
| 19:34:58 | openstack | Launchpad bug 1417667 in OpenStack Compute (nova) "migration/evacuation/rebuild/resize/unshelve of instance with NUMA topology needs to recalculate NUMA topology" [Medium,In progress] - Assigned to sahid (sahid-ferdjaoui) | |
| 19:35:50 | mriedem | that does look misleading, | |
| 19:36:07 | mriedem | cfriesen_ reported that for live migration, and then it looks like a blanket statement was made about other operations, including rebuild, which isn't a move operation | |
| 19:37:01 | eandersson | We did do a lot of testing on this, but honestly don't remember what was actually broken (besides live migration) for us | |
| 19:37:05 | mriedem | eandersson: so https://review.openstack.org/#/q/topic:bug/1417667+(status:open+OR+status:merged) | |
| 19:37:14 | mriedem | should handle all of the cold migrate/evac/resize cases i think | |
| 19:38:08 | mriedem | https://review.openstack.org/#/c/226411/ goes back to liberty | |
| 19:38:40 | mriedem | https://review.openstack.org/#/c/218938/ looks like that is the fix for cold migrate / resize | |
| 19:40:37 | eandersson | btw unrelated by is there a reason why filters and weights weren't made into plugins? | |
| 19:40:44 | mriedem | so i think we can probably mark this as fixed for evac and cold migrate, and open a new bug for tracking the live migration issue, which is being resolved via https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-live-migration.html in rocky | |
| 19:41:00 | mriedem | you can plug those in | |
| 19:41:04 | eandersson | oh | |
| 19:41:29 | eandersson | we have been adding them to the nova/schedulers/filters etc :D | |
| 19:41:35 | mriedem | cfriesen_: if you're around and https://bugs.launchpad.net/nova/+bug/1417667/comments/45 is accurate, we should work on closing out that bug to reflect reality since Liberty | |
| 19:41:35 | openstack | Launchpad bug 1417667 in OpenStack Compute (nova) "migration/evacuation/rebuild/resize/unshelve of instance with NUMA topology needs to recalculate NUMA topology" [Medium,In progress] - Assigned to sahid (sahid-ferdjaoui) | |
| 19:41:49 | mriedem | eandersson: we == ? | |
| 19:41:50 | mriedem | suse? | |