| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-07 | |||
| 19:09:50 | artom | mriedem, also, I'm pretty sure we need loads of underlying stuff for PCI devices to be live-migratable | |
| 19:10:04 | mriedem | jaypipes: or do we have CN RP (root) -> PF RP (child) -> VF RP (grandchild)? | |
| 19:10:08 | artom | We had an internal call this morning and someone brought up live migration with SRIOV | |
| 19:10:37 | artom | And while I still need to do research and ask around, the conclusion was "we need low level stuff, possible even at the kernel level" | |
| 19:11:10 | mnaser | can someone give me some pointers about where to RTFM regarding what happens between the time a new server request comes in and the vm gets a cell assigned? i'm seeing behaviour where VMs will intermittantly (under load) go in BUILD status, with no task | |
| 19:11:12 | dansmith | mriedem: VF is a resource not a provider | |
| 19:11:17 | dansmith | mriedem: a PF is a provider (of VFs) | |
| 19:11:33 | dansmith | PF is also a provider of inventory of 1 PF, in case you want to pass a PF into the guest | |
| 19:11:35 | mnaser | usually i would see BUILD status + SCHEDULING when i see any scheduler issues, but this seems before a cell gets assigned (i think?) | |
| 19:11:49 | mnaser | not looking for an answer, but maybe a pointer to where should i be reading | |
| 19:12:12 | mriedem | dansmith: ok, trying to load https://review.openstack.org/#/c/485522/ into my brain | |
| 19:13:05 | mriedem | mnaser: the build request, which represents the instance until it's created in a cell, is created in the compute API._provision_instances method | |
| 19:13:15 | mriedem | the API casts to conductor's schedule_and_build_instances method, | |
| 19:13:38 | mriedem | which calls the scheduler for hosts and then based on the host selected, finds the cell the host is in and creates the instance in that cell | |
| 19:13:41 | mriedem | then deletes the build request | |
| 19:13:53 | mnaser | mriedem: ok i see, i'll trace from there, i guess something is happening so the instance doesn't even go into scheduling state.. so i'll see what happens across there | |
| 19:14:09 | mnaser | it's strange because it just happens from time to time (but seems to happen more under stress) | |
| 19:14:47 | mriedem | dansmith: was wondering if in this trusted VF thing, with placement, if there would be a way to say a VF is trusted, but if it's a resource and not a provider, then i don't think there is | |
| 19:15:10 | mriedem | it sounds like the thing would be saying the PF has trusted VFs | |
| 19:15:22 | artom | Could "trusted" be a trait? | |
| 19:15:27 | mriedem | artom: that's what i'm saying | |
| 19:15:42 | mriedem | the PF resource provider of VF inventory would have a 'trusted' trait | |
| 19:16:03 | artom | Grr, I keep forgetting only providers have traits | |
| 19:16:18 | dansmith | how would one VF be trusted and not another? | |
| 19:16:20 | mriedem | and then the flavor extra spec says it requires trusted VFs, and the flavor could be in a set of host aggregates for hosts that provide trusted VFs | |
| 19:16:45 | dansmith | either way, if you want to have some VFs with a trait, you expose two providers with the VFs you want, and a trait of trusted | |
| 19:16:58 | dansmith | but yeah, placement is the way to do this I think | |
| 19:17:25 | mriedem | i haven't gotten far enough long in this spec to yet see what is the proposed solution, but i'm assuming it doesn't involve placement | |
| 19:17:36 | mriedem | so i'm working up the context for my questions | |
| 19:17:45 | dansmith | yeah,m doesn't look like it | |
| 19:18:26 | artom | A thing to keep in mind is that the operator decides what's trusted and not | |
| 19:18:34 | artom | There's a way for them to do that in placement, right? | |
| 19:21:14 | mriedem | well, | |
| 19:21:20 | mriedem | presumably | |
| 19:21:28 | mriedem | admins can set traits on providers, yes | |
| 19:21:32 | mriedem | however, | |
| 19:21:52 | mriedem | nova currently doesn't merge in any externally set traits if it's also setting traits on the same provider | |
| 19:21:53 | mriedem | like the compute node | |
| 19:21:57 | mriedem | that came up on friday at the ptg | |
| 19:22:01 | mriedem | remember artom?! | |
| 19:22:23 | artom | Why am I sensing aggression? ;) | |
| 19:22:27 | mriedem | ha | |
| 19:22:32 | artom | But yeah, I remember the merging of traits | |
| 19:23:05 | openstackgerrit | Merged openstack/nova master: Allow 'network' in RequestContext service_catalog https://review.openstack.org/548572 | |
| 19:23:30 | artom | So that... could be a problem. Or a dependency, I should say | |
| 19:24:39 | mriedem | bauzas: ^ merged so can you remove the -W on https://review.openstack.org/#/c/550259/ ? | |
| 19:25:20 | mriedem | artom: well in the case of PFs it's likely not a problem *today* since nova doesn't create PF providers or set traits on them, but with NRP i'm not sure what the plan is for who creates PFs | |
| 19:25:23 | mriedem | nova or neutron or other | |
| 19:25:50 | artom | mriedem, right, but I'm thinking of the trusted VFs thing | |
| 19:26:19 | artom | If the plan is to handle them in placement (which I think is a good idea), then we probably need some dependencies to land first (or semi-concurrently, at least) | |
| 19:26:33 | artom | Well | |
| 19:26:39 | mriedem | artom: oh totally | |
| 19:26:50 | artom | Unless Nova just sets *no* traits on VFs at all | |
| 19:26:59 | mriedem | before i even read it, i'm assuming the spec is not placement, but just a frankenstein of flavor extra specs, host aggregates, and neutron binding profile plumbing | |
| 19:27:01 | artom | So that only the externally set "trusted" trait is, err, set | |
| 19:27:07 | artom | mriedem, oh yeah | |
| 19:27:28 | mriedem | and to say "we should do this the right way (tm) with placement" it would mean it not happening in rocky | |
| 19:28:01 | artom | Well, my downstream hat wants to say "do it now!" but my upstream hat is saying "let's do this properly" | |
| 19:29:21 | dansmith | +1 from me | |
| 19:34:01 | mriedem | i'll take that as dansmith +1ing the 'do it now' part of that | |
| 19:34:11 | mriedem | quick and dirty yeehaw pew pew | |
| 19:35:25 | mnaser | so i've been trying to find how nova sets an instance to SCHEDULING task state.. i see two places where it is actually set and one is in compute/api.py in _populate_instance_for_create which seems like cells v1 code from what i read, the other one is in _do_build_and_run_instance() upon 'RescheduledException' being raised, it seems to reset it back to SCHEDULING | |
| 19:35:31 | mnaser | but i can't find where it gets set for the first time? :\ | |
| 19:36:25 | mnaser | in following _provision_instances down to the conductor schedule_and_build_instances, i dont see anything that changes the task_state | |
| 19:37:11 | mriedem | mnaser: _provision_instances -> create_db_entry_for_new_instance -> _populate_instance_for_create | |
| 19:37:30 | mriedem | the instance returned is serialized and stored in the build_requiest | |
| 19:37:32 | mriedem | *request | |
| 19:37:33 | mnaser | mriedem: so the comment right above create db entry saying "This method remains because cellsv1 uses it in the scheduler" | |
| 19:37:46 | mnaser | is a bit misleading maybe? | |
| 19:38:01 | mriedem | yeah, i think that's more for the create_instance kwarg on the method | |
| 19:38:10 | mnaser | gr, you're right, that's part of the flow too | |
| 19:38:32 | mnaser | create_db_entry_for_new_instance => _populate_instance_for_create => instance.task_state = task_states.SCHEDULING | |
| 19:38:38 | mriedem | right | |
| 19:38:43 | mnaser | so i guess it never reaches that state because the instance is in BUILD/None | |
| 19:39:20 | mriedem | it should be SCHEDULING until the next thing that changes the task_state | |
| 19:39:53 | mnaser | hmm, i see, so maybe the issue here is that something takes it from BUILD/SCHEDULING => BUILD/None and it just stalls out | |
| 19:39:56 | mriedem | which i think is build_and_run_instance in the compute | |
| 19:40:09 | mnaser | but i dont see a compute host listed when i see the instance in that state | |
| 19:40:21 | mriedem | is the instance in cell0? | |
| 19:40:37 | jaypipes | mriedem: typically a VF will be represented as an amount of SRIOV_NET_VF resource class inventory on a resource provider representing the PF. However, it is possible that a resource provider representing the individual VF could be created by the virt driver or neutron agent IFF differences in capabilities/traits differentiate individual VFs from each other. | |
| 19:40:51 | mnaser | mriedem: i can only assume eventually it ends up there once it stalls on BUILD/None for a while, ill investigate a bit more | |
| 19:41:28 | mriedem | should be able to tell by just looking for the instance in the cell0 db | |
| 19:41:46 | mnaser | grepping scheduler logs while this happens too | |
| 19:42:23 | mnaser | it doesn't end up in cell0 | |
| 19:42:56 | mriedem | so there must be a build_request for it in the nova_api db | |
| 19:43:05 | mriedem | query the build_requests table for that instance uuid? | |
| 19:44:06 | mnaser | no build request in nova_api, but the instance is in the cell db (not cell0) with state as 'building' | |
| 19:44:11 | mnaser | until i assume the instance was just deleted | |
| 19:44:23 | mriedem | but it doesn't have a host set on the instance right? | |
| 19:44:36 | mnaser | nope | |
| 19:44:38 | mriedem | so the instance is in cell1, but host is not set, and vm_state is BUILDING | |
| 19:44:41 | mriedem | and task_state is None | |
| 19:44:55 | mnaser | mriedem: http://paste.openstack.org/show/693715/ exactly | |
| 19:45:10 | mnaser | thats "select uuid, vm_state, task_state, host from instances where uuid='58dec7d0-00f3-448b-a7ce-90e4c2b7d4de';" in the nova db | |
| 19:45:17 | mriedem | is there a fault recorded on the instance? | |
| 19:45:58 | mnaser | uh are faults stored in the same table | |
| 19:46:08 | mnaser | nope instance_faults | |
| 19:46:23 | mnaser | nope | |
| 19:46:27 | mnaser | no faults | |
| 19:46:52 | mnaser | the instance stays there forever, im pretty sure its just deleted by the api user and they're reattempting to launch the instance, so the behaviour is non deterministic | |