| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-07 | |||
| 19:16:18 | dansmith | how would one VF be trusted and not another? | |
| 19:16:20 | mriedem | and then the flavor extra spec says it requires trusted VFs, and the flavor could be in a set of host aggregates for hosts that provide trusted VFs | |
| 19:16:45 | dansmith | either way, if you want to have some VFs with a trait, you expose two providers with the VFs you want, and a trait of trusted | |
| 19:16:58 | dansmith | but yeah, placement is the way to do this I think | |
| 19:17:25 | mriedem | i haven't gotten far enough long in this spec to yet see what is the proposed solution, but i'm assuming it doesn't involve placement | |
| 19:17:36 | mriedem | so i'm working up the context for my questions | |
| 19:17:45 | dansmith | yeah,m doesn't look like it | |
| 19:18:26 | artom | A thing to keep in mind is that the operator decides what's trusted and not | |
| 19:18:34 | artom | There's a way for them to do that in placement, right? | |
| 19:21:14 | mriedem | well, | |
| 19:21:20 | mriedem | presumably | |
| 19:21:28 | mriedem | admins can set traits on providers, yes | |
| 19:21:32 | mriedem | however, | |
| 19:21:52 | mriedem | nova currently doesn't merge in any externally set traits if it's also setting traits on the same provider | |
| 19:21:53 | mriedem | like the compute node | |
| 19:21:57 | mriedem | that came up on friday at the ptg | |
| 19:22:01 | mriedem | remember artom?! | |
| 19:22:23 | artom | Why am I sensing aggression? ;) | |
| 19:22:27 | mriedem | ha | |
| 19:22:32 | artom | But yeah, I remember the merging of traits | |
| 19:23:05 | openstackgerrit | Merged openstack/nova master: Allow 'network' in RequestContext service_catalog https://review.openstack.org/548572 | |
| 19:23:30 | artom | So that... could be a problem. Or a dependency, I should say | |
| 19:24:39 | mriedem | bauzas: ^ merged so can you remove the -W on https://review.openstack.org/#/c/550259/ ? | |
| 19:25:20 | mriedem | artom: well in the case of PFs it's likely not a problem *today* since nova doesn't create PF providers or set traits on them, but with NRP i'm not sure what the plan is for who creates PFs | |
| 19:25:23 | mriedem | nova or neutron or other | |
| 19:25:50 | artom | mriedem, right, but I'm thinking of the trusted VFs thing | |
| 19:26:19 | artom | If the plan is to handle them in placement (which I think is a good idea), then we probably need some dependencies to land first (or semi-concurrently, at least) | |
| 19:26:33 | artom | Well | |
| 19:26:39 | mriedem | artom: oh totally | |
| 19:26:50 | artom | Unless Nova just sets *no* traits on VFs at all | |
| 19:26:59 | mriedem | before i even read it, i'm assuming the spec is not placement, but just a frankenstein of flavor extra specs, host aggregates, and neutron binding profile plumbing | |
| 19:27:01 | artom | So that only the externally set "trusted" trait is, err, set | |
| 19:27:07 | artom | mriedem, oh yeah | |
| 19:27:28 | mriedem | and to say "we should do this the right way (tm) with placement" it would mean it not happening in rocky | |
| 19:28:01 | artom | Well, my downstream hat wants to say "do it now!" but my upstream hat is saying "let's do this properly" | |
| 19:29:21 | dansmith | +1 from me | |
| 19:34:01 | mriedem | i'll take that as dansmith +1ing the 'do it now' part of that | |
| 19:34:11 | mriedem | quick and dirty yeehaw pew pew | |
| 19:35:25 | mnaser | so i've been trying to find how nova sets an instance to SCHEDULING task state.. i see two places where it is actually set and one is in compute/api.py in _populate_instance_for_create which seems like cells v1 code from what i read, the other one is in _do_build_and_run_instance() upon 'RescheduledException' being raised, it seems to reset it back to SCHEDULING | |
| 19:35:31 | mnaser | but i can't find where it gets set for the first time? :\ | |
| 19:36:25 | mnaser | in following _provision_instances down to the conductor schedule_and_build_instances, i dont see anything that changes the task_state | |
| 19:37:11 | mriedem | mnaser: _provision_instances -> create_db_entry_for_new_instance -> _populate_instance_for_create | |
| 19:37:30 | mriedem | the instance returned is serialized and stored in the build_requiest | |
| 19:37:32 | mriedem | *request | |
| 19:37:33 | mnaser | mriedem: so the comment right above create db entry saying "This method remains because cellsv1 uses it in the scheduler" | |
| 19:37:46 | mnaser | is a bit misleading maybe? | |
| 19:38:01 | mriedem | yeah, i think that's more for the create_instance kwarg on the method | |
| 19:38:10 | mnaser | gr, you're right, that's part of the flow too | |
| 19:38:32 | mnaser | create_db_entry_for_new_instance => _populate_instance_for_create => instance.task_state = task_states.SCHEDULING | |
| 19:38:38 | mriedem | right | |
| 19:38:43 | mnaser | so i guess it never reaches that state because the instance is in BUILD/None | |
| 19:39:20 | mriedem | it should be SCHEDULING until the next thing that changes the task_state | |
| 19:39:53 | mnaser | hmm, i see, so maybe the issue here is that something takes it from BUILD/SCHEDULING => BUILD/None and it just stalls out | |
| 19:39:56 | mriedem | which i think is build_and_run_instance in the compute | |
| 19:40:09 | mnaser | but i dont see a compute host listed when i see the instance in that state | |
| 19:40:21 | mriedem | is the instance in cell0? | |
| 19:40:37 | jaypipes | mriedem: typically a VF will be represented as an amount of SRIOV_NET_VF resource class inventory on a resource provider representing the PF. However, it is possible that a resource provider representing the individual VF could be created by the virt driver or neutron agent IFF differences in capabilities/traits differentiate individual VFs from each other. | |
| 19:40:51 | mnaser | mriedem: i can only assume eventually it ends up there once it stalls on BUILD/None for a while, ill investigate a bit more | |
| 19:41:28 | mriedem | should be able to tell by just looking for the instance in the cell0 db | |
| 19:41:46 | mnaser | grepping scheduler logs while this happens too | |
| 19:42:23 | mnaser | it doesn't end up in cell0 | |
| 19:42:56 | mriedem | so there must be a build_request for it in the nova_api db | |
| 19:43:05 | mriedem | query the build_requests table for that instance uuid? | |
| 19:44:06 | mnaser | no build request in nova_api, but the instance is in the cell db (not cell0) with state as 'building' | |
| 19:44:11 | mnaser | until i assume the instance was just deleted | |
| 19:44:23 | mriedem | but it doesn't have a host set on the instance right? | |
| 19:44:36 | mnaser | nope | |
| 19:44:38 | mriedem | so the instance is in cell1, but host is not set, and vm_state is BUILDING | |
| 19:44:41 | mriedem | and task_state is None | |
| 19:44:55 | mnaser | mriedem: http://paste.openstack.org/show/693715/ exactly | |
| 19:45:10 | mnaser | thats "select uuid, vm_state, task_state, host from instances where uuid='58dec7d0-00f3-448b-a7ce-90e4c2b7d4de';" in the nova db | |
| 19:45:17 | mriedem | is there a fault recorded on the instance? | |
| 19:45:58 | mnaser | uh are faults stored in the same table | |
| 19:46:08 | mnaser | nope instance_faults | |
| 19:46:23 | mnaser | nope | |
| 19:46:27 | mnaser | no faults | |
| 19:46:52 | mnaser | the instance stays there forever, im pretty sure its just deleted by the api user and they're reattempting to launch the instance, so the behaviour is non deterministic | |
| 19:47:04 | mriedem | yeah so my guess is _do_build_and_run_instance set task_state=None and then something hung or failed | |
| 19:47:24 | mriedem | the instance.host gets set in the resource tracker after it's spawned | |
| 19:48:30 | mriedem | so networking or block device setup maybe failed? or driver.spawn failed | |
| 19:50:13 | mriedem | however, those should either result in a reschedule or the instance being put into ERROR state | |
| 19:50:34 | mriedem | mnaser: probably have to trace the server create request id through the compute logs to find out what happened to i | |
| 19:50:35 | mriedem | *it | |
| 19:50:54 | mnaser | mriedem: actually, i'm looking at recent instance_faults and im seeing some rpc timeouts | |
| 19:50:58 | mriedem | jaypipes: ok (on the VF/PF thing) | |
| 19:51:13 | mnaser | which could very well explain things.. ill dig more, because there's plenty of conductors serving here | |
| 19:51:17 | mriedem | mnaser: so it's possible the state changes to the instance never made it through conductor to the db | |
| 19:51:19 | mnaser | thanks for the small talk through, that gives a hint | |
| 19:51:31 | mnaser | instance.save(expected_task_state=task_states.SPAWNING) | |
| 19:51:31 | mnaser | File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2110, in _build_and_run_instance | |
| 19:51:41 | mnaser | it is *very* possible considering that's the stacktrace that timed out... | |
| 19:52:55 | mriedem | yup, and that's the time that the instance.host would be set and saved off from the RT.instance_claim() | |
| 19:54:10 | mnaser | well at least we know things are breaking the way they're supposed to be.. | |
| 19:55:48 | mnaser | dansmith: i'm not sure, i'm worried about finding that part out :p | |
| 19:55:59 | dansmith | should be any day now.. :) | |
| 19:56:08 | mnaser | our queens upgrades are happening soon | |
| 19:56:31 | mnaser | i'm sure that'll push things | |
| 19:56:32 | mnaser | :P | |
| 19:56:43 | mriedem | mnaser: i'd wait for 17.0.1 | |
| 19:56:44 | dansmith | heh | |