| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-07 | |||
| 19:27:28 | mriedem | and to say "we should do this the right way (tm) with placement" it would mean it not happening in rocky | |
| 19:28:01 | artom | Well, my downstream hat wants to say "do it now!" but my upstream hat is saying "let's do this properly" | |
| 19:29:21 | dansmith | +1 from me | |
| 19:34:01 | mriedem | i'll take that as dansmith +1ing the 'do it now' part of that | |
| 19:34:11 | mriedem | quick and dirty yeehaw pew pew | |
| 19:35:25 | mnaser | so i've been trying to find how nova sets an instance to SCHEDULING task state.. i see two places where it is actually set and one is in compute/api.py in _populate_instance_for_create which seems like cells v1 code from what i read, the other one is in _do_build_and_run_instance() upon 'RescheduledException' being raised, it seems to reset it back to SCHEDULING | |
| 19:35:31 | mnaser | but i can't find where it gets set for the first time? :\ | |
| 19:36:25 | mnaser | in following _provision_instances down to the conductor schedule_and_build_instances, i dont see anything that changes the task_state | |
| 19:37:11 | mriedem | mnaser: _provision_instances -> create_db_entry_for_new_instance -> _populate_instance_for_create | |
| 19:37:30 | mriedem | the instance returned is serialized and stored in the build_requiest | |
| 19:37:32 | mriedem | *request | |
| 19:37:33 | mnaser | mriedem: so the comment right above create db entry saying "This method remains because cellsv1 uses it in the scheduler" | |
| 19:37:46 | mnaser | is a bit misleading maybe? | |
| 19:38:01 | mriedem | yeah, i think that's more for the create_instance kwarg on the method | |
| 19:38:10 | mnaser | gr, you're right, that's part of the flow too | |
| 19:38:32 | mnaser | create_db_entry_for_new_instance => _populate_instance_for_create => instance.task_state = task_states.SCHEDULING | |
| 19:38:38 | mriedem | right | |
| 19:38:43 | mnaser | so i guess it never reaches that state because the instance is in BUILD/None | |
| 19:39:20 | mriedem | it should be SCHEDULING until the next thing that changes the task_state | |
| 19:39:53 | mnaser | hmm, i see, so maybe the issue here is that something takes it from BUILD/SCHEDULING => BUILD/None and it just stalls out | |
| 19:39:56 | mriedem | which i think is build_and_run_instance in the compute | |
| 19:40:09 | mnaser | but i dont see a compute host listed when i see the instance in that state | |
| 19:40:21 | mriedem | is the instance in cell0? | |
| 19:40:37 | jaypipes | mriedem: typically a VF will be represented as an amount of SRIOV_NET_VF resource class inventory on a resource provider representing the PF. However, it is possible that a resource provider representing the individual VF could be created by the virt driver or neutron agent IFF differences in capabilities/traits differentiate individual VFs from each other. | |
| 19:40:51 | mnaser | mriedem: i can only assume eventually it ends up there once it stalls on BUILD/None for a while, ill investigate a bit more | |
| 19:41:28 | mriedem | should be able to tell by just looking for the instance in the cell0 db | |
| 19:41:46 | mnaser | grepping scheduler logs while this happens too | |
| 19:42:23 | mnaser | it doesn't end up in cell0 | |
| 19:42:56 | mriedem | so there must be a build_request for it in the nova_api db | |
| 19:43:05 | mriedem | query the build_requests table for that instance uuid? | |
| 19:44:06 | mnaser | no build request in nova_api, but the instance is in the cell db (not cell0) with state as 'building' | |
| 19:44:11 | mnaser | until i assume the instance was just deleted | |
| 19:44:23 | mriedem | but it doesn't have a host set on the instance right? | |
| 19:44:36 | mnaser | nope | |
| 19:44:38 | mriedem | so the instance is in cell1, but host is not set, and vm_state is BUILDING | |
| 19:44:41 | mriedem | and task_state is None | |
| 19:44:55 | mnaser | mriedem: http://paste.openstack.org/show/693715/ exactly | |
| 19:45:10 | mnaser | thats "select uuid, vm_state, task_state, host from instances where uuid='58dec7d0-00f3-448b-a7ce-90e4c2b7d4de';" in the nova db | |
| 19:45:17 | mriedem | is there a fault recorded on the instance? | |
| 19:45:58 | mnaser | uh are faults stored in the same table | |
| 19:46:08 | mnaser | nope instance_faults | |
| 19:46:23 | mnaser | nope | |
| 19:46:27 | mnaser | no faults | |
| 19:46:52 | mnaser | the instance stays there forever, im pretty sure its just deleted by the api user and they're reattempting to launch the instance, so the behaviour is non deterministic | |
| 19:47:04 | mriedem | yeah so my guess is _do_build_and_run_instance set task_state=None and then something hung or failed | |
| 19:47:24 | mriedem | the instance.host gets set in the resource tracker after it's spawned | |
| 19:48:30 | mriedem | so networking or block device setup maybe failed? or driver.spawn failed | |
| 19:50:13 | mriedem | however, those should either result in a reschedule or the instance being put into ERROR state | |
| 19:50:34 | mriedem | mnaser: probably have to trace the server create request id through the compute logs to find out what happened to i | |
| 19:50:35 | mriedem | *it | |
| 19:50:54 | mnaser | mriedem: actually, i'm looking at recent instance_faults and im seeing some rpc timeouts | |
| 19:50:58 | mriedem | jaypipes: ok (on the VF/PF thing) | |
| 19:51:13 | mnaser | which could very well explain things.. ill dig more, because there's plenty of conductors serving here | |
| 19:51:17 | mriedem | mnaser: so it's possible the state changes to the instance never made it through conductor to the db | |
| 19:51:19 | mnaser | thanks for the small talk through, that gives a hint | |
| 19:51:31 | mnaser | instance.save(expected_task_state=task_states.SPAWNING) | |
| 19:51:31 | mnaser | File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2110, in _build_and_run_instance | |
| 19:51:41 | mnaser | it is *very* possible considering that's the stacktrace that timed out... | |
| 19:52:55 | mriedem | yup, and that's the time that the instance.host would be set and saved off from the RT.instance_claim() | |
| 19:54:10 | mnaser | well at least we know things are breaking the way they're supposed to be.. | |
| 19:55:48 | mnaser | dansmith: i'm not sure, i'm worried about finding that part out :p | |
| 19:55:59 | dansmith | should be any day now.. :) | |
| 19:56:08 | mnaser | our queens upgrades are happening soon | |
| 19:56:31 | mnaser | i'm sure that'll push things | |
| 19:56:32 | mnaser | :P | |
| 19:56:43 | mriedem | mnaser: i'd wait for 17.0.1 | |
| 19:56:44 | dansmith | heh | |
| 19:56:49 | mriedem | which should happen this week or early next | |
| 19:57:01 | mnaser | mriedem: taking your word for it, usually nova is one of the later targets for upgrades | |
| 19:57:10 | mriedem | mnaser: are you going to leave glance behind for v1 support? | |
| 19:57:15 | mnaser | we have a whole ceph cluster to upgrade just before that | |
| 19:57:45 | mnaser | mriedem: we have 1 client that heavily uses v1, and they use the openstack cloud jenkins plugin so i'm probably just going to push up a patch to use the v2 apis | |
| 19:58:00 | mnaser | and once i get them to use it, then we can move on | |
| 19:58:28 | mriedem | i was just thinking about this at the ptg, | |
| 19:58:43 | mriedem | but the direct rbd snapshot stuff for ephemeral rbd disks in the libvirt driver, | |
| 19:58:44 | mnaser | i think v1 still exists in queens though | |
| 19:58:57 | mriedem | that relies on being able to get the location from the image, which i thought was a v1 only thing in glance | |
| 19:59:04 | mriedem | nova dropped support for glance v1 | |
| 19:59:21 | mnaser | direct image location or whatever that is | |
| 19:59:26 | mnaser | it's a thing in glance v2 i'm pretty sure | |
| 20:00:21 | mriedem | whatever this is https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L683 | |
| 20:00:34 | mnaser | https://developer.openstack.org/api-ref/image/v2/index.html#show-image-details | |
| 20:00:42 | mnaser | yeah that is the direct_url and locations[] in the api there | |
| 20:01:17 | mriedem | ok i thought they weren't going to carry that forward to v2 | |
| 20:01:36 | mriedem | but if they did, then i don't have to worry about us breaking the direct snapshot rbd stuff that a lot of people wanted and are using | |
| 20:03:02 | mnaser | mriedem: afaik we're using v2 with nova and direct snapshot is working here .. under pike at least | |
| 20:03:19 | mriedem | ok i'll not worry about this then | |
| 20:03:41 | cfriesen | mnaser: mriedem: if you enable "show_multiple_locations" then the direct snapshot stuff will work. it's just that glance tells you that turning that option on is a security risk | |
| 20:04:20 | mriedem | cfriesen: i know, i just thought it was a v1 only thing | |
| 20:04:24 | hrw | stephenfin: sorry but r14 got wrong in one place during merge and I did not noticed | |
| 20:04:32 | mriedem | nova dropped support for glance v1 in ocata, so it's further back than i thought | |
| 20:04:58 | mnaser | oh yeah, nope, its fine, glance says its a security risk but i'd argue with that in ceph's case .. but i don't have the time to discuss that :D | |
| 20:08:38 | openstackgerrit | Marcin Juszkiewicz proposed openstack/nova master: Allow to configure amount of PCIe ports https://review.openstack.org/545034 | |
| 20:08:50 | hrw | stephenfin, mriedem: fixed rev | |
| 20:11:19 | bauzas | mriedem: dansmith: I'll be on PTO tomorrow afternoon UTC | |
| 20:11:23 | mriedem | artom: so -1 on https://review.openstack.org/#/c/485522/ since the spec doesn't say anything | |
| 20:11:35 | bauzas | in case you need me | |
| 20:11:45 | mriedem | bauzas: i need you right now | |
| 20:11:47 | mriedem | to remove your -W on https://review.openstack.org/#/c/550259/ | |
| 20:12:03 | artom | And to stop forest fires? | |