| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 18:27:18 | smcginnis | FWIW, until the gcc issue gets clarified I think we should block it in requirements too. | |
| 18:27:38 | openstackgerrit | Merged openstack/nova master: Rename auth_uri to www_authenticate_uri https://review.openstack.org/576820 | |
| 18:43:52 | openstackgerrit | sean mooney proposed openstack/nova master: fix disk_bus handeling https://review.openstack.org/584999 | |
| 18:55:04 | openstackgerrit | Merged openstack/nova master: Func test for improper cn local DISK_GB reporting https://review.openstack.org/583646 | |
| 18:56:27 | openstackgerrit | Merged openstack/nova master: [placement] disallow additional fields in allocations https://review.openstack.org/583907 | |
| 19:15:28 | mriedem | efried: if you agree with my comments on https://review.openstack.org/#/c/560317/ i'll update those quick | |
| 19:16:08 | efried | looking | |
| 19:17:52 | mriedem | and i'll probably blacklist greenlet 0.4.14 in a change below this so we can get the powerkvm ci passing again | |
| 19:18:07 | efried | mriedem: I assume https://review.openstack.org/#/c/560317/30/nova/virt/libvirt/driver.py@8940 you meant, "I'd make this an info" | |
| 19:18:33 | efried | mriedem: I'm fine if you make those changes, sure. | |
| 19:18:39 | mriedem | yeah heh | |
| 19:18:43 | mriedem | ok will do | |
| 19:18:58 | efried | kashyap: You good with https://review.openstack.org/#/c/560317/ at this point? | |
| 19:19:12 | mriedem | kashyap is likely done for the day | |
| 19:53:06 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add method to get cpu traits https://review.openstack.org/560317 | |
| 19:53:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: FakeLibvirtFixture: mock get_fs_info https://review.openstack.org/579201 | |
| 19:53:08 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Blacklist greenlet 0.4.14 https://review.openstack.org/585016 | |
| 19:53:56 | mriedem | edmondsw: ^ | |
| 19:55:15 | edmondsw | mriedem +1 | |
| 19:55:35 | edmondsw | mmedvede ^ | |
| 20:06:47 | mnaser | lols here's interesting behaviour | |
| 20:07:05 | mnaser | list doesn't hit cells so vms that were stuck in 'scheduling' are shown when you list the vms in horizon | |
| 20:07:18 | mnaser | but then as horizon tries to 'refresh' them by doing a GET, nova returns a 404 | |
| 20:07:25 | mnaser | so they disappear afterwards | |
| 20:07:46 | dansmith | mnaser: what do you mean by "list doesn't hit cells" ? | |
| 20:08:14 | dansmith | also, GET on a scheduling vm shouldn't return a 404 | |
| 20:08:15 | mnaser | dansmith: well those are instances which seem to have gotten somehow stuck in 'scheduling' status | |
| 20:08:28 | mnaser | ok, let me verify this | |
| 20:08:56 | dansmith | oh, if they're scheduling they're not created in any cell, maybe that's what you mean, but list should still show them from their buildrequest of course | |
| 20:09:04 | dansmith | and GET during/after should all work | |
| 20:09:12 | mnaser | well list seems to show them, but GET gives a 404 afaik | |
| 20:09:36 | mnaser | im double checking the behaviour | |
| 20:09:47 | mnaser | yeah, they show up in list, but GET gives 404 | |
| 20:10:24 | dansmith | something's broken then | |
| 20:10:28 | dansmith | that shouldn't happen | |
| 20:10:43 | mnaser | well i guess the db got in some weird state | |
| 20:10:56 | mnaser | or even if the db is mucked it shouldn't be like this? | |
| 20:11:18 | dansmith | well, depends on the level of mucking I guess, but.. it should never be like this | |
| 20:11:37 | mnaser | so as i understand it, list just grabs stuff from the 'api' db, but the 'get' actually hits the cell | |
| 20:11:43 | dansmith | no | |
| 20:11:58 | dansmith | list grabs stuff from the api and cell databases together | |
| 20:12:11 | mnaser | so it must know the mapping too i guess | |
| 20:12:15 | dansmith | get will pull the mapping from the api db, and if it looks like it is scheduled, will talk to the cell, and if not will return the build request data | |
| 20:12:21 | mnaser | ah okay | |
| 20:12:38 | mnaser | so it's in a state where it's doing neither talk to cell or return build request data | |
| 20:12:50 | dansmith | if it' | |
| 20:12:57 | dansmith | if it is 404'ing a scheduling instance, | |
| 20:13:11 | dansmith | then it's getting the mapping and then not returning the buildrequest or something | |
| 20:14:01 | dansmith | mnaser: all this logic is fairly straightforward here: https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2298-L2357 | |
| 20:14:09 | dansmith | start with L2328 | |
| 20:14:26 | dansmith | we get the mapping (which should always exist once we've returned a uuid to the user) | |
| 20:14:36 | dansmith | maybe check to see if that failed to get written? | |
| 20:14:41 | mnaser | let me check the db mapping | |
| 20:14:56 | dansmith | if there's no mapping but it did get created in a cell, that would cause it to show up in list and not get | |
| 20:15:04 | dansmith | because of L2355 there | |
| 20:15:37 | mnaser | cell_id=NULL in the db | |
| 20:15:51 | dansmith | that's what it should be for scheduling | |
| 20:15:54 | mnaser | so it should def go into the elif | |
| 20:16:08 | dansmith | that triggers L2333 to get it from the api db's buildrequest | |
| 20:16:14 | mnaser | let me check the build request | |
| 20:16:35 | mnaser | no build_request | |
| 20:16:42 | mriedem | i have a patch for this | |
| 20:16:52 | mriedem | but that's listing, not showing | |
| 20:16:52 | mriedem | https://review.openstack.org/#/c/575556/ ? | |
| 20:16:55 | mriedem | sounds similar though | |
| 20:17:03 | mriedem | build request is gone, instance mapping isn't pointing at a cell | |
| 20:17:05 | dansmith | and it's marker | |
| 20:17:16 | dansmith | how does that situation happen? | |
| 20:17:24 | mriedem | do not know | |
| 20:17:27 | mnaser | i mean | |
| 20:17:30 | dansmith | this patch doesn't address his problem, | |
| 20:17:32 | mnaser | rabbitmq had a field day a few days ago for us | |
| 20:17:38 | dansmith | but papers over it for list I guess | |
| 20:17:41 | mnaser | so i would totally attribute it to anything related to that | |
| 20:17:54 | mriedem | i believe jroll was seeing similar | |
| 20:18:00 | dansmith | mnaser: mm, shouldn't really as we don't use rabbit to create the instance in the cell db | |
| 20:18:03 | dansmith | well, | |
| 20:18:05 | mnaser | oh true | |
| 20:18:19 | dansmith | unless we dropped something between superconductors or something | |
| 20:18:25 | mnaser | no superconductor here | |
| 20:18:29 | mnaser | 1 cell, 1 conductor | |
| 20:18:37 | dansmith | s/super// | |
| 20:18:42 | mriedem | last i looked at this, we do have a window between the time we delete the build request and set the instance mapping cell | |
| 20:19:05 | dansmith | mriedem: we should be setting the mapping and then deleting the BR | |
| 20:19:07 | mnaser | wouldn't it be safer to set the instance mapping first and then delete the br? | |
| 20:19:09 | mnaser | yeah ^ | |
| 20:19:12 | dansmith | because that's how the get logic expects it to work | |
| 20:19:19 | dansmith | if the mapping is set, then BR is ignored | |
| 20:19:27 | dansmith | so we can delete it late | |
| 20:19:50 | dansmith | and that's what we're doing | |
| 20:19:57 | mriedem | schedule_and_build_instances in the conductor does set the cell in the instance mapping and then attempt to delete the build request, | |
| 20:20:04 | mriedem | but the build request can be deleted by the api first | |
| 20:20:05 | dansmith | https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155-L1182 | |
| 20:20:09 | dansmith | that's for cell0 | |
| 20:20:20 | dansmith | which presumably is what his situation is if it never scheduled | |
| 20:20:53 | mriedem | not sure why we'd pass here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1178 | |
| 20:21:13 | dansmith | if it got deleted | |
| 20:21:24 | mriedem | the instance mapping? | |
| 20:21:28 | dansmith | on the non-cell0 case we also target the mapping before deleting the BR | |
| 20:21:29 | mriedem | why would that get deleted during scheduling | |