| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-08 | |||
| 20:51:37 | melwitt | jroll: what's your service version for the nova-osapi_compute service? | |
| 20:52:09 | melwitt | probably in nova_cell0.services | |
| 20:52:49 | jroll | right, not deleted | |
| 20:54:10 | jroll | debug logs gave me jack | |
| 20:54:17 | jroll | no records in nova_cell0.services O_o | |
| 20:54:34 | melwitt | must be in a different db then, I don't know where to expect it | |
| 20:55:13 | melwitt | oh, maybe just nova? | |
| 20:55:20 | jroll | 16 in nova yeah | |
| 20:55:25 | melwitt | the non-compute service records are all together. okay | |
| 20:55:56 | jroll | bah, dogs need to go out, bbiab | |
| 20:56:02 | melwitt | k | |
| 20:56:39 | melwitt | so you should be in here https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L2322-L2347 | |
| 21:00:25 | melwitt | oh, wait, it never even gets there. uuugggh | |
| 21:00:56 | melwitt | this is trying to look up the instance long before instance mappings are even begun to be examined | |
| 21:03:55 | jroll | the "bad" one does get that far | |
| 21:04:31 | jroll | but can't tell on the "good" one since there's no traceback or anything on the 404 | |
| 21:05:28 | jroll | I need to take off shortly, can write up something in launchpad for now? | |
| 21:05:34 | jroll | almost seems like two separate bugs | |
| 21:06:02 | melwitt | yes, that would be helpful. thanks. I'll comment there on the bug | |
| 21:06:39 | jroll | ok, I'll make two reports | |
| 21:09:27 | jroll | oh I'm an idiot, that "good" one is deleted, I'm sorry | |
| 21:09:39 | melwitt | oh, heh. that's okay | |
| 21:10:56 | melwitt | the one I'm looking at is the one in your original paste. tracing through to get straight what is happening on that one | |
| 21:11:16 | jroll | ok :) | |
| 21:11:27 | jroll | I'm just going to reopen https://bugs.launchpad.net/nova/+bug/1749167, looks to be the same | |
| 21:11:28 | openstack | Launchpad bug 1749167 in OpenStack Compute (nova) "nova show can not get an instance information, and this instance can be queried from nova list." [Undecided,Expired] | |
| 21:12:51 | melwitt | is it? this bug says they can see the instance with 'nova list' but can't delete it | |
| 21:13:05 | melwitt | your paste shows a failure to 'nova list' at all | |
| 21:16:18 | jroll | oh, good point | |
| 21:16:24 | jroll | ok, I'll open anew | |
| 21:19:56 | melwitt | if the 'nova list' involves deleted instances too, then the bug you linked earlier about lazy-load read_deleted="yes" may well fix your problem | |
| 21:20:39 | melwitt | I guess that would only make sense if you did 'nova list --deleted' | |
| 21:21:01 | jroll | https://bugs.launchpad.net/nova/+bug/1775934 | |
| 21:21:02 | openstack | Launchpad bug 1775934 in OpenStack Compute (nova) "Cannot run "openstack server list" with instance stuck in scheduling state" [Undecided,New] | |
| 21:21:05 | jroll | yeah, this isn't with deleted | |
| 21:21:31 | jroll | writing that I now understand why bauzus was trying to figure out how it got in that state | |
| 21:21:31 | melwitt | k, thanks | |
| 21:21:42 | jroll | I guess we probably don't expect it to work with an invalid cell mapping | |
| 21:22:07 | melwitt | well, if there's an invalid cell_mapping, it's supposed to use the data from nova_api.build_requests | |
| 21:23:04 | melwitt | rather, if InstanceMapping.cell_id=NULL it should be using nova_api.build_requests and using that to list and not failing | |
| 21:23:17 | jroll | well, that's an empty table :/ | |
| 21:23:39 | melwitt | build_requests is empty? | |
| 21:23:57 | jroll | yep | |
| 21:24:17 | melwitt | huh. that is weird | |
| 21:24:23 | jroll | indeed | |
| 21:24:38 | mriedem | pretty sure it doesn't work that way | |
| 21:24:46 | mriedem | this is the same thing surya was talking about on wednesday | |
| 21:24:56 | mriedem | we look for a build request and if not there, we look for an instance mapping with a cell | |
| 21:25:17 | mriedem | if the build request is gone, the instance mapping should have a cell mapping because we delete the build request after updating the instance mapping | |
| 21:25:36 | melwitt | this shows if mapped, we look in the cell db, else we look at build requests https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L2322-L2347 | |
| 21:25:40 | mriedem | https://bugs.launchpad.net/nova/+bug/1773945 | |
| 21:25:41 | openstack | Launchpad bug 1773945 in OpenStack Compute (nova) "nova client servers.list crashes with bad marker" [Medium,Triaged] - Assigned to Surya Seetharaman (tssurya) | |
| 21:26:12 | mriedem | ok, but could be hitting ^ | |
| 21:26:20 | mriedem | where the build request is gone but the instance mapping doesn't have a cell mapping in it | |
| 21:26:28 | melwitt | ah, okay | |
| 21:27:14 | melwitt | jroll: build requests don't get soft-deleted so could be empty if everything got cleaned up properly | |
| 21:27:32 | jroll | oh nice, thanks mriedem | |
| 21:28:01 | melwitt | your traceback is different than that one, but could it be the same root cause? | |
| 21:28:24 | jroll | probably | |
| 21:29:02 | jroll | I have to bounce but pointed internal folks at that bug and the one I filed | |
| 21:29:14 | jroll | I can update monday if they found anything new | |
| 21:29:58 | jroll | thanks for all the help melwitt, mriedem | |
| 21:30:23 | melwitt | based on your traceback, it looks like the instance get_all from compute/api succeeded but then you failed when the REST API code tried to access instance.get_flavor(), it wanted to lazy-load it. but that attribute should have been pre-loaded. so I don't get that yet | |
| 21:31:36 | melwitt | jroll: thanks for letting us know about the bug | |
| 21:32:08 | mriedem | that lazy load failure sounds familiar | |
| 21:34:03 | mriedem | https://github.com/openstack/nova/commit/5a363a0d72e7dd8d79d7e950effc1d8a5fdc801b#diff-1b01e8e37be9d889295e5333b7857139 | |
| 21:34:30 | mriedem | that's likely not it | |
| 21:34:34 | mriedem | probably something else in the api code | |
| 21:38:35 | melwitt | so they get here and fail during flavor lazy-load https://github.com/openstack/nova/blob/stable/ocata/nova/api/openstack/compute/servers.py#L369 but it seems "impossible" that they could have gotten there with cell_id=NULL + build_requests table empty | |
| 21:39:46 | melwitt | oh actually that's get_all so different than what I was looking at get earlier | |
| 21:43:37 | melwitt | okay so that does make sense, the get_all in compute/api would find whatever is in the cell dbs regardless of cell_mapping=None. so that would succeed. but should also have flavor preloaded on the instance objects returned, so why is a lazy-load happening | |
| 21:44:18 | melwitt | nothing should have been created from build requests if the build_requests table is empty... | |
| 21:47:09 | melwitt | maybe he tried that test earlier when there were build requests still. that would make sense, if some items in the instance list were "shell" instances created from build_requests, then instance.get_flavor() would trigger a lazy-load | |
| 21:56:09 | melwitt | jroll: if you could show nova_cell0.instance_extra for that instance it might help too | |
| 21:58:27 | melwitt | maybe not actually. I was trying to think whether the embedded flavor was even there but it might not be relevant | |
| 23:30:43 | openstackgerrit | Chris Dent proposed openstack/nova master: Optional separate database for placement API https://review.openstack.org/362766 | |
| 23:30:44 | openstackgerrit | Chris Dent proposed openstack/nova master: Isolate placement database config https://review.openstack.org/541435 | |
| 23:30:45 | openstackgerrit | Chris Dent proposed openstack/nova master: Ensure that os-traits sync is attempted only at start of process https://review.openstack.org/553857 | |
| 23:31:03 | openstackgerrit | Chris Dent proposed openstack/nova master: Add PLACEMENT_DB_ENABLED=True to the nova-next job https://review.openstack.org/564067 | |
| 23:56:48 | openstackgerrit | Chris Dent proposed openstack/nova master: Extract part of PlacementFixture to placement https://review.openstack.org/568359 | |
| #openstack-nova - 2018-06-09 | |||
| 00:48:45 | vpc | hi | |
| 00:48:49 | vpc | anyone online? | |
| 07:40:14 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480 | |
| 07:40:15 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add microversion for nested allocation candidate https://review.openstack.org/565487 | |
| 07:40:16 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix nits in placement-return-all-resources series https://review.openstack.org/573604 | |
| #openstack-nova - 2018-06-10 | |||
| 01:42:40 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480 | |
| 01:42:41 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add microversion for nested allocation candidate https://review.openstack.org/565487 | |
| 01:42:42 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix nits in placement-return-all-resources series https://review.openstack.org/573604 | |
| 03:46:24 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Return all nested providers in tree https://review.openstack.org/559480 | |
| 03:46:25 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Add microversion for nested allocation candidate https://review.openstack.org/565487 | |
| 03:46:26 | openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix nits in placement-return-all-resources series https://review.openstack.org/573604 | |
| 11:08:09 | openstackgerrit | Merged openstack/nova master: Return all resources in provider_summaries https://review.openstack.org/558045 | |
| 11:22:10 | openstackgerrit | Alvaro Lopez Garcia proposed openstack/nova master: Ensure that periodic reclaim cleans DB deleted instances https://review.openstack.org/323250 | |
| 14:16:15 | mnaser | is it safe to remove the cell_v2 map_instances from openstack-ansible deployment process in queens/rocky assuming no user skips releases when doing upgrades? | |
| 15:48:12 | openstackgerrit | Merged openstack/nova master: Remove mox in test_compute_api.py (3) https://review.openstack.org/568184 | |
| 16:23:25 | openstackgerrit | Maciej Kucia proposed openstack/nova-specs master: Fix Non-unique network names in Servers IPs API response https://review.openstack.org/558125 | |
| 18:09:28 | openstackgerrit | Merged openstack/nova master: Remove mox in libvirt/test_driver.py (4) https://review.openstack.org/570750 | |
| 18:54:29 | openstackgerrit | Chris Dent proposed openstack/nova master: Provide a direct interface to placement https://review.openstack.org/572576 | |
| 22:18:54 | openstackgerrit | Chris Dent proposed openstack/nova master: Provide a direct interface to placement https://review.openstack.org/572576 | |
| 23:56:56 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (1) https://review.openstack.org/572018 | |