| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-17 | |||
| 21:27:53 | efried | okay. Cause I think the major hole with what we're doing now is that we're calling it from within report client, often in ways that may cause us to be ignoring an exception on step 3 of a 10-step process and then proceeding with the rest. | |
| 21:28:23 | dansmith | right | |
| 21:28:24 | dansmith | which is why we shouldn't have it underneath report client | |
| 21:28:28 | efried | ++ | |
| 21:28:42 | efried | unless it's doing a reconnect-and-retry thing. | |
| 21:28:47 | dansmith | sure | |
| 21:28:56 | cfriesen | mriedem: I'm confused why it's not failing...it was trivially easy to reproduce in devstack by adding a single row to the services table. | |
| 21:29:09 | efried | In which case it should probably decorate the get/put/post/delete primitives and nothing else. | |
| 21:33:36 | lyarwood | Does anyone have a simple way to enable debug logging within functional tests? self.flags doesn't appear to work. | |
| 21:34:53 | mriedem | OS_DEBUG=True | |
| 21:35:14 | mriedem | https://github.com/openstack/nova/blob/ad249d8994067b1c255823cb7c1ff4c32593e5ef/tox.ini#L24 | |
| 21:35:36 | lyarwood | thanks! :) | |
| 21:43:39 | cfriesen | mriedem: when I run your testcase locally it's failing in _migrate_server() | |
| 21:44:57 | cfriesen | http://paste.openstack.org/show/719417/ | |
| 21:45:16 | mriedem | cfriesen: in pike? | |
| 21:45:53 | mriedem | because pike doesn't have 204 in this list https://github.com/openstack/nova/blob/87ea686f9f2cc706205d188922bb14272625e7be/nova/tests/functional/api/client.py#L223 | |
| 21:46:04 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Make safe_connect retry or raise https://review.openstack.org/562060 | |
| 21:46:09 | efried | dansmith, mriedem: ^ | |
| 21:46:48 | efried | dansmith, mriedem: There would be a separate, unrelated thingy for the n-api side of the world. | |
| 21:46:55 | cfriesen | mriedem: yes, in pike | |
| 21:48:13 | cfriesen | retrying with that | |
| 21:49:59 | cfriesen | is there a way to stick a pdb breakpoint in there right before the last line? | |
| 21:52:35 | efried | cfriesen: Are you asking how to pdb via stestr? | |
| 21:52:56 | efried | cause I have that answer | |
| 21:53:22 | cfriesen | efried: I want to pdb in a functional test | |
| 21:53:41 | efried | cfriesen: Same as in a unit test. | |
| 21:53:54 | efried | cfriesen: You have to run stestr directly, and with the -n option and the full name of the test case. | |
| 21:54:13 | efried | or you can use remote_pdb | |
| 21:54:44 | openstackgerrit | Jackie Truong proposed openstack/python-novaclient master: Microversion 2.62 - Add trusted_image_certificates https://review.openstack.org/500396 | |
| 21:55:54 | efried | stestr run -n nova.tests.functional.regressions.test_bug_1746509.InstanceListWithServicesTestCase.test_instance_list_deleted_service_with_no_uuid | |
| 21:55:54 | efried | source .tox/func/bin/activate | |
| 21:55:54 | efried | cfriesen: e.g. | |
| 21:56:11 | efried | s/func/functional/ I guess - whatever the venv name is for functional. | |
| 21:59:59 | cfriesen | efried: thanks, running ostestr directly worked | |
| 22:00:20 | efried | cfriesen: ostestr? Are you not in nova? But cool. | |
| 22:00:43 | cfriesen | efried: nova, but stable/pike. it's what tox was using | |
| 22:00:46 | efried | ahh | |
| 22:01:08 | cfriesen | mriedem: running db.service_get_all_by_binary() righte before the call to self.admin_api.get_servers() shows that both services have uuid entries | |
| 22:01:27 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix docs for confirmResize action https://review.openstack.org/562062 | |
| 22:01:59 | mriedem | cfriesen: yes, which they both will for the non-deleted services | |
| 22:02:13 | mriedem | because host1.start() creates a new service record with a uuid | |
| 22:02:25 | mriedem | simulating that you've upgraded to pike and started that compute host with pike code | |
| 22:02:38 | mriedem | after deleting the "newton" compute service / node which didn't have the uuid set | |
| 22:03:53 | cfriesen | mriedem: right, I need the mutated context to read the deleted one. | |
| 22:04:20 | openstackgerrit | Jackie Truong proposed openstack/python-novaclient master: Microversion 2.62 - Add trusted_image_certificates https://review.openstack.org/500396 | |
| 22:06:26 | mriedem | cfriesen: only other thing i could think of is like melwitt was saying yesterday and the backref joinedload on the services table is getting lucky and pulling the non-deleted service when listing instances | |
| 22:06:49 | mriedem | this https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L103 | |
| 22:07:03 | cfriesen | mriedem: the deleted service in the DB has a uuid | |
| 22:07:30 | melwitt | mriedem, cfriesen: I was wrong about that. the instance.services is all services whose host match the instance, and it will try to add a UUID to both of them, so should reliably fail I think | |
| 22:08:46 | mriedem | cfriesen: ok i don't know how that is per https://review.openstack.org/#/c/562041/2/nova/tests/functional/regressions/test_bug_1764556.py@149 | |
| 22:09:09 | mriedem | cfriesen: i guess there needs to be an assertion after the 2nd _migrate_server that the deleted service still doesn't have a uuid | |
| 22:10:48 | cfriesen | mriedem: if I set the uuid to None right before the final line then it fails | |
| 22:12:29 | melwitt | something in the migrate is loading instance.services and generating UUID I guess? | |
| 22:12:44 | cfriesen | seems like...I'm guessing wait_for_state_change() | |
| 22:15:39 | mriedem | idk, at what point is the test mocked out so hard that it masks whether or not we have a bug? | |
| 22:15:45 | mriedem | or that i trust the recreate of it? | |
| 22:17:30 | mriedem | here are some updates, but i've got to run - the 2nd migrate is indeed migrating the deleted service record | |
| 22:17:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Revert "Lazy-load instance attributes with read_deleted=yes" https://review.openstack.org/562051 | |
| 22:17:30 | openstack | bug 1764556 in OpenStack Compute (nova) ""nova list" fails with exception.ServiceNotFound if service is deleted and has no UUID" [Undecided,New] https://launchpad.net/bugs/1764556 | |
| 22:17:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add functional recreate test for bug 1764556 https://review.openstack.org/562041 | |
| 22:18:21 | melwitt | yeah, wouldn't want to mock it. I guess have to determine what exactly is the func test doing differently than the devstack repro and see if we make it the same as the actions needed with devstack | |
| 22:19:15 | cfriesen | melwitt: I wonder if we need to do the delete/migrate/recreate/migrate first and then do the upgrade to pike | |
| 22:19:27 | cfriesen | so that the migrate happens without uuids on the service | |
| 22:20:24 | SamYaple | looking for love in getting the +3 to get this backport merged. https://review.openstack.org/#/c/560088/ | |
| 22:21:07 | melwitt | looking for love in all the wrong places | |
| 22:21:27 | melwitt | dansmith might be able to help you with the backport review ^ | |
| 22:21:56 | SamYaple | story of my life melwitt | |
| 22:22:02 | melwitt | heh | |
| 22:22:05 | cfriesen | melwitt: guess not, my bug report specifically says they added the new compute was added after the upgrade | |
| 22:22:38 | melwitt | cfriesen: yeah, I don't yet understand how we're simulating an upgrade in the func test, that's probably what's messing it up | |
| 22:22:52 | melwitt | (I haven't looked in detail because I'm mired in other stuff right now) | |
| 22:23:01 | cfriesen | melwitt: we're simulating the upgrade by just deleting the uuid on the service | |
| 22:23:12 | melwitt | k, that makes sense | |
| 22:23:57 | cfriesen | melwitt: mriedem_away: I suspect I know what's going on....initially we didn't have the other two fixes from stable/pike so I suspect that the migration after the upgrade wouldn't have updated the UUID, thus leaving the DB in the wonky state. | |
| 22:24:25 | cfriesen | melwitt: mriedem_away: I bet that upgrading to stable/pike probably *would* update the service UUID on the migration. | |
| 22:25:09 | melwitt | cfriesen: but earlier didn't you recreate the bug in devstack with the two fixes already applied? | |
| 22:25:41 | openstackgerrit | Merged openstack/nova master: Remove redundant _do_check_can_live_migrate_destination https://review.openstack.org/561948 | |
| 22:26:18 | cfriesen | melwitt: yes, by manually editing the DB to make an entry without the UUID. and then trying to list servers. I didn't do the migration step. | |
| 22:26:30 | melwitt | ah | |
| 22:26:31 | cfriesen | melwitt: I suspect that the migration step would update the UUID. | |
| 22:26:39 | melwitt | I see | |
| 22:27:05 | cfriesen | if I'm right, then anyone upgrading to current stable/pike wouldn't see this | |
| 22:27:13 | melwitt | yeah | |
| 22:34:28 | dansmith | melwitt: I think SamYaple means "dansmith mergin' my stuff in a timely manner is the story of my life" | |
| 22:34:46 | melwitt | :) | |
| 22:37:29 | SamYaple | dansmith: <3 | |
| 22:40:54 | openstack | bug 1764883 in OpenStack Compute (nova) "Evacuation fails if the source host returns while the migration is still in progress" [Undecided,New] https://launchpad.net/bugs/1764883 | |
| 22:40:54 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP Add regression test for bug #1764883 https://review.openstack.org/562072 | |
| 22:41:17 | lyarwood | dansmith / melwitt ^ FYI, hopefully that's somewhere near what we need | |
| 22:41:30 | melwitt | ack | |
| 22:42:59 | melwitt | lyarwood: also, reminder this thursday is r-1 and we're gonna cut releases for stable branches | |
| 22:45:11 | lyarwood | melwitt: ah cool, thanks for the heads up | |
| 22:46:04 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Report the allocated size of preallocated file based disks https://review.openstack.org/562074 | |
| 22:51:43 | openstackgerrit | Merged openstack/nova master: Deduplicate notification samples Rocky - 4 https://review.openstack.org/560731 | |
| 22:51:51 | openstackgerrit | Merged openstack/nova master: Deduplicate notification samples Rocky - 5 https://review.openstack.org/560734 | |
| 22:52:13 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: libvirt: Report the allocated size of preallocated file based disks https://review.openstack.org/562077 | |
| 22:52:35 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: libvirt: Report the allocated size of preallocated file based disks https://review.openstack.org/562078 | |
| 22:55:46 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/ocata: libvirt: Block swap volume attempts with encrypted volumes prior to Queens https://review.openstack.org/561604 | |
| 23:55:39 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Microversion 2.62 - List/Show all server migration types https://review.openstack.org/430839 | |
| #openstack-nova - 2018-04-18 | |||
| 00:01:36 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/virt/xenapi/test_vmops.py https://review.openstack.org/561846 | |