| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-07 | |||
| 22:13:53 | melwitt | that sounds right though | |
| 22:14:08 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234 | |
| 22:18:53 | mriedem | hmm, ok, i'm trying to figure out http://logs.openstack.org/76/523676/11/check/legacy-tempest-dsvm-cells/1d155c6/logs/screen-n-cpu.txt.gz#_Dec_07_09_02_09_579991 | |
| 22:19:01 | mriedem | the patch creates an instance action in the api | |
| 22:19:13 | mriedem | then when the compute is trying to look it up, it can't find it and blows up | |
| 22:19:26 | mriedem | so i'm guessing, | |
| 22:19:42 | mriedem | the top level api database has the action record, but not the cell | |
| 22:19:45 | mriedem | cell db i mean | |
| 22:20:12 | mriedem | but then i don't know how we have action records working for anything in cellsv1 | |
| 22:20:18 | melwitt | yeah, so you'll want to check if the instance.save() at the top does a sync down to the cell. it probably does, but it can race | |
| 22:20:27 | melwitt | like compute might try to read before the sync makes it to the cell | |
| 22:20:38 | mriedem | this isn't instance.save() | |
| 22:20:46 | melwitt | okay, then it won't sync it | |
| 22:20:54 | mriedem | objects.InstanceAction.action_start | |
| 22:21:28 | melwitt | hm, so yeah does compute rely on reading instance actions in any other case? | |
| 22:21:48 | melwitt | because if so, like you said those shouldn't work either | |
| 22:21:51 | mriedem | oh i think i see something | |
| 22:22:11 | mriedem | so stop_instance is in this list https://github.com/openstack/nova/blob/master/nova/compute/cells_api.py#L51 | |
| 22:22:15 | mriedem | and so is backup_instance | |
| 22:22:35 | mriedem | https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1714 | |
| 22:22:38 | mriedem | stop_instance has that ^ | |
| 22:22:41 | mriedem | backup_instance doesn't | |
| 22:23:10 | melwitt | and you're seeing a fail with backup? | |
| 22:24:45 | mriedem | yeah https://review.openstack.org/#/c/523676/11 | |
| 22:24:46 | melwitt | so returning the response from the instance action vs not | |
| 22:25:04 | melwitt | I didn't realize instance action had a rpc response in any case | |
| 22:25:22 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Deduplicate functional test code https://review.openstack.org/526227 | |
| 22:27:53 | melwitt | yeah it looks like for some reason stop_instance can return something but backup_instance can't in nova/compute/rpcapi.py | |
| 22:28:07 | melwitt | I don't see stop_instance ever returning anything in compute/manager though | |
| 22:29:25 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete https://review.openstack.org/525231 | |
| 22:29:34 | mriedem | melwitt: i think the do_cast thing is something else | |
| 22:29:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (2) https://review.openstack.org/526263 | |
| 22:29:40 | mriedem | red herring | |
| 22:29:43 | melwitt | yeah, it's just saying wait for response vs not | |
| 22:29:53 | mriedem | so i'm not sure what ever creates instance action records in the top level vs cell db | |
| 22:30:05 | melwitt | and looks correct based on backup vs stop | |
| 22:30:28 | melwitt | well, it would be whichever service does the write. if n-api writes it, it will be in the top level db, if n-cpu writes it, it will be in the cell db | |
| 22:30:40 | melwitt | I think only instance records are synced back and forth in cells v1 | |
| 22:31:08 | melwitt | I'm checking to see if anything in backup_instance in compute does a write of instance action records | |
| 22:31:14 | mriedem | but if that's the case, then how does our @wrap_instance_event in the compute not blow up on everything? | |
| 22:31:20 | melwitt | I guess it can't be, else they'd be found | |
| 22:31:32 | mriedem | the action record stuff for backup_instance is new to this patch | |
| 22:31:40 | mriedem | following the same pattern as everywhere else | |
| 22:31:48 | melwitt | okay, I'll look at that then | |
| 22:32:44 | melwitt | yeah, that's just writing it in n-api so it'll go in the top level db only | |
| 22:32:52 | melwitt | I'm not sure how the other actions are currently working | |
| 22:44:44 | mriedem | alaski: you remember anything special about how the hell instance action records get synced between the api cell and the child cells in v1? | |
| 22:47:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add instance action record for backup instances https://review.openstack.org/523676 | |
| 22:53:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [WIP] POC to use neutron port_list when filtering instance by ip https://review.openstack.org/525505 | |
| 23:05:23 | tasker | I'm playing around with the nova-metadata service and every PUT or POST request I send to "openstack/latest/password" is seen by the service as a GET. is this a security feature of the metadata service or is there some configuration issue that I haven't uncovered yet? | |
| 23:06:34 | tasker | I tracked it through the code from api/metadata/password.py:handle_password up through MetadataRequestHandler and it's a GET the whole time. | |
| 23:06:51 | tasker | is something translating it to GET? | |
| 23:08:04 | mriedem | tempest does a POST successfully from the guest to change the password | |
| 23:08:08 | mriedem | i remember because of this bug https://bugs.launchpad.net/nova/+bug/1696848 | |
| 23:08:10 | openstack | Launchpad bug 1696848 in OpenStack Compute (nova) pike "POST /openstack/2013-10-17/password returns 500 if instance is not found" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 23:08:46 | mriedem | http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_server_basic_ops.py#n90 | |
| 23:09:02 | mriedem | tasker: are you passing a body? | |
| 23:09:24 | tasker | yeah. though now that I think about it, it's probably malformed | |
| 23:10:32 | tasker | curl -D - -X POST -d 'cloudnull' http://169.254.169.254/openstack/latest/password | |
| 23:12:21 | tasker | before we go much farther .. this Mitaka at best. I haven't been able to find anything that says it won't work there though. | |
| 23:12:39 | mriedem | metadata api doesn't really ever change | |
| 23:12:43 | mriedem | mikal might have some ideas | |
| 23:12:54 | mriedem | he did give the talk about it in sydney | |
| 23:13:27 | tasker | I can't seem to find anything else on the internet about it not working. as if it not working wasn't even a thing. | |
| 23:13:42 | tasker | I'll keep hacking and tracking seeing if I can figure out what's going on. | |
| 23:13:52 | mriedem | are you using service_metadata_proxy in nova.conf? | |
| 23:14:13 | tasker | yes. set to True | |
| 23:17:01 | tasker | or | |
| 23:17:02 | tasker | oh | |
| 23:17:10 | tasker | and .. that explains a lot. | |
| 23:17:39 | mriedem | not set? | |
| 23:17:47 | tasker | it's set to True. | |
| 23:18:09 | tasker | and I don't use the `neutron_agents` nor its metadata service | |
| 23:18:18 | mriedem | ah | |
| 23:19:04 | tasker | thanks, Matt. that shed a lot of light on what's going on. | |
| 23:19:07 | tasker | back to work! | |
| 23:19:09 | mriedem | o/ | |
| 23:19:12 | tasker | to me. not you. | |
| 23:19:28 | tasker | you do whatever it was you were doing. .) | |
| 23:19:37 | mriedem | working | |
| 23:26:13 | openstackgerrit | Eric Fried proposed openstack/nova master: Traits ops on ProviderTree https://review.openstack.org/521605 | |
| 23:26:14 | openstackgerrit | Eric Fried proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685 | |
| 23:26:14 | openstackgerrit | Eric Fried proposed openstack/nova master: Aggregate ops on ProviderTree https://review.openstack.org/526216 | |
| 23:26:15 | openstackgerrit | Eric Fried proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756 | |
| 23:26:15 | openstackgerrit | Eric Fried proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686 | |
| 23:26:28 | efried | I broke the patchbot. | |
| 23:26:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541 | |
| 23:26:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540 | |
| 23:26:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539 | |
| 23:26:31 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | |
| 23:26:31 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187 | |
| 23:26:31 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098 | |
| 23:26:42 | efried | ah, there it goes. | |
| 23:27:43 | efried | I got `related changes` to scroll :) | |
| 23:30:36 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Get original image_id from volume for volume-backed instance rebuild https://review.openstack.org/526544 | |
| 23:30:36 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Only query BDMs once in API during rebuild https://review.openstack.org/526543 | |
| 23:30:36 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Add regression test for rebuilding a volume-backed server https://review.openstack.org/526542 | |
| 23:31:10 | mriedem | tonyb: here we go https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/newton | |
| 23:31:15 | mriedem | i think that's going to be the end for newton | |
| 23:31:16 | mriedem | for real this time | |