| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-07 | |||
| 21:37:46 | mriedem | melwitt: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Kernel%20panic%20-%20not%20syncing%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 21:38:08 | mriedem | nothing super obvious there, not like a single node provider | |
| 21:38:16 | mriedem | but it's all master branch, so i wonder if we're using a new cirros image in queens | |
| 21:39:15 | melwitt | how do we check that? | |
| 21:39:55 | mriedem | it's in devstack | |
| 21:40:10 | mriedem | https://github.com/openstack-dev/devstack/blob/master/stackrc#L671 | |
| 21:40:25 | mriedem | https://github.com/openstack-dev/devstack/commit/9f2dcd333103553626db1924a019e151e3e7252e | |
| 21:40:28 | melwitt | cool thanks | |
| 21:40:29 | mriedem | that's not new so... | |
| 21:40:51 | mriedem | heh read the commit message there | |
| 21:40:54 | melwitt | just ran logstash for 30d and it looks like this is pretty recent stuff. nothing before 11/27 | |
| 21:41:12 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 21:41:14 | melwitt | haha, huh. | |
| 21:41:48 | mriedem | melwitt: logstash only goes back 10 days | |
| 21:42:01 | melwitt | oh :( | |
| 21:42:36 | melwitt | yeah I see what you mean, all providers hitting the kernel panic though pretty rarely | |
| 21:51:47 | mriedem | melwitt: i assume i'm pulling threads here https://review.openstack.org/#/c/524234/ | |
| 21:52:54 | melwitt | haha, that's okay | |
| 21:53:04 | mriedem | also, might be better to move that into a utility in QUOTAS | |
| 21:53:11 | mriedem | so the callers don't have to know about NoopQuotaDriver being a thing | |
| 21:53:15 | mriedem | QUOTAS.is_noop? | |
| 21:53:19 | melwitt | after fighting with the test coverage I wasn't thinking too clearly. that's my lame excuse | |
| 21:53:39 | mriedem | ok. i don't know what stephenfin's excuse was then. :) | |
| 21:54:11 | mriedem | thinking about bangers and mash during the football game at the pub | |
| 21:55:09 | melwitt | how is is_noop not knowing about the driver? | |
| 21:55:40 | melwitt | honestly the way it was before (the values coming from inside the driver) is the way the caller knows the least about it :P | |
| 21:55:45 | mriedem | i'm saying hide that check within QUOTAS | |
| 21:55:56 | mriedem | true | |
| 21:56:15 | mriedem | so, QUOTAS.get_reserved()? | |
| 21:56:16 | melwitt | so I'm like ¯\_(ツ)_/¯ | |
| 21:56:21 | mriedem | umm | |
| 21:56:29 | mriedem | ^ is reserved for our api response when a cell is down and you list instances | |
| 21:56:51 | melwitt | wwhhaaaaat | |
| 21:57:05 | mriedem | all we have is the uuid in the instance mapping, | |
| 21:57:11 | mriedem | so we're going to use "¯\_(ツ)_/¯" for all other fields | |
| 21:57:15 | melwitt | oh right | |
| 21:57:21 | melwitt | that would be fun | |
| 21:57:28 | mriedem | reserve that patch for april 1 | |
| 22:11:47 | mriedem | melwitt: dansmith: remind me, in the cellsv1 job, is n-api the top-level API or is that the cell API? | |
| 22:12:15 | melwitt | top-level API | |
| 22:12:32 | melwitt | I don't think we run n-api per cell though it is possible to do that (if you want to test a cell in isolation) | |
| 22:12:45 | mriedem | and cell-region is the nova-cells service at the top and cell-child is nova-cells in the cell? | |
| 22:13:03 | melwitt | oh boy, memory cobwebs | |
| 22:13:53 | melwitt | that sounds right though | |
| 22:14:08 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234 | |
| 22:18:53 | mriedem | hmm, ok, i'm trying to figure out http://logs.openstack.org/76/523676/11/check/legacy-tempest-dsvm-cells/1d155c6/logs/screen-n-cpu.txt.gz#_Dec_07_09_02_09_579991 | |
| 22:19:01 | mriedem | the patch creates an instance action in the api | |
| 22:19:13 | mriedem | then when the compute is trying to look it up, it can't find it and blows up | |
| 22:19:26 | mriedem | so i'm guessing, | |
| 22:19:42 | mriedem | the top level api database has the action record, but not the cell | |
| 22:19:45 | mriedem | cell db i mean | |
| 22:20:12 | mriedem | but then i don't know how we have action records working for anything in cellsv1 | |
| 22:20:18 | melwitt | yeah, so you'll want to check if the instance.save() at the top does a sync down to the cell. it probably does, but it can race | |
| 22:20:27 | melwitt | like compute might try to read before the sync makes it to the cell | |
| 22:20:38 | mriedem | this isn't instance.save() | |
| 22:20:46 | melwitt | okay, then it won't sync it | |
| 22:20:54 | mriedem | objects.InstanceAction.action_start | |
| 22:21:28 | melwitt | hm, so yeah does compute rely on reading instance actions in any other case? | |
| 22:21:48 | melwitt | because if so, like you said those shouldn't work either | |
| 22:21:51 | mriedem | oh i think i see something | |
| 22:22:11 | mriedem | so stop_instance is in this list https://github.com/openstack/nova/blob/master/nova/compute/cells_api.py#L51 | |
| 22:22:15 | mriedem | and so is backup_instance | |
| 22:22:35 | mriedem | https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1714 | |
| 22:22:38 | mriedem | stop_instance has that ^ | |
| 22:22:41 | mriedem | backup_instance doesn't | |
| 22:23:10 | melwitt | and you're seeing a fail with backup? | |
| 22:24:45 | mriedem | yeah https://review.openstack.org/#/c/523676/11 | |
| 22:24:46 | melwitt | so returning the response from the instance action vs not | |
| 22:25:04 | melwitt | I didn't realize instance action had a rpc response in any case | |
| 22:25:22 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Deduplicate functional test code https://review.openstack.org/526227 | |
| 22:27:53 | melwitt | yeah it looks like for some reason stop_instance can return something but backup_instance can't in nova/compute/rpcapi.py | |
| 22:28:07 | melwitt | I don't see stop_instance ever returning anything in compute/manager though | |
| 22:29:25 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete https://review.openstack.org/525231 | |
| 22:29:34 | mriedem | melwitt: i think the do_cast thing is something else | |
| 22:29:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (2) https://review.openstack.org/526263 | |
| 22:29:40 | mriedem | red herring | |
| 22:29:43 | melwitt | yeah, it's just saying wait for response vs not | |
| 22:29:53 | mriedem | so i'm not sure what ever creates instance action records in the top level vs cell db | |
| 22:30:05 | melwitt | and looks correct based on backup vs stop | |
| 22:30:28 | melwitt | well, it would be whichever service does the write. if n-api writes it, it will be in the top level db, if n-cpu writes it, it will be in the cell db | |
| 22:30:40 | melwitt | I think only instance records are synced back and forth in cells v1 | |
| 22:31:08 | melwitt | I'm checking to see if anything in backup_instance in compute does a write of instance action records | |
| 22:31:14 | mriedem | but if that's the case, then how does our @wrap_instance_event in the compute not blow up on everything? | |
| 22:31:20 | melwitt | I guess it can't be, else they'd be found | |
| 22:31:32 | mriedem | the action record stuff for backup_instance is new to this patch | |
| 22:31:40 | mriedem | following the same pattern as everywhere else | |
| 22:31:48 | melwitt | okay, I'll look at that then | |
| 22:32:44 | melwitt | yeah, that's just writing it in n-api so it'll go in the top level db only | |
| 22:32:52 | melwitt | I'm not sure how the other actions are currently working | |
| 22:44:44 | mriedem | alaski: you remember anything special about how the hell instance action records get synced between the api cell and the child cells in v1? | |
| 22:47:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add instance action record for backup instances https://review.openstack.org/523676 | |
| 22:53:07 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [WIP] POC to use neutron port_list when filtering instance by ip https://review.openstack.org/525505 | |
| 23:05:23 | tasker | I'm playing around with the nova-metadata service and every PUT or POST request I send to "openstack/latest/password" is seen by the service as a GET. is this a security feature of the metadata service or is there some configuration issue that I haven't uncovered yet? | |
| 23:06:34 | tasker | I tracked it through the code from api/metadata/password.py:handle_password up through MetadataRequestHandler and it's a GET the whole time. | |
| 23:06:51 | tasker | is something translating it to GET? | |
| 23:08:04 | mriedem | tempest does a POST successfully from the guest to change the password | |
| 23:08:08 | mriedem | i remember because of this bug https://bugs.launchpad.net/nova/+bug/1696848 | |
| 23:08:10 | openstack | Launchpad bug 1696848 in OpenStack Compute (nova) pike "POST /openstack/2013-10-17/password returns 500 if instance is not found" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 23:08:46 | mriedem | http://git.openstack.org/cgit/openstack/tempest/tree/tempest/scenario/test_server_basic_ops.py#n90 | |
| 23:09:02 | mriedem | tasker: are you passing a body? | |