| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-07 | |||
| 21:28:10 | mriedem | the console output is all from tempest trying to gather information before the test pukes | |
| 21:28:14 | jaypipes | dansmith: from the API layer, if I want to find which cell a compute node (note: not the service host, but the Ironic baremetal node) was in, how would I do that? do I loop through cells doing a query? | |
| 21:28:56 | mriedem | jaypipes: i think you'd have to | |
| 21:29:11 | mriedem | the host mapping is the compute_node.host, not compute_nodes.hypervisor_hostname which is the node name | |
| 21:29:46 | jaypipes | mriedem: right | |
| 21:33:48 | melwitt | mriedem: this is the change I was thinking of where I had to recheck it about a week ago. looks like the same deal http://logs.openstack.org/22/518022/8/check/legacy-tempest-dsvm-neutron-full/81721fe/job-output.txt.gz#_2017-11-30_21_57_15_827380 | |
| 21:34:26 | mriedem | melwitt: http://logs.openstack.org/22/518022/8/check/legacy-tempest-dsvm-neutron-full/81721fe/job-output.txt.gz#_2017-11-30_21_57_15_838147 | |
| 21:35:02 | melwitt | oh heh | |
| 21:35:41 | melwitt | so not the same | |
| 21:37:46 | mriedem | melwitt: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Kernel%20panic%20-%20not%20syncing%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 21:38:08 | mriedem | nothing super obvious there, not like a single node provider | |
| 21:38:16 | mriedem | but it's all master branch, so i wonder if we're using a new cirros image in queens | |
| 21:39:15 | melwitt | how do we check that? | |
| 21:39:55 | mriedem | it's in devstack | |
| 21:40:10 | mriedem | https://github.com/openstack-dev/devstack/blob/master/stackrc#L671 | |
| 21:40:25 | mriedem | https://github.com/openstack-dev/devstack/commit/9f2dcd333103553626db1924a019e151e3e7252e | |
| 21:40:28 | melwitt | cool thanks | |
| 21:40:29 | mriedem | that's not new so... | |
| 21:40:51 | mriedem | heh read the commit message there | |
| 21:40:54 | melwitt | just ran logstash for 30d and it looks like this is pretty recent stuff. nothing before 11/27 | |
| 21:41:12 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 21:41:14 | melwitt | haha, huh. | |
| 21:41:48 | mriedem | melwitt: logstash only goes back 10 days | |
| 21:42:01 | melwitt | oh :( | |
| 21:42:36 | melwitt | yeah I see what you mean, all providers hitting the kernel panic though pretty rarely | |
| 21:51:47 | mriedem | melwitt: i assume i'm pulling threads here https://review.openstack.org/#/c/524234/ | |
| 21:52:54 | melwitt | haha, that's okay | |
| 21:53:04 | mriedem | also, might be better to move that into a utility in QUOTAS | |
| 21:53:11 | mriedem | so the callers don't have to know about NoopQuotaDriver being a thing | |
| 21:53:15 | mriedem | QUOTAS.is_noop? | |
| 21:53:19 | melwitt | after fighting with the test coverage I wasn't thinking too clearly. that's my lame excuse | |
| 21:53:39 | mriedem | ok. i don't know what stephenfin's excuse was then. :) | |
| 21:54:11 | mriedem | thinking about bangers and mash during the football game at the pub | |
| 21:55:09 | melwitt | how is is_noop not knowing about the driver? | |
| 21:55:40 | melwitt | honestly the way it was before (the values coming from inside the driver) is the way the caller knows the least about it :P | |
| 21:55:45 | mriedem | i'm saying hide that check within QUOTAS | |
| 21:55:56 | mriedem | true | |
| 21:56:15 | mriedem | so, QUOTAS.get_reserved()? | |
| 21:56:16 | melwitt | so I'm like ¯\_(ツ)_/¯ | |
| 21:56:21 | mriedem | umm | |
| 21:56:29 | mriedem | ^ is reserved for our api response when a cell is down and you list instances | |
| 21:56:51 | melwitt | wwhhaaaaat | |
| 21:57:05 | mriedem | all we have is the uuid in the instance mapping, | |
| 21:57:11 | mriedem | so we're going to use "¯\_(ツ)_/¯" for all other fields | |
| 21:57:15 | melwitt | oh right | |
| 21:57:21 | melwitt | that would be fun | |
| 21:57:28 | mriedem | reserve that patch for april 1 | |
| 22:11:47 | mriedem | melwitt: dansmith: remind me, in the cellsv1 job, is n-api the top-level API or is that the cell API? | |
| 22:12:15 | melwitt | top-level API | |
| 22:12:32 | melwitt | I don't think we run n-api per cell though it is possible to do that (if you want to test a cell in isolation) | |
| 22:12:45 | mriedem | and cell-region is the nova-cells service at the top and cell-child is nova-cells in the cell? | |
| 22:13:03 | melwitt | oh boy, memory cobwebs | |
| 22:13:53 | melwitt | that sounds right though | |
| 22:14:08 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234 | |
| 22:18:53 | mriedem | hmm, ok, i'm trying to figure out http://logs.openstack.org/76/523676/11/check/legacy-tempest-dsvm-cells/1d155c6/logs/screen-n-cpu.txt.gz#_Dec_07_09_02_09_579991 | |
| 22:19:01 | mriedem | the patch creates an instance action in the api | |
| 22:19:13 | mriedem | then when the compute is trying to look it up, it can't find it and blows up | |
| 22:19:26 | mriedem | so i'm guessing, | |
| 22:19:42 | mriedem | the top level api database has the action record, but not the cell | |
| 22:19:45 | mriedem | cell db i mean | |
| 22:20:12 | mriedem | but then i don't know how we have action records working for anything in cellsv1 | |
| 22:20:18 | melwitt | yeah, so you'll want to check if the instance.save() at the top does a sync down to the cell. it probably does, but it can race | |
| 22:20:27 | melwitt | like compute might try to read before the sync makes it to the cell | |
| 22:20:38 | mriedem | this isn't instance.save() | |
| 22:20:46 | melwitt | okay, then it won't sync it | |
| 22:20:54 | mriedem | objects.InstanceAction.action_start | |
| 22:21:28 | melwitt | hm, so yeah does compute rely on reading instance actions in any other case? | |
| 22:21:48 | melwitt | because if so, like you said those shouldn't work either | |
| 22:21:51 | mriedem | oh i think i see something | |
| 22:22:11 | mriedem | so stop_instance is in this list https://github.com/openstack/nova/blob/master/nova/compute/cells_api.py#L51 | |
| 22:22:15 | mriedem | and so is backup_instance | |
| 22:22:35 | mriedem | https://github.com/openstack/nova/blob/master/nova/cells/messaging.py#L1714 | |
| 22:22:38 | mriedem | stop_instance has that ^ | |
| 22:22:41 | mriedem | backup_instance doesn't | |
| 22:23:10 | melwitt | and you're seeing a fail with backup? | |
| 22:24:45 | mriedem | yeah https://review.openstack.org/#/c/523676/11 | |
| 22:24:46 | melwitt | so returning the response from the instance action vs not | |
| 22:25:04 | melwitt | I didn't realize instance action had a rpc response in any case | |
| 22:25:22 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: Deduplicate functional test code https://review.openstack.org/526227 | |
| 22:27:53 | melwitt | yeah it looks like for some reason stop_instance can return something but backup_instance can't in nova/compute/rpcapi.py | |
| 22:28:07 | melwitt | I don't see stop_instance ever returning anything in compute/manager though | |
| 22:29:25 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete https://review.openstack.org/525231 | |
| 22:29:34 | mriedem | melwitt: i think the do_cast thing is something else | |
| 22:29:38 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (2) https://review.openstack.org/526263 | |
| 22:29:40 | mriedem | red herring | |
| 22:29:43 | melwitt | yeah, it's just saying wait for response vs not | |
| 22:29:53 | mriedem | so i'm not sure what ever creates instance action records in the top level vs cell db | |
| 22:30:05 | melwitt | and looks correct based on backup vs stop | |
| 22:30:28 | melwitt | well, it would be whichever service does the write. if n-api writes it, it will be in the top level db, if n-cpu writes it, it will be in the cell db | |
| 22:30:40 | melwitt | I think only instance records are synced back and forth in cells v1 | |
| 22:31:08 | melwitt | I'm checking to see if anything in backup_instance in compute does a write of instance action records | |
| 22:31:14 | mriedem | but if that's the case, then how does our @wrap_instance_event in the compute not blow up on everything? | |
| 22:31:20 | melwitt | I guess it can't be, else they'd be found | |
| 22:31:32 | mriedem | the action record stuff for backup_instance is new to this patch | |
| 22:31:40 | mriedem | following the same pattern as everywhere else | |
| 22:31:48 | melwitt | okay, I'll look at that then | |
| 22:32:44 | melwitt | yeah, that's just writing it in n-api so it'll go in the top level db only | |
| 22:32:52 | melwitt | I'm not sure how the other actions are currently working | |
| 22:44:44 | mriedem | alaski: you remember anything special about how the hell instance action records get synced between the api cell and the child cells in v1? | |
| 22:47:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add instance action record for backup instances https://review.openstack.org/523676 | |