| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-07 | |||
| 20:49:29 | mriedem | keep in mind that anything running in those threads that logs anything could lock you up | |
| 20:49:48 | imacdonn | yes, saw some stuff about that | |
| 20:49:50 | mriedem | so if you have a GMR when things are locked, i'd look for any libvirt driver/host methods in the thread dump, | |
| 20:49:53 | mriedem | and see if those do loging | |
| 20:49:55 | mriedem | *logging | |
| 20:50:22 | imacdonn | GMR is at https://pastebin.com/1jfgdurJ | |
| 20:50:29 | mriedem | because i'm sure we don't do a good job of auditing stuff like that | |
| 20:50:31 | imacdonn | getLibVersion() call at line 659 | |
| 20:51:00 | imacdonn | victim thread waiting to acquire() lock around line 777 | |
| 20:55:36 | imacdonn | seems like getLibVersion() isn't asking much of libvirt ... don't think it'd even have to talk to the hypervisor ... | |
| 20:56:45 | mriedem | well, it's making a connection to libvirt i believe | |
| 20:56:58 | mriedem | so it's not like the libvirt-python package version or something | |
| 20:57:11 | mriedem | nova meeting in 3 minutes | |
| 20:57:42 | imacdonn | it'd make a native library call at least .. dunno if it'd have to connect to libvirtd .. OK, I'll shut up for now ;) | |
| 20:58:03 | mriedem | true yeah | |
| 20:59:21 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (usage) https://review.openstack.org/520603 | |
| 20:59:37 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (inventory) https://review.openstack.org/520613 | |
| 21:00:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (aggregate) https://review.openstack.org/520608 | |
| 21:00:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (resource_class) https://review.openstack.org/520611 | |
| 21:00:04 | mriedem | melwitt: might want to check out v | |
| 21:00:05 | mriedem | https://bugs.launchpad.net/nova/+bug/1737011 | |
| 21:00:06 | openstack | Launchpad bug 1737011 in OpenStack Compute (nova) "ServerActionsTestJSON.test_reboot_server_hard failed to ssh into instance" [Undecided,New] | |
| 21:00:15 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (trait) https://review.openstack.org/520605 | |
| 21:00:32 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add x-openstack-request-id in API ref https://review.openstack.org/523007 | |
| 21:00:44 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add 'Location' parameters in API ref https://review.openstack.org/521541 | |
| 21:01:14 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Fix getting placement request ID https://review.openstack.org/523606 | |
| 21:01:57 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for resource class API https://review.openstack.org/524506 | |
| 21:02:09 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for traits API https://review.openstack.org/524094 | |
| 21:03:03 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400 | |
| 21:03:28 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234 | |
| 21:03:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add API and nova-manage tests that use the NoopQuotaDriver https://review.openstack.org/526270 | |
| 21:13:44 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | |
| 21:14:42 | openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | |
| 21:16:31 | melwitt | mriedem: is that different than our usual sporadic ssh timeouts? | |
| 21:19:33 | mriedem | melwitt: i don't think ssh timeouts are all that sporadic anymore | |
| 21:19:35 | mriedem | andreaf: ^? | |
| 21:20:19 | melwitt | okay, I saw one recently that I had to recheck so I thought they were still going on. I guess I should look at that one and compare | |
| 21:21:38 | mriedem | i don't see cloud-init run at all | |
| 21:24:23 | mriedem | force_config_drive = True | |
| 21:24:32 | mriedem | so we force a config drive to inject keys and such | |
| 21:27:01 | mriedem | we don't even get to the point of rebooting the serer | |
| 21:27:02 | mriedem | *server | |
| 21:27:08 | mriedem | it's trying to ssh into the guest to check uptime before that | |
| 21:27:10 | mriedem | and that's what fails | |
| 21:27:11 | mriedem | http://logs.openstack.org/83/526183/1/gate/legacy-tempest-dsvm-py35/166f0c9/job-output.txt.gz#_2017-12-07_13_37_58_430147 | |
| 21:27:32 | melwitt | good eye | |
| 21:28:10 | mriedem | the console output is all from tempest trying to gather information before the test pukes | |
| 21:28:14 | jaypipes | dansmith: from the API layer, if I want to find which cell a compute node (note: not the service host, but the Ironic baremetal node) was in, how would I do that? do I loop through cells doing a query? | |
| 21:28:56 | mriedem | jaypipes: i think you'd have to | |
| 21:29:11 | mriedem | the host mapping is the compute_node.host, not compute_nodes.hypervisor_hostname which is the node name | |
| 21:29:46 | jaypipes | mriedem: right | |
| 21:33:48 | melwitt | mriedem: this is the change I was thinking of where I had to recheck it about a week ago. looks like the same deal http://logs.openstack.org/22/518022/8/check/legacy-tempest-dsvm-neutron-full/81721fe/job-output.txt.gz#_2017-11-30_21_57_15_827380 | |
| 21:34:26 | mriedem | melwitt: http://logs.openstack.org/22/518022/8/check/legacy-tempest-dsvm-neutron-full/81721fe/job-output.txt.gz#_2017-11-30_21_57_15_838147 | |
| 21:35:02 | melwitt | oh heh | |
| 21:35:41 | melwitt | so not the same | |
| 21:37:46 | mriedem | melwitt: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Kernel%20panic%20-%20not%20syncing%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 21:38:08 | mriedem | nothing super obvious there, not like a single node provider | |
| 21:38:16 | mriedem | but it's all master branch, so i wonder if we're using a new cirros image in queens | |
| 21:39:15 | melwitt | how do we check that? | |
| 21:39:55 | mriedem | it's in devstack | |
| 21:40:10 | mriedem | https://github.com/openstack-dev/devstack/blob/master/stackrc#L671 | |
| 21:40:25 | mriedem | https://github.com/openstack-dev/devstack/commit/9f2dcd333103553626db1924a019e151e3e7252e | |
| 21:40:28 | melwitt | cool thanks | |
| 21:40:29 | mriedem | that's not new so... | |
| 21:40:51 | mriedem | heh read the commit message there | |
| 21:40:54 | melwitt | just ran logstash for 30d and it looks like this is pretty recent stuff. nothing before 11/27 | |
| 21:41:12 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: vSCSI https://review.openstack.org/526094 | |
| 21:41:14 | melwitt | haha, huh. | |
| 21:41:48 | mriedem | melwitt: logstash only goes back 10 days | |
| 21:42:01 | melwitt | oh :( | |
| 21:42:36 | melwitt | yeah I see what you mean, all providers hitting the kernel panic though pretty rarely | |
| 21:51:47 | mriedem | melwitt: i assume i'm pulling threads here https://review.openstack.org/#/c/524234/ | |
| 21:52:54 | melwitt | haha, that's okay | |
| 21:53:04 | mriedem | also, might be better to move that into a utility in QUOTAS | |
| 21:53:11 | mriedem | so the callers don't have to know about NoopQuotaDriver being a thing | |
| 21:53:15 | mriedem | QUOTAS.is_noop? | |
| 21:53:19 | melwitt | after fighting with the test coverage I wasn't thinking too clearly. that's my lame excuse | |
| 21:53:39 | mriedem | ok. i don't know what stephenfin's excuse was then. :) | |
| 21:54:11 | mriedem | thinking about bangers and mash during the football game at the pub | |
| 21:55:09 | melwitt | how is is_noop not knowing about the driver? | |
| 21:55:40 | melwitt | honestly the way it was before (the values coming from inside the driver) is the way the caller knows the least about it :P | |
| 21:55:45 | mriedem | i'm saying hide that check within QUOTAS | |
| 21:55:56 | mriedem | true | |
| 21:56:15 | mriedem | so, QUOTAS.get_reserved()? | |
| 21:56:16 | melwitt | so I'm like ¯\_(ツ)_/¯ | |
| 21:56:21 | mriedem | umm | |
| 21:56:29 | mriedem | ^ is reserved for our api response when a cell is down and you list instances | |
| 21:56:51 | melwitt | wwhhaaaaat | |
| 21:57:05 | mriedem | all we have is the uuid in the instance mapping, | |
| 21:57:11 | mriedem | so we're going to use "¯\_(ツ)_/¯" for all other fields | |
| 21:57:15 | melwitt | oh right | |
| 21:57:21 | melwitt | that would be fun | |
| 21:57:28 | mriedem | reserve that patch for april 1 | |
| 22:11:47 | mriedem | melwitt: dansmith: remind me, in the cellsv1 job, is n-api the top-level API or is that the cell API? | |
| 22:12:15 | melwitt | top-level API | |
| 22:12:32 | melwitt | I don't think we run n-api per cell though it is possible to do that (if you want to test a cell in isolation) | |
| 22:12:45 | mriedem | and cell-region is the nova-cells service at the top and cell-child is nova-cells in the cell? | |
| 22:13:03 | melwitt | oh boy, memory cobwebs | |
| 22:13:53 | melwitt | that sounds right though | |
| 22:14:08 | openstackgerrit | melanie witt proposed openstack/nova master: Follow up on removing old-style quotas code https://review.openstack.org/524234 | |