| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-13 | |||
| 13:46:44 | mriedem | stephenfin: if you agree with this, i'll rev that change quick and just fast approve it https://review.openstack.org/#/c/571375/4/api-ref/source/servers-admin-action.inc@70 | |
| 13:47:32 | stephenfin | mriedem: Fine by me | |
| 13:49:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix bug to api-ref https://review.openstack.org/571375 | |
| 13:49:06 | mriedem | thanks | |
| 13:51:17 | SpamapS | I'm sitting here watching my nova-compute process wait for input on stdin and I'm wondering what I've done wrong here... | |
| 13:52:10 | mriedem | nova-compute is now a vending machine program from cs 201 | |
| 13:52:15 | dansmith | sudo? | |
| 13:52:15 | SpamapS | http://paste.openstack.org/show/723383/ | |
| 13:52:22 | SpamapS | That's all it logs. | |
| 13:52:46 | SpamapS | there's no sudo running no.. the nova-compute process itself is looping on select(0,....) | |
| 13:53:38 | SpamapS | http://paste.openstack.org/show/723384/ | |
| 13:53:39 | SpamapS | stracing it | |
| 13:55:21 | dansmith | get gmr and see where it's sitting | |
| 13:55:25 | dansmith | I've never seen that before, fwiw | |
| 13:55:30 | SpamapS | doesn't log anything | |
| 13:55:36 | dansmith | gmr doesn't? | |
| 13:55:43 | SpamapS | no.. sent it USR1 and USR2 | |
| 13:56:12 | dansmith | wow, must be stuck super early | |
| 13:56:59 | SpamapS | http://paste.openstack.org/show/723385/ | |
| 13:57:08 | SpamapS | it actually did log that when I sent it USR1 just now | |
| 13:57:10 | SpamapS | just that | |
| 13:57:30 | SpamapS | I wonder if I send it a SIGINT if it will at least give me a keyboard interrupt stack trace | |
| 13:59:19 | SpamapS | http://paste.openstack.org/show/723386/ ooo... SIGINT did actually make a clue | |
| 14:00:46 | SpamapS | but .. .there's no amqp sockets open and no traffic trying so..... still so confused | |
| 14:01:18 | dansmith | SpamapS: is fd 0 really stdin for that? maybe stdin is closed and 0 was used for a socket or something? | |
| 14:02:05 | dansmith | ls -l /proc/$pid/fd/0 | |
| 14:02:39 | SpamapS | nova-comp 7 nova 0u CHR 136,2 0t0 5 /dev/pts/2 | |
| 14:02:51 | dansmith | bizarre | |
| 14:02:54 | SpamapS | yeah | |
| 14:03:11 | SpamapS | I wonder if it's some bug in kolla-ansible | |
| 14:03:13 | SpamapS | made bad containers | |
| 14:04:21 | SpamapS | ohh! gmr is actually spraying, to stdout | |
| 14:04:23 | SpamapS | which is the docker logs | |
| 14:05:33 | dansmith | it goes to stderr, which is probably the same for the container | |
| 14:05:39 | dansmith | you were looking in the logs? | |
| 14:06:21 | SpamapS | http://paste.openstack.org/show/723389/ | |
| 14:06:32 | SpamapS | Yeah I thought the log I was looking at was stderr | |
| 14:06:49 | SpamapS | there's a greenthread in there with no traceback | |
| 14:07:04 | SpamapS | but I think what's going on is, rmq tries, and fails, and is sleeping a bit? | |
| 14:07:24 | SpamapS | and that select on 0 is actually just eventlet's "got nothin to do" loop | |
| 14:08:00 | dansmith | makes sense | |
| 14:08:20 | SpamapS | yep that's it | |
| 14:08:24 | SpamapS | it backs off quick | |
| 14:08:34 | SpamapS | my tcpdump wasn't running before I restarted nova-compute | |
| 14:08:52 | SpamapS | figured it'd still be doing stuff immediately after startup... | |
| 14:09:19 | mriedem | odyssey4me: osa doesn't run the nova-status upgrade check during upgrades huh? | |
| 14:09:32 | mriedem | odyssey4me: are you aware of any existing patches for that? if not, i'll see if i can hack something in | |
| 14:10:45 | SpamapS | dansmith: thanks for the hints | |
| 14:10:59 | dansmith | I didn't do anything, but sure | |
| 14:13:24 | openstackgerrit | Dan Smith proposed openstack/nova master: Implement file backed memory for instances in libvirt https://review.openstack.org/567876 | |
| 14:19:52 | jaypipes | mnaser: hyper scale. | |
| 14:21:16 | openstackgerrit | Eric Fried proposed openstack/nova master: Enforce placement minimum in nova.cmd.status https://review.openstack.org/568965 | |
| 14:26:54 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] replace deprecated accept.best_match https://review.openstack.org/575127 | |
| 14:27:27 | mriedem | FFU bombs away! https://review.openstack.org/#/c/575125/ | |
| 14:28:02 | efried | dansmith: cellsv1 job gate failure: testtools.matchers._impl.MismatchError: set(['VM-2', 'VM-1']) != set([u'VM']) <== known? | |
| 14:28:27 | dansmith | not to me | |
| 14:28:29 | mriedem | no, efried link | |
| 14:28:46 | efried | http://logs.openstack.org/75/486475/38/gate/nova-cells-v1/2c2a566/job-output.txt.gz#_2018-06-13_13_26_17_111992 | |
| 14:29:54 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22self.assertEqual(set(%5B'VM-1'%2C%20'VM-2'%5D)%2C%20server_names)%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 14:30:21 | efried | same hit http://logs.openstack.org/40/568840/14/check/nova-cells-v1/1f17903/job-output.txt.gz#_2018-06-13_13_18_29_085017 and http://logs.openstack.org/67/567267/15/check/nova-cells-v1/0717ac7/job-output.txt.gz#_2018-06-13_13_27_34_890271 | |
| 14:30:39 | efried | okay, yah | |
| 14:30:42 | mriedem | hmm, check queue only | |
| 14:30:43 | mriedem | 4 changes | |
| 14:30:52 | mriedem | last 24 hours | |
| 14:30:58 | mriedem | is it a specific change related to multiple create? | |
| 14:31:14 | mriedem | https://review.openstack.org/#/c/570847/ | |
| 14:31:23 | mriedem | if it's all in that series, then it's a bug / race / leak in that series | |
| 14:31:33 | efried | No, the first link was in the xen series | |
| 14:31:43 | mriedem | yeah they are different | |
| 14:31:47 | mriedem | hmm, tempest change in the last 24 hours? | |
| 14:32:11 | mriedem | DING DING DING https://github.com/openstack/tempest/commit/306be6ce35652ca6273745b692aed3e681b30d8c | |
| 14:32:25 | mriedem | https://review.openstack.org/#/c/569199/ | |
| 14:33:16 | mriedem | gmann: fyi ^ | |
| 14:33:22 | mriedem | i guess we could just blacklist this test for cells v1.... | |
| 14:33:37 | dansmith | why does it fail for cellsv1? | |
| 14:33:58 | mriedem | well it's intermittent, so assuming some kind of time delay | |
| 14:34:14 | dansmith | but | |
| 14:34:21 | dansmith | it's not missing one it's missing both yeah? | |
| 14:34:32 | mriedem | i haven't dug into the actual test changes | |
| 14:34:40 | mriedem | opening a bug first | |
| 14:35:41 | openstack | Launchpad bug 1776684 in OpenStack Compute (nova) "MultipleCreateTestJSON.test_multiple_create intermittently fails for cells v1 due to server name check change" [High,Confirmed] | |
| 14:35:41 | mriedem | https://bugs.launchpad.net/nova/+bug/1776684 | |
| 14:37:27 | mriedem | new_name = '%s-%d' % (original_name, index + 1) | |
| 14:37:27 | mriedem | original_name = instance.display_name | |
| 14:37:27 | mriedem | if num_instances > 1 and self.cell_type != 'api': | |
| 14:37:37 | mriedem | so...this is probably going to be a hard 100% failure | |
| 14:37:51 | dansmith | yeah that makes more sense | |
| 14:39:29 | mriedem | revert https://review.openstack.org/#/c/575132/ | |
| 14:39:36 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add status field to version document at / https://review.openstack.org/575117 | |
| 14:40:27 | dansmith | mriedem: you don't think it's legit to have a test for that and that cellsv1 is the odd man out? | |
| 14:40:54 | dansmith | or do you think that's a detail of the api we shouldn't depend on | |
| 14:40:55 | dansmith | / | |
| 14:41:15 | mriedem | forgot about the blacklist, we can just do that | |
| 14:41:57 | mriedem | i think it's likely not a great test wrinkle to tempest, | |
| 14:42:13 | mriedem | because tempest is branchless and before dropping the multi create instance name template, it was configurable in the API | |
| 14:42:21 | mriedem | so this could cause some deployments to fail that test if they have modified that config option | |
| 14:42:28 | dansmith | ah that's fair | |
| 14:44:12 | mriedem | https://review.openstack.org/#/c/499612/ | |
| 14:44:17 | mriedem | so that was just removed in queens | |
| 14:44:29 | mriedem | so tempest could now fail on anything using a non-standard name template before queens | |