| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-13 | |||
| 14:04:23 | SpamapS | which is the docker logs | |
| 14:05:33 | dansmith | it goes to stderr, which is probably the same for the container | |
| 14:05:39 | dansmith | you were looking in the logs? | |
| 14:06:21 | SpamapS | http://paste.openstack.org/show/723389/ | |
| 14:06:32 | SpamapS | Yeah I thought the log I was looking at was stderr | |
| 14:06:49 | SpamapS | there's a greenthread in there with no traceback | |
| 14:07:04 | SpamapS | but I think what's going on is, rmq tries, and fails, and is sleeping a bit? | |
| 14:07:24 | SpamapS | and that select on 0 is actually just eventlet's "got nothin to do" loop | |
| 14:08:00 | dansmith | makes sense | |
| 14:08:20 | SpamapS | yep that's it | |
| 14:08:24 | SpamapS | it backs off quick | |
| 14:08:34 | SpamapS | my tcpdump wasn't running before I restarted nova-compute | |
| 14:08:52 | SpamapS | figured it'd still be doing stuff immediately after startup... | |
| 14:09:19 | mriedem | odyssey4me: osa doesn't run the nova-status upgrade check during upgrades huh? | |
| 14:09:32 | mriedem | odyssey4me: are you aware of any existing patches for that? if not, i'll see if i can hack something in | |
| 14:10:45 | SpamapS | dansmith: thanks for the hints | |
| 14:10:59 | dansmith | I didn't do anything, but sure | |
| 14:13:24 | openstackgerrit | Dan Smith proposed openstack/nova master: Implement file backed memory for instances in libvirt https://review.openstack.org/567876 | |
| 14:19:52 | jaypipes | mnaser: hyper scale. | |
| 14:21:16 | openstackgerrit | Eric Fried proposed openstack/nova master: Enforce placement minimum in nova.cmd.status https://review.openstack.org/568965 | |
| 14:26:54 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] replace deprecated accept.best_match https://review.openstack.org/575127 | |
| 14:27:27 | mriedem | FFU bombs away! https://review.openstack.org/#/c/575125/ | |
| 14:28:02 | efried | dansmith: cellsv1 job gate failure: testtools.matchers._impl.MismatchError: set(['VM-2', 'VM-1']) != set([u'VM']) <== known? | |
| 14:28:27 | dansmith | not to me | |
| 14:28:29 | mriedem | no, efried link | |
| 14:28:46 | efried | http://logs.openstack.org/75/486475/38/gate/nova-cells-v1/2c2a566/job-output.txt.gz#_2018-06-13_13_26_17_111992 | |
| 14:29:54 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22self.assertEqual(set(%5B'VM-1'%2C%20'VM-2'%5D)%2C%20server_names)%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 14:30:21 | efried | same hit http://logs.openstack.org/40/568840/14/check/nova-cells-v1/1f17903/job-output.txt.gz#_2018-06-13_13_18_29_085017 and http://logs.openstack.org/67/567267/15/check/nova-cells-v1/0717ac7/job-output.txt.gz#_2018-06-13_13_27_34_890271 | |
| 14:30:39 | efried | okay, yah | |
| 14:30:42 | mriedem | hmm, check queue only | |
| 14:30:43 | mriedem | 4 changes | |
| 14:30:52 | mriedem | last 24 hours | |
| 14:30:58 | mriedem | is it a specific change related to multiple create? | |
| 14:31:14 | mriedem | https://review.openstack.org/#/c/570847/ | |
| 14:31:23 | mriedem | if it's all in that series, then it's a bug / race / leak in that series | |
| 14:31:33 | efried | No, the first link was in the xen series | |
| 14:31:43 | mriedem | yeah they are different | |
| 14:31:47 | mriedem | hmm, tempest change in the last 24 hours? | |
| 14:32:11 | mriedem | DING DING DING https://github.com/openstack/tempest/commit/306be6ce35652ca6273745b692aed3e681b30d8c | |
| 14:32:25 | mriedem | https://review.openstack.org/#/c/569199/ | |
| 14:33:16 | mriedem | gmann: fyi ^ | |
| 14:33:22 | mriedem | i guess we could just blacklist this test for cells v1.... | |
| 14:33:37 | dansmith | why does it fail for cellsv1? | |
| 14:33:58 | mriedem | well it's intermittent, so assuming some kind of time delay | |
| 14:34:14 | dansmith | but | |
| 14:34:21 | dansmith | it's not missing one it's missing both yeah? | |
| 14:34:32 | mriedem | i haven't dug into the actual test changes | |
| 14:34:40 | mriedem | opening a bug first | |
| 14:35:41 | mriedem | https://bugs.launchpad.net/nova/+bug/1776684 | |
| 14:35:41 | openstack | Launchpad bug 1776684 in OpenStack Compute (nova) "MultipleCreateTestJSON.test_multiple_create intermittently fails for cells v1 due to server name check change" [High,Confirmed] | |
| 14:37:27 | mriedem | if num_instances > 1 and self.cell_type != 'api': | |
| 14:37:27 | mriedem | original_name = instance.display_name | |
| 14:37:27 | mriedem | new_name = '%s-%d' % (original_name, index + 1) | |
| 14:37:37 | mriedem | so...this is probably going to be a hard 100% failure | |
| 14:37:51 | dansmith | yeah that makes more sense | |
| 14:39:29 | mriedem | revert https://review.openstack.org/#/c/575132/ | |
| 14:39:36 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add status field to version document at / https://review.openstack.org/575117 | |
| 14:40:27 | dansmith | mriedem: you don't think it's legit to have a test for that and that cellsv1 is the odd man out? | |
| 14:40:54 | dansmith | or do you think that's a detail of the api we shouldn't depend on | |
| 14:40:55 | dansmith | / | |
| 14:41:15 | mriedem | forgot about the blacklist, we can just do that | |
| 14:41:57 | mriedem | i think it's likely not a great test wrinkle to tempest, | |
| 14:42:13 | mriedem | because tempest is branchless and before dropping the multi create instance name template, it was configurable in the API | |
| 14:42:21 | mriedem | so this could cause some deployments to fail that test if they have modified that config option | |
| 14:42:28 | dansmith | ah that's fair | |
| 14:44:12 | mriedem | https://review.openstack.org/#/c/499612/ | |
| 14:44:17 | mriedem | so that was just removed in queens | |
| 14:44:29 | mriedem | so tempest could now fail on anything using a non-standard name template before queens | |
| 14:45:53 | dansmith | aye | |
| 14:46:13 | mriedem | if we still had eol, this tempest change really shouldn't be made until nova's pike-eol | |
| 14:46:28 | mriedem | so, we could probably revert on those grounds alone | |
| 14:48:54 | mriedem | restored the revert | |
| 14:53:57 | mriedem | melwitt: actually looks like test_multiple_create is something in tempest that isn't microversion-specific and isn't in the interop guidelines | |
| 14:54:00 | mriedem | so that's a candidate | |
| 14:54:25 | mriedem | it's not backend specific, not really configurable either | |
| 14:59:31 | mgagne | mriedem: so I'm mostly at a point where I can test your patch. I had to upgrade Ironic to Queens too in order to be able to start nova-compute (which requires a recent Ironic API) to populate placement with provider resources. | |
| 15:00:06 | mriedem | yeah ironic always goes before nova | |
| 15:00:08 | mgagne | BUT I find something surprising. Could it be that nova-compute already auto-heal allocations on startup? | |
| 15:00:26 | openstackgerrit | Eric Fried proposed openstack/nova master: Update nova-status & docs: require placement 1.25 https://review.openstack.org/575140 | |
| 15:00:32 | mriedem | mgagne: computes < pike would create allocatiosn | |
| 15:00:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Enforce placement minimum in nova.cmd.status https://review.openstack.org/568965 | |
| 15:00:43 | mriedem | well, newton and ocata computes configured to talk to placement would create allocations | |
| 15:00:53 | mriedem | mgagne: but i thought you were going directly from mitaka to queens? | |
| 15:01:08 | mgagne | 17.0.4 is queens right? | |
| 15:01:22 | mgagne | mriedem: yes I am, running queens now | |
| 15:01:28 | mriedem | 17.0.5 at this point, but yeah | |
| 15:01:45 | mriedem | i wonder if you have any stale nova-compute services in the cell db? | |
| 15:02:01 | mgagne | 17.0.5 already? oh well | |
| 15:02:13 | mriedem | because just one will trigger the auto-heal https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1217 | |
| 15:02:27 | mriedem | mgagne: check to see if you have any services records in the cell db that are version < 22 | |
| 15:02:46 | mriedem | OH | |
| 15:02:52 | mriedem | if these are ironic computes, then those auto-heal | |
| 15:02:56 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1225 | |
| 15:02:59 | mgagne | hmm, yes I'm hitting that code path | |
| 15:03:02 | melwitt | mriedem: okay, thanks | |
| 15:03:02 | mgagne | "Compute driver requires allocation refresh. Will auto-correct allocations to handle Ocata-style assumptions." | |
| 15:03:15 | mriedem | mgagne: yeah that's for your ironic nodes | |
| 15:03:24 | mgagne | hilarious... | |
| 15:03:31 | mriedem | mgagne: but how about libvirt nodes? | |
| 15:04:12 | mgagne | we don't use caching with libvirt in our product... well, we do but in a private cloud elsewhere | |