| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-13 | |||
| 14:28:29 | mriedem | no, efried link | |
| 14:28:46 | efried | http://logs.openstack.org/75/486475/38/gate/nova-cells-v1/2c2a566/job-output.txt.gz#_2018-06-13_13_26_17_111992 | |
| 14:29:54 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22self.assertEqual(set(%5B'VM-1'%2C%20'VM-2'%5D)%2C%20server_names)%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 14:30:21 | efried | same hit http://logs.openstack.org/40/568840/14/check/nova-cells-v1/1f17903/job-output.txt.gz#_2018-06-13_13_18_29_085017 and http://logs.openstack.org/67/567267/15/check/nova-cells-v1/0717ac7/job-output.txt.gz#_2018-06-13_13_27_34_890271 | |
| 14:30:39 | efried | okay, yah | |
| 14:30:42 | mriedem | hmm, check queue only | |
| 14:30:43 | mriedem | 4 changes | |
| 14:30:52 | mriedem | last 24 hours | |
| 14:30:58 | mriedem | is it a specific change related to multiple create? | |
| 14:31:14 | mriedem | https://review.openstack.org/#/c/570847/ | |
| 14:31:23 | mriedem | if it's all in that series, then it's a bug / race / leak in that series | |
| 14:31:33 | efried | No, the first link was in the xen series | |
| 14:31:43 | mriedem | yeah they are different | |
| 14:31:47 | mriedem | hmm, tempest change in the last 24 hours? | |
| 14:32:11 | mriedem | DING DING DING https://github.com/openstack/tempest/commit/306be6ce35652ca6273745b692aed3e681b30d8c | |
| 14:32:25 | mriedem | https://review.openstack.org/#/c/569199/ | |
| 14:33:16 | mriedem | gmann: fyi ^ | |
| 14:33:22 | mriedem | i guess we could just blacklist this test for cells v1.... | |
| 14:33:37 | dansmith | why does it fail for cellsv1? | |
| 14:33:58 | mriedem | well it's intermittent, so assuming some kind of time delay | |
| 14:34:14 | dansmith | but | |
| 14:34:21 | dansmith | it's not missing one it's missing both yeah? | |
| 14:34:32 | mriedem | i haven't dug into the actual test changes | |
| 14:34:40 | mriedem | opening a bug first | |
| 14:35:41 | mriedem | https://bugs.launchpad.net/nova/+bug/1776684 | |
| 14:35:41 | openstack | Launchpad bug 1776684 in OpenStack Compute (nova) "MultipleCreateTestJSON.test_multiple_create intermittently fails for cells v1 due to server name check change" [High,Confirmed] | |
| 14:37:27 | mriedem | if num_instances > 1 and self.cell_type != 'api': | |
| 14:37:27 | mriedem | original_name = instance.display_name | |
| 14:37:27 | mriedem | new_name = '%s-%d' % (original_name, index + 1) | |
| 14:37:37 | mriedem | so...this is probably going to be a hard 100% failure | |
| 14:37:51 | dansmith | yeah that makes more sense | |
| 14:39:29 | mriedem | revert https://review.openstack.org/#/c/575132/ | |
| 14:39:36 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add status field to version document at / https://review.openstack.org/575117 | |
| 14:40:27 | dansmith | mriedem: you don't think it's legit to have a test for that and that cellsv1 is the odd man out? | |
| 14:40:54 | dansmith | or do you think that's a detail of the api we shouldn't depend on | |
| 14:40:55 | dansmith | / | |
| 14:41:15 | mriedem | forgot about the blacklist, we can just do that | |
| 14:41:57 | mriedem | i think it's likely not a great test wrinkle to tempest, | |
| 14:42:13 | mriedem | because tempest is branchless and before dropping the multi create instance name template, it was configurable in the API | |
| 14:42:21 | mriedem | so this could cause some deployments to fail that test if they have modified that config option | |
| 14:42:28 | dansmith | ah that's fair | |
| 14:44:12 | mriedem | https://review.openstack.org/#/c/499612/ | |
| 14:44:17 | mriedem | so that was just removed in queens | |
| 14:44:29 | mriedem | so tempest could now fail on anything using a non-standard name template before queens | |
| 14:45:53 | dansmith | aye | |
| 14:46:13 | mriedem | if we still had eol, this tempest change really shouldn't be made until nova's pike-eol | |
| 14:46:28 | mriedem | so, we could probably revert on those grounds alone | |
| 14:48:54 | mriedem | restored the revert | |
| 14:53:57 | mriedem | melwitt: actually looks like test_multiple_create is something in tempest that isn't microversion-specific and isn't in the interop guidelines | |
| 14:54:00 | mriedem | so that's a candidate | |
| 14:54:25 | mriedem | it's not backend specific, not really configurable either | |
| 14:59:31 | mgagne | mriedem: so I'm mostly at a point where I can test your patch. I had to upgrade Ironic to Queens too in order to be able to start nova-compute (which requires a recent Ironic API) to populate placement with provider resources. | |
| 15:00:06 | mriedem | yeah ironic always goes before nova | |
| 15:00:08 | mgagne | BUT I find something surprising. Could it be that nova-compute already auto-heal allocations on startup? | |
| 15:00:26 | openstackgerrit | Eric Fried proposed openstack/nova master: Update nova-status & docs: require placement 1.25 https://review.openstack.org/575140 | |
| 15:00:32 | mriedem | mgagne: computes < pike would create allocatiosn | |
| 15:00:39 | openstackgerrit | Eric Fried proposed openstack/nova master: Enforce placement minimum in nova.cmd.status https://review.openstack.org/568965 | |
| 15:00:43 | mriedem | well, newton and ocata computes configured to talk to placement would create allocations | |
| 15:00:53 | mriedem | mgagne: but i thought you were going directly from mitaka to queens? | |
| 15:01:08 | mgagne | 17.0.4 is queens right? | |
| 15:01:22 | mgagne | mriedem: yes I am, running queens now | |
| 15:01:28 | mriedem | 17.0.5 at this point, but yeah | |
| 15:01:45 | mriedem | i wonder if you have any stale nova-compute services in the cell db? | |
| 15:02:01 | mgagne | 17.0.5 already? oh well | |
| 15:02:13 | mriedem | because just one will trigger the auto-heal https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1217 | |
| 15:02:27 | mriedem | mgagne: check to see if you have any services records in the cell db that are version < 22 | |
| 15:02:46 | mriedem | OH | |
| 15:02:52 | mriedem | if these are ironic computes, then those auto-heal | |
| 15:02:56 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L1225 | |
| 15:02:59 | mgagne | hmm, yes I'm hitting that code path | |
| 15:03:02 | melwitt | mriedem: okay, thanks | |
| 15:03:02 | mgagne | "Compute driver requires allocation refresh. Will auto-correct allocations to handle Ocata-style assumptions." | |
| 15:03:15 | mriedem | mgagne: yeah that's for your ironic nodes | |
| 15:03:24 | mgagne | hilarious... | |
| 15:03:31 | mriedem | mgagne: but how about libvirt nodes? | |
| 15:04:12 | mgagne | we don't use caching with libvirt in our product... well, we do but in a private cloud elsewhere | |
| 15:04:35 | mgagne | and without cells but I guess it's irrelevant | |
| 15:05:17 | mgagne | so I would have to test this specific case. so back to tests | |
| 15:08:16 | mriedem | mgagne: so you use cells v1 + ironic + caching scheduler, | |
| 15:08:29 | mriedem | but not cells v1, libvirt and filter scheduler elsewhere? | |
| 15:09:41 | mriedem | mgagne: also note that in queens, the IronicHostManager and Exact* filters are deprecated for ironic | |
| 15:09:49 | mriedem | actually pike, but ... | |
| 15:15:21 | openstackgerrit | Eric Fried proposed openstack/nova master: Update nova-status & docs: require placement 1.25 https://review.openstack.org/575140 | |
| 15:15:22 | openstackgerrit | Eric Fried proposed openstack/nova master: Enforce placement minimum in nova.cmd.status https://review.openstack.org/568965 | |
| 15:17:11 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 15:17:12 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add notification support for trusted_certs https://review.openstack.org/563269 | |
| 15:17:13 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Add certificate validation docs https://review.openstack.org/560158 | |
| 15:17:14 | openstackgerrit | Brianna Poulos proposed openstack/nova master: WIP: Add trusted certs to feature support matrix docs https://review.openstack.org/574890 | |
| 15:17:15 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Remove max_size parameter from fake_libvirt_utils.fetch_*image methods https://review.openstack.org/574911 | |
| 15:19:27 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add status and links fields to version document at / https://review.openstack.org/575117 | |
| 15:20:19 | mriedem | gibi: i don't know when bdm.no_device is useful :) | |
| 15:20:36 | mriedem | i think it's some legacy bdm v1 / ec2 carryover | |
| 15:20:55 | gibi | mriedem: me neither so fair enough :) | |
| 15:21:21 | mgagne | mriedem: yes, that's right: CellsV1 + Ironic + Caching. CellsV1 + libvirt + filter elsewhere. (and no cells + libvirt + Caching in our private cloud) | |
| 15:27:01 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] replace deprecated accept.best_match https://review.openstack.org/575127 | |
| 15:30:52 | melwitt | I'm going to be out next week and need someone to run the nova meeting June 21 at 21:00 UTC. is there anyone who can help with that? | |
| 15:34:30 | mriedem | i'm out wed-fri next week so not it | |
| 15:34:54 | mriedem | i nominate dansmith or efried | |
| 15:35:55 | openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Add status and links fields to version document at / https://review.openstack.org/575117 | |
| 15:37:15 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: Allocation.consumer field https://review.openstack.org/565405 | |