| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 19:41:39 | artom | mriedem, only backportable ones | |
| 19:41:46 | mriedem | are you asking about the serial changing on an existing guest? | |
| 19:41:50 | artom | mriedem, yeah | |
| 19:42:02 | mriedem | so change the default to 'auto' like it was? | |
| 19:42:29 | artom | mriedem, well, not sure it makes sense to go back at this point | |
| 19:42:43 | mriedem | there was quite a bit of discussion around that blueprint, including being able to override the host config with image meta or flavors but that was dropped | |
| 19:42:50 | eandersson | btw launched_on is is always null | |
| 19:42:51 | artom | But... there's an impact, right? The default becomes 'unique', instance is hard-rebooted, boom, new serial | |
| 19:42:55 | artom | Linux is probably OK, but Windows... | |
| 19:43:46 | artom | So lyarwood just updated our proposed backport to not change the default | |
| 19:43:51 | artom | Do... do we need to do the same upstream? | |
| 19:43:57 | mriedem | there is an upgrade release note. i'd have to dig into all of the comments on that patch to determine if we changed the default later or not for good reason | |
| 19:44:10 | mriedem | that was released in stein | |
| 19:44:16 | mriedem | changing the default from 2 releases back would be weird | |
| 19:44:17 | artom | Right, so it's already out there | |
| 19:44:20 | artom | I know | |
| 19:45:07 | efried | mriedem: see gibi's comment at the bottom of https://review.opendev.org/#/c/696992/ -- is it possible the bw test was removed when you refactored to get rid of tempest-slow? | |
| 19:46:12 | eandersson | I think this is a new stuckin BUILD bug because I can't even find the instance in placement | |
| 19:48:58 | mriedem | efried: it should be run in the nova-next job | |
| 19:49:14 | efried | okay, will look for it there | |
| 19:49:40 | mriedem | i don't see it, checking something | |
| 19:49:42 | artom | I guess there's less impact when it's in a new release | |
| 19:49:55 | mriedem | artom: well we don't backport features upstream for a reason so sure | |
| 19:50:01 | artom | But in an existing release we definitely can't change the defaults | |
| 19:50:29 | mriedem | hey, how you want to break your enterprise users on queens is up to you :) | |
| 19:50:45 | artom | In a way that we can fix for loads of $$ | |
| 19:50:50 | mriedem | job security | |
| 19:50:58 | artom | 'zactly | |
| 19:52:51 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418 | |
| 19:52:51 | openstackgerrit | Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991 | |
| 19:52:52 | openstackgerrit | Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946 | |
| 19:52:52 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992 | |
| 19:52:55 | mriedem | efried: so nova-next should run tempest api compute and scenario tests: | |
| 19:52:55 | mriedem | tempest_test_regex: ^tempest\.(scenario|api\.compute) | |
| 19:53:00 | mriedem | except these scenario tests: | |
| 19:53:05 | mriedem | tempest_black_regex: ^tempest.scenario.test_network | |
| 19:53:20 | mriedem | but that shouldn't hit on tempest/scenario/test_minbw_allocation_placement | |
| 19:54:47 | mriedem | it does get run in tempest-slow in train though yeah https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/job-output.txt#73583 | |
| 19:55:13 | eandersson | I am wondering if we are hitting some race conndition | |
| 19:55:33 | eandersson | Because we create 2 VMs ever 10 minutes and never get VMs stuck in BUILDING. | |
| 19:55:42 | eandersson | but our customers hit this very often with aggressive terraform deployments | |
| 19:55:54 | mriedem | efried: so we went from this in tempest-slow: | |
| 19:55:54 | mriedem | slow-serial run-test: commands[1] | tempest run --serial --regex '\[.*\bslow\b.*\]' --concurrency=2 --black-regex= | |
| 19:55:57 | mriedem | to this in nova-next: | |
| 19:56:02 | mriedem | all run-test: commands[1] | tempest run --regex '^tempest\.(scenario|api\.compute)' --concurrency=4 '--black-regex=^tempest.scenario.test_network' | |
| 19:56:22 | mriedem | i'm not sure why test_minbw_allocation_placement would be filtered out | |
| 19:56:51 | mriedem | oh i see | |
| 19:56:51 | mriedem | https://zuul.opendev.org/t/openstack/build/66f29bf5f12449059e82d24db5aff47a/log/job-output.txt#79328 | |
| 19:57:06 | mriedem | {2} setUpClass (tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest) ... SKIPPED: Skipped as no physnet is available in config for placement based QoS allocation. | |
| 19:57:35 | efried | o | |
| 19:57:51 | efried | "in config" like the devstack config? | |
| 19:58:01 | mriedem | tempest config | |
| 19:58:14 | mriedem | https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/tempest/scenario/test_minbw_allocation_placement.py#L72 | |
| 19:58:55 | mriedem | https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/controller/logs/tempest_conf.txt.gz#81 | |
| 19:59:03 | mriedem | the tempest-slow job has that, the nova-next job does not | |
| 19:59:57 | mriedem | https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/.zuul.yaml#L290 | |
| 20:00:01 | mriedem | so we need that in the nova-next job | |
| 20:00:26 | efried | neat | |
| 20:00:59 | mriedem | report a bug and i can fix that up in a bit or push your own change | |
| 20:01:12 | efried | and probably the stuff above it too. | |
| 20:01:25 | mriedem | yeah i suppose | |
| 20:01:37 | mriedem | https://github.com/openstack/tempest/commit/c87a06b3c29427dc8f2513047c804e0410b4b99c | |
| 20:01:41 | mriedem | whatever was added in there | |
| 20:02:04 | mriedem | actually you're in luck, | |
| 20:02:11 | eandersson | I created a bug, will add more info if/when I find it. https://bugs.launchpad.net/nova/+bug/1854992 | |
| 20:02:11 | openstack | Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,New] | |
| 20:02:20 | mriedem | the nova-next job already sets that shit up b/c the post-test hook runs heal_allocations on a instance with a port that has bw | |
| 20:02:24 | mriedem | you just need tempest.conf updated | |
| 20:04:55 | eandersson | My best guess at the moment is RabbitMQ related issues (e.g. we have hit bugs in RabbitMQ where bindings exists, but are broken) | |
| 20:07:12 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063 | |
| 20:07:25 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1854993 | |
| 20:07:25 | openstack | Launchpad bug 1854993 in OpenStack Compute (nova) "QoS bandwidth tempest test no longer running" [Undecided,New] | |
| 20:09:24 | efried | fixing... | |
| 20:09:54 | openstackgerrit | Merged openstack/nova master: Add a way to exit early from a wait_for_instance_event() https://review.opendev.org/695985 | |
| 20:10:00 | artom | dansmith, answering my own earlier question, looks like we actually save the request_spec in the conductor only if the resize/migration succeeded | |
| 20:10:00 | openstackgerrit | Merged openstack/nova master: docs: Change order of PCI configuration steps https://review.opendev.org/694521 | |
| 20:10:07 | openstackgerrit | Merged openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522 | |
| 20:10:14 | openstackgerrit | Merged openstack/nova master: Suppress policy deprecated warnings in tests https://review.opendev.org/676670 | |
| 20:10:30 | artom | Except... that doens't work, because the conductor then casts to the computes to do the work | |
| 20:10:44 | artom | So if something fails in prep_resize or resize_instance, we'll never know | |
| 20:11:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180 | |
| 20:11:33 | efried | mriedem: ^ | |
| 20:11:36 | efried | gibi: ^ | |
| 20:15:16 | mriedem | efried: hammered you | |
| 20:16:48 | mriedem | https://youtu.be/otCpCn0l4Wo?t=110 | |
| 20:16:50 | mriedem | break it down! | |
| 20:17:19 | dansmith | artom: right, that's why I said if it happens too late | |
| 20:19:27 | openstackgerrit | Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180 | |
| 20:19:33 | artom | dansmith, right, just caught up with you | |
| 20:20:19 | artom | Mind you | |
| 20:22:20 | mriedem | doing a thing in the api, failing in compute, and being out of whack isn't a new problem | |
| 20:22:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418 | |
| 20:22:31 | openstackgerrit | Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991 | |
| 20:22:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946 | |
| 20:22:32 | mriedem | s/api/controller/ | |
| 20:22:32 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992 | |
| 20:22:48 | artom | mriedem, so do we have a "best practice" kind of thing to handle it? | |
| 20:22:56 | artom | Or it's all case by case? | |
| 20:23:18 | mriedem | you're worried that we change the request spec to use the new flavor in the api/conductor, cast to compute to do the resize, it fails and the instance is using the old flavor and the request spec is using the new flavor, right? | |
| 20:23:31 | mriedem | and then when cold migrating that server, the request spec is incorrectly using the new flavor that the instance isn't actually using | |