| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 19:32:35 | mriedem | makes my little wip hackaroo i was going to do this afternoon not so simple, but anyway | |
| 19:32:36 | mriedem | thanks | |
| 19:33:09 | artom | dansmith, I think so | |
| 19:33:11 | dustinc | https://docs.openstack.org/openstacksdk/latest/user/microversions.html | |
| 19:33:11 | mordred | mriedem: one day we'll get to the point where people will add support for a thing to sdk when they add it to a service and life will be magical | |
| 19:33:19 | artom | (Sorry, got ADHD'ed away to another thing) | |
| 19:33:21 | mordred | we are not at that day today | |
| 19:33:37 | dansmith | artom: might be valid to refresh the reqspec from the instance before we go into an operation like that | |
| 19:34:08 | mriedem | mordred: and osc et al :) | |
| 19:34:45 | mriedem | efried: note the ..note:: rendering at the bottom of https://docs.openstack.org/openstacksdk/latest/user/microversions.html - tickles your rst ocd?! | |
| 19:35:12 | efried | aaaaaagh! | |
| 19:36:35 | artom | Hey, so I'm backporting https://review.opendev.org/#/c/619953/10/nova/conf/libvirt.py@719 internally | |
| 19:36:42 | artom | And wondering about changing the default to 'unique' | |
| 19:36:44 | efried | mordred: mriedem: https://review.opendev.org/697168 | |
| 19:36:52 | efried | ...and, you suck. | |
| 19:37:03 | artom | Can't imagine Windows would appreciate its sysinfo_serial changing under it | |
| 19:37:52 | artom | Am I just being paranoid about this? mriedem, dansmith, ^^ you guys are the approvers do you recall talking about this at all? | |
| 19:38:21 | mordred | efried: +A | |
| 19:40:14 | eandersson | mriedem I think there are at least a handful of bugs causing it for us | |
| 19:40:23 | eandersson | difficult to get a handle on the causes | |
| 19:41:26 | mriedem | artom: backporting features huh | |
| 19:41:39 | artom | mriedem, only backportable ones | |
| 19:41:46 | mriedem | are you asking about the serial changing on an existing guest? | |
| 19:41:50 | artom | mriedem, yeah | |
| 19:42:02 | mriedem | so change the default to 'auto' like it was? | |
| 19:42:29 | artom | mriedem, well, not sure it makes sense to go back at this point | |
| 19:42:43 | mriedem | there was quite a bit of discussion around that blueprint, including being able to override the host config with image meta or flavors but that was dropped | |
| 19:42:50 | eandersson | btw launched_on is is always null | |
| 19:42:51 | artom | But... there's an impact, right? The default becomes 'unique', instance is hard-rebooted, boom, new serial | |
| 19:42:55 | artom | Linux is probably OK, but Windows... | |
| 19:43:46 | artom | So lyarwood just updated our proposed backport to not change the default | |
| 19:43:51 | artom | Do... do we need to do the same upstream? | |
| 19:43:57 | mriedem | there is an upgrade release note. i'd have to dig into all of the comments on that patch to determine if we changed the default later or not for good reason | |
| 19:44:10 | mriedem | that was released in stein | |
| 19:44:16 | mriedem | changing the default from 2 releases back would be weird | |
| 19:44:17 | artom | Right, so it's already out there | |
| 19:44:20 | artom | I know | |
| 19:45:07 | efried | mriedem: see gibi's comment at the bottom of https://review.opendev.org/#/c/696992/ -- is it possible the bw test was removed when you refactored to get rid of tempest-slow? | |
| 19:46:12 | eandersson | I think this is a new stuckin BUILD bug because I can't even find the instance in placement | |
| 19:48:58 | mriedem | efried: it should be run in the nova-next job | |
| 19:49:14 | efried | okay, will look for it there | |
| 19:49:40 | mriedem | i don't see it, checking something | |
| 19:49:42 | artom | I guess there's less impact when it's in a new release | |
| 19:49:55 | mriedem | artom: well we don't backport features upstream for a reason so sure | |
| 19:50:01 | artom | But in an existing release we definitely can't change the defaults | |
| 19:50:29 | mriedem | hey, how you want to break your enterprise users on queens is up to you :) | |
| 19:50:45 | artom | In a way that we can fix for loads of $$ | |
| 19:50:50 | mriedem | job security | |
| 19:50:58 | artom | 'zactly | |
| 19:52:51 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418 | |
| 19:52:51 | openstackgerrit | Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991 | |
| 19:52:52 | openstackgerrit | Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946 | |
| 19:52:52 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992 | |
| 19:52:55 | mriedem | efried: so nova-next should run tempest api compute and scenario tests: | |
| 19:52:55 | mriedem | tempest_test_regex: ^tempest\.(scenario|api\.compute) | |
| 19:53:00 | mriedem | except these scenario tests: | |
| 19:53:05 | mriedem | tempest_black_regex: ^tempest.scenario.test_network | |
| 19:53:20 | mriedem | but that shouldn't hit on tempest/scenario/test_minbw_allocation_placement | |
| 19:54:47 | mriedem | it does get run in tempest-slow in train though yeah https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/job-output.txt#73583 | |
| 19:55:13 | eandersson | I am wondering if we are hitting some race conndition | |
| 19:55:33 | eandersson | Because we create 2 VMs ever 10 minutes and never get VMs stuck in BUILDING. | |
| 19:55:42 | eandersson | but our customers hit this very often with aggressive terraform deployments | |
| 19:55:54 | mriedem | efried: so we went from this in tempest-slow: | |
| 19:55:54 | mriedem | slow-serial run-test: commands[1] | tempest run --serial --regex '\[.*\bslow\b.*\]' --concurrency=2 --black-regex= | |
| 19:55:57 | mriedem | to this in nova-next: | |
| 19:56:02 | mriedem | all run-test: commands[1] | tempest run --regex '^tempest\.(scenario|api\.compute)' --concurrency=4 '--black-regex=^tempest.scenario.test_network' | |
| 19:56:22 | mriedem | i'm not sure why test_minbw_allocation_placement would be filtered out | |
| 19:56:51 | mriedem | oh i see | |
| 19:56:51 | mriedem | https://zuul.opendev.org/t/openstack/build/66f29bf5f12449059e82d24db5aff47a/log/job-output.txt#79328 | |
| 19:57:06 | mriedem | {2} setUpClass (tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest) ... SKIPPED: Skipped as no physnet is available in config for placement based QoS allocation. | |
| 19:57:35 | efried | o | |
| 19:57:51 | efried | "in config" like the devstack config? | |
| 19:58:01 | mriedem | tempest config | |
| 19:58:14 | mriedem | https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/tempest/scenario/test_minbw_allocation_placement.py#L72 | |
| 19:58:55 | mriedem | https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/controller/logs/tempest_conf.txt.gz#81 | |
| 19:59:03 | mriedem | the tempest-slow job has that, the nova-next job does not | |
| 19:59:57 | mriedem | https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/.zuul.yaml#L290 | |
| 20:00:01 | mriedem | so we need that in the nova-next job | |
| 20:00:26 | efried | neat | |
| 20:00:59 | mriedem | report a bug and i can fix that up in a bit or push your own change | |
| 20:01:12 | efried | and probably the stuff above it too. | |
| 20:01:25 | mriedem | yeah i suppose | |
| 20:01:37 | mriedem | https://github.com/openstack/tempest/commit/c87a06b3c29427dc8f2513047c804e0410b4b99c | |
| 20:01:41 | mriedem | whatever was added in there | |
| 20:02:04 | mriedem | actually you're in luck, | |
| 20:02:11 | eandersson | I created a bug, will add more info if/when I find it. https://bugs.launchpad.net/nova/+bug/1854992 | |
| 20:02:11 | openstack | Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,New] | |
| 20:02:20 | mriedem | the nova-next job already sets that shit up b/c the post-test hook runs heal_allocations on a instance with a port that has bw | |
| 20:02:24 | mriedem | you just need tempest.conf updated | |
| 20:04:55 | eandersson | My best guess at the moment is RabbitMQ related issues (e.g. we have hit bugs in RabbitMQ where bindings exists, but are broken) | |
| 20:07:12 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063 | |
| 20:07:25 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1854993 | |
| 20:07:25 | openstack | Launchpad bug 1854993 in OpenStack Compute (nova) "QoS bandwidth tempest test no longer running" [Undecided,New] | |
| 20:09:24 | efried | fixing... | |
| 20:09:54 | openstackgerrit | Merged openstack/nova master: Add a way to exit early from a wait_for_instance_event() https://review.opendev.org/695985 | |
| 20:10:00 | artom | dansmith, answering my own earlier question, looks like we actually save the request_spec in the conductor only if the resize/migration succeeded | |
| 20:10:00 | openstackgerrit | Merged openstack/nova master: docs: Change order of PCI configuration steps https://review.opendev.org/694521 | |
| 20:10:07 | openstackgerrit | Merged openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522 | |
| 20:10:14 | openstackgerrit | Merged openstack/nova master: Suppress policy deprecated warnings in tests https://review.opendev.org/676670 | |
| 20:10:30 | artom | Except... that doens't work, because the conductor then casts to the computes to do the work | |