Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-12-03
19:45:07 efried mriedem: see gibi's comment at the bottom of https://review.opendev.org/#/c/696992/ -- is it possible the bw test was removed when you refactored to get rid of tempest-slow?
19:46:12 eandersson I think this is a new stuckin BUILD bug because I can't even find the instance in placement
19:48:58 mriedem efried: it should be run in the nova-next job
19:49:14 efried okay, will look for it there
19:49:40 mriedem i don't see it, checking something
19:49:42 artom I guess there's less impact when it's in a new release
19:49:55 mriedem artom: well we don't backport features upstream for a reason so sure
19:50:01 artom But in an existing release we definitely can't change the defaults
19:50:29 mriedem hey, how you want to break your enterprise users on queens is up to you :)
19:50:45 artom In a way that we can fix for loads of $$
19:50:50 mriedem job security
19:50:58 artom 'zactly
19:52:51 openstackgerrit Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418
19:52:51 openstackgerrit Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991
19:52:52 openstackgerrit Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946
19:52:52 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992
19:52:55 mriedem efried: so nova-next should run tempest api compute and scenario tests:
19:52:55 mriedem tempest_test_regex: ^tempest\.(scenario|api\.compute)
19:53:00 mriedem except these scenario tests:
19:53:05 mriedem tempest_black_regex: ^tempest.scenario.test_network
19:53:20 mriedem but that shouldn't hit on tempest/scenario/test_minbw_allocation_placement
19:54:47 mriedem it does get run in tempest-slow in train though yeah https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/job-output.txt#73583
19:55:13 eandersson I am wondering if we are hitting some race conndition
19:55:33 eandersson Because we create 2 VMs ever 10 minutes and never get VMs stuck in BUILDING.
19:55:42 eandersson but our customers hit this very often with aggressive terraform deployments
19:55:54 mriedem efried: so we went from this in tempest-slow:
19:55:54 mriedem slow-serial run-test: commands[1] | tempest run --serial --regex '\[.*\bslow\b.*\]' --concurrency=2 --black-regex=
19:55:57 mriedem to this in nova-next:
19:56:02 mriedem all run-test: commands[1] | tempest run --regex '^tempest\.(scenario|api\.compute)' --concurrency=4 '--black-regex=^tempest.scenario.test_network'
19:56:22 mriedem i'm not sure why test_minbw_allocation_placement would be filtered out
19:56:51 mriedem oh i see
19:56:51 mriedem https://zuul.opendev.org/t/openstack/build/66f29bf5f12449059e82d24db5aff47a/log/job-output.txt#79328
19:57:06 mriedem {2} setUpClass (tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest) ... SKIPPED: Skipped as no physnet is available in config for placement based QoS allocation.
19:57:35 efried o
19:57:51 efried "in config" like the devstack config?
19:58:01 mriedem tempest config
19:58:14 mriedem https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/tempest/scenario/test_minbw_allocation_placement.py#L72
19:58:55 mriedem https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/controller/logs/tempest_conf.txt.gz#81
19:59:03 mriedem the tempest-slow job has that, the nova-next job does not
19:59:57 mriedem https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/.zuul.yaml#L290
20:00:01 mriedem so we need that in the nova-next job
20:00:26 efried neat
20:00:59 mriedem report a bug and i can fix that up in a bit or push your own change
20:01:12 efried and probably the stuff above it too.
20:01:25 mriedem yeah i suppose
20:01:37 mriedem https://github.com/openstack/tempest/commit/c87a06b3c29427dc8f2513047c804e0410b4b99c
20:01:41 mriedem whatever was added in there
20:02:04 mriedem actually you're in luck,
20:02:11 eandersson I created a bug, will add more info if/when I find it. https://bugs.launchpad.net/nova/+bug/1854992
20:02:11 openstack Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,New]
20:02:20 mriedem the nova-next job already sets that shit up b/c the post-test hook runs heal_allocations on a instance with a port that has bw
20:02:24 mriedem you just need tempest.conf updated
20:04:55 eandersson My best guess at the moment is RabbitMQ related issues (e.g. we have hit bugs in RabbitMQ where bindings exists, but are broken)
20:07:12 openstackgerrit Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063
20:07:25 efried mriedem: https://bugs.launchpad.net/nova/+bug/1854993
20:07:25 openstack Launchpad bug 1854993 in OpenStack Compute (nova) "QoS bandwidth tempest test no longer running" [Undecided,New]
20:09:24 efried fixing...
20:09:54 openstackgerrit Merged openstack/nova master: Add a way to exit early from a wait_for_instance_event() https://review.opendev.org/695985
20:10:00 artom dansmith, answering my own earlier question, looks like we actually save the request_spec in the conductor only if the resize/migration succeeded
20:10:00 openstackgerrit Merged openstack/nova master: docs: Change order of PCI configuration steps https://review.opendev.org/694521
20:10:07 openstackgerrit Merged openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522
20:10:14 openstackgerrit Merged openstack/nova master: Suppress policy deprecated warnings in tests https://review.opendev.org/676670
20:10:30 artom Except... that doens't work, because the conductor then casts to the computes to do the work
20:10:44 artom So if something fails in prep_resize or resize_instance, we'll never know
20:11:29 openstackgerrit Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180
20:11:33 efried mriedem: ^
20:11:36 efried gibi: ^
20:15:16 mriedem efried: hammered you
20:16:48 mriedem https://youtu.be/otCpCn0l4Wo?t=110
20:16:50 mriedem break it down!
20:17:19 dansmith artom: right, that's why I said if it happens too late
20:19:27 openstackgerrit Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180
20:19:33 artom dansmith, right, just caught up with you
20:20:19 artom Mind you
20:22:20 mriedem doing a thing in the api, failing in compute, and being out of whack isn't a new problem
20:22:31 openstackgerrit Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418
20:22:31 openstackgerrit Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991
20:22:32 openstackgerrit Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946
20:22:32 mriedem s/api/controller/
20:22:32 openstackgerrit Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992
20:22:48 artom mriedem, so do we have a "best practice" kind of thing to handle it?
20:22:56 artom Or it's all case by case?
20:23:18 mriedem you're worried that we change the request spec to use the new flavor in the api/conductor, cast to compute to do the resize, it fails and the instance is using the old flavor and the request spec is using the new flavor, right?
20:23:31 mriedem and then when cold migrating that server, the request spec is incorrectly using the new flavor that the instance isn't actually using
20:24:24 artom mriedem, yes to both
20:24:47 artom And really it's gibi that's worried - he brought it up on the review
20:25:16 mriedem there isn't really a best practice for that. the compute can't/shouldn't get an update the request spec. there is no periodic in the controller services that is healing the request spec for failed resizes.
20:25:45 mriedem note that in revert_resize in the API, we also update the request spec to match the source host and then cast off to do the things in compute
20:25:51 mriedem if those things fail, the request spec could be out of sync again
20:26:13 mriedem as time has gone on we've persisted less and less of the original crap we used to store in the request spec
20:26:22 mriedem so it was just a blob to pass things from the api to the scheduler
20:26:39 mriedem the flavor is tricky though since it is useful for things like down-cell api responses where we don't have the instance record
20:27:17 mriedem tl;dr is there a best practice for avoiding split brain? heal things periodically? use etcd? :)
20:27:21 artom (so what's in the api_db for instances? just a cell mapping?)
20:27:31 mriedem cell/instance mappings and request spec
20:27:36 dansmith mriedem: earlier I said it's probably legit to refresh the reqspec from the instance's flavor before we go into a scheduling operation for an existing instance
20:27:39 mriedem server group members
20:28:00 dansmith that would solve the problem I think
20:28:15 dansmith leaving only cell down showing some outdated instance info guesses, which wouldn't be the end of the world
20:28:16 artom dansmith, that's kinda where I'm leaning - if we take the definition of request spec to be what mriedem said "pass scheduling-related things from API to scheduler"

Earlier   Later