| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 19:53:05 | mriedem | tempest_black_regex: ^tempest.scenario.test_network | |
| 19:53:20 | mriedem | but that shouldn't hit on tempest/scenario/test_minbw_allocation_placement | |
| 19:54:47 | mriedem | it does get run in tempest-slow in train though yeah https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/job-output.txt#73583 | |
| 19:55:13 | eandersson | I am wondering if we are hitting some race conndition | |
| 19:55:33 | eandersson | Because we create 2 VMs ever 10 minutes and never get VMs stuck in BUILDING. | |
| 19:55:42 | eandersson | but our customers hit this very often with aggressive terraform deployments | |
| 19:55:54 | mriedem | efried: so we went from this in tempest-slow: | |
| 19:55:54 | mriedem | slow-serial run-test: commands[1] | tempest run --serial --regex '\[.*\bslow\b.*\]' --concurrency=2 --black-regex= | |
| 19:55:57 | mriedem | to this in nova-next: | |
| 19:56:02 | mriedem | all run-test: commands[1] | tempest run --regex '^tempest\.(scenario|api\.compute)' --concurrency=4 '--black-regex=^tempest.scenario.test_network' | |
| 19:56:22 | mriedem | i'm not sure why test_minbw_allocation_placement would be filtered out | |
| 19:56:51 | mriedem | oh i see | |
| 19:56:51 | mriedem | https://zuul.opendev.org/t/openstack/build/66f29bf5f12449059e82d24db5aff47a/log/job-output.txt#79328 | |
| 19:57:06 | mriedem | {2} setUpClass (tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest) ... SKIPPED: Skipped as no physnet is available in config for placement based QoS allocation. | |
| 19:57:35 | efried | o | |
| 19:57:51 | efried | "in config" like the devstack config? | |
| 19:58:01 | mriedem | tempest config | |
| 19:58:14 | mriedem | https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/tempest/scenario/test_minbw_allocation_placement.py#L72 | |
| 19:58:55 | mriedem | https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/controller/logs/tempest_conf.txt.gz#81 | |
| 19:59:03 | mriedem | the tempest-slow job has that, the nova-next job does not | |
| 19:59:57 | mriedem | https://github.com/openstack/tempest/blob/3eb3c29e979fd3f13c205d62119748952d63054a/.zuul.yaml#L290 | |
| 20:00:01 | mriedem | so we need that in the nova-next job | |
| 20:00:26 | efried | neat | |
| 20:00:59 | mriedem | report a bug and i can fix that up in a bit or push your own change | |
| 20:01:12 | efried | and probably the stuff above it too. | |
| 20:01:25 | mriedem | yeah i suppose | |
| 20:01:37 | mriedem | https://github.com/openstack/tempest/commit/c87a06b3c29427dc8f2513047c804e0410b4b99c | |
| 20:01:41 | mriedem | whatever was added in there | |
| 20:02:04 | mriedem | actually you're in luck, | |
| 20:02:11 | eandersson | I created a bug, will add more info if/when I find it. https://bugs.launchpad.net/nova/+bug/1854992 | |
| 20:02:11 | openstack | Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,New] | |
| 20:02:20 | mriedem | the nova-next job already sets that shit up b/c the post-test hook runs heal_allocations on a instance with a port that has bw | |
| 20:02:24 | mriedem | you just need tempest.conf updated | |
| 20:04:55 | eandersson | My best guess at the moment is RabbitMQ related issues (e.g. we have hit bugs in RabbitMQ where bindings exists, but are broken) | |
| 20:07:12 | openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Boot from volume instance rescue https://review.opendev.org/694063 | |
| 20:07:25 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1854993 | |
| 20:07:25 | openstack | Launchpad bug 1854993 in OpenStack Compute (nova) "QoS bandwidth tempest test no longer running" [Undecided,New] | |
| 20:09:24 | efried | fixing... | |
| 20:09:54 | openstackgerrit | Merged openstack/nova master: Add a way to exit early from a wait_for_instance_event() https://review.opendev.org/695985 | |
| 20:10:00 | artom | dansmith, answering my own earlier question, looks like we actually save the request_spec in the conductor only if the resize/migration succeeded | |
| 20:10:00 | openstackgerrit | Merged openstack/nova master: docs: Change order of PCI configuration steps https://review.opendev.org/694521 | |
| 20:10:07 | openstackgerrit | Merged openstack/nova master: docs: Clarify configuration steps for PF devices https://review.opendev.org/694522 | |
| 20:10:14 | openstackgerrit | Merged openstack/nova master: Suppress policy deprecated warnings in tests https://review.opendev.org/676670 | |
| 20:10:30 | artom | Except... that doens't work, because the conductor then casts to the computes to do the work | |
| 20:10:44 | artom | So if something fails in prep_resize or resize_instance, we'll never know | |
| 20:11:29 | openstackgerrit | Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180 | |
| 20:11:33 | efried | mriedem: ^ | |
| 20:11:36 | efried | gibi: ^ | |
| 20:15:16 | mriedem | efried: hammered you | |
| 20:16:48 | mriedem | https://youtu.be/otCpCn0l4Wo?t=110 | |
| 20:16:50 | mriedem | break it down! | |
| 20:17:19 | dansmith | artom: right, that's why I said if it happens too late | |
| 20:19:27 | openstackgerrit | Eric Fried proposed openstack/nova master: Add QoS tempest config so bw tests run https://review.opendev.org/697180 | |
| 20:19:33 | artom | dansmith, right, just caught up with you | |
| 20:20:19 | artom | Mind you | |
| 20:22:20 | mriedem | doing a thing in the api, failing in compute, and being out of whack isn't a new problem | |
| 20:22:31 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418 | |
| 20:22:31 | openstackgerrit | Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991 | |
| 20:22:32 | openstackgerrit | Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946 | |
| 20:22:32 | mriedem | s/api/controller/ | |
| 20:22:32 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992 | |
| 20:22:48 | artom | mriedem, so do we have a "best practice" kind of thing to handle it? | |
| 20:22:56 | artom | Or it's all case by case? | |
| 20:23:18 | mriedem | you're worried that we change the request spec to use the new flavor in the api/conductor, cast to compute to do the resize, it fails and the instance is using the old flavor and the request spec is using the new flavor, right? | |
| 20:23:31 | mriedem | and then when cold migrating that server, the request spec is incorrectly using the new flavor that the instance isn't actually using | |
| 20:24:24 | artom | mriedem, yes to both | |
| 20:24:47 | artom | And really it's gibi that's worried - he brought it up on the review | |
| 20:25:16 | mriedem | there isn't really a best practice for that. the compute can't/shouldn't get an update the request spec. there is no periodic in the controller services that is healing the request spec for failed resizes. | |
| 20:25:45 | mriedem | note that in revert_resize in the API, we also update the request spec to match the source host and then cast off to do the things in compute | |
| 20:25:51 | mriedem | if those things fail, the request spec could be out of sync again | |
| 20:26:13 | mriedem | as time has gone on we've persisted less and less of the original crap we used to store in the request spec | |
| 20:26:22 | mriedem | so it was just a blob to pass things from the api to the scheduler | |
| 20:26:39 | mriedem | the flavor is tricky though since it is useful for things like down-cell api responses where we don't have the instance record | |
| 20:27:17 | mriedem | tl;dr is there a best practice for avoiding split brain? heal things periodically? use etcd? :) | |
| 20:27:21 | artom | (so what's in the api_db for instances? just a cell mapping?) | |
| 20:27:31 | mriedem | cell/instance mappings and request spec | |
| 20:27:36 | dansmith | mriedem: earlier I said it's probably legit to refresh the reqspec from the instance's flavor before we go into a scheduling operation for an existing instance | |
| 20:27:39 | mriedem | server group members | |
| 20:28:00 | dansmith | that would solve the problem I think | |
| 20:28:15 | dansmith | leaving only cell down showing some outdated instance info guesses, which wouldn't be the end of the world | |
| 20:28:16 | artom | dansmith, that's kinda where I'm leaning - if we take the definition of request spec to be what mriedem said "pass scheduling-related things from API to scheduler" | |
| 20:28:28 | dansmith | artom: yup | |
| 20:28:53 | artom | Then yeah, "load it up from latest instance makes sense' | |
| 20:29:20 | dansmith | there's another much more edge case where that makes sense too, | |
| 20:29:24 | artom | Although that makes one wonder "what's the point of request_spec in the first place, if everything it in instance anyways" | |
| 20:29:38 | dansmith | which is you restore either database from a non-coordinated backup after a disaster or failed upgrade | |
| 20:29:57 | dansmith | artom: you mean reqspec.flavor? | |
| 20:30:09 | dansmith | artom: before an instance exists, that's where we hold the flavor, | |
| 20:30:11 | artom | dansmith, yeah, and request_spec.numa_topology | |
| 20:30:11 | mriedem | artom: it's not all instance | |
| 20:30:21 | mriedem | scheduler hints, forced hosts/nodes, etc | |
| 20:30:24 | dansmith | artom: and if the cell is down, then we use the stuff in reqspec as a cache | |
| 20:30:26 | mriedem | requested destinations | |
| 20:30:45 | artom | I see | |
| 20:30:46 | dansmith | mriedem: right, hence why I asked, I assume he meant the duplicative parts of reqspec, which is just flavor, AFAIK | |
| 20:30:54 | artom | dansmith, numa_topology | |
| 20:31:01 | mriedem | it's an objectified and persisted version of the old filter_properties stuff, and some of it getting persisted always has caused a lot of problems | |
| 20:31:13 | dansmith | artom: I thought the instance numa topo ended up getting fleshed out by the virt driver, | |
| 20:31:26 | dansmith | artom: where reqspec's copy was kinda the planned topo, which would still be valid to keep I tink | |
| 20:31:37 | artom | dansmith, so there's 2 things there: 1 is the instance numa topology, which can be got from flavor and image meta | |