| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-12-03 | |||
| 19:07:12 | dansmith | artom: compute can't reset the reqspec, so if it happens late enough, that won't happen | |
| 19:07:53 | dansmith | er, s/happens/fails/ | |
| 19:08:55 | dansmith | I think the only downside to that is if we do something that looks at the reqspec and not the instance, not sure what that would be | |
| 19:17:55 | artom | Sorry, was on a call | |
| 19:18:54 | artom | Context is https://review.opendev.org/#/c/662522/13 | |
| 19:19:56 | artom | So I guess scheduling looks at request spec | |
| 19:21:00 | eandersson | btw major annoyance at the moment is when instances get stuck in building | |
| 19:21:11 | eandersson | often you don't know which compute they are stuck buidling against | |
| 19:24:26 | mriedem | check the launched_on field on the instance - that wasn't cleaned up until recently | |
| 19:24:39 | mriedem | if instances are getting stuck in BUILD status somewhere when we failed, report a bug | |
| 19:25:41 | mriedem | like you did with bug 1837955 | |
| 19:25:41 | openstack | bug 1837955 in OpenStack Compute (nova) stein "MaxRetriesExceeded sometime fails with messaging exception" [Medium,Fix committed] https://launchpad.net/bugs/1837955 - Assigned to Matt Riedemann (mriedem) | |
| 19:26:51 | mriedem | efried: dustinc: what baremetal api version is used when we get a node's details now? https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L226 | |
| 19:27:18 | mriedem | since we're not using the client_wrapper there i can't tell if it's 1.46 https://github.com/openstack/nova/blob/master/nova/virt/ironic/client_wrapper.py#L35 or something else | |
| 19:27:32 | efried | negotiated by sdk, no? | |
| 19:27:39 | dansmith | artom: so you mean a failed resize followed by a migration would use the new flavor as part of scheduling yeah? | |
| 19:27:41 | mriedem | based on the fields requested? | |
| 19:28:00 | efried | no, based on what's available | |
| 19:28:24 | efried | I think sdk does version discovery and uses the latest they mutually understand. mordred? | |
| 19:28:48 | mordred | uh | |
| 19:28:50 | mordred | reading | |
| 19:28:51 | dustinc | I don't know, I never had to deal with it so I _assume_ it is the most recent common version | |
| 19:29:26 | mriedem | is there a way to tell if the sdk will use 1.50+ to get the node.owner field? | |
| 19:29:52 | mriedem | ah yes | |
| 19:29:56 | mriedem | _max_microversion = '1.52' | |
| 19:29:57 | mordred | yes - that's right. it does discovery and gets the latest that sdk and the remote both understand | |
| 19:30:15 | mordred | so yeah - shuold be 1.50+ - assuming the ironic supports that | |
| 19:30:19 | mriedem | ah but the sdk node object doesn't have an explicit owner property so i'd have to add that | |
| 19:30:49 | mordred | yah. dtantsur|afk ^^ | |
| 19:30:51 | efried | mriedem: right -- the sdk is kind of supposed to abstract away "microversion" and instead let you think about "feature" -- in this case node.owner. | |
| 19:31:25 | mriedem | sure. i didn't know it hard-coded all of the properties though. | |
| 19:31:31 | efried | so if you want $feature at $microversion, sdk has to be plumbed to a) use $microversion, and b) expose $feature. Then nova has to require the release of sdk that does that, etc. | |
| 19:31:39 | efried | oh. Yeah :) | |
| 19:32:35 | mriedem | makes my little wip hackaroo i was going to do this afternoon not so simple, but anyway | |
| 19:32:36 | mriedem | thanks | |
| 19:33:09 | artom | dansmith, I think so | |
| 19:33:11 | dustinc | https://docs.openstack.org/openstacksdk/latest/user/microversions.html | |
| 19:33:11 | mordred | mriedem: one day we'll get to the point where people will add support for a thing to sdk when they add it to a service and life will be magical | |
| 19:33:19 | artom | (Sorry, got ADHD'ed away to another thing) | |
| 19:33:21 | mordred | we are not at that day today | |
| 19:33:37 | dansmith | artom: might be valid to refresh the reqspec from the instance before we go into an operation like that | |
| 19:34:08 | mriedem | mordred: and osc et al :) | |
| 19:34:45 | mriedem | efried: note the ..note:: rendering at the bottom of https://docs.openstack.org/openstacksdk/latest/user/microversions.html - tickles your rst ocd?! | |
| 19:35:12 | efried | aaaaaagh! | |
| 19:36:35 | artom | Hey, so I'm backporting https://review.opendev.org/#/c/619953/10/nova/conf/libvirt.py@719 internally | |
| 19:36:42 | artom | And wondering about changing the default to 'unique' | |
| 19:36:44 | efried | mordred: mriedem: https://review.opendev.org/697168 | |
| 19:36:52 | efried | ...and, you suck. | |
| 19:37:03 | artom | Can't imagine Windows would appreciate its sysinfo_serial changing under it | |
| 19:37:52 | artom | Am I just being paranoid about this? mriedem, dansmith, ^^ you guys are the approvers do you recall talking about this at all? | |
| 19:38:21 | mordred | efried: +A | |
| 19:40:14 | eandersson | mriedem I think there are at least a handful of bugs causing it for us | |
| 19:40:23 | eandersson | difficult to get a handle on the causes | |
| 19:41:26 | mriedem | artom: backporting features huh | |
| 19:41:39 | artom | mriedem, only backportable ones | |
| 19:41:46 | mriedem | are you asking about the serial changing on an existing guest? | |
| 19:41:50 | artom | mriedem, yeah | |
| 19:42:02 | mriedem | so change the default to 'auto' like it was? | |
| 19:42:29 | artom | mriedem, well, not sure it makes sense to go back at this point | |
| 19:42:43 | mriedem | there was quite a bit of discussion around that blueprint, including being able to override the host config with image meta or flavors but that was dropped | |
| 19:42:50 | eandersson | btw launched_on is is always null | |
| 19:42:51 | artom | But... there's an impact, right? The default becomes 'unique', instance is hard-rebooted, boom, new serial | |
| 19:42:55 | artom | Linux is probably OK, but Windows... | |
| 19:43:46 | artom | So lyarwood just updated our proposed backport to not change the default | |
| 19:43:51 | artom | Do... do we need to do the same upstream? | |
| 19:43:57 | mriedem | there is an upgrade release note. i'd have to dig into all of the comments on that patch to determine if we changed the default later or not for good reason | |
| 19:44:10 | mriedem | that was released in stein | |
| 19:44:16 | mriedem | changing the default from 2 releases back would be weird | |
| 19:44:17 | artom | Right, so it's already out there | |
| 19:44:20 | artom | I know | |
| 19:45:07 | efried | mriedem: see gibi's comment at the bottom of https://review.opendev.org/#/c/696992/ -- is it possible the bw test was removed when you refactored to get rid of tempest-slow? | |
| 19:46:12 | eandersson | I think this is a new stuckin BUILD bug because I can't even find the instance in placement | |
| 19:48:58 | mriedem | efried: it should be run in the nova-next job | |
| 19:49:14 | efried | okay, will look for it there | |
| 19:49:40 | mriedem | i don't see it, checking something | |
| 19:49:42 | artom | I guess there's less impact when it's in a new release | |
| 19:49:55 | mriedem | artom: well we don't backport features upstream for a reason so sure | |
| 19:50:01 | artom | But in an existing release we definitely can't change the defaults | |
| 19:50:29 | mriedem | hey, how you want to break your enterprise users on queens is up to you :) | |
| 19:50:45 | artom | In a way that we can fix for loads of $$ | |
| 19:50:50 | mriedem | job security | |
| 19:50:58 | artom | 'zactly | |
| 19:52:51 | openstackgerrit | Eric Fried proposed openstack/nova master: Use Placement 1.34 (string suffixes & mappings) https://review.opendev.org/696418 | |
| 19:52:51 | openstackgerrit | Eric Fried proposed openstack/nova master: refactor: RequestGroup.is_empty() and .strip_zeros() https://review.opendev.org/696991 | |
| 19:52:52 | openstackgerrit | Eric Fried proposed openstack/nova master: Tie requester_id to RequestGroup suffix https://review.opendev.org/696946 | |
| 19:52:52 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Use provider mappings from Placement (mostly) https://review.opendev.org/696992 | |
| 19:52:55 | mriedem | efried: so nova-next should run tempest api compute and scenario tests: | |
| 19:52:55 | mriedem | tempest_test_regex: ^tempest\.(scenario|api\.compute) | |
| 19:53:00 | mriedem | except these scenario tests: | |
| 19:53:05 | mriedem | tempest_black_regex: ^tempest.scenario.test_network | |
| 19:53:20 | mriedem | but that shouldn't hit on tempest/scenario/test_minbw_allocation_placement | |
| 19:54:47 | mriedem | it does get run in tempest-slow in train though yeah https://zuul.opendev.org/t/openstack/build/ddb1fb60455d4f7681a9a377aaef63ab/log/job-output.txt#73583 | |
| 19:55:13 | eandersson | I am wondering if we are hitting some race conndition | |
| 19:55:33 | eandersson | Because we create 2 VMs ever 10 minutes and never get VMs stuck in BUILDING. | |
| 19:55:42 | eandersson | but our customers hit this very often with aggressive terraform deployments | |
| 19:55:54 | mriedem | efried: so we went from this in tempest-slow: | |
| 19:55:54 | mriedem | slow-serial run-test: commands[1] | tempest run --serial --regex '\[.*\bslow\b.*\]' --concurrency=2 --black-regex= | |
| 19:55:57 | mriedem | to this in nova-next: | |
| 19:56:02 | mriedem | all run-test: commands[1] | tempest run --regex '^tempest\.(scenario|api\.compute)' --concurrency=4 '--black-regex=^tempest.scenario.test_network' | |
| 19:56:22 | mriedem | i'm not sure why test_minbw_allocation_placement would be filtered out | |