Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-01
15:19:02 sean-k-mooney so random quest. would peole object to an api to list the currently enabled schduler filters? specifically to enabel tempest and other multicloud services to detect what schuler featres they can expect
15:19:12 dansmith the strongest argument I've seen for it is that AWS has it and thus the standalone EC2 thing needs to be able to proxy that in
15:19:16 sean-k-mooney *question however it could become a quest
15:19:18 dansmith but afaik, that's pretty much dead these days
15:19:30 dansmith sean-k-mooney: yes I would object
15:19:40 mriedem "because AWS and Alibaba have it" is something i hear every week
15:20:05 sean-k-mooney dansmith: because we are exposing configuration via the api or somethign else
15:20:51 dansmith sean-k-mooney: it would literally be an api call that would make an rpc call to scheduler to return a chunk of config, which shouldn't be visible externally anyway. and if you're running multiple schedulers, which do you cal?
15:20:52 mriedem the only reason i could see for doing something like that (scheduler filters and such) is to tell users, via the api, which hints are available
15:21:01 johnthetubaguy sean-k-mooney: discovery of available scheduler hints was something we once said we would consider, which is a bit different
15:21:08 dansmith yep
15:21:16 mriedem right, it would only be feasible if it was a list of hints
15:21:24 mriedem which is totally pluggable btw
15:21:39 sean-k-mooney johnthetubaguy: its related to this tempest change https://review.openstack.org/#/c/570207/12
15:21:43 johnthetubaguy yeah, that was the downside, in the general case, it means nothing useful
15:22:02 sean-k-mooney johnthetubaguy: the issue i have with the change is it require use to keep the nova and tempest default in sync
15:22:03 dansmith sean-k-mooney: tempest has always been blackbox, requiring you to tell it the nova side scheduler config for this reason
15:22:16 artom sean-k-mooney, don't you dare bring more people into this. I will fly to Ireland and cut you, I swear.
15:22:28 artom We already can't agree downstream
15:22:33 dansmith tempest is a testing/validation tool.. keeping the two configs in sync is a few lines of bash
15:22:36 mriedem right, devstack configures the filters in both nova and tempest
15:22:40 dansmith right
15:22:51 sean-k-mooney dansmith: the issue is making triplo do that
15:23:06 mriedem devstack also adds the same/different host filtesr which aren't in the default enabled_filters list for nova
15:23:09 dansmith sean-k-mooney: s/bash/puppet/
15:23:28 mriedem for any nfv ci, they'd also need to configure to numa/pci filters
15:23:28 mriedem etc
15:23:30 sean-k-mooney dansmith: ya i know its just triplo is a pain to make work instead of devstack
15:23:44 dansmith sean-k-mooney: adding an api to nova to work around tripleo not being able to communicate config to another module is INSANITY
15:23:51 sean-k-mooney mriedem: yes today they only need to enable it in nova however
15:24:18 johnthetubaguy I think sdague convinced me about this in the past, you don't want auto discovery, you want to tell the test system what you expect to happen, else there be dragons
15:24:32 artom dansmith, it's not communicate per se - if tripleo doens't set the nova value, it shouldn't have to set the corresponding tempest value
15:25:06 dansmith artom: find another way
15:25:12 dansmith seriously.
15:25:22 johnthetubaguy matching defaults?
15:25:23 artom dansmith, my other way is https://review.openstack.org/#/c/570207/12
15:25:23 sean-k-mooney johnthetubaguy: ya well it was jut a taught my main issue with artom change is that we would have to keep it in sync if we add filter in the futre to the default set
15:25:28 artom johnthetubaguy, that's what ^^ is
15:25:39 artom But apparently everyone is literally willing to fight to the death over this.
15:25:44 dansmith I am
15:25:50 dansmith let's do it.
15:26:14 artom I only have this bluetooth mouse :(
15:26:22 dansmith forfeit?
15:26:34 sean-k-mooney johnthetubaguy: the interop benift is really only a side effect and i dont feel that strongly that its a good thing
15:26:40 artom Pfft, as if. I'm making brass knuckles. Wireless ones.
15:29:46 johnthetubaguy artom: curious, when nova changes a default in its config, what happens to the rest of the tempest settings?
15:31:00 artom johnthetubaguy, you mean for other config options where Tempest uses values from Nova? Good question - gmann was saying on that review that they just update Tempest, but I'd need to look for concrete examples
15:31:53 johnthetubaguy artom: cool, that is what I assumed. I know its branchless, but the default is just a helping hand.
15:32:33 artom johnthetubaguy, yeah, I grok that it can't be perfect, I figured at least making it match what Nova has in master is a good first step.
15:33:16 artom johnthetubaguy, because the previous default of 'all' is... well, it's a handy "feature" for CIs, because they can just enable any filter in Nova and Tempest just runs with it
15:33:39 artom But it becomes a problem if a filter *hasn't* been enabled in Nova, Tempest will still try to run with it.
15:52:51 mriedem this is fun https://bugs.launchpad.net/nova/+bug/1800508
15:52:51 openstack Launchpad bug 1800508 in OpenStack Compute (nova) "Missing exception handling mechanism in 'schedule_and_build_instances' for DBError at line 1180 of nova/conductor/manager.py" [Low,New]
15:53:01 mriedem "nova should set the instance to error state when nova fails to insert the instance into the db"
15:54:01 sean-k-mooney mriedem: am wait if nova cant insert the instacne into the db what is it setting error on?
15:54:08 artom Chicken, meet again. Cart, meet horse.
15:54:17 artom "again"? I mean egg
15:57:57 mriedem the only thing i could think there is we could try updating the instance within the build request, but that's pretty shitty
15:59:04 sean-k-mooney mriedem: in this case however it seams like they are booting with an invalid flavor id right?
15:59:14 mriedem no
15:59:22 mriedem he's injecting some kind of fault into the code
15:59:24 mriedem to trigger the db error
15:59:35 mriedem if you try to boot with an invalid flavor id, you'll get a 404 in the api looking up the flavor
16:00:16 sean-k-mooney oh ok i was trying to figure out how the create a flavor with id 1E+22 but then failed to boot with that flavor
16:00:18 mriedem so, i mean, your cell db could drop right when we're trying to create the server i guess, that would do it as well
16:00:24 mriedem but are we going to handle that scenario everywhere in nova?
16:01:11 sean-k-mooney ok right so in that case the insnace would be in the api db but fail to insert into the cell db
16:02:08 sean-k-mooney me moved the instance staus into the api db recnetly right so ya in that case we coudl set error on the api db i guess but there are a tone of other edgecase like that we dont handel
16:04:18 sean-k-mooney mriedem: the other thing we coudl do is have a periodic task that just updates the status of perptually building instance to error after some time e.g a day or rety limit*build timeout or something
16:05:14 mriedem this guy has been busy https://bugs.launchpad.net/nova/+bug/1800204 https://bugs.launchpad.net/nova/+bug/1799949
16:05:14 openstack Launchpad bug 1800204 in OpenStack Compute (nova) "n-cpu.service consuming 100% of CPU indeterminately" [Undecided,New]
16:05:15 openstack Launchpad bug 1799949 in OpenStack Compute (nova) "VM instance building forever when an RPC error occurs" [Undecided,New]
16:06:20 sean-k-mooney mriedem: actull https://bugs.launchpad.net/nova/+bug/1800204 seams familar there was a similar bug report a few monts back around the rocky release
16:06:20 openstack Launchpad bug 1800204 in OpenStack Compute (nova) "n-cpu.service consuming 100% of CPU indeterminately" [Undecided,New]
16:13:15 sean-k-mooney oh wait that n-cpu not the conductor never mind
16:16:10 sean-k-mooney mriedem: do you think we will actully adress any of those bugs.
16:16:34 stephenfin artom: So, do I need to review https://code.engineering.redhat.com/gerrit/#/c/154627/2 yet?
16:16:45 sean-k-mooney stephenfin: wrong irc
16:16:51 stephenfin ta :)
16:19:14 mriedem sean-k-mooney: probably not
16:19:35 mriedem unless there is a more obvious way to create those faults with injecting code into the path and blow up the system
16:19:42 mriedem *without
16:22:10 sean-k-mooney mriedem: i was just debating if we shoudl triage them as incomplete or wontfix unless a different way to reporduce can be provided
16:23:48 mriedem i marked one of them as opinion
16:25:01 johnthetubaguy FWIW, I always wanted to be able to "timeout" tasks to try and catch that pending forever case. They caused me endless pain at Rackspace (I think mostly in the migrate/resize code path). The difference was they were more expected / user triggered errors.
16:25:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Add --before to nova-manage db archive_deleted_rows https://review.openstack.org/556751
16:26:07 mriedem johnthetubaguy: how much of that was resolved with service user tokens though?
16:26:14 mriedem or the long_rpc_timeout we have since rocky
16:26:23 mriedem which we're using now in the live migration flows that do rpc calls
16:27:17 johnthetubaguy mriedem: yeah, I saw that go in. Although most of those cases it went to Error (eventually) when it didn't have to.
16:27:34 mriedem that's a different bug then
16:29:05 sean-k-mooney johnthetubaguy: i have seen this happen with rabbitmq restarte in the past where when perstiency was disabled on instance build and a few other cases.
16:29:40 sean-k-mooney i never really considerd that a nova bug however because i cased the issue by restarting rabbit
16:31:04 sean-k-mooney johnthetubaguy: but ya its proably more complcated then jsut set to error after x time as some request could still be in flight
16:36:00 melwitt mriedem: yes, it completely slipped my mind :( and I'm not done going through the entire list of the schedule yet
16:40:12 mriedem melwitt: i added several sessions in there based on my schedule
16:41:09 melwitt ok, thank you. that's helpful
16:47:07 johnthetubaguy sean-k-mooney: yeah, its hard to get right
16:56:53 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: API microversion bump for handling-down-cell https://review.openstack.org/591657
16:56:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810

Earlier   Later