Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-11-01
15:22:33 dansmith tempest is a testing/validation tool.. keeping the two configs in sync is a few lines of bash
15:22:36 mriedem right, devstack configures the filters in both nova and tempest
15:22:40 dansmith right
15:22:51 sean-k-mooney dansmith: the issue is making triplo do that
15:23:06 mriedem devstack also adds the same/different host filtesr which aren't in the default enabled_filters list for nova
15:23:09 dansmith sean-k-mooney: s/bash/puppet/
15:23:28 mriedem etc
15:23:28 mriedem for any nfv ci, they'd also need to configure to numa/pci filters
15:23:30 sean-k-mooney dansmith: ya i know its just triplo is a pain to make work instead of devstack
15:23:44 dansmith sean-k-mooney: adding an api to nova to work around tripleo not being able to communicate config to another module is INSANITY
15:23:51 sean-k-mooney mriedem: yes today they only need to enable it in nova however
15:24:18 johnthetubaguy I think sdague convinced me about this in the past, you don't want auto discovery, you want to tell the test system what you expect to happen, else there be dragons
15:24:32 artom dansmith, it's not communicate per se - if tripleo doens't set the nova value, it shouldn't have to set the corresponding tempest value
15:25:06 dansmith artom: find another way
15:25:12 dansmith seriously.
15:25:22 johnthetubaguy matching defaults?
15:25:23 sean-k-mooney johnthetubaguy: ya well it was jut a taught my main issue with artom change is that we would have to keep it in sync if we add filter in the futre to the default set
15:25:23 artom dansmith, my other way is https://review.openstack.org/#/c/570207/12
15:25:28 artom johnthetubaguy, that's what ^^ is
15:25:39 artom But apparently everyone is literally willing to fight to the death over this.
15:25:44 dansmith I am
15:25:50 dansmith let's do it.
15:26:14 artom I only have this bluetooth mouse :(
15:26:22 dansmith forfeit?
15:26:34 sean-k-mooney johnthetubaguy: the interop benift is really only a side effect and i dont feel that strongly that its a good thing
15:26:40 artom Pfft, as if. I'm making brass knuckles. Wireless ones.
15:29:46 johnthetubaguy artom: curious, when nova changes a default in its config, what happens to the rest of the tempest settings?
15:31:00 artom johnthetubaguy, you mean for other config options where Tempest uses values from Nova? Good question - gmann was saying on that review that they just update Tempest, but I'd need to look for concrete examples
15:31:53 johnthetubaguy artom: cool, that is what I assumed. I know its branchless, but the default is just a helping hand.
15:32:33 artom johnthetubaguy, yeah, I grok that it can't be perfect, I figured at least making it match what Nova has in master is a good first step.
15:33:16 artom johnthetubaguy, because the previous default of 'all' is... well, it's a handy "feature" for CIs, because they can just enable any filter in Nova and Tempest just runs with it
15:33:39 artom But it becomes a problem if a filter *hasn't* been enabled in Nova, Tempest will still try to run with it.
15:52:51 openstack Launchpad bug 1800508 in OpenStack Compute (nova) "Missing exception handling mechanism in 'schedule_and_build_instances' for DBError at line 1180 of nova/conductor/manager.py" [Low,New]
15:52:51 mriedem this is fun https://bugs.launchpad.net/nova/+bug/1800508
15:53:01 mriedem "nova should set the instance to error state when nova fails to insert the instance into the db"
15:54:01 sean-k-mooney mriedem: am wait if nova cant insert the instacne into the db what is it setting error on?
15:54:08 artom Chicken, meet again. Cart, meet horse.
15:54:17 artom "again"? I mean egg
15:57:57 mriedem the only thing i could think there is we could try updating the instance within the build request, but that's pretty shitty
15:59:04 sean-k-mooney mriedem: in this case however it seams like they are booting with an invalid flavor id right?
15:59:14 mriedem no
15:59:22 mriedem he's injecting some kind of fault into the code
15:59:24 mriedem to trigger the db error
15:59:35 mriedem if you try to boot with an invalid flavor id, you'll get a 404 in the api looking up the flavor
16:00:16 sean-k-mooney oh ok i was trying to figure out how the create a flavor with id 1E+22 but then failed to boot with that flavor
16:00:18 mriedem so, i mean, your cell db could drop right when we're trying to create the server i guess, that would do it as well
16:00:24 mriedem but are we going to handle that scenario everywhere in nova?
16:01:11 sean-k-mooney ok right so in that case the insnace would be in the api db but fail to insert into the cell db
16:02:08 sean-k-mooney me moved the instance staus into the api db recnetly right so ya in that case we coudl set error on the api db i guess but there are a tone of other edgecase like that we dont handel
16:04:18 sean-k-mooney mriedem: the other thing we coudl do is have a periodic task that just updates the status of perptually building instance to error after some time e.g a day or rety limit*build timeout or something
16:05:14 openstack Launchpad bug 1800204 in OpenStack Compute (nova) "n-cpu.service consuming 100% of CPU indeterminately" [Undecided,New]
16:05:14 mriedem this guy has been busy https://bugs.launchpad.net/nova/+bug/1800204 https://bugs.launchpad.net/nova/+bug/1799949
16:05:15 openstack Launchpad bug 1799949 in OpenStack Compute (nova) "VM instance building forever when an RPC error occurs" [Undecided,New]
16:06:20 openstack Launchpad bug 1800204 in OpenStack Compute (nova) "n-cpu.service consuming 100% of CPU indeterminately" [Undecided,New]
16:06:20 sean-k-mooney mriedem: actull https://bugs.launchpad.net/nova/+bug/1800204 seams familar there was a similar bug report a few monts back around the rocky release
16:13:15 sean-k-mooney oh wait that n-cpu not the conductor never mind
16:16:10 sean-k-mooney mriedem: do you think we will actully adress any of those bugs.
16:16:34 stephenfin artom: So, do I need to review https://code.engineering.redhat.com/gerrit/#/c/154627/2 yet?
16:16:45 sean-k-mooney stephenfin: wrong irc
16:16:51 stephenfin ta :)
16:19:14 mriedem sean-k-mooney: probably not
16:19:35 mriedem unless there is a more obvious way to create those faults with injecting code into the path and blow up the system
16:19:42 mriedem *without
16:22:10 sean-k-mooney mriedem: i was just debating if we shoudl triage them as incomplete or wontfix unless a different way to reporduce can be provided
16:23:48 mriedem i marked one of them as opinion
16:25:01 johnthetubaguy FWIW, I always wanted to be able to "timeout" tasks to try and catch that pending forever case. They caused me endless pain at Rackspace (I think mostly in the migrate/resize code path). The difference was they were more expected / user triggered errors.
16:25:06 openstackgerrit Matt Riedemann proposed openstack/nova master: Add --before to nova-manage db archive_deleted_rows https://review.openstack.org/556751
16:26:07 mriedem johnthetubaguy: how much of that was resolved with service user tokens though?
16:26:14 mriedem or the long_rpc_timeout we have since rocky
16:26:23 mriedem which we're using now in the live migration flows that do rpc calls
16:27:17 johnthetubaguy mriedem: yeah, I saw that go in. Although most of those cases it went to Error (eventually) when it didn't have to.
16:27:34 mriedem that's a different bug then
16:29:05 sean-k-mooney johnthetubaguy: i have seen this happen with rabbitmq restarte in the past where when perstiency was disabled on instance build and a few other cases.
16:29:40 sean-k-mooney i never really considerd that a nova bug however because i cased the issue by restarting rabbit
16:31:04 sean-k-mooney johnthetubaguy: but ya its proably more complcated then jsut set to error after x time as some request could still be in flight
16:36:00 melwitt mriedem: yes, it completely slipped my mind :( and I'm not done going through the entire list of the schedule yet
16:40:12 mriedem melwitt: i added several sessions in there based on my schedule
16:41:09 melwitt ok, thank you. that's helpful
16:47:07 johnthetubaguy sean-k-mooney: yeah, its hard to get right
16:56:53 openstackgerrit Matt Riedemann proposed openstack/nova master: WIP: API microversion bump for handling-down-cell https://review.openstack.org/591657
16:56:54 openstackgerrit Matt Riedemann proposed openstack/nova master: Add DownCellFixture https://review.openstack.org/614810
16:56:57 mriedem tssurya: ^
16:57:10 tssurya looking, thanks
17:04:43 tssurya mriedem: okay I am going to write the tests here https://review.openstack.org/#/c/591657/12/nova/tests/functional/api_sample_tests/test_servers.py based on your fixture
17:13:08 openstackgerrit Merged openstack/nova master: Make ResourceTracker.tracked_instances a set https://review.openstack.org/608781
17:18:31 dansmith mriedem: tssurya I'm explaining the host-status concept to someone right now, and why an instance state doesn't go to STOPPED just because the compute node is down
17:18:47 dansmith mriedem: I wonder if it would make sense to integrate the use of the UNKNOWN state we're adding here with that feature,
17:18:59 dansmith so that in the same microversion, instances with a down host show up as UNKNOWN as well
17:25:59 tssurya dansmith: you mean you want to add a new "UNKNOWN" vm_state ?
17:26:20 dansmith tssurya: you're already doing that from the external view right now
17:26:25 tssurya yea
17:26:39 dansmith we would do a similar thing for real instances we can look up just fine, but which have down hosts
17:27:05 dansmith the only problem would be that right now UNKNOWN means "the rest of the instance details aren't there" which would be slightly more ambiguous in this case
17:28:27 tssurya hmm, makes sense to make the instance state UNKNOWN since we don't know the host state, I mean I guess "UNKNOWN" could mean unknown details/state right ?
17:28:54 belmoreira dansmith mriedem should placement/nova issues be discussed here or in placement channel
17:28:57 cfriesen dansmith: for what it's worth, in our environment if a compute node goes down an external entity sets all of the instances to the "error" state, until they were automatically recovered.
17:29:30 dansmith that seems like an improper use of the error state to me
17:29:48 dansmith not to mention that nova on its own won't know whether they're still up and fine or not
17:29:50 cfriesen if the host is "down", then we fence it off and force a reboot. those instances are guaranteed to be toast
17:29:53 dansmith which is why we don't call them "stopped"

Earlier   Later