| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-07 | |||
| 15:42:20 | superdan | cool | |
| 15:42:26 | superdan | well, not cool, but.. | |
| 15:42:48 | mriedem | about as cool as a kick in the pants | |
| 15:42:58 | mriedem | maybe another nail in the shelve coffin | |
| 15:43:26 | superdan | meh, anyone that uses shelve is probably used to shelve being broken | |
| 15:43:39 | mriedem | "Days since shelve failed: 0" | |
| 15:43:43 | mriedem | ^ in their office | |
| 15:44:21 | superdan | hah yeah | |
| 15:48:03 | mriedem | hmm, in the multi-cell job, the cpu config database is also pointing at cell0 | |
| 15:48:04 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 15:48:24 | mriedem | but it should go through conductor when the compute manager queries for the list of bdms right? | |
| 15:48:29 | mriedem | and cell conductor should be using http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz ? | |
| 15:48:50 | superdan | yeah | |
| 15:49:22 | superdan | really we should unset the db connection for both main and api in that cpu config, | |
| 15:49:28 | superdan | but the way I generate it is kinda weird | |
| 15:49:56 | mriedem | the [database] one for sure, | |
| 15:50:07 | mriedem | but the api_database one is needed for the retry and affinity crap | |
| 15:50:14 | superdan | not in compute | |
| 15:50:17 | mriedem | well, at least affinity | |
| 15:50:38 | mriedem | sure it is, the compute manager pulls server groups from the api db right? | |
| 15:51:00 | superdan | no, it would ask conductor like anything else | |
| 15:51:20 | superdan | and conductor in cell1 is configured to not be able to find the api db: http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:51:33 | superdan | which means it can't retry in that configuration nor do the affinity check, | |
| 15:51:34 | mriedem | so how does the affinity stuff not all blow up? | |
| 15:51:45 | superdan | which is why that workaround config was added and the devstack patch sets it | |
| 15:51:53 | mriedem | oh | |
| 15:52:10 | superdan | mriedem: got a bug for the offload target thing yet? | |
| 15:52:16 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:52:22 | mriedem | oops | |
| 15:52:31 | mriedem | i meant, if not CONF.workarounds.disable_group_policy_check_upcall: | |
| 15:52:47 | mriedem | ok | |
| 15:53:00 | openstack | Launchpad bug 1702932 in OpenStack Compute (nova) "Unshelving an offloaded server with volume attachments may not attach to the guest in multi-cell env" [Undecided,New] | |
| 15:53:00 | mriedem | https://bugs.launchpad.net/nova/+bug/1702932 | |
| 15:53:24 | superdan | thanks | |
| 15:53:24 | mriedem | i'll crank out another one for the unshelve limit to same cell thing | |
| 15:54:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix targeting cell when attaching a volume to a shelved instance https://review.openstack.org/481683 | |
| 15:54:48 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565 | |
| 15:54:56 | superdan | here's that first fix and updated canary ^ | |
| 15:56:23 | mriedem | https://bugs.launchpad.net/nova/+bug/1702935 | |
| 15:56:24 | openstack | Launchpad bug 1702935 in OpenStack Compute (nova) "Unshelving an offloaded server does not restrict to the original cell" [Undecided,New] | |
| 16:01:14 | mriedem | cool https://review.openstack.org/#/c/481683/ is easier than i thought, but makes sense. still really wondering how this wasn't caught in tempest | |
| 16:12:21 | mriedem | so normal attach volume works to create the volume bdm in the same cell as the instance because when we lookup the instance, set target the context | |
| 16:13:08 | mriedem | and we create that bdm in the cell compute service | |
| 16:34:57 | openstack | Launchpad bug 1702959 in OpenStack Compute (nova) "MigrationNotFound in multi-cell setup doing server external events processing" [High,Triaged] | |
| 16:34:57 | mriedem | another one https://bugs.launchpad.net/nova/+bug/1702959 | |
| 16:38:13 | mriedem | i'll take a crack at that one | |
| 16:38:54 | dfisher | anybody know of attach/detach issues with current nova/cinder (via devstack) bits? I can't seem to get a simple volume to attach to a simple Ubuntu or Cirros VM | |
| 16:39:16 | mriedem | no known issues | |
| 16:39:22 | dfisher | hmm, ok. | |
| 16:39:28 | dfisher | thanks | |
| 16:43:21 | superdan | mriedem: is that one resolved by the patch that is up already? | |
| 16:43:42 | superdan | mriedem: https://review.openstack.org/#/c/445142/ | |
| 16:46:00 | clarkb | mriedem: http://logs.openstack.org/31/476631/18/gate/gate-nova-tox-functional-ubuntu-xenial/2cafb17/console.html#_2017-07-07_15_29_18_416631 known issue? that reset the gate not too long ago. Looks like test is attempting admin request and failing without admin rights | |
| 16:50:23 | mriedem | clarkb: i'm not aware of that one | |
| 16:50:47 | mriedem | superdan: maybe given https://review.openstack.org/#/c/445142/10/nova/compute/api.py@4263 | |
| 16:51:07 | mriedem | superdan: are you ok with me making the fleetify change depend on that patch as well? | |
| 16:51:19 | superdan | mriedem: sure | |
| 16:56:27 | openstackgerrit | melanie witt proposed openstack/nova master: Use ConsoleConnection object to generate authorizations https://review.openstack.org/325414 | |
| 16:56:27 | openstackgerrit | melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381 | |
| 16:56:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614 | |
| 16:56:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add console connection object https://review.openstack.org/320063 | |
| 16:56:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add console_auth_token_get() method to DB API https://review.openstack.org/481700 | |
| 16:56:29 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 16:56:41 | mriedem | ok, fleeitify updated, will check out the api logs when that's done to see if the migration not found is gone | |
| 16:58:33 | mriedem | clarkb: have you checked logstash? | |
| 17:00:11 | mriedem | probably can't really | |
| 17:00:23 | clarkb | mriedem: not yet trying to diagnose problems with my desktop :/ | |
| 17:00:42 | mriedem | in general we've had very weird failures like that with the nova functional tests, | |
| 17:01:03 | mriedem | something is making a request to the placement service and another is making a request to the compute api service, and the response goes the wrong way, | |
| 17:01:07 | mriedem | assuming eventlet weirdness | |
| 17:05:45 | mriedem | aha | |
| 17:05:57 | mriedem | vdrok: found that weird sql connection thing in the fleetify patch | |
| 17:06:08 | openstackgerrit | John Griffith proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285 | |
| 17:06:15 | mriedem | http://logs.openstack.org/56/477556/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/ee3e9b6/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_05_20_44_19_705646 | |
| 17:06:41 | mriedem | the subnode n-cpu is doing the server group affinity check, which blows up b/c the cell conductor doesn't have an api db connection, | |
| 17:06:52 | mriedem | the nova-cpu.conf on the subnode is missing the config option to disable that upcall | |
| 17:10:38 | vdrok | mriedem: cool, thanks! Do we need to set it in ironic plugin or it's going to be handled on Nova side? | |
| 17:10:50 | mriedem | not sure, it needs to be handled in https://review.openstack.org/#/c/477556/4/lib/nova though | |
| 17:11:18 | mriedem | i think this is where the nova-cpu.conf is used https://review.openstack.org/#/c/477556/4/lib/nova@818 | |
| 17:13:33 | vdrok | mriedem: aha, ok. | |
| 17:15:05 | mriedem | i don't know what nova.conf it's copying though, or where that is configured | |
| 17:17:30 | mriedem | ah i think i know where to set it | |
| 17:17:36 | mriedem | if dan is ok with me touching his change | |
| 17:19:22 | mriedem | superdan: i think i know how to fix the subnode nova.conf thing for the upcall in https://review.openstack.org/#/c/477556/ - ok with me doing that? | |
| 17:20:58 | superdan | mriedem: not sure what you mean "fix the upcall" | |
| 17:21:11 | mriedem | the subnode n-cpu conf isn't configured to disable the server group upcall | |
| 17:21:17 | mriedem | i know where to fix it in the fleetify change | |
| 17:21:21 | mriedem | asking permission | |
| 17:21:23 | superdan | it's not? | |
| 17:21:26 | mriedem | f no | |
| 17:21:44 | superdan | ohh, subnode | |
| 17:21:45 | superdan | gotcha | |
| 17:21:47 | superdan | yes please | |
| 17:21:57 | mriedem | aye aye | |
| 17:22:20 | mriedem | is it weird when i can see the mail man out of the corner of my eye and look at him and he nods at me cross my yard to drop something off? | |
| 17:22:27 | mriedem | should we have such a familiar relationship at this point? | |
| 17:23:41 | mriedem | ok it's up https://review.openstack.org/#/c/477556/4..5/lib/nova | |
| 17:23:43 | mriedem | and now it's taco time | |
| 17:24:31 | smcginnis | mriedem: But it's not Tuesday. | |
| 17:28:17 | openstackgerrit | Mathieu Gagné proposed openstack/nova master: Add ability to signal and perform online volume size change https://review.openstack.org/454322 | |