| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-07-07 | |||
| 15:27:07 | openstackgerrit | Merged openstack/nova master: Support tag instances when boot(3/4) https://review.openstack.org/459593 | |
| 15:31:01 | superdan | mriedem: was there a bug for this shelved issue? | |
| 15:31:12 | mriedem | no, i can create one | |
| 15:31:22 | superdan | thanks | |
| 15:33:53 | mriedem | you know, | |
| 15:33:54 | mriedem | another thing, | |
| 15:34:16 | mriedem | when we shelve offload a server, we never clean up it's InstanceMapping, | |
| 15:34:30 | mriedem | and when we unshelve it, conductor asks the scheduler for a new host, which could be in a different cell, | |
| 15:34:40 | mriedem | but we never create a new instance mapping, or update the existing instance mapping, for that new cell | |
| 15:35:17 | superdan | mriedem: we can't move it between cells, | |
| 15:35:27 | superdan | so if that's true, we need to be asking the scheduler to limit to one cell, | |
| 15:35:29 | superdan | like we do for resize | |
| 15:35:38 | mriedem | i don't see anything in conductor that's doing that | |
| 15:35:49 | mriedem | is it in the req spec? | |
| 15:35:52 | superdan | yep | |
| 15:35:58 | superdan | reqspec.destination.cell | |
| 15:38:58 | mriedem | ok this https://github.com/openstack/nova/blob/56cd608d3a199dcb02ac2ae071ff3057241259da/nova/conductor/tasks/migrate.py#L63-L66 | |
| 15:39:09 | mriedem | that's the only place we set destination.cell from what i can tell | |
| 15:40:11 | superdan | yep | |
| 15:41:08 | mriedem | ok i'll open a separate bug for that | |
| 15:42:20 | superdan | cool | |
| 15:42:26 | superdan | well, not cool, but.. | |
| 15:42:48 | mriedem | about as cool as a kick in the pants | |
| 15:42:58 | mriedem | maybe another nail in the shelve coffin | |
| 15:43:26 | superdan | meh, anyone that uses shelve is probably used to shelve being broken | |
| 15:43:39 | mriedem | "Days since shelve failed: 0" | |
| 15:43:43 | mriedem | ^ in their office | |
| 15:44:21 | superdan | hah yeah | |
| 15:48:03 | mriedem | hmm, in the multi-cell job, the cpu config database is also pointing at cell0 | |
| 15:48:04 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova-cpu.conf.txt.gz | |
| 15:48:24 | mriedem | but it should go through conductor when the compute manager queries for the list of bdms right? | |
| 15:48:29 | mriedem | and cell conductor should be using http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz ? | |
| 15:48:50 | superdan | yeah | |
| 15:49:22 | superdan | really we should unset the db connection for both main and api in that cpu config, | |
| 15:49:28 | superdan | but the way I generate it is kinda weird | |
| 15:49:56 | mriedem | the [database] one for sure, | |
| 15:50:07 | mriedem | but the api_database one is needed for the retry and affinity crap | |
| 15:50:14 | superdan | not in compute | |
| 15:50:17 | mriedem | well, at least affinity | |
| 15:50:38 | mriedem | sure it is, the compute manager pulls server groups from the api db right? | |
| 15:51:00 | superdan | no, it would ask conductor like anything else | |
| 15:51:20 | superdan | and conductor in cell1 is configured to not be able to find the api db: http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:51:33 | superdan | which means it can't retry in that configuration nor do the affinity check, | |
| 15:51:34 | mriedem | so how does the affinity stuff not all blow up? | |
| 15:51:45 | superdan | which is why that workaround config was added and the devstack patch sets it | |
| 15:51:53 | mriedem | oh | |
| 15:52:10 | superdan | mriedem: got a bug for the offload target thing yet? | |
| 15:52:16 | mriedem | http://logs.openstack.org/65/473565/4/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/fcfedba/logs/etc/nova/nova_cell1.conf.txt.gz | |
| 15:52:22 | mriedem | oops | |
| 15:52:31 | mriedem | i meant, if not CONF.workarounds.disable_group_policy_check_upcall: | |
| 15:52:47 | mriedem | ok | |
| 15:53:00 | openstack | Launchpad bug 1702932 in OpenStack Compute (nova) "Unshelving an offloaded server with volume attachments may not attach to the guest in multi-cell env" [Undecided,New] | |
| 15:53:00 | mriedem | https://bugs.launchpad.net/nova/+bug/1702932 | |
| 15:53:24 | superdan | thanks | |
| 15:53:24 | mriedem | i'll crank out another one for the unshelve limit to same cell thing | |
| 15:54:48 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix targeting cell when attaching a volume to a shelved instance https://review.openstack.org/481683 | |
| 15:54:48 | openstackgerrit | Dan Smith proposed openstack/nova master: DNM: Canary https://review.openstack.org/473565 | |
| 15:54:56 | superdan | here's that first fix and updated canary ^ | |
| 15:56:23 | mriedem | https://bugs.launchpad.net/nova/+bug/1702935 | |
| 15:56:24 | openstack | Launchpad bug 1702935 in OpenStack Compute (nova) "Unshelving an offloaded server does not restrict to the original cell" [Undecided,New] | |
| 16:01:14 | mriedem | cool https://review.openstack.org/#/c/481683/ is easier than i thought, but makes sense. still really wondering how this wasn't caught in tempest | |
| 16:12:21 | mriedem | so normal attach volume works to create the volume bdm in the same cell as the instance because when we lookup the instance, set target the context | |
| 16:13:08 | mriedem | and we create that bdm in the cell compute service | |
| 16:34:57 | openstack | Launchpad bug 1702959 in OpenStack Compute (nova) "MigrationNotFound in multi-cell setup doing server external events processing" [High,Triaged] | |
| 16:34:57 | mriedem | another one https://bugs.launchpad.net/nova/+bug/1702959 | |
| 16:38:13 | mriedem | i'll take a crack at that one | |
| 16:38:54 | dfisher | anybody know of attach/detach issues with current nova/cinder (via devstack) bits? I can't seem to get a simple volume to attach to a simple Ubuntu or Cirros VM | |
| 16:39:16 | mriedem | no known issues | |
| 16:39:22 | dfisher | hmm, ok. | |
| 16:39:28 | dfisher | thanks | |
| 16:43:21 | superdan | mriedem: is that one resolved by the patch that is up already? | |
| 16:43:42 | superdan | mriedem: https://review.openstack.org/#/c/445142/ | |
| 16:46:00 | clarkb | mriedem: http://logs.openstack.org/31/476631/18/gate/gate-nova-tox-functional-ubuntu-xenial/2cafb17/console.html#_2017-07-07_15_29_18_416631 known issue? that reset the gate not too long ago. Looks like test is attempting admin request and failing without admin rights | |
| 16:50:23 | mriedem | clarkb: i'm not aware of that one | |
| 16:50:47 | mriedem | superdan: maybe given https://review.openstack.org/#/c/445142/10/nova/compute/api.py@4263 | |
| 16:51:07 | mriedem | superdan: are you ok with me making the fleetify change depend on that patch as well? | |
| 16:51:19 | superdan | mriedem: sure | |
| 16:56:27 | openstackgerrit | melanie witt proposed openstack/nova master: Use ConsoleConnection object to generate authorizations https://review.openstack.org/325414 | |
| 16:56:27 | openstackgerrit | melanie witt proposed openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381 | |
| 16:56:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add access_url_base to console_auth_tokens table https://review.openstack.org/334614 | |
| 16:56:28 | openstackgerrit | melanie witt proposed openstack/nova master: Add console connection object https://review.openstack.org/320063 | |
| 16:56:29 | openstackgerrit | melanie witt proposed openstack/nova master: Add console_auth_token_get() method to DB API https://review.openstack.org/481700 | |
| 16:56:29 | openstackgerrit | melanie witt proposed openstack/nova master: Convert websocketproxy to use db for token validation https://review.openstack.org/333990 | |
| 16:56:41 | mriedem | ok, fleeitify updated, will check out the api logs when that's done to see if the migration not found is gone | |
| 16:58:33 | mriedem | clarkb: have you checked logstash? | |
| 17:00:11 | mriedem | probably can't really | |
| 17:00:23 | clarkb | mriedem: not yet trying to diagnose problems with my desktop :/ | |
| 17:00:42 | mriedem | in general we've had very weird failures like that with the nova functional tests, | |
| 17:01:03 | mriedem | something is making a request to the placement service and another is making a request to the compute api service, and the response goes the wrong way, | |
| 17:01:07 | mriedem | assuming eventlet weirdness | |
| 17:05:45 | mriedem | aha | |
| 17:05:57 | mriedem | vdrok: found that weird sql connection thing in the fleetify patch | |
| 17:06:08 | openstackgerrit | John Griffith proposed openstack/nova master: Implement new attach Cinder flow https://review.openstack.org/330285 | |
| 17:06:15 | mriedem | http://logs.openstack.org/56/477556/3/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial-nv/ee3e9b6/logs/subnode-2/screen-n-cpu.txt.gz#_Jul_05_20_44_19_705646 | |
| 17:06:41 | mriedem | the subnode n-cpu is doing the server group affinity check, which blows up b/c the cell conductor doesn't have an api db connection, | |
| 17:06:52 | mriedem | the nova-cpu.conf on the subnode is missing the config option to disable that upcall | |
| 17:10:38 | vdrok | mriedem: cool, thanks! Do we need to set it in ironic plugin or it's going to be handled on Nova side? | |
| 17:10:50 | mriedem | not sure, it needs to be handled in https://review.openstack.org/#/c/477556/4/lib/nova though | |
| 17:11:18 | mriedem | i think this is where the nova-cpu.conf is used https://review.openstack.org/#/c/477556/4/lib/nova@818 | |
| 17:13:33 | vdrok | mriedem: aha, ok. | |