| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-06 | |||
| 14:15:43 | mriedem | but yes i don't expect how placement handles traits to change much | |
| 14:15:50 | cdent | it is in the sense that we'll never raise the minimum | |
| 14:15:54 | mriedem | and yes how they are incorporated into nova is the messy part | |
| 14:16:08 | mriedem | or could be messy | |
| 14:16:27 | mriedem | right now traits-based scheduling from queens is pretty straight forward | |
| 14:16:48 | mriedem | 'put me on a host with trait=foo, got it' | |
| 14:22:30 | mriedem | mmedvede: i got a pkvm ci failure setting up devstack here https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/20/324720/5/check/tempest-dsvm-full-xenial/b59cb03/devstacklog.txt.gz | |
| 14:22:41 | mriedem | Connecting to cirros-builder.novalocal (cirros-builder.novalocal)|9.114.111.40|:80... failed: No route to host. | |
| 14:23:26 | mriedem | jgwentworth: fyi, i had emailed the virtuozzo team about their CI failing at least a week ago, so it's a known issue | |
| 14:23:42 | mriedem | ImagePropertiesFilter always kicks out whatever their trying to use for an image | |
| 14:25:55 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Skip placement on rebuild in same host https://review.openstack.org/546357 | |
| 14:33:15 | odyssey4me | howdy everyone, thanks again for your help so far - we have an upgrade task failing which might be due to new changes in rocky... would this be a change in something that needs to happen? http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/result/b9fe5b27-00e6-455b-9a10-71b2046663c5/ | |
| 14:33:33 | edmondsw | fyi, someone is trying to add suppz support to the libvirt driver, but seems to be doing that off on their own | |
| 14:33:36 | edmondsw | https://ask.openstack.org/en/question/113484/add-support-for-a-new-driver-in-nova/ | |
| 14:33:45 | edmondsw | I tried to point them in the right direction | |
| 14:34:29 | jgwentworth | mriedem: okay, so virtuozzo CI currently broken and being worked on | |
| 14:36:13 | mriedem | lyarwood: so in https://review.openstack.org/#/c/544238/4/nova/virt/libvirt/driver.py we also can't do the swap if both volumes are encrypted? | |
| 14:36:52 | mriedem | jgwentworth: it's a known issue, i'm not sure how actively they are working to fix it | |
| 14:38:17 | mriedem | odyssey4me: that looks like the new 'disabled' colum in the nova_api.cell_mappings table, has the nova_api db been synced before running this? | |
| 14:38:27 | lyarwood | mriedem: correct | |
| 14:38:48 | mriedem | tssurya: ^ | |
| 14:39:53 | odyssey4me | mriedem well, that's a queens to rocky upgrade, so yes - it was done in the queens deployment | |
| 14:40:18 | kashyap | lyarwood: Probably that should point also be documented as a note in the code? (With current limitations.) | |
| 14:40:27 | mriedem | odyssey4me: if you're upgrading to rocky and you already created the cell0 mapping in queens, you don't need to do it again | |
| 14:41:25 | mriedem | odyssey4me: although, map_cell0 should be smarter, it has a check for DBDuplicateEntry for running it when it already exists | |
| 14:41:43 | jgwentworth | mriedem: okay. would you say it's too soon to mail the ML to describe the issue and ask if anyone on their side can look into it? | |
| 14:42:09 | mriedem | jgwentworth: i emailed the vz CI email a couple of weeks ago, and got a reply this week, but deleted that reply | |
| 14:42:22 | mriedem | i'd give it maybe another week? | |
| 14:42:38 | jgwentworth | k, will do. thanks | |
| 14:42:57 | mriedem | odyssey4me: so i don't think we're getting the duplicate entry b/c it's trying to create the cell0 mapping in the api db, and that fails because the object code doesn't match the api db schema | |
| 14:43:31 | mriedem | odyssey4me: which tells me that nova-manage api_db sync hasn't been run on rocky yet | |
| 14:43:35 | openstack | Launchpad bug 1739593 in OpenStack Security Advisory "Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack (CVE-2017-18191)" [Undecided,Incomplete] | |
| 14:43:35 | lyarwood | mriedem: thanks, btw https://review.openstack.org/#/c/543569/ is the change covering this all prior to Queens where we would hit https://bugs.launchpad.net/nova/+bug/1739593, it would be great to land this in stable/ocata and pike sometime soon to close that all out. | |
| 14:44:04 | mriedem | lyarwood: i've tried to purge that from my memory | |
| 14:44:09 | odyssey4me | mriedem hmm, IIRC on an upgrade the steps are different - we have this now, so you're right: https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml | |
| 14:45:34 | lyarwood | mriedem: you and me both, definitely a mistake on my part to label this as a security issue | |
| 14:46:10 | mriedem | odyssey4me: so https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml is getting run all over again on an upgrade from queens to rocky? | |
| 14:46:49 | odyssey4me | mriedem yes, that sequence is run for greenfield, or upgrade - but the result of 'Get the api_db version' would be different if it was an upgrade | |
| 14:47:06 | mriedem | odyssey4me: oh i see the problem https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml#L29 | |
| 14:47:16 | mriedem | yeah looking at http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/reports/68ffa953-cc0e-494b-89b9-dcf7ca4a8b59.html | |
| 14:47:29 | mriedem | so "Perform a Nova API DB sync" doesn't get run | |
| 14:47:30 | mriedem | that's a problem | |
| 14:47:53 | odyssey4me | ok, but that gets run later | |
| 14:48:03 | odyssey4me | IIRC this relates to the difference in procedure for greenfield vs upgrade | |
| 14:48:20 | odyssey4me | this might only have been an ocata/pike thing though, when cells v2 was all new | |
| 14:48:24 | mriedem | how did you upgrade from ocata to pike, or pike to queens? | |
| 14:48:34 | mriedem | i guess we haven't added a new cell mappings field sine then | |
| 14:48:36 | mriedem | so that's why | |
| 14:48:53 | odyssey4me | this same procedure is used - ocata->pike: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/pike/tasks/nova_db_setup.yml | |
| 14:49:11 | odyssey4me | pike->queens: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/queens/tasks/nova_db_setup.yml | |
| 14:49:29 | mriedem | yeah, we just have'nt added a new cell_mappings column in queens or pike | |
| 14:49:36 | odyssey4me | aha, ok | |
| 14:49:41 | mriedem | so, we can fix this in one of two ways: | |
| 14:49:56 | odyssey4me | so this conditional set is only important for the newton->ocata upgrade? | |
| 14:50:02 | mriedem | 1. nova-manage cell_v2 map_cell0 can do a db query for cell0 rather than rely on a DBDuplicateEntry | |
| 14:50:10 | mriedem | 2. you can run nova-manage api_db sync unconditionally | |
| 14:50:48 | odyssey4me | we're very happy to shorten and simplify the steps taken, so option 2 sounds good to me | |
| 14:51:29 | odyssey4me | I'll push a patch up shortly. I'll ping you to verify the review if that's ok? | |
| 14:51:31 | mriedem | fwiw, grenade also runs map_cell0 on upgrade and cell0 is already mapped http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_34_277 | |
| 14:51:48 | mriedem | sure. i'm not ansible guy, but i can see if the non-voting upgrade job is green or not :) | |
| 14:52:33 | odyssey4me | :) hopefully we can make it as readable as a shell task procedure | |
| 14:52:42 | mriedem | oh looky here http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_25_031 | |
| 14:52:49 | mriedem | grenade hits the same thing but doesn't fail for some reason | |
| 14:56:11 | tssurya | sorry was afk, that doesn't look goo | |
| 14:56:14 | tssurya | good* | |
| 14:57:40 | odyssey4me | mriedem maybe someone couldn't figure it out and added || true ;) | |
| 14:58:41 | odyssey4me | jokes aside, that's quite serious... but I'm glad we're not alone, and that we could find a red flag :) | |
| 14:58:48 | mriedem | it's hitting htis https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/upgrade.sh#L74 | |
| 14:59:09 | mriedem | oh i know why it's not failing | |
| 14:59:17 | mriedem | the nova-manage command doesn't return a non-0 | |
| 14:59:27 | mriedem | that's why it dumps out the breadcrumb | |
| 14:59:33 | mriedem | "Has "nova-manage api_db sync" been run?" | |
| 15:00:03 | openstackgerrit | Merged openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | |
| 15:00:05 | tssurya | mriedem: is this because api DB was not synced before running db sync ? | |
| 15:00:26 | mriedem | yes | |
| 15:00:46 | tssurya | ah ok | |
| 15:04:47 | mriedem | the order in grenade for the db sync matches what we have in our upgrade docs too https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process | |
| 15:07:50 | openstack | Launchpad bug 1761775 in grenade "InternalError during nova-manage db sync when upgrading to Rocky" [Undecided,Confirmed] | |
| 15:07:50 | mriedem | tssurya: odyssey4me: superdan: jgwentworth: https://bugs.launchpad.net/grenade/+bug/1761775 | |
| 15:07:58 | mriedem | i listed 2 options in there for fixing grenade | |
| 15:08:13 | mriedem | trying to channel what sdague would say... | |
| 15:08:40 | tssurya | hmm question.. why wasn't this an issue before me adding a new column to cell_mappings ? | |
| 15:08:40 | fried_rice | mriedem: Care to +W https://review.openstack.org/#/c/555961/ (lower-constraints.txt) quick before we break it again? | |
| 15:09:12 | tssurya | I mean no other column change in api_db before has needed a api sync before db sync situation ? | |
| 15:09:29 | odyssey4me | mriedem yeah, that does seem a bit chicken-and-egg | |
| 15:09:55 | cdent | finucannot: +many on the -W stuff. pain now for less pain later | |
| 15:10:09 | mriedem | tssurya: we just haven't modified the cell_mappings table in forever | |
| 15:10:41 | tssurya | mriedem: cell_mappings yes, what about other tables ? | |
| 15:10:57 | mriedem | db sync doesn't hit other tables | |
| 15:11:01 | mriedem | in the api db | |
| 15:11:11 | jgwentworth | mriedem: interesting, I always thought the order was api_db first in the upgrade docs. I would have thought that one should be done first and I can't immediately think of why it's not | |
| 15:11:17 | superdan | mriedem: oh is it trying to read the cell mappings to do the cell databases before it has sync'd the api db? | |
| 15:11:27 | superdan | yeah I'm not sure why api isn't first | |
| 15:11:32 | finucannot | cdent: Yup, only way to do it, unfortunately | |
| 15:11:35 | mriedem | superdan: nova-manage db sync by default tries to also sync cell0- | |
| 15:11:37 | tssurya | mriedem: ah heh yes, anyways I thought api_sync was run before db sync until now | |
| 15:11:59 | superdan | mriedem: and it looks up the mapping for it before it has sync'd the api db? | |
| 15:12:18 | tssurya | superdan: yes | |
| 15:12:32 | mriedem | yeah | |
| 15:12:58 | superdan | er, I guess t's just that we call db sync before api_db sync in grenade you mean | |