Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
13:40:54 cdent jroll: meh, if people want that kind of rope, maybe that's what the rope should do?
13:41:05 jroll cdent: not when they leave me to clean up the mess :P
13:41:14 cdent traits are effectively the same thing as labels in k8s, aren't they?
13:41:15 jroll I mean my specific downstream, to be clear :)
13:41:32 cdent that's a good point. If it's you on the hook to suffer, slap away
14:02:39 leakypipes finucannot: morning sir! :)
14:11:32 kashyap mriedem: I think you're the closest to ACK this: https://review.openstack.org/#/c/544238/ ("libvirt: Block swapping to an encrypted volume when using QEMU to decrypt")
14:11:39 kashyap Trivial diff-stat, though.
14:14:30 mriedem cdent: the other thing with giving that rope is people build solutions off it, and then when the time comes that we want to change anything related to it, we break those solutions
14:14:47 mriedem and if it's fairly nebulous we don't really know how anyone is using it
14:15:02 cdent traits are part of the (immutable?) api of placement
14:15:08 cdent so how it is used in placement is stuck
14:15:16 cdent how it is used in nova, though, that's different
14:15:33 mriedem with microversions the placement api isn't immutable
14:15:34 cdent (not disagreeing with you)
14:15:43 mriedem but yes i don't expect how placement handles traits to change much
14:15:50 cdent it is in the sense that we'll never raise the minimum
14:15:54 mriedem and yes how they are incorporated into nova is the messy part
14:16:08 mriedem or could be messy
14:16:27 mriedem right now traits-based scheduling from queens is pretty straight forward
14:16:48 mriedem 'put me on a host with trait=foo, got it'
14:22:30 mriedem mmedvede: i got a pkvm ci failure setting up devstack here https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/20/324720/5/check/tempest-dsvm-full-xenial/b59cb03/devstacklog.txt.gz
14:22:41 mriedem Connecting to cirros-builder.novalocal (cirros-builder.novalocal)|9.114.111.40|:80... failed: No route to host.
14:23:26 mriedem jgwentworth: fyi, i had emailed the virtuozzo team about their CI failing at least a week ago, so it's a known issue
14:23:42 mriedem ImagePropertiesFilter always kicks out whatever their trying to use for an image
14:25:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip placement on rebuild in same host https://review.openstack.org/546357
14:33:15 odyssey4me howdy everyone, thanks again for your help so far - we have an upgrade task failing which might be due to new changes in rocky... would this be a change in something that needs to happen? http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/result/b9fe5b27-00e6-455b-9a10-71b2046663c5/
14:33:33 edmondsw fyi, someone is trying to add suppz support to the libvirt driver, but seems to be doing that off on their own
14:33:36 edmondsw https://ask.openstack.org/en/question/113484/add-support-for-a-new-driver-in-nova/
14:33:45 edmondsw I tried to point them in the right direction
14:34:29 jgwentworth mriedem: okay, so virtuozzo CI currently broken and being worked on
14:36:13 mriedem lyarwood: so in https://review.openstack.org/#/c/544238/4/nova/virt/libvirt/driver.py we also can't do the swap if both volumes are encrypted?
14:36:52 mriedem jgwentworth: it's a known issue, i'm not sure how actively they are working to fix it
14:38:17 mriedem odyssey4me: that looks like the new 'disabled' colum in the nova_api.cell_mappings table, has the nova_api db been synced before running this?
14:38:27 lyarwood mriedem: correct
14:38:48 mriedem tssurya: ^
14:39:53 odyssey4me mriedem well, that's a queens to rocky upgrade, so yes - it was done in the queens deployment
14:40:18 kashyap lyarwood: Probably that should point also be documented as a note in the code? (With current limitations.)
14:40:27 mriedem odyssey4me: if you're upgrading to rocky and you already created the cell0 mapping in queens, you don't need to do it again
14:41:25 mriedem odyssey4me: although, map_cell0 should be smarter, it has a check for DBDuplicateEntry for running it when it already exists
14:41:43 jgwentworth mriedem: okay. would you say it's too soon to mail the ML to describe the issue and ask if anyone on their side can look into it?
14:42:09 mriedem jgwentworth: i emailed the vz CI email a couple of weeks ago, and got a reply this week, but deleted that reply
14:42:22 mriedem i'd give it maybe another week?
14:42:38 jgwentworth k, will do. thanks
14:42:57 mriedem odyssey4me: so i don't think we're getting the duplicate entry b/c it's trying to create the cell0 mapping in the api db, and that fails because the object code doesn't match the api db schema
14:43:31 mriedem odyssey4me: which tells me that nova-manage api_db sync hasn't been run on rocky yet
14:43:35 openstack Launchpad bug 1739593 in OpenStack Security Advisory "Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack (CVE-2017-18191)" [Undecided,Incomplete]
14:43:35 lyarwood mriedem: thanks, btw https://review.openstack.org/#/c/543569/ is the change covering this all prior to Queens where we would hit https://bugs.launchpad.net/nova/+bug/1739593, it would be great to land this in stable/ocata and pike sometime soon to close that all out.
14:44:04 mriedem lyarwood: i've tried to purge that from my memory
14:44:09 odyssey4me mriedem hmm, IIRC on an upgrade the steps are different - we have this now, so you're right: https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml
14:45:34 lyarwood mriedem: you and me both, definitely a mistake on my part to label this as a security issue
14:46:10 mriedem odyssey4me: so https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml is getting run all over again on an upgrade from queens to rocky?
14:46:49 odyssey4me mriedem yes, that sequence is run for greenfield, or upgrade - but the result of 'Get the api_db version' would be different if it was an upgrade
14:47:06 mriedem odyssey4me: oh i see the problem https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml#L29
14:47:16 mriedem yeah looking at http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/reports/68ffa953-cc0e-494b-89b9-dcf7ca4a8b59.html
14:47:29 mriedem so "Perform a Nova API DB sync" doesn't get run
14:47:30 mriedem that's a problem
14:47:53 odyssey4me ok, but that gets run later
14:48:03 odyssey4me IIRC this relates to the difference in procedure for greenfield vs upgrade
14:48:20 odyssey4me this might only have been an ocata/pike thing though, when cells v2 was all new
14:48:24 mriedem how did you upgrade from ocata to pike, or pike to queens?
14:48:34 mriedem i guess we haven't added a new cell mappings field sine then
14:48:36 mriedem so that's why
14:48:53 odyssey4me this same procedure is used - ocata->pike: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/pike/tasks/nova_db_setup.yml
14:49:11 odyssey4me pike->queens: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/queens/tasks/nova_db_setup.yml
14:49:29 mriedem yeah, we just have'nt added a new cell_mappings column in queens or pike
14:49:36 odyssey4me aha, ok
14:49:41 mriedem so, we can fix this in one of two ways:
14:49:56 odyssey4me so this conditional set is only important for the newton->ocata upgrade?
14:50:02 mriedem 1. nova-manage cell_v2 map_cell0 can do a db query for cell0 rather than rely on a DBDuplicateEntry
14:50:10 mriedem 2. you can run nova-manage api_db sync unconditionally
14:50:48 odyssey4me we're very happy to shorten and simplify the steps taken, so option 2 sounds good to me
14:51:29 odyssey4me I'll push a patch up shortly. I'll ping you to verify the review if that's ok?
14:51:31 mriedem fwiw, grenade also runs map_cell0 on upgrade and cell0 is already mapped http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_34_277
14:51:48 mriedem sure. i'm not ansible guy, but i can see if the non-voting upgrade job is green or not :)
14:52:33 odyssey4me :) hopefully we can make it as readable as a shell task procedure
14:52:42 mriedem oh looky here http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_25_031
14:52:49 mriedem grenade hits the same thing but doesn't fail for some reason
14:56:11 tssurya sorry was afk, that doesn't look goo
14:56:14 tssurya good*
14:57:40 odyssey4me mriedem maybe someone couldn't figure it out and added || true ;)
14:58:41 odyssey4me jokes aside, that's quite serious... but I'm glad we're not alone, and that we could find a red flag :)
14:58:48 mriedem it's hitting htis https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/upgrade.sh#L74
14:59:09 mriedem oh i know why it's not failing
14:59:17 mriedem the nova-manage command doesn't return a non-0
14:59:27 mriedem that's why it dumps out the breadcrumb
14:59:33 mriedem "Has "nova-manage api_db sync" been run?"
15:00:03 openstackgerrit Merged openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384
15:00:05 tssurya mriedem: is this because api DB was not synced before running db sync ?
15:00:26 mriedem yes
15:00:46 tssurya ah ok
15:04:47 mriedem the order in grenade for the db sync matches what we have in our upgrade docs too https://docs.openstack.org/nova/latest/user/upgrade.html#rolling-upgrade-process
15:07:50 openstack Launchpad bug 1761775 in grenade "InternalError during nova-manage db sync when upgrading to Rocky" [Undecided,Confirmed]
15:07:50 mriedem tssurya: odyssey4me: superdan: jgwentworth: https://bugs.launchpad.net/grenade/+bug/1761775
15:07:58 mriedem i listed 2 options in there for fixing grenade
15:08:13 mriedem trying to channel what sdague would say...
15:08:40 tssurya hmm question.. why wasn't this an issue before me adding a new column to cell_mappings ?
15:08:40 fried_rice mriedem: Care to +W https://review.openstack.org/#/c/555961/ (lower-constraints.txt) quick before we break it again?
15:09:12 tssurya I mean no other column change in api_db before has needed a api sync before db sync situation ?
15:09:29 odyssey4me mriedem yeah, that does seem a bit chicken-and-egg

Earlier   Later