Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-06
13:31:02 mriedem so i'm not sure how we leverage placement-req-filter with ironic node aggregates in placement yet
13:31:07 sean-k-mooney jroll: no worries just giving one option. i think we agree this is a valid usecase that should be supported. how is a question that can be adressed later
13:31:54 mriedem not sure if traits would get us there, traits are the new metadata bastard child
13:33:12 openstackgerrit Tyler Blakeslee proposed openstack/nova master: Use six.text_type instead of str in NovaException __repr__ https://review.openstack.org/559169
13:33:16 jroll mriedem: yeah, makes sense, thank you
13:33:47 sean-k-mooney mriedem: i think jay would argue this is not a qualitive capablity of the resouce accounting so traits are not the best way but use it would work with traits
13:34:15 sean-k-mooney e.g. custom_rack1 but again that is not user facing
13:37:31 mriedem jroll: i think i get what you meant by "alloc candidates aren't exposed to the user" - in that there is no user-facing compute API to translate a request for node aggregates (in placement) to the nova scheduler
13:38:34 cdent mriedem: traits can take us anywhere we want to go, if we're willing. Such is the danger of traits.
13:38:52 finucannot leakypipes: I was literally just tab completing your name. Morning o/
13:38:56 jroll mriedem: yes, exactly. maybe there's a new api eventually, maybe instance groups affinity moves to use placement aggregates, maybe it's traits. dunno, need to think about it
13:39:13 finucannot leakypipes: Does this sound like something you could tackle today? https://review.openstack.org/#/c/553072/
13:39:30 mriedem jroll: yeah, needs some thought - gut feeling is traits isn't the answer here
13:39:34 jroll agree
13:39:36 mriedem munging aggregates with traits i mean
13:39:54 mriedem cdent: yes that's readily apparent already :)
13:40:03 mriedem bbiab
13:40:54 cdent jroll: meh, if people want that kind of rope, maybe that's what the rope should do?
13:41:05 jroll cdent: not when they leave me to clean up the mess :P
13:41:14 cdent traits are effectively the same thing as labels in k8s, aren't they?
13:41:15 jroll I mean my specific downstream, to be clear :)
13:41:32 cdent that's a good point. If it's you on the hook to suffer, slap away
14:02:39 leakypipes finucannot: morning sir! :)
14:11:32 kashyap mriedem: I think you're the closest to ACK this: https://review.openstack.org/#/c/544238/ ("libvirt: Block swapping to an encrypted volume when using QEMU to decrypt")
14:11:39 kashyap Trivial diff-stat, though.
14:14:30 mriedem cdent: the other thing with giving that rope is people build solutions off it, and then when the time comes that we want to change anything related to it, we break those solutions
14:14:47 mriedem and if it's fairly nebulous we don't really know how anyone is using it
14:15:02 cdent traits are part of the (immutable?) api of placement
14:15:08 cdent so how it is used in placement is stuck
14:15:16 cdent how it is used in nova, though, that's different
14:15:33 mriedem with microversions the placement api isn't immutable
14:15:34 cdent (not disagreeing with you)
14:15:43 mriedem but yes i don't expect how placement handles traits to change much
14:15:50 cdent it is in the sense that we'll never raise the minimum
14:15:54 mriedem and yes how they are incorporated into nova is the messy part
14:16:08 mriedem or could be messy
14:16:27 mriedem right now traits-based scheduling from queens is pretty straight forward
14:16:48 mriedem 'put me on a host with trait=foo, got it'
14:22:30 mriedem mmedvede: i got a pkvm ci failure setting up devstack here https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/nova/20/324720/5/check/tempest-dsvm-full-xenial/b59cb03/devstacklog.txt.gz
14:22:41 mriedem Connecting to cirros-builder.novalocal (cirros-builder.novalocal)|9.114.111.40|:80... failed: No route to host.
14:23:26 mriedem jgwentworth: fyi, i had emailed the virtuozzo team about their CI failing at least a week ago, so it's a known issue
14:23:42 mriedem ImagePropertiesFilter always kicks out whatever their trying to use for an image
14:25:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Skip placement on rebuild in same host https://review.openstack.org/546357
14:33:15 odyssey4me howdy everyone, thanks again for your help so far - we have an upgrade task failing which might be due to new changes in rocky... would this be a change in something that needs to happen? http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/result/b9fe5b27-00e6-455b-9a10-71b2046663c5/
14:33:33 edmondsw fyi, someone is trying to add suppz support to the libvirt driver, but seems to be doing that off on their own
14:33:36 edmondsw https://ask.openstack.org/en/question/113484/add-support-for-a-new-driver-in-nova/
14:33:45 edmondsw I tried to point them in the right direction
14:34:29 jgwentworth mriedem: okay, so virtuozzo CI currently broken and being worked on
14:36:13 mriedem lyarwood: so in https://review.openstack.org/#/c/544238/4/nova/virt/libvirt/driver.py we also can't do the swap if both volumes are encrypted?
14:36:52 mriedem jgwentworth: it's a known issue, i'm not sure how actively they are working to fix it
14:38:17 mriedem odyssey4me: that looks like the new 'disabled' colum in the nova_api.cell_mappings table, has the nova_api db been synced before running this?
14:38:27 lyarwood mriedem: correct
14:38:48 mriedem tssurya: ^
14:39:53 odyssey4me mriedem well, that's a queens to rocky upgrade, so yes - it was done in the queens deployment
14:40:18 kashyap lyarwood: Probably that should point also be documented as a note in the code? (With current limitations.)
14:40:27 mriedem odyssey4me: if you're upgrading to rocky and you already created the cell0 mapping in queens, you don't need to do it again
14:41:25 mriedem odyssey4me: although, map_cell0 should be smarter, it has a check for DBDuplicateEntry for running it when it already exists
14:41:43 jgwentworth mriedem: okay. would you say it's too soon to mail the ML to describe the issue and ask if anyone on their side can look into it?
14:42:09 mriedem jgwentworth: i emailed the vz CI email a couple of weeks ago, and got a reply this week, but deleted that reply
14:42:22 mriedem i'd give it maybe another week?
14:42:38 jgwentworth k, will do. thanks
14:42:57 mriedem odyssey4me: so i don't think we're getting the duplicate entry b/c it's trying to create the cell0 mapping in the api db, and that fails because the object code doesn't match the api db schema
14:43:31 mriedem odyssey4me: which tells me that nova-manage api_db sync hasn't been run on rocky yet
14:43:35 openstack Launchpad bug 1739593 in OpenStack Security Advisory "Swapping encrypted volumes can lead to data loss and a possible compute host DOS attack (CVE-2017-18191)" [Undecided,Incomplete]
14:43:35 lyarwood mriedem: thanks, btw https://review.openstack.org/#/c/543569/ is the change covering this all prior to Queens where we would hit https://bugs.launchpad.net/nova/+bug/1739593, it would be great to land this in stable/ocata and pike sometime soon to close that all out.
14:44:04 mriedem lyarwood: i've tried to purge that from my memory
14:44:09 odyssey4me mriedem hmm, IIRC on an upgrade the steps are different - we have this now, so you're right: https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml
14:45:34 lyarwood mriedem: you and me both, definitely a mistake on my part to label this as a security issue
14:46:10 mriedem odyssey4me: so https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml is getting run all over again on an upgrade from queens to rocky?
14:46:49 odyssey4me mriedem yes, that sequence is run for greenfield, or upgrade - but the result of 'Get the api_db version' would be different if it was an upgrade
14:47:06 mriedem odyssey4me: oh i see the problem https://github.com/openstack/openstack-ansible-os_nova/blob/e3e7c1f00bd7155dc051ba093fc872751776f769/tasks/nova_db_setup.yml#L29
14:47:16 mriedem yeah looking at http://logs.openstack.org/78/558078/2/check/openstack-ansible-upgrade-ubuntu-xenial-nv/3dc2358/logs/ara-report/reports/68ffa953-cc0e-494b-89b9-dcf7ca4a8b59.html
14:47:29 mriedem so "Perform a Nova API DB sync" doesn't get run
14:47:30 mriedem that's a problem
14:47:53 odyssey4me ok, but that gets run later
14:48:03 odyssey4me IIRC this relates to the difference in procedure for greenfield vs upgrade
14:48:20 odyssey4me this might only have been an ocata/pike thing though, when cells v2 was all new
14:48:24 mriedem how did you upgrade from ocata to pike, or pike to queens?
14:48:34 mriedem i guess we haven't added a new cell mappings field sine then
14:48:36 mriedem so that's why
14:48:53 odyssey4me this same procedure is used - ocata->pike: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/pike/tasks/nova_db_setup.yml
14:49:11 odyssey4me pike->queens: https://github.com/openstack/openstack-ansible-os_nova/blob/stable/queens/tasks/nova_db_setup.yml
14:49:29 mriedem yeah, we just have'nt added a new cell_mappings column in queens or pike
14:49:36 odyssey4me aha, ok
14:49:41 mriedem so, we can fix this in one of two ways:
14:49:56 odyssey4me so this conditional set is only important for the newton->ocata upgrade?
14:50:02 mriedem 1. nova-manage cell_v2 map_cell0 can do a db query for cell0 rather than rely on a DBDuplicateEntry
14:50:10 mriedem 2. you can run nova-manage api_db sync unconditionally
14:50:48 odyssey4me we're very happy to shorten and simplify the steps taken, so option 2 sounds good to me
14:51:29 odyssey4me I'll push a patch up shortly. I'll ping you to verify the review if that's ok?
14:51:31 mriedem fwiw, grenade also runs map_cell0 on upgrade and cell0 is already mapped http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_34_277
14:51:48 mriedem sure. i'm not ansible guy, but i can see if the non-voting upgrade job is green or not :)
14:52:33 odyssey4me :) hopefully we can make it as readable as a shell task procedure
14:52:42 mriedem oh looky here http://logs.openstack.org/38/544238/4/check/neutron-grenade/d101bba/logs/grenade.sh.txt.gz#_2018-04-05_09_41_25_031
14:52:49 mriedem grenade hits the same thing but doesn't fail for some reason
14:56:11 tssurya sorry was afk, that doesn't look goo
14:56:14 tssurya good*
14:57:40 odyssey4me mriedem maybe someone couldn't figure it out and added || true ;)
14:58:41 odyssey4me jokes aside, that's quite serious... but I'm glad we're not alone, and that we could find a red flag :)
14:58:48 mriedem it's hitting htis https://github.com/openstack-dev/grenade/blob/master/projects/60_nova/upgrade.sh#L74

Earlier   Later