Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-04-24
17:02:27 jroll dansmith: but anyway, I thought there was still an opportunity for a scheduler race, glad to here that's gone
17:02:31 dansmith melwitt: gotta make sure you have a post-spawn-failure method to unreserve too
17:02:38 TheJulia like, ironic ends up being a central broker of baremetal across datacenters *blink* *blink*
17:02:41 dansmith jroll: not if you're doing the resource class thang
17:03:20 melwitt dansmith: ah, yeah
17:03:20 jroll TheJulia: I'm aware people do crazy things with ironic, like using it to manage VMs, I'm not saying it doesn't happen. I'm just saying I'm not sure we should support nova being used with ironic without owning it :)
17:03:42 jroll dansmith: sweet.
17:03:59 TheJulia dansmith: I think we already rip the instance_uuid in the cleanup
17:04:04 TheJulia rip it off the node that is
17:04:18 openstackgerrit Ed Leafe proposed openstack/nova master: Address issues raised in adding member_of to GET /a-c https://review.openstack.org/554357
17:04:18 dansmith TheJulia: not if we fail before we called spawn I imagine
17:04:18 TheJulia so we should be covered if there is a failure
17:04:33 jroll dansmith: ++
17:04:44 dansmith TheJulia: I'm saying if we do a pre-spawn method that could have allocated resources, and then we fail to talk to glance or neutron, we have to make sure we let the virt driver know we were joking
17:04:46 TheJulia dansmith: that is a good point... I thought it still got called but maybe not
17:04:53 TheJulia oh jeeze, that is another way we could be orphaning vifs
17:06:22 jroll yeah, it does not, only if we fail spawn(). whomp whomp
17:06:43 jroll TheJulia: this ensures we don't orphan vifs: https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2198
17:06:43 TheJulia :(
17:07:22 melwitt TheJulia, jroll: so what would y'all rather do? pre-spawn and have to deal with failure cleanup cases or do something to allow the host binding to be done separately from plug_vifs? or another solution I'm not thinking of?
17:07:50 TheJulia jroll: if InstanceDeployFailure is added to https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2192 then that would prevent the case I'm seeing in a couple different cases
17:08:33 jroll melwitt: I don't think we can split the host-binding from the plug_vifs, really. I think either way we have to do a pre-spawn, this started with talking about a new method vs hacking it into plug_vifs :)
17:09:03 melwitt okay
17:09:03 TheJulia what jroll said :)
17:09:16 jroll TheJulia: that only comes from spawn(), right?
17:09:33 jroll maybe not :/
17:09:56 efried mriedem (or anyone): Are you aware of an recent changes in nova or elsewhere that would make instance.uuid stop responding to isinstance(str)?
17:10:17 melwitt yeah, I guess duh, the methods that were added have a cleanup with it already too https://review.openstack.org/#/c/468353/19/nova/virt/driver.py
17:10:20 TheJulia jroll: I just spotted https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2214
17:11:11 mriedem efried: nope
17:11:16 efried k
17:11:17 jroll efried: uuid.uuid4() or whatever will yield some object rather than a string, we tend to use uuidutils.generate<tab>, could be it
17:11:30 jroll TheJulia: ah, yeah
17:11:49 dansmith jroll: instance.uuid will coerce it to a string thouigh
17:11:57 jroll ah ok
17:12:17 efried yeah, see, that's what seems not to be happening anymore in a certain code path for us.
17:15:38 openstackgerrit melanie witt proposed openstack/nova stable/queens: DNM: testing the ceph job https://review.openstack.org/563175
17:16:02 gibi Kevin_Zheng: I have an approved bp that could be interesting for you if you have time: https://blueprints.launchpad.net/nova/+spec/add-full-traceback-to-error-notifications
17:17:11 gibi Kevin_Zheng: it is totally OK if you don't have time either, I just wanted advertise the opportunity
17:20:01 jroll melwitt: dansmith: jaypipes: thanks for the help, btw :)
17:20:21 openstackgerrit Merged openstack/nova master: Add periodic task to clean expired console tokens https://review.openstack.org/325381
17:20:40 jaypipes jroll: I feel like I did nothing but add confusion to the situation, honestly.
17:21:04 jroll jaypipes: thanks for trying to help? :P
17:21:05 dansmith jroll: jaypipes needs a hug
17:21:12 jroll or a pug
17:21:20 melwitt pug hug
17:21:26 dansmith or a pug hug
17:22:23 TheJulia Pug hugs sound far better than cat cuddles... and walking away bleeding
17:24:23 jroll jaypipes: do aussies also cheer you up? https://i.imgur.com/Yswxtpy.jpg
17:24:32 melwitt heh. not our cat. we think he might be part ragdoll
17:24:34 dansmith jroll: I think he only likes ugly dogs
17:24:42 jroll ha, fair
17:25:05 jaypipes jroll: ++ aussie's are a good cheerup.
17:25:11 jroll :)
17:25:18 melwitt jroll: that's adorbz
17:25:26 dansmith jaypipes: but. but.. their eyeballs are fully contained within their heads...
17:25:53 jroll melwitt: thanks, make sure you pass it along to james :)
17:25:59 TheJulia It is side conversations like ^^^ why I love this community
17:26:27 melwitt jroll: forwarding it now
17:27:03 jaypipes dansmith: details, details...
17:27:20 jaypipes dansmith: they are furry, fluffy and perfect.
17:27:31 jaypipes dansmith: even if they are missing googly eyes.
17:27:45 jaypipes heh
17:32:51 TheJulia jroll: Any thoughts on continuing your poc forward or just rewrite mine?
17:33:23 jroll TheJulia: I could do it, or you could take it over, or you could rewrite yours, whatever you'd like to do :)
17:34:56 TheJulia I've got it on the brain so I'll pickup yours and continue it along
17:36:23 jroll awesome, thanks! happy to review if needed
17:36:43 jroll TheJulia: fyi I probably picked up your bug from patchset 1 where you needed the patch wrapped in []
17:37:27 TheJulia my brain is currently trying to think of any other ways we could be leaking vifs and thus blocking future deploys outside of tearing down an instance and I'm not thinking of any
17:37:39 openstackgerrit Jay Pipes proposed openstack/os-traits master: Add compute capabilities traits https://review.openstack.org/546713
17:37:58 TheJulia jroll: easy fix :)
17:38:11 jaypipes artom_, gibi, efried: addressed the rewording in https://review.openstack.org/#/c/546713/.
17:38:14 melwitt mriedem: why do you think we can't drop the API code until T? I thought once all computes are upgraded to rocky, then things are guaranteed to all be using the database backend, so removal would be in Stein? https://review.openstack.org/#/c/325414/31/nova/compute/api.py@3613
17:38:25 jroll TheJulia: yep, just a heads up so you don't wait an hour for CI to find it :)
17:39:14 gibi jaypipes: thanks, +2
17:39:31 mriedem melwitt: i might have been overthinking it
17:40:18 melwitt k, just trying to make sure there's not something I'm missing
17:40:33 mriedem replied to clarify
17:41:16 melwitt cool thanks
17:43:32 sean-k-mooney melwitt: sorry i was in meeting for the last 4 hours. did ye resolve the issue ye were having?
17:44:11 melwitt sean-k-mooney: yeah, I think so. we were talking about, when is a port guaranteed to have an IP allocated to it? maybe never, but it sounded like after host binding it has to?
17:45:46 openstackgerrit Merged openstack/nova master: Extract generate_hostid method into utils.py https://review.openstack.org/555282
17:45:52 sean-k-mooney melwitt: right so with multi segment, l3 routed networks the allocation of the ip happens only when the port is bound to a specific host. for the general case the port will get an ip when it is created but you can have ports that only have mac adresses
17:46:36 melwitt sean-k-mooney: gotcha. thanks
17:46:37 sean-k-mooney melwitt: so after you do port binding it should have an ip if it is connected to a neutron network that has a subnet
17:47:28 openstackgerrit Balazs Gibizer proposed openstack/nova-specs master: Network bandwidth resource provider https://review.openstack.org/502306
17:49:26 melwitt sean-k-mooney: okay, so it's not ever necessarily guaranteed (would a neutron network ever not have a subnet?). we were looking at some code in the ironic driver where they need to get the IP in order to hand it to the volume connector (some volume backends require the IP before attaching). and we were wondering, at what point could one reliably query for the IP
17:50:42 sean-k-mooney melwitt: neutron allows your to have interfaces with no ips. nova used to require one for a neutron port to be used with an vm/instance but that requrement was removed.
17:51:03 sean-k-mooney melwitt: the reason it was removed was to cater for non ip networking e.g. infinaband over ethernet
17:51:07 melwitt I see
17:52:27 sean-k-mooney melwitt: it sound like in the ironic case there is a hard requirement for an ip presuable you are using it to configre iscsi targest for the baremtal not or something like that?
17:53:57 melwitt sean-k-mooney: yeah must be a hard requirement. it's already doing it like this https://review.openstack.org/#/c/468353/19/nova/virt/ironic/driver.py but yeah need the IP to set in the volume connector
17:55:07 sean-k-mooney in general l2 only connectivity is not common in neutron so this is an extream edgecase.
17:55:33 melwitt and the problem was that they have to do their equivalent of "plug_vifs" first in order to get the IP and that's happening outside their node reserve lock and there are races happening. so at first we were talking about "can we get the IP before plugging the vif" but it ended up not mattering and the solution needs to be to reserve the node first before doing anything with the networking
17:58:28 openstackgerrit Merged openstack/nova master: Address nits in I00d29e9fd80e6b8f7ba3bbd8e82dde9d4cb1522f https://review.openstack.org/561493
18:00:36 sean-k-mooney from a ironic perspective i think its prefectly resonable to scope the cinder volume support to jsut instance with ips at a first pass. volumes attached over infinaband is something in the storage space where we might not have ips but that is less common.
18:01:35 melwitt yeah, that's what they've done
18:05:10 openstackgerrit Merged openstack/nova stable/ocata: Add regression test for persisted RequestSpec.retry from failed resize https://review.openstack.org/560162
18:05:17 openstackgerrit Merged openstack/nova stable/ocata: Don't persist RequestSpec.retry https://review.openstack.org/560167
18:05:23 openstackgerrit Merged openstack/nova stable/ocata: only increment disk address unit for scsi devices https://review.openstack.org/561613

Earlier   Later