Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-03-07
19:46:52 mnaser the instance stays there forever, im pretty sure its just deleted by the api user and they're reattempting to launch the instance, so the behaviour is non deterministic
19:47:04 mriedem yeah so my guess is _do_build_and_run_instance set task_state=None and then something hung or failed
19:47:24 mriedem the instance.host gets set in the resource tracker after it's spawned
19:48:30 mriedem so networking or block device setup maybe failed? or driver.spawn failed
19:50:13 mriedem however, those should either result in a reschedule or the instance being put into ERROR state
19:50:34 mriedem mnaser: probably have to trace the server create request id through the compute logs to find out what happened to i
19:50:35 mriedem *it
19:50:54 mnaser mriedem: actually, i'm looking at recent instance_faults and im seeing some rpc timeouts
19:50:58 mriedem jaypipes: ok (on the VF/PF thing)
19:51:13 mnaser which could very well explain things.. ill dig more, because there's plenty of conductors serving here
19:51:17 mriedem mnaser: so it's possible the state changes to the instance never made it through conductor to the db
19:51:19 mnaser thanks for the small talk through, that gives a hint
19:51:31 mnaser File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2110, in _build_and_run_instance
19:51:31 mnaser instance.save(expected_task_state=task_states.SPAWNING)
19:51:41 mnaser it is *very* possible considering that's the stacktrace that timed out...
19:52:55 mriedem yup, and that's the time that the instance.host would be set and saved off from the RT.instance_claim()
19:54:10 mnaser well at least we know things are breaking the way they're supposed to be..
19:55:48 mnaser dansmith: i'm not sure, i'm worried about finding that part out :p
19:55:59 dansmith should be any day now.. :)
19:56:08 mnaser our queens upgrades are happening soon
19:56:31 mnaser i'm sure that'll push things
19:56:32 mnaser :P
19:56:43 mriedem mnaser: i'd wait for 17.0.1
19:56:44 dansmith heh
19:56:49 mriedem which should happen this week or early next
19:57:01 mnaser mriedem: taking your word for it, usually nova is one of the later targets for upgrades
19:57:10 mriedem mnaser: are you going to leave glance behind for v1 support?
19:57:15 mnaser we have a whole ceph cluster to upgrade just before that
19:57:45 mnaser mriedem: we have 1 client that heavily uses v1, and they use the openstack cloud jenkins plugin so i'm probably just going to push up a patch to use the v2 apis
19:58:00 mnaser and once i get them to use it, then we can move on
19:58:28 mriedem i was just thinking about this at the ptg,
19:58:43 mriedem but the direct rbd snapshot stuff for ephemeral rbd disks in the libvirt driver,
19:58:44 mnaser i think v1 still exists in queens though
19:58:57 mriedem that relies on being able to get the location from the image, which i thought was a v1 only thing in glance
19:59:04 mriedem nova dropped support for glance v1
19:59:21 mnaser direct image location or whatever that is
19:59:26 mnaser it's a thing in glance v2 i'm pretty sure
20:00:21 mriedem whatever this is https://github.com/openstack/devstack-plugin-ceph/blob/master/devstack/lib/ceph#L683
20:00:34 mnaser https://developer.openstack.org/api-ref/image/v2/index.html#show-image-details
20:00:42 mnaser yeah that is the direct_url and locations[] in the api there
20:01:17 mriedem ok i thought they weren't going to carry that forward to v2
20:01:36 mriedem but if they did, then i don't have to worry about us breaking the direct snapshot rbd stuff that a lot of people wanted and are using
20:03:02 mnaser mriedem: afaik we're using v2 with nova and direct snapshot is working here .. under pike at least
20:03:19 mriedem ok i'll not worry about this then
20:03:41 cfriesen mnaser: mriedem: if you enable "show_multiple_locations" then the direct snapshot stuff will work. it's just that glance tells you that turning that option on is a security risk
20:04:20 mriedem cfriesen: i know, i just thought it was a v1 only thing
20:04:24 hrw stephenfin: sorry but r14 got wrong in one place during merge and I did not noticed
20:04:32 mriedem nova dropped support for glance v1 in ocata, so it's further back than i thought
20:04:58 mnaser oh yeah, nope, its fine, glance says its a security risk but i'd argue with that in ceph's case .. but i don't have the time to discuss that :D
20:08:38 openstackgerrit Marcin Juszkiewicz proposed openstack/nova master: Allow to configure amount of PCIe ports https://review.openstack.org/545034
20:08:50 hrw stephenfin, mriedem: fixed rev
20:11:19 bauzas mriedem: dansmith: I'll be on PTO tomorrow afternoon UTC
20:11:23 mriedem artom: so -1 on https://review.openstack.org/#/c/485522/ since the spec doesn't say anything
20:11:35 bauzas in case you need me
20:11:45 mriedem bauzas: i need you right now
20:11:47 mriedem to remove your -W on https://review.openstack.org/#/c/550259/
20:12:03 artom And to stop forest fires?
20:12:10 artom Am I mixing up my references?
20:12:17 artom mriedem, fair, I expected nothing less
20:12:31 bauzas mriedem: just did
20:12:54 bauzas mriedem: I was looking at today's discussions
20:13:56 mriedem dansmith: can you +W https://review.openstack.org/#/c/550259/
20:14:00 mriedem that backport
20:14:01 artom mriedem, to be fair, maybe there just isn't that much to do in Nova for trusted VFs (if doing it without placement), in which case maybe it would have needed to be a Neutron spec?
20:14:15 mriedem artom: well, look at 'work items'
20:14:22 artom In any case, I still agree that we're better off tracking them in placement
20:14:23 mriedem "something something pci request spec, and libvirt driver, done!"
20:14:30 openstackgerrit Surya Seetharaman proposed openstack/nova-specs master: Support disabling a cell https://review.openstack.org/546684
20:14:41 mriedem don't look behind the curtain
20:14:54 artom But it's cloud!
20:15:07 artom You're not *supposed* to look behind the curtain
20:15:15 mriedem ew https://review.openstack.org/#/c/458513/11/nova/network/linux_net.py
20:15:36 mriedem ^ should happen in os-vif i'm pretty sure
20:15:39 artom The hell, we did that in nova-network?!
20:15:40 dansmith mriedem: yes
20:15:50 mriedem artom: that's common 'networking' code
20:15:57 artom Ah, sorry
20:16:31 mikal So are we deleting nova-network this release? Can I ignore that stuff for the purposes of privsep?
20:16:50 artom I think we said we're getting rid of cellsv1 first?
20:17:01 mriedem mikal: yes
20:17:07 mriedem artom: other way around
20:17:09 dansmith artom: other way around
20:17:10 dansmith heh
20:17:13 mriedem artom: you were there, in the room!
20:17:28 artom Oh so other way round?
20:17:31 mriedem so i think i get what the trusted VFs thing is now
20:17:31 mikal I'd like to be core for just the one review to +2 the removal of nova-net please. For old times sake.
20:17:43 mriedem 1. create port in neutron with a trusted attribute
20:17:56 mriedem 2. nova takes that off the port binding profile and shoves into the pci request spec,
20:18:19 artom (Oh right, your action item was to make cellsv1 work with Neutron)
20:18:26 mriedem 3. libvirt driver reads that off the pci request spec and shells out to set the vf to trusted on the host
20:18:42 mriedem why do we need the virt driver doing this?
20:18:49 dansmith mriedem: but in reality, it should be "2. nova takes that and shoves it into traits"
20:19:01 dansmith mriedem: then placement weeds everything we can't land on for us
20:19:18 mriedem dansmith: yeah that's what i was thinking the spec (and asked) - how to get go through scheduling to find a host that handles this type of port
20:19:23 mriedem since we don't have network-aware scheduling
20:19:33 mriedem jesus words
20:20:40 bauzas is efried still on PTO ?
20:21:10 bauzas I was considering to work on a spec for providing NUMA affinity using nested RPs, and AFAIR he told me there was already a spec
20:21:21 bauzas am I wrong?

Earlier   Later