Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
13:25:27 mriedem we don't show that field from the instance.flavor
13:25:58 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/views/servers.py#n349
13:26:56 gibi mriedem: then the API is not broken. cool. You are way faster to find these things in the code than me.
13:27:07 gibi mriedem: I agree that it needs a separate bug
13:27:24 gibi mriedem: thanks for reporting it
13:27:36 mriedem https://bugs.launchpad.net/nova/+bug/1789423
13:27:36 openstack Launchpad bug 1789423 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset is_public field" [Undecided,New]
13:28:17 mriedem lyarwood: speaking of which, can you hit this? https://review.openstack.org/#/c/580525/
13:32:02 lyarwood mriedem: yup looking
13:34:22 gibi jaypipes: thanks for the mail about the consumer gen handling in nova I think efried_doc's and your comment together helps me redo the patch series (now I have to find the time to do it).
13:42:04 jaypipes gibi: np
14:07:24 kosamara Hi efried, are you around?
14:09:59 openstackgerrit Jay Pipes proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:10:32 jaypipes stephenfin: can you +2/W https://review.openstack.org/#/c/597111/ please?
14:11:11 stephenfin jaypipes: Sure, done
14:11:36 jaypipes stephenfin: danke
14:11:48 jaypipes kosamara: ^^
14:11:54 jaypipes kosamara: thx mate
14:20:26 kosamara jaypipes: thanks!
14:23:48 kosamara efried: I'd like to take up your offer :) From the 4 major things to address on the spec, could you focus on 2,3,4 so that I can focus on 1 for libvirt?
14:24:54 kosamara efried: I'll also see if I can carry over content from your OOT spec for powervm there. And I think alex_xu's questions are part of the "cyborg intersection".
14:43:48 mriedem tommylikehu: you had a question about volumes?
14:44:05 tommylikehu yeah, I have a question regarding deleting operations, When deleting instance and its related volumes, there could be a period of time that those volumes' statues can be available, right? I mean right after the volume is detached
14:44:37 mriedem tommylikehu: i believe so yes
14:44:47 mriedem b/c you can't delete an in-use volume
14:45:05 mriedem i think you can force delete an in-use volume though....
14:46:01 mriedem but nova doesn't force delete and it's an admin-only API, and historically nova just relies on the user context token to do the detach/delete of the volume, we don't use configured admin credentials for that
14:46:45 tommylikehu we got a bug report from our customers saying that that period could be dangerous since other operations are not prohibitted
14:47:26 Tahvok Hey guys! Asked here before, but sean-k-mooney told me to wait 'till nova conductor cores show up online. We see some spikes of cpu usage of nova-conductor processes (every second or two): http://paste.openstack.org/show/728925/.
14:47:40 openstackgerrit Merged openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:48:33 efried kosamara: Hi, I'm here now. Ack, I'll write some words for 2,3,4.
14:48:54 efried kosamara: Would you like me to upload new patch sets to the spec, or dump the content somewhere for you to pull in?
14:49:42 tommylikehu mriedem: :)
14:49:42 Tahvok We have a big environment, with 160 compute hosts, and 2 controller (adding a third by the end of this week). I've enabled debugging on nova-conductor service, and saw that it's receiving around 30~120 messages per second. So I'm trying to investigate what could be throwing so much messages each second.. Our cloud is not very active, we have around 1 instance coming up/deleted every hour or so
14:50:23 kosamara efried: either works for me, but I think many patch sets can tangle up the discussion. Of course, this is my first spec, so you know.
14:51:07 efried kosamara: Meh, patch sets are no big deal; often they can help provide history/context of the evolution of a thing. As long as comments aren't lost along the way, it's fine.
14:51:14 mriedem tommylikehu: "since other operations are not prohibitted" ?
14:51:45 mriedem tommylikehu: meaning, the customer thinks something could reserve the volume in that split second while we're deleting the instance and then fail to delete the volume?
14:51:46 efried kosamara: Just need to avoid stepping on each other. So we should just both check in with each other before posting a new patch set to make sure the other doesn't have local edits pending.
14:51:48 mriedem and thus leave it orphaned
14:52:10 tommylikehu mriedem: yeah
14:52:19 mriedem tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?
14:52:32 kosamara efried: I've also got pending changes on the "done" things. I'll post them tomorrow.
14:53:16 efried kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in?
14:53:22 kosamara cool
14:53:31 mriedem tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server
14:54:04 tommylikehu mriedem: can we do something to protect this process?
14:54:23 tommylikehu mriedem: something like what we do when attaching volume
14:54:25 mriedem tommylikehu: is this an actual issue someone ran into or they are just doing some kind of audit?
14:55:04 mriedem and what client software is waiting a millisecond to attach a volume while we're deleting it from another server?
14:55:31 mriedem this seems extremely low priority
14:55:37 tommylikehu mriedem: personally I think it's the second case
14:55:49 mriedem ok; never tell them about ports then :)
14:56:04 mriedem because you can attach/detach those to servers out of band all you like
14:56:48 mriedem PUT /v2.0/ports/{port_id} with a new device_id - now it's my port yay!
14:57:37 mriedem tommylikehu: so like i said above, nova could use the force_delete API if we're configured with cinder admin creds, but i'd consider it very low priority
14:58:18 tommylikehu mriedem: oooook, thanks:)
15:00:30 mriedem tommylikehu: i guess start by reporting a bug to nova
15:00:33 mriedem so it's on the books
15:01:27 dansmith mriedem: tommylikehu the concern is that a volume nova is going to delete becomes available for a second before being deleted?
15:02:18 tommylikehu dansmith: yes
15:02:41 mriedem trump could get his grubby hands all of ma volumes
15:02:47 mriedem *all over
15:02:55 dansmith and what, something could attach those and block the delete?
15:03:02 mriedem yeah i guess
15:03:17 dansmith but the only entity that could do that already owns the thing
15:03:24 mriedem note: that doesn't block the server delete,
15:03:26 dansmith so, who cares?
15:03:34 mriedem and we already fail to delete the volume if it has snapshots
15:03:40 mriedem so yeah, this is like the lowest of priorities
15:04:01 dansmith or you could say it's working as designed and thus not a bug, which would be my preference
15:04:12 tommylikehu lol
15:04:39 mriedem that works for me
15:04:48 mriedem as i said above, "(9:52:21 AM) mriedem: tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?"
15:05:11 dansmith this would be like complaining that something that is able to hardlink the image file on disk before nova deletes it can still read the data
15:05:20 dansmith and the response to that is "yup. that's how that works"
15:05:46 dansmith I mean, just MHO of course
15:06:45 mriedem we have insanely bigger fish to fry so yes
15:06:50 mriedem like the port thing i already mentioned
15:07:33 dansmith how much bigger is insanely bigger?
15:07:40 dansmith like bigger^2?
15:10:30 mriedem hyperbole sized bigger
15:19:10 dansmith tssurya: mriedem: don't we want this to be under the big stack of down-cell patches so we can merge and backport it? https://review.openstack.org/#/c/592428/2
15:20:17 dansmith oh sorry I guess it is
15:20:18 dansmith nevermind
15:24:54 openstackgerrit Merged openstack/nova stable/ocata: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580525
15:27:28 dansmith melwitt: if you want to hit this last patch in my series, it'll make the down cell stuff soon able to be based on master: https://review.openstack.org/#/c/594577/11
15:33:33 sean-k-mooney dansmith: Tahvok was asking about periodic spike in nova-conductor cpu usage in a cloud of ~160 compute nodes eairler. bejond the periodic jobs i was not sure what would be likely to cause the condoctor to be processing ~100 rpc messages a second. any toughts?
15:33:58 dansmith sean-k-mooney: sounds like a support (not dev) question
15:34:27 dansmith sean-k-mooney: but yeah, conductor pretty much just answers to nova-compute on an idle cloud, so it'd be periodics from compute nodes
15:34:54 sean-k-mooney dansmith: perhapes but do we expect 18 nova-condoctor worker treads to spike to 100% usage every 1-2 seconds http://paste.openstack.org/show/728925/
15:35:43 dansmith we should totally call it condoctor
15:35:59 sean-k-mooney :)
15:37:09 Tahvok lol
15:37:13 dansmith I assume that was a rhetorical question, but 160 computes configured to run some periodics every minute could certainly generate a fair bit of traffic
15:37:39 sean-k-mooney i was wondering if we should consider introducing intetional jitter in the perodic jobs to maybe spread when the jobs are running on each node
15:37:50 dansmith that is already done
15:37:52 Tahvok dansmith: but 100 messages a second? Don't they sync every minute or so? That should generate around 2.5 message a second.. Not a 100..
15:38:04 openstackgerrit Merged openstack/nova master: Make monkey patch work in uWSGI mode https://review.openstack.org/592285
15:38:27 dansmith Tahvok: they sync as often as you have them configured for.. without knowing what the messages are, who is sending them, etc, it's hard to say what the problem is

Earlier   Later