Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
13:21:08 mriedem this isn't a failure on flavor resources, or shouldn't be, it's embedded flavors in the instance which were originally migrated from the instance system_metadata
13:21:57 mriedem this is the problem http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/flavors.py#n52
13:22:09 mriedem is_public isn't in there, so it wasn't stored in the embeded instance.flavor
13:22:15 mriedem so we'll have to default to is_public=True
13:22:20 mriedem if it's not in the embedded flavor
13:24:32 mriedem i left a comment on the bug
13:24:40 gibi mriedem: OK, I have to look at the API code showing the embeded flavor to see how is_public is handled there
13:24:50 mriedem i'll report a new bug for is_public since this other one is already closed
13:25:27 mriedem we don't show that field from the instance.flavor
13:25:58 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/views/servers.py#n349
13:26:56 gibi mriedem: then the API is not broken. cool. You are way faster to find these things in the code than me.
13:27:07 gibi mriedem: I agree that it needs a separate bug
13:27:24 gibi mriedem: thanks for reporting it
13:27:36 mriedem https://bugs.launchpad.net/nova/+bug/1789423
13:27:36 openstack Launchpad bug 1789423 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset is_public field" [Undecided,New]
13:28:17 mriedem lyarwood: speaking of which, can you hit this? https://review.openstack.org/#/c/580525/
13:32:02 lyarwood mriedem: yup looking
13:34:22 gibi jaypipes: thanks for the mail about the consumer gen handling in nova I think efried_doc's and your comment together helps me redo the patch series (now I have to find the time to do it).
13:42:04 jaypipes gibi: np
14:07:24 kosamara Hi efried, are you around?
14:09:59 openstackgerrit Jay Pipes proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:10:32 jaypipes stephenfin: can you +2/W https://review.openstack.org/#/c/597111/ please?
14:11:11 stephenfin jaypipes: Sure, done
14:11:36 jaypipes stephenfin: danke
14:11:48 jaypipes kosamara: ^^
14:11:54 jaypipes kosamara: thx mate
14:20:26 kosamara jaypipes: thanks!
14:23:48 kosamara efried: I'd like to take up your offer :) From the 4 major things to address on the spec, could you focus on 2,3,4 so that I can focus on 1 for libvirt?
14:24:54 kosamara efried: I'll also see if I can carry over content from your OOT spec for powervm there. And I think alex_xu's questions are part of the "cyborg intersection".
14:43:48 mriedem tommylikehu: you had a question about volumes?
14:44:05 tommylikehu yeah, I have a question regarding deleting operations, When deleting instance and its related volumes, there could be a period of time that those volumes' statues can be available, right? I mean right after the volume is detached
14:44:37 mriedem tommylikehu: i believe so yes
14:44:47 mriedem b/c you can't delete an in-use volume
14:45:05 mriedem i think you can force delete an in-use volume though....
14:46:01 mriedem but nova doesn't force delete and it's an admin-only API, and historically nova just relies on the user context token to do the detach/delete of the volume, we don't use configured admin credentials for that
14:46:45 tommylikehu we got a bug report from our customers saying that that period could be dangerous since other operations are not prohibitted
14:47:26 Tahvok Hey guys! Asked here before, but sean-k-mooney told me to wait 'till nova conductor cores show up online. We see some spikes of cpu usage of nova-conductor processes (every second or two): http://paste.openstack.org/show/728925/.
14:47:40 openstackgerrit Merged openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:48:33 efried kosamara: Hi, I'm here now. Ack, I'll write some words for 2,3,4.
14:48:54 efried kosamara: Would you like me to upload new patch sets to the spec, or dump the content somewhere for you to pull in?
14:49:42 tommylikehu mriedem: :)
14:49:42 Tahvok We have a big environment, with 160 compute hosts, and 2 controller (adding a third by the end of this week). I've enabled debugging on nova-conductor service, and saw that it's receiving around 30~120 messages per second. So I'm trying to investigate what could be throwing so much messages each second.. Our cloud is not very active, we have around 1 instance coming up/deleted every hour or so
14:50:23 kosamara efried: either works for me, but I think many patch sets can tangle up the discussion. Of course, this is my first spec, so you know.
14:51:07 efried kosamara: Meh, patch sets are no big deal; often they can help provide history/context of the evolution of a thing. As long as comments aren't lost along the way, it's fine.
14:51:14 mriedem tommylikehu: "since other operations are not prohibitted" ?
14:51:45 mriedem tommylikehu: meaning, the customer thinks something could reserve the volume in that split second while we're deleting the instance and then fail to delete the volume?
14:51:46 efried kosamara: Just need to avoid stepping on each other. So we should just both check in with each other before posting a new patch set to make sure the other doesn't have local edits pending.
14:51:48 mriedem and thus leave it orphaned
14:52:10 tommylikehu mriedem: yeah
14:52:19 mriedem tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?
14:52:32 kosamara efried: I've also got pending changes on the "done" things. I'll post them tomorrow.
14:53:16 efried kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in?
14:53:22 kosamara cool
14:53:31 mriedem tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server
14:54:04 tommylikehu mriedem: can we do something to protect this process?
14:54:23 tommylikehu mriedem: something like what we do when attaching volume
14:54:25 mriedem tommylikehu: is this an actual issue someone ran into or they are just doing some kind of audit?
14:55:04 mriedem and what client software is waiting a millisecond to attach a volume while we're deleting it from another server?
14:55:31 mriedem this seems extremely low priority
14:55:37 tommylikehu mriedem: personally I think it's the second case
14:55:49 mriedem ok; never tell them about ports then :)
14:56:04 mriedem because you can attach/detach those to servers out of band all you like
14:56:48 mriedem PUT /v2.0/ports/{port_id} with a new device_id - now it's my port yay!
14:57:37 mriedem tommylikehu: so like i said above, nova could use the force_delete API if we're configured with cinder admin creds, but i'd consider it very low priority
14:58:18 tommylikehu mriedem: oooook, thanks:)
15:00:30 mriedem tommylikehu: i guess start by reporting a bug to nova
15:00:33 mriedem so it's on the books
15:01:27 dansmith mriedem: tommylikehu the concern is that a volume nova is going to delete becomes available for a second before being deleted?
15:02:18 tommylikehu dansmith: yes
15:02:41 mriedem trump could get his grubby hands all of ma volumes
15:02:47 mriedem *all over
15:02:55 dansmith and what, something could attach those and block the delete?
15:03:02 mriedem yeah i guess
15:03:17 dansmith but the only entity that could do that already owns the thing
15:03:24 mriedem note: that doesn't block the server delete,
15:03:26 dansmith so, who cares?
15:03:34 mriedem and we already fail to delete the volume if it has snapshots
15:03:40 mriedem so yeah, this is like the lowest of priorities
15:04:01 dansmith or you could say it's working as designed and thus not a bug, which would be my preference
15:04:12 tommylikehu lol
15:04:39 mriedem that works for me
15:04:48 mriedem as i said above, "(9:52:21 AM) mriedem: tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?"
15:05:11 dansmith this would be like complaining that something that is able to hardlink the image file on disk before nova deletes it can still read the data
15:05:20 dansmith and the response to that is "yup. that's how that works"
15:05:46 dansmith I mean, just MHO of course
15:06:45 mriedem we have insanely bigger fish to fry so yes
15:06:50 mriedem like the port thing i already mentioned
15:07:33 dansmith how much bigger is insanely bigger?
15:07:40 dansmith like bigger^2?
15:10:30 mriedem hyperbole sized bigger
15:19:10 dansmith tssurya: mriedem: don't we want this to be under the big stack of down-cell patches so we can merge and backport it? https://review.openstack.org/#/c/592428/2
15:20:17 dansmith oh sorry I guess it is
15:20:18 dansmith nevermind
15:24:54 openstackgerrit Merged openstack/nova stable/ocata: Default embedded instance.flavor.disabled attribute https://review.openstack.org/580525
15:27:28 dansmith melwitt: if you want to hit this last patch in my series, it'll make the down cell stuff soon able to be based on master: https://review.openstack.org/#/c/594577/11
15:33:33 sean-k-mooney dansmith: Tahvok was asking about periodic spike in nova-conductor cpu usage in a cloud of ~160 compute nodes eairler. bejond the periodic jobs i was not sure what would be likely to cause the condoctor to be processing ~100 rpc messages a second. any toughts?
15:33:58 dansmith sean-k-mooney: sounds like a support (not dev) question
15:34:27 dansmith sean-k-mooney: but yeah, conductor pretty much just answers to nova-compute on an idle cloud, so it'd be periodics from compute nodes
15:34:54 sean-k-mooney dansmith: perhapes but do we expect 18 nova-condoctor worker treads to spike to 100% usage every 1-2 seconds http://paste.openstack.org/show/728925/
15:35:43 dansmith we should totally call it condoctor

Earlier   Later