Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-08-28
11:18:09 sean-k-mooney Tahvok: i unfortuetly dont have that data to hand.
11:24:32 sean-k-mooney Tahvok: we did some sacle testing back 2 years ago https://review.openstack.org/#/c/352101/ that im quickly checking.
11:25:59 sean-k-mooney Tahvok: that was done with a cloud with 3 contolers and about 230 compute nodes. if i remember correctly the total cpu usage on the contolers was in low 10% for the majority of the testing
12:02:08 stephenfin sean-k-mooney: bauzas would be my other suggestion but he's still on vacation
12:07:51 sean-k-mooney stephenfin: ya he came to mind. Tahvok i would suggest asking again in an hour or so. the us based cores that work on the conductor will be online then and perhaps can give a better answer
12:08:28 Tahvok sean-k-mooney: ok, thanks!
12:42:04 openstackgerrit Konstantinos Samaras-Tsakiris proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
12:47:38 openstackgerrit Merged openstack/nova master: Deprecate Core/Ram/DiskFilter https://review.openstack.org/596502
13:12:36 gibi mriedem: hi! Do you have topics for the today's notification subteam meeting?
13:13:17 mriedem nope
13:15:40 gibi mriedem: cool, then I will cancel
13:15:48 mriedem wfm
13:17:56 gibi mriedem: we got a report that not just the flavor.disabled is missing in some old embedded falvors but flavor.is_public as well. https://bugs.launchpad.net/nova/+bug/1739325 I will look at it a bit later but I guess a solution will be similar than for flavor.disabled
13:17:56 openstack Launchpad bug 1739325 in OpenStack Compute (nova) ocata "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
13:19:23 mriedem gibi: i suppose,
13:19:34 mriedem difference between disabled and is_public is disabled isn't in any API but is_public is
13:20:23 gibi so I have to check that what happens on the API if such old flavor is present and fix possible failures there as well
13:20:46 mriedem well,
13:21:08 mriedem this isn't a failure on flavor resources, or shouldn't be, it's embedded flavors in the instance which were originally migrated from the instance system_metadata
13:21:57 mriedem this is the problem http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/flavors.py#n52
13:22:09 mriedem is_public isn't in there, so it wasn't stored in the embeded instance.flavor
13:22:15 mriedem so we'll have to default to is_public=True
13:22:20 mriedem if it's not in the embedded flavor
13:24:32 mriedem i left a comment on the bug
13:24:40 gibi mriedem: OK, I have to look at the API code showing the embeded flavor to see how is_public is handled there
13:24:50 mriedem i'll report a new bug for is_public since this other one is already closed
13:25:27 mriedem we don't show that field from the instance.flavor
13:25:58 mriedem http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/views/servers.py#n349
13:26:56 gibi mriedem: then the API is not broken. cool. You are way faster to find these things in the code than me.
13:27:07 gibi mriedem: I agree that it needs a separate bug
13:27:24 gibi mriedem: thanks for reporting it
13:27:36 mriedem https://bugs.launchpad.net/nova/+bug/1789423
13:27:36 openstack Launchpad bug 1789423 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset is_public field" [Undecided,New]
13:28:17 mriedem lyarwood: speaking of which, can you hit this? https://review.openstack.org/#/c/580525/
13:32:02 lyarwood mriedem: yup looking
13:34:22 gibi jaypipes: thanks for the mail about the consumer gen handling in nova I think efried_doc's and your comment together helps me redo the patch series (now I have to find the time to do it).
13:42:04 jaypipes gibi: np
14:07:24 kosamara Hi efried, are you around?
14:09:59 openstackgerrit Jay Pipes proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:10:32 jaypipes stephenfin: can you +2/W https://review.openstack.org/#/c/597111/ please?
14:11:11 stephenfin jaypipes: Sure, done
14:11:36 jaypipes stephenfin: danke
14:11:48 jaypipes kosamara: ^^
14:11:54 jaypipes kosamara: thx mate
14:20:26 kosamara jaypipes: thanks!
14:23:48 kosamara efried: I'd like to take up your offer :) From the 4 major things to address on the spec, could you focus on 2,3,4 so that I can focus on 1 for libvirt?
14:24:54 kosamara efried: I'll also see if I can carry over content from your OOT spec for powervm there. And I think alex_xu's questions are part of the "cyborg intersection".
14:43:48 mriedem tommylikehu: you had a question about volumes?
14:44:05 tommylikehu yeah, I have a question regarding deleting operations, When deleting instance and its related volumes, there could be a period of time that those volumes' statues can be available, right? I mean right after the volume is detached
14:44:37 mriedem tommylikehu: i believe so yes
14:44:47 mriedem b/c you can't delete an in-use volume
14:45:05 mriedem i think you can force delete an in-use volume though....
14:46:01 mriedem but nova doesn't force delete and it's an admin-only API, and historically nova just relies on the user context token to do the detach/delete of the volume, we don't use configured admin credentials for that
14:46:45 tommylikehu we got a bug report from our customers saying that that period could be dangerous since other operations are not prohibitted
14:47:26 Tahvok Hey guys! Asked here before, but sean-k-mooney told me to wait 'till nova conductor cores show up online. We see some spikes of cpu usage of nova-conductor processes (every second or two): http://paste.openstack.org/show/728925/.
14:47:40 openstackgerrit Merged openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111
14:48:33 efried kosamara: Hi, I'm here now. Ack, I'll write some words for 2,3,4.
14:48:54 efried kosamara: Would you like me to upload new patch sets to the spec, or dump the content somewhere for you to pull in?
14:49:42 tommylikehu mriedem: :)
14:49:42 Tahvok We have a big environment, with 160 compute hosts, and 2 controller (adding a third by the end of this week). I've enabled debugging on nova-conductor service, and saw that it's receiving around 30~120 messages per second. So I'm trying to investigate what could be throwing so much messages each second.. Our cloud is not very active, we have around 1 instance coming up/deleted every hour or so
14:50:23 kosamara efried: either works for me, but I think many patch sets can tangle up the discussion. Of course, this is my first spec, so you know.
14:51:07 efried kosamara: Meh, patch sets are no big deal; often they can help provide history/context of the evolution of a thing. As long as comments aren't lost along the way, it's fine.
14:51:14 mriedem tommylikehu: "since other operations are not prohibitted" ?
14:51:45 mriedem tommylikehu: meaning, the customer thinks something could reserve the volume in that split second while we're deleting the instance and then fail to delete the volume?
14:51:46 efried kosamara: Just need to avoid stepping on each other. So we should just both check in with each other before posting a new patch set to make sure the other doesn't have local edits pending.
14:51:48 mriedem and thus leave it orphaned
14:52:10 tommylikehu mriedem: yeah
14:52:19 mriedem tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?
14:52:32 kosamara efried: I've also got pending changes on the "done" things. I'll post them tomorrow.
14:53:16 efried kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in?
14:53:22 kosamara cool
14:53:31 mriedem tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server
14:54:04 tommylikehu mriedem: can we do something to protect this process?
14:54:23 tommylikehu mriedem: something like what we do when attaching volume
14:54:25 mriedem tommylikehu: is this an actual issue someone ran into or they are just doing some kind of audit?
14:55:04 mriedem and what client software is waiting a millisecond to attach a volume while we're deleting it from another server?
14:55:31 mriedem this seems extremely low priority
14:55:37 tommylikehu mriedem: personally I think it's the second case
14:55:49 mriedem ok; never tell them about ports then :)
14:56:04 mriedem because you can attach/detach those to servers out of band all you like
14:56:48 mriedem PUT /v2.0/ports/{port_id} with a new device_id - now it's my port yay!
14:57:37 mriedem tommylikehu: so like i said above, nova could use the force_delete API if we're configured with cinder admin creds, but i'd consider it very low priority
14:58:18 tommylikehu mriedem: oooook, thanks:)
15:00:30 mriedem tommylikehu: i guess start by reporting a bug to nova
15:00:33 mriedem so it's on the books
15:01:27 dansmith mriedem: tommylikehu the concern is that a volume nova is going to delete becomes available for a second before being deleted?
15:02:18 tommylikehu dansmith: yes
15:02:41 mriedem trump could get his grubby hands all of ma volumes
15:02:47 mriedem *all over
15:02:55 dansmith and what, something could attach those and block the delete?
15:03:02 mriedem yeah i guess
15:03:17 dansmith but the only entity that could do that already owns the thing
15:03:24 mriedem note: that doesn't block the server delete,
15:03:26 dansmith so, who cares?
15:03:34 mriedem and we already fail to delete the volume if it has snapshots
15:03:40 mriedem so yeah, this is like the lowest of priorities
15:04:01 dansmith or you could say it's working as designed and thus not a bug, which would be my preference
15:04:12 tommylikehu lol
15:04:39 mriedem that works for me
15:04:48 mriedem as i said above, "(9:52:21 AM) mriedem: tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah?"

Earlier   Later