| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-28 | |||
| 11:05:20 | Tahvok | And we don't have ceilometer, so rabbit handles only the basic openstack services | |
| 11:09:23 | sean-k-mooney | they do. but that does not seam that high to me for a cloud of your size. | |
| 11:13:10 | Tahvok | sean-k-mooney: so 100+ messages a second is normal for this size? | |
| 11:18:09 | sean-k-mooney | Tahvok: i unfortuetly dont have that data to hand. | |
| 11:24:32 | sean-k-mooney | Tahvok: we did some sacle testing back 2 years ago https://review.openstack.org/#/c/352101/ that im quickly checking. | |
| 11:25:59 | sean-k-mooney | Tahvok: that was done with a cloud with 3 contolers and about 230 compute nodes. if i remember correctly the total cpu usage on the contolers was in low 10% for the majority of the testing | |
| 12:02:08 | stephenfin | sean-k-mooney: bauzas would be my other suggestion but he's still on vacation | |
| 12:07:51 | sean-k-mooney | stephenfin: ya he came to mind. Tahvok i would suggest asking again in an hour or so. the us based cores that work on the conductor will be online then and perhaps can give a better answer | |
| 12:08:28 | Tahvok | sean-k-mooney: ok, thanks! | |
| 12:42:04 | openstackgerrit | Konstantinos Samaras-Tsakiris proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111 | |
| 12:47:38 | openstackgerrit | Merged openstack/nova master: Deprecate Core/Ram/DiskFilter https://review.openstack.org/596502 | |
| 13:12:36 | gibi | mriedem: hi! Do you have topics for the today's notification subteam meeting? | |
| 13:13:17 | mriedem | nope | |
| 13:15:40 | gibi | mriedem: cool, then I will cancel | |
| 13:15:48 | mriedem | wfm | |
| 13:17:56 | gibi | mriedem: we got a report that not just the flavor.disabled is missing in some old embedded falvors but flavor.is_public as well. https://bugs.launchpad.net/nova/+bug/1739325 I will look at it a bit later but I guess a solution will be similar than for flavor.disabled | |
| 13:17:56 | openstack | Launchpad bug 1739325 in OpenStack Compute (nova) ocata "Server operations fail to complete with versioned notifications if payload contains unset non-nullable fields" [Medium,In progress] - Assigned to Matt Riedemann (mriedem) | |
| 13:19:23 | mriedem | gibi: i suppose, | |
| 13:19:34 | mriedem | difference between disabled and is_public is disabled isn't in any API but is_public is | |
| 13:20:23 | gibi | so I have to check that what happens on the API if such old flavor is present and fix possible failures there as well | |
| 13:20:46 | mriedem | well, | |
| 13:21:08 | mriedem | this isn't a failure on flavor resources, or shouldn't be, it's embedded flavors in the instance which were originally migrated from the instance system_metadata | |
| 13:21:57 | mriedem | this is the problem http://git.openstack.org/cgit/openstack/nova/tree/nova/compute/flavors.py#n52 | |
| 13:22:09 | mriedem | is_public isn't in there, so it wasn't stored in the embeded instance.flavor | |
| 13:22:15 | mriedem | so we'll have to default to is_public=True | |
| 13:22:20 | mriedem | if it's not in the embedded flavor | |
| 13:24:32 | mriedem | i left a comment on the bug | |
| 13:24:40 | gibi | mriedem: OK, I have to look at the API code showing the embeded flavor to see how is_public is handled there | |
| 13:24:50 | mriedem | i'll report a new bug for is_public since this other one is already closed | |
| 13:25:27 | mriedem | we don't show that field from the instance.flavor | |
| 13:25:58 | mriedem | http://git.openstack.org/cgit/openstack/nova/tree/nova/api/openstack/compute/views/servers.py#n349 | |
| 13:26:56 | gibi | mriedem: then the API is not broken. cool. You are way faster to find these things in the code than me. | |
| 13:27:07 | gibi | mriedem: I agree that it needs a separate bug | |
| 13:27:24 | gibi | mriedem: thanks for reporting it | |
| 13:27:36 | mriedem | https://bugs.launchpad.net/nova/+bug/1789423 | |
| 13:27:36 | openstack | Launchpad bug 1789423 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications if payload contains unset is_public field" [Undecided,New] | |
| 13:28:17 | mriedem | lyarwood: speaking of which, can you hit this? https://review.openstack.org/#/c/580525/ | |
| 13:32:02 | lyarwood | mriedem: yup looking | |
| 13:34:22 | gibi | jaypipes: thanks for the mail about the consumer gen handling in nova I think efried_doc's and your comment together helps me redo the patch series (now I have to find the time to do it). | |
| 13:42:04 | jaypipes | gibi: np | |
| 14:07:24 | kosamara | Hi efried, are you around? | |
| 14:09:59 | openstackgerrit | Jay Pipes proposed openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111 | |
| 14:10:32 | jaypipes | stephenfin: can you +2/W https://review.openstack.org/#/c/597111/ please? | |
| 14:11:11 | stephenfin | jaypipes: Sure, done | |
| 14:11:36 | jaypipes | stephenfin: danke | |
| 14:11:48 | jaypipes | kosamara: ^^ | |
| 14:11:54 | jaypipes | kosamara: thx mate | |
| 14:20:26 | kosamara | jaypipes: thanks! | |
| 14:23:48 | kosamara | efried: I'd like to take up your offer :) From the 4 major things to address on the spec, could you focus on 2,3,4 so that I can focus on 1 for libvirt? | |
| 14:24:54 | kosamara | efried: I'll also see if I can carry over content from your OOT spec for powervm there. And I think alex_xu's questions are part of the "cyborg intersection". | |
| 14:43:48 | mriedem | tommylikehu: you had a question about volumes? | |
| 14:44:05 | tommylikehu | yeah, I have a question regarding deleting operations, When deleting instance and its related volumes, there could be a period of time that those volumes' statues can be available, right? I mean right after the volume is detached | |
| 14:44:37 | mriedem | tommylikehu: i believe so yes | |
| 14:44:47 | mriedem | b/c you can't delete an in-use volume | |
| 14:45:05 | mriedem | i think you can force delete an in-use volume though.... | |
| 14:46:01 | mriedem | but nova doesn't force delete and it's an admin-only API, and historically nova just relies on the user context token to do the detach/delete of the volume, we don't use configured admin credentials for that | |
| 14:46:45 | tommylikehu | we got a bug report from our customers saying that that period could be dangerous since other operations are not prohibitted | |
| 14:47:26 | Tahvok | Hey guys! Asked here before, but sean-k-mooney told me to wait 'till nova conductor cores show up online. We see some spikes of cpu usage of nova-conductor processes (every second or two): http://paste.openstack.org/show/728925/. | |
| 14:47:40 | openstackgerrit | Merged openstack/os-traits master: Add CUDA versions 8 and 9 https://review.openstack.org/597111 | |
| 14:48:33 | efried | kosamara: Hi, I'm here now. Ack, I'll write some words for 2,3,4. | |
| 14:48:54 | efried | kosamara: Would you like me to upload new patch sets to the spec, or dump the content somewhere for you to pull in? | |
| 14:49:42 | tommylikehu | mriedem: :) | |
| 14:49:42 | Tahvok | We have a big environment, with 160 compute hosts, and 2 controller (adding a third by the end of this week). I've enabled debugging on nova-conductor service, and saw that it's receiving around 30~120 messages per second. So I'm trying to investigate what could be throwing so much messages each second.. Our cloud is not very active, we have around 1 instance coming up/deleted every hour or so | |
| 14:50:23 | kosamara | efried: either works for me, but I think many patch sets can tangle up the discussion. Of course, this is my first spec, so you know. | |
| 14:51:07 | efried | kosamara: Meh, patch sets are no big deal; often they can help provide history/context of the evolution of a thing. As long as comments aren't lost along the way, it's fine. | |
| 14:51:14 | mriedem | tommylikehu: "since other operations are not prohibitted" ? | |
| 14:51:45 | mriedem | tommylikehu: meaning, the customer thinks something could reserve the volume in that split second while we're deleting the instance and then fail to delete the volume? | |
| 14:51:46 | efried | kosamara: Just need to avoid stepping on each other. So we should just both check in with each other before posting a new patch set to make sure the other doesn't have local edits pending. | |
| 14:51:48 | mriedem | and thus leave it orphaned | |
| 14:52:10 | tommylikehu | mriedem: yeah | |
| 14:52:19 | mriedem | tommylikehu: if something else reserves the volume in the interim, then clearly it wanted it yeah? | |
| 14:52:32 | kosamara | efried: I've also got pending changes on the "done" things. I'll post them tomorrow. | |
| 14:53:16 | efried | kosamara: Okay. How about I just compose content and post it to a pastebin for you to pull in? | |
| 14:53:22 | kosamara | cool | |
| 14:53:31 | mriedem | tommylikehu: unless you're aware of some other way to atomically delete a volume, this is just a known issue. the only thing i can see nova doing is using force_delete with cinder admin creds (if nova is configured for those) to delete the volume while it's attached to the server | |
| 14:54:04 | tommylikehu | mriedem: can we do something to protect this process? | |
| 14:54:23 | tommylikehu | mriedem: something like what we do when attaching volume | |
| 14:54:25 | mriedem | tommylikehu: is this an actual issue someone ran into or they are just doing some kind of audit? | |
| 14:55:04 | mriedem | and what client software is waiting a millisecond to attach a volume while we're deleting it from another server? | |
| 14:55:31 | mriedem | this seems extremely low priority | |
| 14:55:37 | tommylikehu | mriedem: personally I think it's the second case | |
| 14:55:49 | mriedem | ok; never tell them about ports then :) | |
| 14:56:04 | mriedem | because you can attach/detach those to servers out of band all you like | |
| 14:56:48 | mriedem | PUT /v2.0/ports/{port_id} with a new device_id - now it's my port yay! | |
| 14:57:37 | mriedem | tommylikehu: so like i said above, nova could use the force_delete API if we're configured with cinder admin creds, but i'd consider it very low priority | |
| 14:58:18 | tommylikehu | mriedem: oooook, thanks:) | |
| 15:00:30 | mriedem | tommylikehu: i guess start by reporting a bug to nova | |
| 15:00:33 | mriedem | so it's on the books | |
| 15:01:27 | dansmith | mriedem: tommylikehu the concern is that a volume nova is going to delete becomes available for a second before being deleted? | |
| 15:02:18 | tommylikehu | dansmith: yes | |
| 15:02:41 | mriedem | trump could get his grubby hands all of ma volumes | |
| 15:02:47 | mriedem | *all over | |
| 15:02:55 | dansmith | and what, something could attach those and block the delete? | |
| 15:03:02 | mriedem | yeah i guess | |
| 15:03:17 | dansmith | but the only entity that could do that already owns the thing | |
| 15:03:24 | mriedem | note: that doesn't block the server delete, | |
| 15:03:26 | dansmith | so, who cares? | |
| 15:03:34 | mriedem | and we already fail to delete the volume if it has snapshots | |
| 15:03:40 | mriedem | so yeah, this is like the lowest of priorities | |
| 15:04:01 | dansmith | or you could say it's working as designed and thus not a bug, which would be my preference | |