| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-04 | |||
| 15:16:02 | kashyap | gibi: Not that I know of | |
| 15:16:08 | gibi | OK. thanks for joining | |
| 15:16:53 | kashyap | I'm curious to hear stephenfin's topic of "documentation is awful" (line-474) - please take some notes on it on the Etherpad :) | |
| 15:17:39 | gibi | kashyap: ack | |
| 15:30:23 | sean-k-mooney | https://docs.openstack.org/nova/latest/configuration/config.html#filter_scheduler.max_io_ops_per_host | |
| 15:56:20 | melwitt | sean-k-mooney: just curious, how can you know there's not enough swap configured before any instances are booted so you know the flavor? | |
| 15:56:29 | CeeMac | hi sean-k-mooney and melwitt ; regarding or chat yesterday around shelving, we've done some testing today and it would appear that even in shelved_offloaded state an instance is accumulating usage and the resources still appear allocated in the project quota | |
| 15:57:09 | CeeMac | im back to being perplexed | |
| 15:57:24 | melwitt | CeeMac: oh... this is sounding familiar :( | |
| 15:57:43 | CeeMac | well, that doesnt sound good :/ | |
| 15:58:06 | CeeMac | i'm running rocky btw | |
| 15:58:51 | melwitt | long ago, when the shelve API was added I am recalling that we don't release quota while instance is shelved. but I didn't realize that would cause it to count as the simple tenant usage. but it makes sense that it would | |
| 15:59:27 | melwitt | the reasoning behind that decision was, not to let the user ever be in a position where they cannot unshelve for lack of quota | |
| 16:00:46 | CeeMac | yeah, i was actually having that conversation with my colleagues around implications to consider when shelving | |
| 16:01:40 | CeeMac | i guess, at least in the way I'm ingesting the data, it depends on how horizon is building its usage information for Admin | Overview | Usage summary | |
| 16:03:56 | CeeMac | let me see if I can pull some data from the openstack cli | |
| 16:16:53 | CeeMac | apparently I cant /fail | |
| 16:17:24 | melwitt | CeeMac: trying to get the same data that horizon shows you? | |
| 16:17:35 | CeeMac | yeah for the instances specifically | |
| 16:17:42 | CeeMac | i see the project usage in general | |
| 16:17:57 | CeeMac | although it wont show any details for the admin project | |
| 16:18:09 | CeeMac | which is odd (maybe) | |
| 16:18:20 | CeeMac | openstack usage show/list I've tried | |
| 16:18:45 | melwitt | ok I was about to ask, that should be the right command | |
| 16:18:52 | CeeMac | I could have sworn i'd seen something about instance usage when I was reading up yesterday, but I can't find it now if I did | |
| 16:19:02 | CeeMac | yeah, list only has some projects | |
| 16:19:10 | CeeMac | i'll maybe try with a different rc file, one sec | |
| 16:22:58 | CeeMac | yeah, thats not helped. I'll retry the test in demo project as I can see that | |
| 16:23:50 | CeeMac | although openstack usage show only gives the totals, not per instance | |
| 16:29:02 | CeeMac | ok I'm stumped | |
| 16:29:27 | melwitt | I need to look a deeper at it because obviously horizon is getting it somehow, to be able to tell you how to do it with the cli | |
| 16:31:11 | CeeMac | no worries | |
| 16:32:43 | CeeMac | i think maybe the actual api is more capable than what is implemented in the cli just now, from what I remember seeing reading through api-ref last night | |
| 16:32:55 | CeeMac | direct call api stuff is a bit beyond me at the moment though | |
| 16:35:10 | gmann | stephenfin: this one i was talking, i added index page for tempest plugins having some link on tempest do "Stable Branch * Policy" - https://docs.openstack.org/tempest/latest/plugins/index.html | |
| 17:02:39 | dansmith | stephenfin: your picture is dark so I can't tell if you were receiving my good-natured jabbing properly | |
| 17:03:02 | stephenfin | Heh, all good. See Zoom chat | |
| 17:03:24 | dansmith | ah heh yeah | |
| 17:23:30 | sean-k-mooney | melwitt: sorry didnt see that till now. total_ram * allocation ratio - reserved should <= total_ram + swap | |
| 17:24:02 | sean-k-mooney | so that would be the init_host check | |
| 17:24:24 | sean-k-mooney | the per vm check is if i add up the MEMORY_MB for all vms on this host is it larger then total ram | |
| 17:24:28 | sean-k-mooney | if so warn | |
| 17:25:00 | sean-k-mooney | the less agressive warning would be to warn on total_ram + swap | |
| 17:25:38 | sean-k-mooney | instead of > then total ram but once your are over total ram you are in OOM killer terrorty | |
| 17:26:58 | melwitt | sean-k-mooney: ah I see, thanks | |
| 17:27:23 | sean-k-mooney | melwitt: do you rememebr the nova bug for that transport thing i borught up at the end | |
| 17:27:42 | sean-k-mooney | melwitt: i remember debuging it with you at the time | |
| 17:27:52 | melwitt | yeah vaguely | |
| 17:28:01 | melwitt | it's the mandatory flag for oslo messaging | |
| 17:28:05 | sean-k-mooney | yes | |
| 17:28:28 | melwitt | I don't remember if we opened a bug about that specifically or what bug we referred to when talking about it | |
| 17:28:33 | sean-k-mooney | so it looks like there is an example of using it here | |
| 17:28:35 | sean-k-mooney | https://github.com/Gsantomaggio/rabbitmq-utils/blob/master/openstack/mandatory_test/mandatory_client_fail.py#L40-L54 | |
| 17:29:32 | sean-k-mooney | well i guess https://github.com/Gsantomaggio/rabbitmq-utils/blob/master/openstack/mandatory_test/mandatory_test.py#L35-L45 | |
| 17:29:36 | sean-k-mooney | is a better example | |
| 17:30:46 | sean-k-mooney | but what im not clear on is shoudl we just be setting it on every call or just when we do a topic send | |
| 17:31:16 | sean-k-mooney | the examples seam to be for topic sends | |
| 17:32:34 | melwitt | yeah, prob have the ask oslo peeps | |
| 17:32:40 | melwitt | cause I dunno either | |
| 17:33:35 | sean-k-mooney | i guess i could propose a patch and try adding it to things and see what breaks | |
| 17:33:42 | openstack | Launchpad bug 1849741 in oslo.messaging "TransportOptions are not documented" [Medium,Triaged] | |
| 17:33:42 | sean-k-mooney | https://bugs.launchpad.net/oslo.messaging/+bug/1849741 | |
| 17:33:44 | sean-k-mooney | that does not help however | |
| 17:34:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Use 'Exception.__traceback__' for versioned notifications https://review.opendev.org/733668 | |
| 17:34:33 | openstack | bug 1881455 in OpenStack Compute (nova) "migrate server reporting list index out of bound" [Undecided,New] https://launchpad.net/bugs/1881455 | |
| 17:34:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reproducer for bug #1881455 https://review.opendev.org/733667 | |
| 17:39:41 | sean-k-mooney | stephenfin: by the way i don tthink you have to use Exception.__traceback__ directly anymore | |
| 17:39:51 | sean-k-mooney | infact that is an anti patteren i think | |
| 17:40:38 | openstack | Launchpad bug 1854992 in OpenStack Compute (nova) "Frequent instances stuck in BUILD with no apparent failure" [Undecided,Incomplete] | |
| 17:40:38 | melwitt | sean-k-mooney: could it have been this? https://bugs.launchpad.net/nova/+bug/1854992 | |
| 17:40:56 | sean-k-mooney | melwitt: yep that was the nova bug | |
| 17:41:15 | melwitt | ok, can you write a note on it so we can have more hope to find it next time xD | |
| 17:41:15 | sean-k-mooney | well i think it was the second one | |
| 17:41:41 | melwitt | the only reason I found that was because I thought I remembered erik reporting it | |
| 17:42:02 | sean-k-mooney | yep erik is from blizzard right | |
| 17:42:06 | melwitt | yeah | |
| 17:42:15 | sean-k-mooney | options = oslo_messaging.TransportOptions(at_least_once=True) | |
| 17:42:17 | sean-k-mooney | client = oslo_messaging.RPCClient(transport, | |
| 17:42:20 | sean-k-mooney | target, | |
| 17:42:21 | sean-k-mooney | transport_options=options) | |
| 17:42:23 | sean-k-mooney | so i think that is all we have to do | |
| 17:43:08 | melwitt | I wonder if there's a way we could test that it does something | |
| 17:43:22 | sean-k-mooney | where we do this https://github.com/openstack/nova/blob/master/nova/rpc.py#L208-L212 | |
| 17:43:28 | sean-k-mooney | and well the other RPC clients | |
| 17:44:21 | sean-k-mooney | that is what https://github.com/Gsantomaggio/rabbitmq-utils/tree/master/openstack/mandatory_test is trying to do | |
| 17:45:08 | sean-k-mooney | when it fails to deliver a message to the queue | |
| 17:45:13 | sean-k-mooney | we will get a oslo_messaging.exceptions.MessageUndeliverable | |
| 17:45:18 | sean-k-mooney | exception | |
| 17:45:35 | melwitt | yeah I mean, how do you create that env where it would have raised MessageUndeliverable? | |
| 17:45:45 | melwitt | just stop rabbitmq process or? | |
| 17:46:50 | melwitt | just curious how to repro and see it fix the problem | |
| 17:46:58 | sean-k-mooney | no you need to send a message to a queue that does not exist | |
| 17:47:21 | melwitt | ok, I don't know how to do that I guess | |
| 17:47:51 | sean-k-mooney | we could create an exchange without any queue and jsut send a message to a random queue name | |
| 17:47:52 | melwitt | or is that the 'foo' in the example | |
| 17:48:06 | sean-k-mooney | ya so we can sed to foo | |
| 17:48:09 | sean-k-mooney | but not create foo | |
| 17:48:12 | melwitt | ok. shows how much I know about rabbitmq :P | |
| 17:48:29 | sean-k-mooney | well i think that is how this works | |
| 17:49:51 | melwitt | cool. I'm just thinking about a one-off local test just so we know we set the option in the right place and get the improved behavior. since the unit tests that go with the patch will only be checking "did you pass mandatory at this call site" | |