| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-09-30 | |||
| 10:52:02 | sean-k-mooney | is there anything in the instance directory we care about by the way | |
| 10:52:16 | sean-k-mooney | config drive maybe? | |
| 10:52:30 | lyarwood | yeah that the console log | |
| 10:52:35 | sean-k-mooney | the console log not really | |
| 10:52:36 | lyarwood | and maybe other things, I forget | |
| 10:52:45 | sean-k-mooney | it will be cleared when the vm is started again | |
| 10:52:58 | lyarwood | right but this path is also used during LM | |
| 10:53:12 | sean-k-mooney | ah ok | |
| 10:53:36 | sean-k-mooney | ya so config option of rpc are the only things we can do then | |
| 10:54:23 | sean-k-mooney | well in the rbd case there is one other thing we could do but its a bit of a hack | |
| 10:54:41 | sean-k-mooney | mount the instance dir in an rbd volumn | |
| 10:55:31 | sean-k-mooney | or on cephfs but nova doing that is a bit of a hack | |
| 10:55:40 | sean-k-mooney | its really something the installer should do | |
| 10:59:21 | lyarwood | sean-k-mooney: yeah this is all just debt from the early days of Nova | |
| 10:59:34 | lyarwood | sean-k-mooney: we could rework this now or just live with it | |
| 11:07:16 | noonedeadpunk | hey everyone! question: is it possible that during resize siolated aggregates are just ignored? | |
| 11:08:02 | noonedeadpunk | as I have a host which is in the isolated agrregate, and vm's are not created on it. but, during vm resize it catched it... | |
| 11:08:49 | noonedeadpunk | it's on train though | |
| 11:09:29 | ralonsoh | frickler, sorry again. What do you mean with "not in ovs"? | |
| 11:09:48 | ralonsoh | related to http://paste.openstack.org/show/798558/ | |
| 11:10:15 | ralonsoh | maybe I'm wrong but this command is executed from nova-compute | |
| 11:10:47 | ralonsoh | I checked oslo_concurrency but this is just a library and fix should be in nova too | |
| 11:11:05 | sean-k-mooney | noonedeadpunk: i dont think it should change the behavior | |
| 11:11:19 | sean-k-mooney | noonedeadpunk: which form of isolated aggreates are you using | |
| 11:11:35 | sean-k-mooney | noonedeadpunk: tenant isolation or the placement/traits based isolated aggreates | |
| 11:12:11 | noonedeadpunk | placement traits | |
| 11:12:20 | noonedeadpunk | but once you said that... | |
| 11:12:26 | sean-k-mooney | we shoudl be using the traits request form the new flavor | |
| 11:12:30 | noonedeadpunk | how can I make tenant isolation?:) | |
| 11:13:09 | noonedeadpunk | so right now aggregate looks like this in nova http://paste.openstack.org/show/798568/ | |
| 11:13:23 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py | |
| 11:13:25 | noonedeadpunk | and no flavors or images have this trait set.. | |
| 11:13:58 | sean-k-mooney | ralonsoh: that traceback i think is an eventlet issue | |
| 11:14:22 | ralonsoh | sean-k-mooney, yes, related to https://bugs.launchpad.net/nova/+bug/1863021 | |
| 11:14:22 | openstack | Launchpad bug 1863021 in OpenStack Object Storage (swift) "[SRU] eventlet monkey patch results in assert len(_active) == 1 AssertionError" [Undecided,In progress] - Assigned to Chris MacNaughton (chris.macnaughton) | |
| 11:14:30 | ralonsoh | but that should be already fixed in Nova | |
| 11:14:39 | noonedeadpunk | sean-k-mooney: I guess tenant aggregation will allow other tenants to create instances in this aggregate? | |
| 11:14:45 | sean-k-mooney | ralonsoh: ya so the approch they are takign to fix it is a hack | |
| 11:14:51 | noonedeadpunk | *tenant isolation | |
| 11:14:58 | sean-k-mooney | ralonsoh: the real fix is to fix eventlets | |
| 11:15:03 | ralonsoh | sean-k-mooney, I know, but works for now | |
| 11:15:32 | ralonsoh | sean-k-mooney, what I don't understand is why this is failing when calling processutils.execute | |
| 11:15:47 | sean-k-mooney | ralonsoh: because olso need to be fixed | |
| 11:15:56 | ralonsoh | sean-k-mooney, but this is just a library | |
| 11:16:01 | sean-k-mooney | ralonsoh: im pretty sure i sent an email about this a few months ago | |
| 11:16:11 | sean-k-mooney | ralonsoh: yes | |
| 11:16:19 | ralonsoh | sean-k-mooney, sorry, I don't get it | |
| 11:16:21 | sean-k-mooney | ralonsoh: this need to be fixed in the libs too | |
| 11:16:42 | sean-k-mooney | we use processutils.execute comes form oslo | |
| 11:16:51 | ralonsoh | sean-k-mooney, but, and I could be wrong, when we call .execute, the fix is called | |
| 11:17:08 | ralonsoh | we are calling this from a monkey patched process with the "fix" | |
| 11:17:09 | sean-k-mooney | not if its executed via privsep | |
| 11:17:17 | ralonsoh | oh my... | |
| 11:17:21 | ralonsoh | ok, got it!!! | |
| 11:17:53 | sean-k-mooney | noonedeadpunk: you enabled the placment prefilter for this in the nova.conf too right | |
| 11:19:16 | sean-k-mooney | noonedeadpunk: https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.enable_isolated_aggregate_filtering | |
| 11:19:25 | noonedeadpunk | sean-k-mooney: I think I just set `enable_isolated_aggregate_filtering` | |
| 11:19:30 | noonedeadpunk | yeah | |
| 11:19:56 | noonedeadpunk | and actually VMs are not created on host. the only one which is wrong placed for the last month is this one | |
| 11:20:02 | sean-k-mooney | noonedeadpunk: by the way you can also do tenant isolation with placement too | |
| 11:20:05 | noonedeadpunk | which was spawned there after resize | |
| 11:20:40 | sean-k-mooney | noonedeadpunk: did you way your using train | |
| 11:21:24 | noonedeadpunk | sorry didn't get the question:( | |
| 11:21:38 | sean-k-mooney | what release are you running train? | |
| 11:22:45 | sean-k-mooney | i start typeing "by the way are you using train" the swaped to "did you say your using train" in my head and type a mix of the two hehe | |
| 11:23:35 | sean-k-mooney | there is nothing here that is resize specific | |
| 11:23:37 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/train/nova/scheduler/request_filter.py#L31-L88 | |
| 11:23:46 | noonedeadpunk | it's a... 20.2.1.dev1, which to be exact 118ee682571a4bd41c8009dbe2e47fdd1f85a630 | |
| 11:24:26 | sean-k-mooney | noonedeadpunk: can you check for that "'Request filter %r took %.1f seconds'," debug message if you have debug logs | |
| 11:24:57 | sean-k-mooney | the simpelst way to check would be to boot a vm and set the schduler into debug mode then resize it | |
| 11:25:17 | sean-k-mooney | and confirm the isolate_aggregates prefilter is run | |
| 11:25:41 | sean-k-mooney | but they shoudl all be run https://github.com/openstack/nova/blob/stable/train/nova/scheduler/request_filter.py#L221-L237 | |
| 11:34:03 | noonedeadpunk | sean-k-mooney: I thiink I have only info level, not debug one... | |
| 11:34:44 | sean-k-mooney | thats ok. with a quick look i dont see any reason that it woudl be skipped | |
| 11:34:54 | noonedeadpunk | I see... | |
| 11:35:11 | sean-k-mooney | if it is being skipped for some reason then a functional test should be able to repoduces it | |
| 11:36:08 | sean-k-mooney | this logic should be driver independent | |
| 11:38:47 | noonedeadpunk | have the same ALL_REQUEST_FILTERS here | |
| 11:39:05 | noonedeadpunk | sean-k-mooney: ok, thanks for your help, will try to reproduce it and catch logs | |
| 11:40:01 | sean-k-mooney | if you do feel free to file a bug with the steps. we shoudl be able to trun it into a functional regression test and then fix it but im not aware of something that would cause it to break currently | |
| 11:40:22 | sean-k-mooney | that does not mean it cant just not seeing it quickly | |
| 11:46:22 | tobias-urdin | sean-k-mooney: sorry was afk for a while, should it be treated as a bug? | |
| 11:47:35 | sean-k-mooney | tobias-urdin: no we previusly had a bug for this for rbd then meged a skip and reverted it | |
| 11:47:58 | sean-k-mooney | tobias-urdin: the use of rbd does not put the instance state dir on shared storage | |
| 11:48:08 | sean-k-mooney | tobias-urdin: this could be a blueprint/spec | |
| 11:48:14 | sean-k-mooney | but its a new feature not a bug | |
| 11:48:41 | tobias-urdin | sean-k-mooney: ok, thanks :) | |
| 11:48:57 | sean-k-mooney | tobias-urdin: its one i would not mind seeing added by the way. i think it would be useful | |
| 11:49:27 | tobias-urdin | sean-k-mooney: i agree | |
| 11:55:32 | noonedeadpunk | sean-k-mooney: looking at code, I have one question - should I need another aggregate which would include the rest of the hosts? | |
| 11:56:00 | sean-k-mooney | noonedeadpunk: off the top of my head i dont think so | |
| 11:56:14 | noonedeadpunk | ok, just to double check, as I don't have one | |
| 11:56:29 | noonedeadpunk | I mean I have the only aggregate with required trait | |
| 11:56:52 | sean-k-mooney | all the rest will be in the implcit nova aggreate | |
| 11:59:20 | noonedeadpunk | was just thinking how this works. https://github.com/openstack/nova/blob/stable/train/nova/scheduler/request_filter.py#L74-L76 Like we get list of aggregates without the trait? | |
| 12:02:15 | sean-k-mooney | yes so we first get all the required traits from the instance | |
| 12:03:06 | sean-k-mooney | then we all the aggreates that have require traits that are not in the set we require for the instnace | |
| 12:03:20 | sean-k-mooney | we then add those as forbiden aggreates to the placment query | |
| 12:03:42 | noonedeadpunk | ah, ok, yeah | |
| 12:03:45 | sean-k-mooney | so since you have no required traits in the flaovr or image for that instance | |