| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-07-31 | |||
| 19:03:01 | sean-k-mooney | in that case you tell nova that the root disk should not be create using the compute nodes default driver and instad should be provide by cinder | |
| 19:03:16 | nafiux | Oh, so, due my current setup for learning, I will leverage cinder for both bfv and additional volumes as needed | |
| 19:03:17 | sean-k-mooney | you can optionl specifythat that volume should be deleted when the vm is or not | |
| 19:04:05 | sean-k-mooney | i wasnt following the full conversation but you can use cinder for both | |
| 19:04:22 | nafiux | Ok, yea, is just because I have a lot of CPU & Memory but few hard disk | |
| 19:04:40 | sean-k-mooney | the default backend for non bfv guest is to create a local file on the compute node for the root disk of the instance | |
| 19:05:12 | sean-k-mooney | you can also tell the compute node to use a differnt default image backend such as ceph | |
| 19:05:46 | sean-k-mooney | in which case nova will create a remote ceph volume out of band of cinder | |
| 19:06:16 | nafiux | Ok, I need to read a lot :D cinder, ceph, and will back later with more specific questions, this is really amazing. | |
| 19:06:18 | sean-k-mooney | other peopl in a similar situation somethime mount the default location where libvirt stores its disk image on nfs | |
| 19:07:32 | sean-k-mooney | so as mriedem said there are lots of way to do it and people do many different things but its better to keep it simple when learning | |
| 19:08:13 | nafiux | Agree. Thanks for the support. | |
| 19:11:28 | mriedem | nafiux: more info on cinder in their docs https://docs.openstack.org/cinder/latest/ | |
| 19:11:46 | mriedem | there is also the ops guide which is older but relevant https://docs.openstack.org/operations-guide/ | |
| 19:12:41 | sean-k-mooney | or if you like video content there are way more videos then you will have time to watch on teh offical youtube channel | |
| 19:12:43 | sean-k-mooney | https://www.youtube.com/user/OpenStackFoundation | |
| 19:13:17 | nafiux | Sure! I will take a look on this too: https://docs.openstack.org/arch-design/design-storage/design-storage-concepts.html | |
| 19:47:42 | efried_rollin | mriedem: responded. TL;DR: a Set would be better, but a list will work just fine. | |
| 19:49:33 | efried | I guess that's not FUPable since it's an OVO/RPC thing. No takesy backseys. | |
| 19:49:46 | efried | so if you feel strongly about it you should -1 | |
| 19:52:52 | mriedem | done | |
| 19:53:16 | mriedem | i've made more comments up that series but non-voting b/c i'm a coward | |
| 19:53:27 | mriedem | i'd really like dan to review that series but he's out this week | |
| 19:54:48 | efried | mriedem: FYI Dan was involved earlier, which is how we ended up with the query and filter in their current form. | |
| 19:55:21 | mriedem | yeah i knew he blew up the single megapatch | |
| 19:56:04 | efried | but also was the one who suggested the db function as it now exists. | |
| 19:56:14 | efried | much nicer than it was before | |
| 19:56:43 | mriedem | b/c it's mostly a copy of the one he added for his earlier (first) pre-filters | |
| 19:57:08 | mriedem | fwiw most of my comments are "can you document this?" | |
| 19:57:52 | mriedem | i also feel like forbidden is the wrong word for the user-facing side of this feature, it's really about host isolation | |
| 19:57:57 | mriedem | but it's probably too late to change that now | |
| 19:58:25 | mriedem | unless i'm just too slow to connect the dots, which is possible | |
| 19:58:42 | mriedem | lots of negative logic in this feature makes it hard for me to reason about | |
| 19:58:56 | mriedem | don't not give me the hosts i don't not want | |
| 19:59:04 | efried | I'll have another look at the documentation, but I thought it was ultimately explained the right way for users. | |
| 19:59:17 | efried | The logic is necessarily hairy and compound-negative-confusing | |
| 19:59:25 | efried | but the ux winds up being really crisp and clean imo | |
| 20:02:58 | efried | ...you're right, the doc https://review.opendev.org/#/c/667952/10/doc/source/reference/forbidden-aggregates.rst still talks about forbidden aggregates. Though really, the explanation is very good, and "forbidden aggregates" is really just used as the name-of-feature. | |
| 20:16:30 | mriedem | yeah i'm not -1 on it, it's just a bit confusing | |
| 20:16:33 | mriedem | some of the wording | |
| 20:18:15 | efried | I think you've got a valid point that we could s/forbidden/isolated/ and it would be more comprehensible. The docs could be fixed later, but if we want to rename the conf opt we should do it now. | |
| 20:21:25 | prometheanfire | sean-k-mooney: ya, looks like it's lxml https://review.opendev.org/673848 | |
| 20:23:06 | mriedem | efried: question in https://review.opendev.org/#/c/670182/1 | |
| 20:24:23 | efried | mriedem: responded | |
| 20:29:24 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Fix python3 compatibility of rbd get_fsid https://review.opendev.org/664513 | |
| 20:31:07 | mriedem | amorin: are you still working on https://review.opendev.org/#/c/667294/ ? | |
| 20:45:14 | openstackgerrit | Merged openstack/nova master: neutron: log something more useful in _get_instance_nw_info https://review.opendev.org/673694 | |
| 21:43:21 | efried | mriedem: RequestContext question for ya | |
| 21:43:56 | efried | I've been tracking where nova-compute talks to Placement to do things like creating the compute node RP. | |
| 21:44:42 | efried | IIUC those interactions use a RequestContext that gets created when the service starts up, because they're done in a periodic, not based on a request coming from e.g. conductor or whatever. | |
| 21:45:00 | efried | which would be fine, except... | |
| 21:45:13 | efried | it doesn't seem to be the same context that's being used for logging. | |
| 21:45:25 | mriedem | b/c the request id changes etc | |
| 21:45:34 | mriedem | which is because of a thread local store for the context way up in oslo.context | |
| 21:45:36 | mriedem | and it's mega confusing | |
| 21:45:53 | efried | rightright, but it's a *different* context? | |
| 21:46:02 | mriedem | as mentioned https://bugs.launchpad.net/nova/+bug/1835958/comments/7 | |
| 21:46:02 | openstack | Launchpad bug 1835958 in OpenStack Compute (nova) "Nova sync power state on large clusters causes poor performance" [Undecided,New] | |
| 21:46:21 | efried | viz: http://logs.openstack.org/86/672986/2/check/nova-live-migration/0171fa9/logs/screen-n-cpu.txt.gz#_Jul_30_22_09_35_334853 | |
| 21:46:22 | efried | here ^ you can see that the log line is showing one request_id, but the context we're using for the placement request has a different request_id. | |
| 21:47:58 | mriedem | req-428963d8-bc71-4823-8e35-97db13886690 is just some admin context created when the compute service starts up right? | |
| 21:48:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L1321 | |
| 21:48:33 | mriedem | so you're using a random context there with a random request_id | |
| 21:48:50 | efried | req-42... is the placement-side local request_id, just ignore that one. | |
| 21:48:51 | mriedem | because https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L273 | |
| 21:49:20 | mriedem | i was looking at that one because | |
| 21:49:20 | mriedem | Jul 30 22:09:35.334349 ubuntu-bionic-rax-ord-0009499845 nova-compute[28986]: INFO nova.scheduler.client.report [None req-fb42d5e6-1c0d-41d0-88c5-a4f1ac127a41 None None] [req-428963d8-bc71-4823-8e35-97db13886690] Created resource provider record via placement API for resource provider with UUID 5dcb9522-0513-440a-80db-d985b5d45704 and name ubuntu-bionic-rax-ord-0009499845. | |
| 21:49:35 | efried | req-fb42d5e6-1c0d-41d0-88c5-a4f1ac127a41 is the oslo.context one that oslo.log is paying attention to | |
| 21:49:35 | efried | req-8a7e2cc5-3870-4118-b55f-4533d7a4de80 is the one in the RequestContext being used for the placement communication. | |
| 21:50:07 | efried | yeah, I know, I had to track that one down as well. I started this whole thing because I was wondering why tf we needed to print that reqid at all. | |
| 21:50:37 | efried | it's the one being sent *back* to us in the placement response headers, which turns out to be the (not-global!) request_id on the placement side | |
| 21:50:51 | efried | and because of this whole snafu, it's currently the only way for us to correlate the requests on the nova & placement sides | |
| 21:51:04 | efried | but that flies in the face of the entire global_request_id concept | |
| 21:51:38 | efried | if we get global_request_id working, we don't need that at all because we'll be able to correlate with global_request_id, which is the whole friggin point. | |
| 21:51:48 | efried | so my question is this | |
| 21:52:21 | mriedem | idk what oslo.log is doing really but i know things get wonky with this https://github.com/openstack/oslo.context/blob/master/oslo_context/context.py#L300 | |
| 21:52:38 | efried | what about if, when we create the RequestContext on compute startup, we a) set its global_request_id, and b) pass overwrite=True so oslo.context replaces it in the threadlocal | |
| 21:53:15 | efried | yeah, b) would call update_store and replace that. | |
| 21:54:08 | efried | I think having talked through this I've got enough to do that ^ in a WIP and see what shakes out. | |
| 21:55:55 | mriedem | ....ok | |
| 21:56:12 | openstackgerrit | Merged openstack/nova master: Move adding vlans to interfaces to privsep. https://review.opendev.org/635436 | |
| 21:56:30 | mriedem | i know melwitt https://review.opendev.org/#/c/524306/ and gibi https://review.opendev.org/#/c/542891/ have muddled in this and it's a mess | |
| 21:56:36 | mriedem | so i guess welcome to the tarpits | |
| 21:57:37 | efried | mriedem: I see that get_context() and get_admin_context() explicitly set overwrite=False. The former even says it's to not mess with logging. Why?? | |
| 21:57:56 | mriedem | oh eric, i could tell you but that would be too easy | |
| 21:58:47 | mriedem | if you can hunt down vishy from 2012 maybe he can answer https://review.opendev.org/#/c/2923/ | |
| 21:59:51 | mriedem | or i guess alaski from 2016 https://review.opendev.org/#/c/378941/ | |
| 22:01:01 | mriedem | the admin context stuff is also a bit legacy from when db apis were explicitly blocked unless you had an admin context, which alex_xu removed a long time ago | |
| 22:01:09 | mriedem | but still affects project_id filtering in the model_query | |
| 22:01:15 | mriedem | as melwitt can probably go into detail about | |
| 22:01:25 | mriedem | as for the logging and thread local stuff, ....idk | |
| 22:05:51 | openstackgerrit | Eric Fried proposed openstack/nova master: Store a RequestContext on service to reuse for periodic tasks https://review.opendev.org/524306 | |
| 22:05:52 | openstackgerrit | Eric Fried proposed openstack/nova master: Poison context usage in periodic tasks https://review.opendev.org/542891 | |
| 22:07:35 | efried | melwitt, gibi_off: FYI, rebased & resolved merge conflicts ^. I'm going to throw one on top that overwrites the threadlocal context and creates a global_request_id so our logging might be sane. We'll see what explodes. | |
| 22:07:39 | efried | mriedem: fyi ^ | |
| 22:10:53 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Generate and log global_request_id properly https://review.opendev.org/673924 | |
| 22:13:57 | openstackgerrit | Eric Fried proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.opendev.org/636508 | |
| 22:13:58 | openstackgerrit | Eric Fried proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.opendev.org/639280 | |
| 22:13:58 | openstackgerrit | Eric Fried proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.opendev.org/639281 | |
| 22:13:59 | openstackgerrit | Eric Fried proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.opendev.org/639282 | |