| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-21 | |||
| 10:14:50 | bauzas | you have another issue | |
| 10:15:09 | alexe9191 | What could that be? | |
| 10:15:32 | bauzas | I dunno, get the GMR | |
| 10:15:40 | alexe9191 | I see a ton of those messages `Update host state with aggregates` | |
| 10:15:54 | bauzas | how many aggregates do you have ? | |
| 10:15:57 | alexe9191 | We also have about 900 hosts and more than 10K vms | |
| 10:16:04 | alexe9191 | one moment | |
| 10:16:25 | alexe9191 | 9 | |
| 10:16:57 | alexe9191 | I also see a lot of those "Update host state with service dict" | |
| 10:22:53 | bauzas | 9 can't be an issue | |
| 10:23:26 | bauzas | again, please do a GMR and see what threads run | |
| 10:23:40 | alexe9191 | I am honestly not sure what you mean when you say do a GMR? :) | |
| 10:24:24 | alexe9191 | ah `kill -USR2 8675 ` | |
| 10:24:32 | alexe9191 | that wouldn't terminate the process right ? | |
| 10:37:35 | alexe9191 | Nova did not generate any report | |
| 10:42:09 | sean-k-mooney | alexe9191: it will more or less kille the process | |
| 10:42:18 | sean-k-mooney | the report is stored in the nova log | |
| 10:42:24 | sean-k-mooney | where ever you redirect that too | |
| 10:42:33 | sean-k-mooney | to fix the process you will need to restart it | |
| 10:43:58 | alexe9191 | I did send the USR2 signal but the logs where empty | |
| 10:44:05 | alexe9191 | well not empty I did not see any GMR logs | |
| 10:44:15 | sean-k-mooney | what process did you send it to | |
| 10:44:19 | alexe9191 | nova-scheduler | |
| 10:44:23 | alexe9191 | and also nova-api as a test | |
| 10:44:43 | sean-k-mooney | nova-api wont work if its running under appache or uwsgi | |
| 10:44:43 | alexe9191 | I am actually wondering if having debug enabled is causing an overhead? | |
| 10:44:47 | sean-k-mooney | but the shcheduler should | |
| 10:44:56 | sean-k-mooney | debug logging | |
| 10:44:59 | alexe9191 | it's not it's running under dumb-init. All of the processes are running in containers. | |
| 10:45:05 | sean-k-mooney | am if you have slow disks then it could | |
| 10:45:17 | sean-k-mooney | oh this is kolla | |
| 10:45:25 | alexe9191 | Yeah more or less, kolla containers | |
| 10:45:30 | sean-k-mooney | so dumb-init is likely not passing the signal | |
| 10:45:48 | alexe9191 | hmmmm | |
| 10:45:53 | sean-k-mooney | that said if you used the pid of the python process not dumb-init it should work | |
| 10:46:04 | alexe9191 | Let me try it out again | |
| 10:50:39 | alexe9191 | Nothing | |
| 10:50:48 | alexe9191 | I do however see a lot of those in the logs "Lock "host_instance" acquired by "nova.scheduler.host_manager.sync_instance_info"" | |
| 10:51:16 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Add regression test for bug #1908075 https://review.opendev.org/c/openstack/nova/+/810191 | |
| 11:13:54 | alexe9191 | and now after about 90 minutes it's done | |
| 11:30:37 | gibi | bauzas: could you please triage this vgpu bug https://bugs.launchpad.net/nova/+bug/1943933 ? | |
| 11:33:10 | sean-k-mooney | i think the way our config is ment to work is we list the adress of the parent | |
| 11:33:37 | sean-k-mooney | and then report the quantity of the mdev type that can be created for that device | |
| 11:34:05 | gibi | ohh so the reported wants to partially report the possible mdevs from a physical device | |
| 11:34:20 | sean-k-mooney | i think so | |
| 11:34:33 | sean-k-mooney | which i dont think we support | |
| 11:34:57 | sean-k-mooney | they basically want to treat it like we do for sriov i think where you can enable indiviugual vfs | |
| 11:35:16 | gibi | so this would be a new feature | |
| 11:35:19 | sean-k-mooney | i could be reading into the report too much but i think that is what they were expecting | |
| 11:35:49 | sean-k-mooney | gibi: ya basically a host_reserved_mdev parmater or somehting | |
| 11:36:14 | gibi | OK. I let bauzas respond but I think I see what you see in the report | |
| 11:36:19 | sean-k-mooney | e.g. enable all mdevs of this type form this parent but reserve x for host use | |
| 11:36:27 | gibi | bauzas: also ther is another vgpu bug to triage https://bugs.launchpad.net/nova/+bug/1944031 | |
| 11:37:18 | sean-k-mooney | the second one likely is correct but never personally used vgpu so not sure | |
| 11:37:34 | gibi | yeah bauzas has an environment to reproduce :) | |
| 11:37:37 | gibi | hence my ping | |
| 12:03:59 | bauzas | gibi: sean-k-mooney: ack, will look at both | |
| 12:04:08 | gibi | bauzas: thanks | |
| 12:17:02 | gibi | sean-k-mooney: this feels like a networking / neutron bug but I'm not certain. cloud you check it please https://bugs.launchpad.net/nova/+bug/1944083 ? | |
| 12:27:53 | sean-k-mooney | gibi: the only /32 that i can think of that we install is the one for the metadata service | |
| 12:28:15 | sean-k-mooney | gibi: but ya i think this is all contoled more or less on the neutron side in combindation with cloud-init | |
| 12:28:48 | sean-k-mooney | we might store some to the network info in the metadata which will be used by cloud-init | |
| 12:29:00 | sean-k-mooney | but i think this is more or less out of our contol | |
| 12:30:15 | sean-k-mooney | lets add neutron and see what they think | |
| 12:31:34 | gibi | sean-k-mooney: thanks for the analysis | |
| 12:31:41 | gibi | I agree to involve neutron | |
| 12:32:56 | sean-k-mooney | the closet thing i can tink of is https://github.com/openstack/nova/blob/master/nova/virt/interfaces.template | |
| 12:33:37 | sean-k-mooney | we do list the dhcp server per interface https://github.com/openstack/nova/blob/master/nova/virt/interfaces.template#L21 | |
| 12:33:52 | sean-k-mooney | but we are not adding any routes here | |
| 12:39:40 | sean-k-mooney | we do set the default route via the gateway in the metadata https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/netutils.py#L326-L348 | |
| 12:39:55 | sean-k-mooney | and populate any addtional neutron routes https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/netutils.py#L103-L121 | |
| 12:40:37 | sean-k-mooney | but ya as far as i can see there is noting on the nova side that would install /32 routes to the dns servers | |
| 12:40:45 | opendevreview | norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220 | |
| 12:54:13 | opendevreview | Merged openstack/nova master: Add missing __init__.py in nova/db/api https://review.opendev.org/c/openstack/nova/+/809980 | |
| 12:54:32 | opendevreview | Merged openstack/nova stable/xena: [stable-only]Update .gitreview for stable/xena https://review.opendev.org/c/openstack/nova/+/809759 | |
| 12:54:47 | opendevreview | Merged openstack/nova stable/xena: [stable-only]Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/nova/+/809760 | |
| 13:14:58 | opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Add missing __init__.py in nova/db/api https://review.opendev.org/c/openstack/nova/+/810192 | |
| 13:15:45 | gibi | elodilles, lyarwood, bauzas: we need this in xena for RC2 ^^ | |
| 13:16:09 | bauzas | done | |
| 13:16:24 | gibi | thanks | |
| 13:16:44 | lyarwood | ACK'd | |
| 13:17:25 | gibi | cool | |
| 13:17:27 | gibi | this was fast :) | |
| 13:17:39 | lyarwood | I'm going to be AFK for the meeting btw, dental checkup for the first time in a long time. | |
| 13:18:24 | gibi | lyarwood: ack, thanks for the headsup | |
| 13:18:35 | gibi | and I hope the dentis will be painless | |
| 13:21:25 | opendevreview | Lee Yarwood proposed openstack/nova-specs master: Store and allow libvirt instance device buses and models to be updated https://review.opendev.org/c/openstack/nova-specs/+/810235 | |
| 13:22:19 | lyarwood | I'm British so the normal and in my case correct stereotypes apply, it's not going to be fun. | |
| 13:24:25 | bauzas | lyarwood: we need a release liaison that I'll ask in today's meeting | |
| 13:24:35 | bauzas | lyarwood: if you are interested in this role, tell me | |
| 13:24:47 | gibi | this reminds me to book a dentis checkup too | |
| 13:25:02 | bauzas | gibi: booked since 6 months | |
| 13:25:18 | gibi | I'm not rushing :) | |
| 13:25:29 | kashyap | gibi: Yikes; I've been avoiding it | |
| 13:25:35 | kashyap | I know I should | |
| 13:26:33 | bauzas | "I have a tooth rage", "sure, we have a slot in Feb 2022, works for you ?" | |
| 13:26:47 | bauzas | longer than delivering a car | |
| 13:36:56 | gibi | as far as I see I can book a dentist slot for this thursday morning | |
| 13:37:05 | gibi | that feels too close :/ | |
| 13:40:01 | kashyap | gibi: Don't you prefer your band-aid to be quickly removed? | |