| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-21 | |||
| 09:44:07 | gibi | bauzas: on stable we should not bump major version of a dependency I think | |
| 09:44:23 | bauzas | for a .z release, yes | |
| 09:44:53 | gibi | can we do that for a .y. release? | |
| 09:45:15 | bauzas | lemme try to look at the Openstack semver rules | |
| 09:45:24 | bauzas | that's not really a "stable policy" AFAIK | |
| 09:45:37 | bauzas | at least, I can't find anything in https://docs.openstack.org/project-team-guide/stable-branches.html | |
| 09:46:02 | bauzas | there it is https://docs.openstack.org/pbr/latest/user/semver.html#semantic-versioning-specification-semver | |
| 09:48:52 | bauzas | alas, can't find anything specific | |
| 09:49:24 | bauzas | we won't technically change placement API if we bump the minimum | |
| 09:49:54 | bauzas | gibi: I guess we should raise this question to the wider community, maybe | |
| 09:50:11 | bauzas | and see whether they freak out about upgrading our deps in a .y release | |
| 09:50:52 | alexe9191 | anyone on the scheduling issue :) ? | |
| 09:54:02 | opendevreview | Merged openstack/placement master: Bump min decorator to 4.0.0 https://review.opendev.org/c/openstack/placement/+/810001 | |
| 09:54:17 | bauzas | alexe9191: we superseded the idea to have multiple workers by telling we should rather use Placement | |
| 09:54:48 | bauzas | alexe9191: for the CPU usage, try to get a GMR | |
| 09:55:17 | bauzas | alexe9191: https://docs.openstack.org/nova/latest/reference/gmr.html | |
| 09:55:35 | bauzas | so you should see the greenlets and greenthreads concurrently running | |
| 09:56:17 | alexe9191 | I am still on the way to upgrade to rocky and to use the placement fully. I can not drop the sceduler in newton as far as I know | |
| 09:56:36 | bauzas | nova-scheduler will continue to exist | |
| 09:56:55 | bauzas | it's just that scheduler will call out placement for getting a list of candidates before running | |
| 09:57:15 | bauzas | which reduces the amount of complexity by adding a single-lock mechanism | |
| 09:58:11 | alexe9191 | We will start using the placement UI as recommended. However we need to upgrade first. We are coming from Kilo and need to transit first at newton before we continue the migrations. | |
| 09:59:11 | alexe9191 | The scheduler is running at 100% cpu for about an hour now and updating host stats/aggregates etc as I can see in the log files. And as I understand, this needs to be done first before the scheduler starts scheduling anything by either using placement or using the filters. | |
| 09:59:32 | alexe9191 | So my question is, would adding more workers speed up the process of building that cache? or it's just a matter of waiting | |
| 10:02:16 | gibi | bauzas: good idea to start a discussion around this in the wider community | |
| 10:14:46 | bauzas | alexe9191: honestly, updating the cache shouldn't take one hour | |
| 10:14:50 | bauzas | you have another issue | |
| 10:15:09 | alexe9191 | What could that be? | |
| 10:15:32 | bauzas | I dunno, get the GMR | |
| 10:15:40 | alexe9191 | I see a ton of those messages `Update host state with aggregates` | |
| 10:15:54 | bauzas | how many aggregates do you have ? | |
| 10:15:57 | alexe9191 | We also have about 900 hosts and more than 10K vms | |
| 10:16:04 | alexe9191 | one moment | |
| 10:16:25 | alexe9191 | 9 | |
| 10:16:57 | alexe9191 | I also see a lot of those "Update host state with service dict" | |
| 10:22:53 | bauzas | 9 can't be an issue | |
| 10:23:26 | bauzas | again, please do a GMR and see what threads run | |
| 10:23:40 | alexe9191 | I am honestly not sure what you mean when you say do a GMR? :) | |
| 10:24:24 | alexe9191 | ah `kill -USR2 8675 ` | |
| 10:24:32 | alexe9191 | that wouldn't terminate the process right ? | |
| 10:37:35 | alexe9191 | Nova did not generate any report | |
| 10:42:09 | sean-k-mooney | alexe9191: it will more or less kille the process | |
| 10:42:18 | sean-k-mooney | the report is stored in the nova log | |
| 10:42:24 | sean-k-mooney | where ever you redirect that too | |
| 10:42:33 | sean-k-mooney | to fix the process you will need to restart it | |
| 10:43:58 | alexe9191 | I did send the USR2 signal but the logs where empty | |
| 10:44:05 | alexe9191 | well not empty I did not see any GMR logs | |
| 10:44:15 | sean-k-mooney | what process did you send it to | |
| 10:44:19 | alexe9191 | nova-scheduler | |
| 10:44:23 | alexe9191 | and also nova-api as a test | |
| 10:44:43 | alexe9191 | I am actually wondering if having debug enabled is causing an overhead? | |
| 10:44:43 | sean-k-mooney | nova-api wont work if its running under appache or uwsgi | |
| 10:44:47 | sean-k-mooney | but the shcheduler should | |
| 10:44:56 | sean-k-mooney | debug logging | |
| 10:44:59 | alexe9191 | it's not it's running under dumb-init. All of the processes are running in containers. | |
| 10:45:05 | sean-k-mooney | am if you have slow disks then it could | |
| 10:45:17 | sean-k-mooney | oh this is kolla | |
| 10:45:25 | alexe9191 | Yeah more or less, kolla containers | |
| 10:45:30 | sean-k-mooney | so dumb-init is likely not passing the signal | |
| 10:45:48 | alexe9191 | hmmmm | |
| 10:45:53 | sean-k-mooney | that said if you used the pid of the python process not dumb-init it should work | |
| 10:46:04 | alexe9191 | Let me try it out again | |
| 10:50:39 | alexe9191 | Nothing | |
| 10:50:48 | alexe9191 | I do however see a lot of those in the logs "Lock "host_instance" acquired by "nova.scheduler.host_manager.sync_instance_info"" | |
| 10:51:16 | opendevreview | Vlad Gusev proposed openstack/nova stable/stein: Add regression test for bug #1908075 https://review.opendev.org/c/openstack/nova/+/810191 | |
| 11:13:54 | alexe9191 | and now after about 90 minutes it's done | |
| 11:30:37 | gibi | bauzas: could you please triage this vgpu bug https://bugs.launchpad.net/nova/+bug/1943933 ? | |
| 11:33:10 | sean-k-mooney | i think the way our config is ment to work is we list the adress of the parent | |
| 11:33:37 | sean-k-mooney | and then report the quantity of the mdev type that can be created for that device | |
| 11:34:05 | gibi | ohh so the reported wants to partially report the possible mdevs from a physical device | |
| 11:34:20 | sean-k-mooney | i think so | |
| 11:34:33 | sean-k-mooney | which i dont think we support | |
| 11:34:57 | sean-k-mooney | they basically want to treat it like we do for sriov i think where you can enable indiviugual vfs | |
| 11:35:16 | gibi | so this would be a new feature | |
| 11:35:19 | sean-k-mooney | i could be reading into the report too much but i think that is what they were expecting | |
| 11:35:49 | sean-k-mooney | gibi: ya basically a host_reserved_mdev parmater or somehting | |
| 11:36:14 | gibi | OK. I let bauzas respond but I think I see what you see in the report | |
| 11:36:19 | sean-k-mooney | e.g. enable all mdevs of this type form this parent but reserve x for host use | |
| 11:36:27 | gibi | bauzas: also ther is another vgpu bug to triage https://bugs.launchpad.net/nova/+bug/1944031 | |
| 11:37:18 | sean-k-mooney | the second one likely is correct but never personally used vgpu so not sure | |
| 11:37:34 | gibi | yeah bauzas has an environment to reproduce :) | |
| 11:37:37 | gibi | hence my ping | |
| 12:03:59 | bauzas | gibi: sean-k-mooney: ack, will look at both | |
| 12:04:08 | gibi | bauzas: thanks | |
| 12:17:02 | gibi | sean-k-mooney: this feels like a networking / neutron bug but I'm not certain. cloud you check it please https://bugs.launchpad.net/nova/+bug/1944083 ? | |
| 12:27:53 | sean-k-mooney | gibi: the only /32 that i can think of that we install is the one for the metadata service | |
| 12:28:15 | sean-k-mooney | gibi: but ya i think this is all contoled more or less on the neutron side in combindation with cloud-init | |
| 12:28:48 | sean-k-mooney | we might store some to the network info in the metadata which will be used by cloud-init | |
| 12:29:00 | sean-k-mooney | but i think this is more or less out of our contol | |
| 12:30:15 | sean-k-mooney | lets add neutron and see what they think | |
| 12:31:34 | gibi | sean-k-mooney: thanks for the analysis | |
| 12:31:41 | gibi | I agree to involve neutron | |
| 12:32:56 | sean-k-mooney | the closet thing i can tink of is https://github.com/openstack/nova/blob/master/nova/virt/interfaces.template | |
| 12:33:37 | sean-k-mooney | we do list the dhcp server per interface https://github.com/openstack/nova/blob/master/nova/virt/interfaces.template#L21 | |
| 12:33:52 | sean-k-mooney | but we are not adding any routes here | |
| 12:39:40 | sean-k-mooney | we do set the default route via the gateway in the metadata https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/netutils.py#L326-L348 | |
| 12:39:55 | sean-k-mooney | and populate any addtional neutron routes https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/nova/virt/netutils.py#L103-L121 | |
| 12:40:37 | sean-k-mooney | but ya as far as i can see there is noting on the nova side that would install /32 routes to the dns servers | |
| 12:40:45 | opendevreview | norman shen proposed openstack/nova master: Recreate mdev devices according to placement https://review.opendev.org/c/openstack/nova/+/810220 | |
| 12:54:13 | opendevreview | Merged openstack/nova master: Add missing __init__.py in nova/db/api https://review.opendev.org/c/openstack/nova/+/809980 | |