| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-10-03 | |||
| 16:51:39 | dansmith | atmark: so first thing to do is figure out what's wrong with those two computes. my guess is that during the upgrade, they got their transport_url pointed at the cell0/superconductor instead of the cell conductor | |
| 16:53:33 | atmark | I see find that in nova.conf right ? | |
| 16:53:44 | dansmith | yep | |
| 16:54:16 | atmark | the transport_url in nova.conf points to rabbitmq | |
| 16:55:17 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246 | |
| 16:56:00 | dansmith | atmark: right it always points at rabbit.. I'm not familiar with kolla but are you running two sets of conductors, one for the top-level and one for the cells? | |
| 16:56:12 | dansmith | easy way to tell is compare your transport_url to a compute that is not borked :) | |
| 16:56:13 | atmark | in the controller i can see transport_url points to nova_cnco1 vhost of rabbitmq | |
| 16:56:54 | dansmith | start with the transport_url on the computes, and compare one of the broken ones to a non-broken one | |
| 16:57:44 | atmark | dansmith: ok, the transport_url on broken computes doesn't have nova_cnco1 vhost | |
| 16:58:07 | dansmith | okay, that's the problem, so fix that and restart nova-compute there | |
| 17:01:14 | atmark | just a sec | |
| 17:14:15 | atmark | dansmith: done | |
| 17:15:05 | atmark | took a while, had to the push change through kolla-ansible | |
| 17:15:10 | dansmith | atmark: okay is it happier on startup now? there might be more work to clean up, but it might also have just worked it out | |
| 17:15:18 | dansmith | by it I mean the compute service in its log | |
| 17:16:36 | dansmith | now that they're not pointing at the wrong transport (you fixed both right?) you can clear the service and node records from cell0's database which should resolve the service list duplication | |
| 17:16:52 | dansmith | if the computes are happy in the log, you probably need to do the placement heal part now | |
| 17:18:56 | atmark | yes, both are fixed. Still seeing `Failed to retrieve allocations for resource provider c017*` on compute01's nova-compute.log https://paste.openstack.org/show/b8j8iC0eV6TOK1QLVBfN/ | |
| 17:19:00 | atmark | is that expected? | |
| 17:19:59 | dansmith | atmark: yeah, so what happened is you deleted some stuff, the confused computes re-created their providers with the wrong uuid (the new one in the cell0) and now that it's back, it wants to use/recreate the old one | |
| 17:20:23 | dansmith | atmark: so probably best to nuke the allocations *and* the RP for each, then run the heal and then get them started back up | |
| 17:20:49 | dansmith | it's why I said "you shouldn't have deleted the allocations" earlier, despite the doc saying it's appropriate in some circumstances | |
| 17:20:58 | dansmith | but it's cool, this is confusing and messed up obviously :) | |
| 17:24:22 | atmark | clear the service and node records from cell0's database - I'm safe to remove these records https://paste.openstack.org/show/bxfAMF8OBcUCz2GsHiin/ ? | |
| 17:26:05 | atmark | should I remove entry or just update the deleted field ? | |
| 17:26:48 | dansmith | atmark: you can just set the deleted field to id if you want (that's what service delete does) but really they should go away.. but fine to just mark as deleted to test if you want | |
| 17:40:25 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246 | |
| 17:47:20 | atmark | dansmith: For compute25 - I didn't have to do anything. compute01 - I deleted the RP, restarted nova-compute and then ran heal so issue is now resolved. I learned something new. Thanks a lot. | |
| 17:47:48 | dansmith | atmark: awesome, glad to hear it.. so you cleaned up the service *and* node records from cell0 right? | |
| 17:47:49 | atmark | If I unset alloactions instead of alloactions, will I face the same issue? https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html#solution | |
| 17:48:25 | atmark | dansmith: yes, I decided to remove the records instead of updating deleted field. | |
| 17:48:29 | dansmith | cool | |
| 17:48:52 | dansmith | atmark: not sure what you're asking about. unset .. "allocations instead of allocations" | |
| 17:49:31 | atmark | "allocation unset instead of allocation remove" | |
| 17:50:49 | dansmith | atmark: no I think it would have ended up the same way | |
| 17:56:38 | atmark | is it safe to remove these records in instances table in nova_cell0? https://paste.openstack.org/show/bP2VO4c7Gh6Tb4SIm5Wi/ | |
| 17:59:19 | dansmith | yes on the compute nodes.. the instances were all deleted in the last dump, so they belong there and will be cleaned after an archive | |
| 17:59:29 | dansmith | instances *do* belong in cell0, if they failed the schedule | |
| 18:10:18 | atmark | got it. thanks again | |
| 18:19:58 | opendevreview | Merged openstack/nova stable/zed: Fix failed count for anti-affinity check https://review.opendev.org/c/openstack/nova/+/885344 | |
| 19:30:23 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246 | |
| #openstack-nova - 2023-10-04 | |||
| 10:03:38 | auniyal | Hi sean-k-mooney, reminder ping for https://review.opendev.org/q/topic:bug%252F1996732 | |
| 10:05:24 | auniyal | can you also please quick lookon https://review.opendev.org/q/topic:bug%252F2008102 only 2 patches are pending to merge | |
| 10:16:55 | auniyal | thanks sean-k-mooney | |
| 11:58:17 | opendevreview | Merged openstack/nova stable/xena: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/882921 | |
| 15:24:20 | opendevreview | Dan Smith proposed openstack/nova master: WIP: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246 | |
| 15:24:34 | opendevreview | Dan Smith proposed openstack/nova master: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246 | |
| 17:09:49 | frickler | anyone willing tp push a small status check URL fix over the edge? https://review.opendev.org/c/openstack/nova/+/895770 | |
| 17:09:55 | frickler | *to | |
| 17:11:14 | dansmith | frickler: the new one gives me a 404 | |
| 17:11:18 | dansmith | https://docs.openstack.org/nova/latest/admin/hw_machine_type.html | |
| 17:11:40 | dansmith | second new one seems right tho | |
| 17:15:58 | frickler | humm, I admit I only checked the second one because that's what I stumbled about and then just duplicated the change for the other URL | |
| 17:16:07 | frickler | let me try to find a proper URL | |
| 17:17:11 | frickler | ah, dashes instead of underscores, will amend, thx for noticing this | |
| 17:18:26 | opendevreview | Dr. Jens Harbott proposed openstack/nova master: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/895770 | |
| 17:19:07 | frickler | dansmith: ^^ | |
| 17:19:53 | dansmith | frickler: got it | |
| 22:46:31 | opendevreview | Merged openstack/nova master: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/895770 | |
| 23:16:34 | opendevreview | melanie witt proposed openstack/nova master: testing: Detect log message string formatting bugs https://review.opendev.org/c/openstack/nova/+/897370 | |
| #openstack-nova - 2023-10-05 | |||
| 02:36:45 | opendevreview | Dr. Jens Harbott proposed openstack/nova stable/2023.2: Fix URLs in status check results https://review.opendev.org/c/openstack/nova/+/897297 | |
| 13:29:21 | opendevreview | sean mooney proposed openstack/nova stable/wallaby: fix sr-iov support on Cavium ThunderX hosts. https://review.opendev.org/c/openstack/nova/+/897300 | |
| 13:44:03 | opendevreview | Dan Smith proposed openstack/nova master: Warn if we find compute services in cell0 https://review.opendev.org/c/openstack/nova/+/897246 | |
| 13:48:06 | opendevreview | Merged openstack/nova stable/yoga: Fix failed count for anti-affinity check https://review.opendev.org/c/openstack/nova/+/885345 | |
| 14:14:11 | bauzas | fwiw, live OIL session on Youtube about Bobcat now https://www.youtube.com/watch?v=q7WDncK3YuM | |
| 18:11:55 | auniyal | Hi dansmith, can you please review this https://review.opendev.org/c/openstack/nova/+/857339, its pending from quite sometime, I do have few comments from bauzas already. | |
| 18:54:16 | opendevreview | sean mooney proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | |
| 23:19:03 | opendevreview | melanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | |
| 23:19:03 | opendevreview | melanie witt proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | |
| 23:50:01 | opendevreview | melanie witt proposed openstack/nova-specs master: Re-propose spec for ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/897502 | |
| 23:52:42 | opendevreview | melanie witt proposed openstack/nova-specs master: iRe-propose spec for ephemeral encryption for libvirt https://review.opendev.org/c/openstack/nova-specs/+/897503 | |
| 23:53:24 | opendevreview | melanie witt proposed openstack/nova-specs master: Re-propose spec for ephemeral encryption for libvirt https://review.opendev.org/c/openstack/nova-specs/+/897503 | |
| #openstack-nova - 2023-10-06 | |||
| 00:20:00 | opendevreview | sean mooney proposed openstack/nova master: add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | |
| 01:38:38 | opendevreview | sean mooney proposed openstack/nova master: add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | |
| 06:03:11 | opendevreview | Amit Uniyal proposed openstack/nova master: Adds server show in helpers https://review.opendev.org/c/openstack/nova/+/893584 | |
| 06:03:12 | opendevreview | Amit Uniyal proposed openstack/nova master: Refactor vol attachment api call for rest of the tests https://review.opendev.org/c/openstack/nova/+/893587 | |
| 06:03:12 | opendevreview | Amit Uniyal proposed openstack/nova master: Refactor volumeAttachment for regressions https://review.opendev.org/c/openstack/nova/+/893585 | |
| 07:07:36 | zigo | Quick Alembic question, not sure if this channel is the best place to ask, but asking anyways. Where is Alembic storing the current revision of the db, and where's the ordered list of db-migration stored? | |
| 07:07:36 | zigo | Hi there! | |
| 07:25:55 | zigo | Ah no, I'm searching the info about sqlalchemy-migrate in fact... | |
| 07:26:07 | zigo | (we're trying to upgrade from Victoria to Bobcat...) | |
| 07:37:38 | frickler | if you see doc build failures please check https://bugs.launchpad.net/nova/+bug/2026345 | |
| 08:14:38 | bauzas | zigo: with alembic, that's a bit different but you can find them | |
| 08:15:04 | zigo | Yeah, I found the version in the db, but can't find the version in the code ... | |
| 08:17:56 | auniyal | frickler, I am seeing this today, even though my change do not have any doc update, thanks for sharing bug. | |
| 08:21:01 | frickler | zigo: there was a big cleanup in https://review.opendev.org/c/openstack/nova/+/872428 which might affect you? | |
| 08:22:29 | frickler | are you trying to go in one big step from V to B? | |
| 08:37:55 | zigo | frickler: No, we're currently designing our upgrade script to script it all... | |
| 08:38:09 | zigo | And we'd like to add db version checks on each step. | |
| 08:40:13 | zigo | So we need this check for both alembic and sqla-migrate... | |
| 08:41:59 | zigo | python3 manage.py version | |
| 08:41:59 | zigo | to get the current version in the *db* for alembic, and: | |
| 08:41:59 | zigo | SELECT version_num FROM alembic_version; | |
| 08:41:59 | zigo | So far, I have: | |
| 08:42:00 | zigo | to get the *code* version of sqla-migrate. I need to find out how to fetch: | |
| 08:42:02 | zigo | The *code* version for Alembic and the *db* version for sqla-migrate ... | |
| 08:56:48 | bauzas | we recently packed the versions | |
| 08:56:53 | bauzas | https://github.com/openstack/nova/tree/master/nova/db/main/migrations/versions | |
| 08:56:58 | bauzas | for the cell DB | |