Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-14
15:01:33 gibi lyarwood: ack
15:29:15 artom stephenfin, when you have time, mind taking another look at https://review.opendev.org/c/openstack/nova-specs/+/765551 ? It feels awefully short to me, but I think I've covered everything...?
15:29:36 gibi lyarwood: as far as I see https://review.opendev.org/c/openstack/nova/+/754702 does not need to depend on the libvirt bump as our current minimum libvirt (5.0.0) already bigger than what is needed (4.1.0) for this patch
15:44:01 andrewbogott melwitt: on the offhand chance that you are around… I'm in the midst of a db migration from rocky to stein and the migration is stuck in a loop. the stuck phase seems to be from your patch https://review.opendev.org/c/openstack/nova/+/633351 (which I can see you refactored quite a lot in T so I'm wondering if this is a known issue)
15:44:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: DNM try to replace retry with libvirt event in detach https://review.opendev.org/c/openstack/nova/+/770246
15:44:13 andrewbogott obviously you're under no obligation to provide me with tech support :)
15:47:20 sean-k-mooney andrewbogott: do you have an error
15:47:46 andrewbogott sean-k-mooney: stdout says this, forever: "50 rows matched query populate_user_id, 18 migrated"
15:47:49 andrewbogott error log says...
15:48:02 andrewbogott https://www.irccloud.com/pastebin/pNsHlrU0/
15:48:09 andrewbogott ^ also forever
15:48:19 andrewbogott hm, which doesn't seem to be from the same migration phase as stdout is reporting
15:48:57 sean-k-mooney ya that for keypairs
15:49:49 andrewbogott seems so, although I don't know what gives with stdout logging a different phase
15:49:51 sean-k-mooney so it kind of sounds like it was not in a consitent stae when you start the online migrations
15:50:15 andrewbogott It does
15:50:15 gibi nova meeting starts in 10 minutes in #openstack-meeting-3
15:51:09 andrewbogott Although I've no idea how it got here, this is a cloud that's been running since Mitaka
15:52:40 andrewbogott Is the 'keypair' referred to there the user-defined keypairs for ssh access?
15:52:51 andrewbogott Because our setup doesn't use those at all, I'd expect it to be unset in all cases.
15:52:58 andrewbogott (we have a PAM setup for ssh)
15:53:25 andrewbogott sean-k-mooney: any chance that we're just a use case that wasn't anticipated by that code?
15:59:27 lyarwood gibi: ah sorry, it's likely left over from the previous bump cleanup and I rebased it into this series by mistake
15:59:38 lyarwood gibi: feel free to pull that out and rebase it under your changes
16:00:11 gibi lyarwood: ack
16:00:11 gibi lyarwood: I think I will do it but not today
16:00:30 lyarwood gibi: np
16:06:18 sean-k-mooney andrewbogott: its the user keypair yes
16:06:42 sean-k-mooney not an unique constratint or forien key at the db level
16:06:55 andrewbogott and yet the migration fails if it's unset :/
16:07:10 andrewbogott I don't see keypairs in my db though, I see mostly NULLS and some json elsewhere
16:07:14 andrewbogott https://www.irccloud.com/pastebin/CIh6rCTi/
16:07:15 sean-k-mooney yep so it looke like you missed a previous migration to populate tehm
16:09:54 andrewbogott maybe. My test cloud has the same weird state. Lots of nulls, a few json objects.
16:10:23 andrewbogott anyway I can insert the empty json object into my null records
16:13:31 andrewbogott ok, I've confirmed that the keypair issue is not the thing that's breaking my migration. It fails gracefully if there are no keypairs, as it should.
16:29:23 lyarwood melwitt: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/711625 - so this works if I enable c-bak and c-vol on the subnode, fun.
16:43:09 sean-k-mooney bauzas: so lookign that the cyborg vgpu spec the last version i reviewd was version 9
16:43:16 sean-k-mooney bauzas: what was your outstanding issue
16:43:51 bauzas sean-k-mooney: the 'other deployer impact' section, starting PS12 AFAICR
16:44:11 sean-k-mooney it looks like many of my comments were adress but i need to review it porperly
16:44:18 sean-k-mooney ah the advataghes stuff
16:44:32 bauzas and the fact that they weren't using the same methods for modifying the XML, but eventually the last revision fixes it
16:44:53 sean-k-mooney ya
16:45:03 sean-k-mooney that is one of the the thigns i rememebred
16:45:03 bauzas anyway, /me needs to stop working early
16:45:17 sean-k-mooney i porably would just drop the comparison form the spec
16:45:29 sean-k-mooney in the other deployer impact section
16:45:42 sean-k-mooney and jsut leave the note about not configuring the same device in both services
16:45:43 bauzas sean-k-mooney: that's my point indeed
16:45:48 bauzas yeah
16:46:16 sean-k-mooney ok ill leave the same comment after i do a full review
16:46:32 bauzas my other concern would be to maybe use the same module for creating both inventories
16:46:46 bauzas but we could do it as a follow-up
16:46:53 bauzas like, creating a common package
16:47:13 bauzas for both the cyborg agent and the nova-libvirt code
16:47:29 bauzas this way, gibi's concerns about maintenance would be fixed
16:47:47 bauzas if both services would eventually share the same code
16:47:59 bauzas anyway, /me needs to run
16:48:20 sean-k-mooney as in a shared lib
16:48:37 sean-k-mooney we could but im not sure we want to create a new repo at this point in the cycle
16:49:00 sean-k-mooney given the pacakging work that would entail
16:49:18 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP zuul: Add nova-live-migration-ceph job https://review.opendev.org/c/openstack/nova/+/768466
16:50:08 sean-k-mooney bauzas: i tought gibis main concern in relation to code duplciation was the rest of the code for tracking and assigning mdevs
16:50:19 sean-k-mooney although i may have misread that
16:52:20 sean-k-mooney andrewbogott: by the way did you do the normal db migration before running the online ones?
16:52:32 sean-k-mooney andrewbogott: e.g. nova-manage db sync
16:52:35 andrewbogott I did
16:53:13 andrewbogott sean-k-mooney: I've now determined that the keypair errors were a red herring. It might be that the whole thing was a red herring and stdout was just overly verbose but I need to circle back.
16:53:26 andrewbogott the keypair migration throws a log warning about missing keypairs but moves on as it should
16:53:48 sean-k-mooney andrewbogott: so it might be related to this
16:53:50 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/633351/31/nova/objects/instance_mapping.py#250
16:54:02 sean-k-mooney we skipi instance that dont have instance cell mappings
16:54:29 sean-k-mooney so before you run that you need to ensure all instance have the cell mappings set
16:54:29 andrewbogott yeah, I think that's likely the issue. I need to get out of my deployment window and then I'll circle back and debug :)
16:55:01 andrewbogott hm, I thought that the cell mapping thing was handled by an upgrade script many versions ago
16:55:09 andrewbogott but since I only have one cell it should be easy to fix by hand in the db
16:56:14 sean-k-mooney it could be deleted but unachived/purged isntances
16:56:28 sean-k-mooney which is why it suggest nova-manage db archive_deleted_rows --purge
16:57:19 andrewbogott yeah, I ran that but it didn't do anything (didn't report anything on stdout and also didn't affect behavior of the migration)
17:01:17 sean-k-mooney ya so i would guess you jsut need to add the mappings and hopefully it will correct the isseu you have
17:19:16 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING https://review.opendev.org/c/openstack/nova/+/754702
17:19:18 lyarwood gibi: ^ fixed, sorry about that.
17:24:24 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING https://review.opendev.org/c/openstack/nova/+/754702
17:36:49 lyarwood does anyone recall where we list the supported distros for a release?
17:37:06 openstackgerrit Stephen Finucane proposed openstack/nova-specs master: Add spec for configurable-instance-hostnames https://review.opendev.org/c/openstack/nova-specs/+/770842
17:37:08 stephenfin lyarwood: that was in the wiki
17:37:24 stephenfin actually, no - I'm thinking of the libvirt version matrix
17:38:04 lyarwood https://github.com/openstack/governance/blob/2bdd9cff00fb40b2f95b66cad47ae1cfd14a2f1b/reference/project-testing-interface.rst#linux-distributions is all I can find
17:38:16 stephenfin artom: Looking at your spec now, but just FYI I've a super low priority proposal for a '--hostname' parameter here https://review.opendev.org/c/openstack/nova-specs/+/770842 Had a look earlier and it's pretty easy
17:38:19 lyarwood and yeah our distro matrix in the wiki just lists versions
17:38:27 stephenfin lyarwood: fungi would probably know
17:38:33 stephenfin lyarwood++ good call
17:38:57 fungi it's in governance, getting it now
17:39:48 fungi https://governance.openstack.org/tc/reference/runtimes/wallaby.html
17:39:49 lyarwood oh sorry I see the runtime link at the bottom now, helps if I view the hosted page instead of the source
17:39:57 lyarwood fungi: many thanks
17:40:00 fungi yw
17:42:41 sean-k-mooney fungi: that does not cover libvirt verions
17:42:46 sean-k-mooney or qemu

Earlier   Later