Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-01-14
14:19:05 stephenfin but I just haven't had time to write a fixture or whatever to do that
14:20:34 lyarwood just dumping this here while I work on something downstream
14:20:45 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Bump MIN_{LIBVIRT,QEMU}_VERSION and NEXT_MIN_{LIBVIRT,QEMU}_VERSION https://review.opendev.org/c/openstack/nova/+/754700
14:20:46 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_{LIBVIRT,QEMU}_BLOCKDEV https://review.opendev.org/c/openstack/nova/+/754701
14:20:46 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_LIBVIRT_VIR_ERR_DEVICE_MISSING https://review.opendev.org/c/openstack/nova/+/754702
14:20:47 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_LIBVIRT_VTPM https://review.opendev.org/c/openstack/nova/+/754703
14:20:47 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP libvirt: Remove MIN_LIBVIRT_S390X_CPU_COMPARE https://review.opendev.org/c/openstack/nova/+/754704
14:20:51 lyarwood .. git-review really is slow since the update to gerrit ..
14:21:03 stephenfin I find it varies
14:21:11 stephenfin It's usually fast after a restart
14:21:28 stephenfin *feels fast
14:21:31 stephenfin I've no data to back up that assertion :-\
14:23:03 stephenfin We're getting close to API microversion 2.100. It could be another cycle or two, but that's coming
14:30:10 sean-k-mooney at some point we might go to nova v4
14:41:40 dansmith lyarwood: has seemed painfully slow to me as well
14:41:46 dansmith longer for multi-patch uploads
14:47:35 gmann brinzhang0: no, your patch is all ok as it is and fixed the 500 ->403, mine is to improve it as general when other API use this exception further with unit tests
14:48:34 gmann brinzhang0: to avoid conflict I made mine depends on top of your otherwise they both can go independent also
14:54:26 sean-k-mooney dansmith: i havent really noticed much of a change but i havent done many multi patch updates recently
14:59:58 gibi lyarwood: thanks for https://review.opendev.org/c/openstack/nova/+/754702 I will review that and rebase the libvirt event patch top of that
15:00:31 lyarwood gibi: I'll clean that series up later once I'm done with something downstream
15:00:38 lyarwood and np :)
15:01:33 gibi lyarwood: ack
15:29:15 artom stephenfin, when you have time, mind taking another look at https://review.opendev.org/c/openstack/nova-specs/+/765551 ? It feels awefully short to me, but I think I've covered everything...?
15:29:36 gibi lyarwood: as far as I see https://review.opendev.org/c/openstack/nova/+/754702 does not need to depend on the libvirt bump as our current minimum libvirt (5.0.0) already bigger than what is needed (4.1.0) for this patch
15:44:01 andrewbogott melwitt: on the offhand chance that you are around… I'm in the midst of a db migration from rocky to stein and the migration is stuck in a loop. the stuck phase seems to be from your patch https://review.opendev.org/c/openstack/nova/+/633351 (which I can see you refactored quite a lot in T so I'm wondering if this is a known issue)
15:44:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: DNM try to replace retry with libvirt event in detach https://review.opendev.org/c/openstack/nova/+/770246
15:44:13 andrewbogott obviously you're under no obligation to provide me with tech support :)
15:47:20 sean-k-mooney andrewbogott: do you have an error
15:47:46 andrewbogott sean-k-mooney: stdout says this, forever: "50 rows matched query populate_user_id, 18 migrated"
15:47:49 andrewbogott error log says...
15:48:02 andrewbogott https://www.irccloud.com/pastebin/pNsHlrU0/
15:48:09 andrewbogott ^ also forever
15:48:19 andrewbogott hm, which doesn't seem to be from the same migration phase as stdout is reporting
15:48:57 sean-k-mooney ya that for keypairs
15:49:49 andrewbogott seems so, although I don't know what gives with stdout logging a different phase
15:49:51 sean-k-mooney so it kind of sounds like it was not in a consitent stae when you start the online migrations
15:50:15 andrewbogott It does
15:50:15 gibi nova meeting starts in 10 minutes in #openstack-meeting-3
15:51:09 andrewbogott Although I've no idea how it got here, this is a cloud that's been running since Mitaka
15:52:40 andrewbogott Is the 'keypair' referred to there the user-defined keypairs for ssh access?
15:52:51 andrewbogott Because our setup doesn't use those at all, I'd expect it to be unset in all cases.
15:52:58 andrewbogott (we have a PAM setup for ssh)
15:53:25 andrewbogott sean-k-mooney: any chance that we're just a use case that wasn't anticipated by that code?
15:59:27 lyarwood gibi: ah sorry, it's likely left over from the previous bump cleanup and I rebased it into this series by mistake
15:59:38 lyarwood gibi: feel free to pull that out and rebase it under your changes
16:00:11 gibi lyarwood: ack
16:00:11 gibi lyarwood: I think I will do it but not today
16:00:30 lyarwood gibi: np
16:06:18 sean-k-mooney andrewbogott: its the user keypair yes
16:06:42 sean-k-mooney not an unique constratint or forien key at the db level
16:06:55 andrewbogott and yet the migration fails if it's unset :/
16:07:10 andrewbogott I don't see keypairs in my db though, I see mostly NULLS and some json elsewhere
16:07:14 andrewbogott https://www.irccloud.com/pastebin/CIh6rCTi/
16:07:15 sean-k-mooney yep so it looke like you missed a previous migration to populate tehm
16:09:54 andrewbogott maybe. My test cloud has the same weird state. Lots of nulls, a few json objects.
16:10:23 andrewbogott anyway I can insert the empty json object into my null records
16:13:31 andrewbogott ok, I've confirmed that the keypair issue is not the thing that's breaking my migration. It fails gracefully if there are no keypairs, as it should.
16:29:23 lyarwood melwitt: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/711625 - so this works if I enable c-bak and c-vol on the subnode, fun.
16:43:09 sean-k-mooney bauzas: so lookign that the cyborg vgpu spec the last version i reviewd was version 9
16:43:16 sean-k-mooney bauzas: what was your outstanding issue
16:43:51 bauzas sean-k-mooney: the 'other deployer impact' section, starting PS12 AFAICR
16:44:11 sean-k-mooney it looks like many of my comments were adress but i need to review it porperly
16:44:18 sean-k-mooney ah the advataghes stuff
16:44:32 bauzas and the fact that they weren't using the same methods for modifying the XML, but eventually the last revision fixes it
16:44:53 sean-k-mooney ya
16:45:03 sean-k-mooney that is one of the the thigns i rememebred
16:45:03 bauzas anyway, /me needs to stop working early
16:45:17 sean-k-mooney i porably would just drop the comparison form the spec
16:45:29 sean-k-mooney in the other deployer impact section
16:45:42 sean-k-mooney and jsut leave the note about not configuring the same device in both services
16:45:43 bauzas sean-k-mooney: that's my point indeed
16:45:48 bauzas yeah
16:46:16 sean-k-mooney ok ill leave the same comment after i do a full review
16:46:32 bauzas my other concern would be to maybe use the same module for creating both inventories
16:46:46 bauzas but we could do it as a follow-up
16:46:53 bauzas like, creating a common package
16:47:13 bauzas for both the cyborg agent and the nova-libvirt code
16:47:29 bauzas this way, gibi's concerns about maintenance would be fixed
16:47:47 bauzas if both services would eventually share the same code
16:47:59 bauzas anyway, /me needs to run
16:48:20 sean-k-mooney as in a shared lib
16:48:37 sean-k-mooney we could but im not sure we want to create a new repo at this point in the cycle
16:49:00 sean-k-mooney given the pacakging work that would entail
16:49:18 openstackgerrit Lee Yarwood proposed openstack/nova master: WIP zuul: Add nova-live-migration-ceph job https://review.opendev.org/c/openstack/nova/+/768466
16:50:08 sean-k-mooney bauzas: i tought gibis main concern in relation to code duplciation was the rest of the code for tracking and assigning mdevs
16:50:19 sean-k-mooney although i may have misread that
16:52:20 sean-k-mooney andrewbogott: by the way did you do the normal db migration before running the online ones?
16:52:32 sean-k-mooney andrewbogott: e.g. nova-manage db sync
16:52:35 andrewbogott I did
16:53:13 andrewbogott sean-k-mooney: I've now determined that the keypair errors were a red herring. It might be that the whole thing was a red herring and stdout was just overly verbose but I need to circle back.
16:53:26 andrewbogott the keypair migration throws a log warning about missing keypairs but moves on as it should
16:53:48 sean-k-mooney andrewbogott: so it might be related to this
16:53:50 sean-k-mooney https://review.opendev.org/c/openstack/nova/+/633351/31/nova/objects/instance_mapping.py#250
16:54:02 sean-k-mooney we skipi instance that dont have instance cell mappings
16:54:29 sean-k-mooney so before you run that you need to ensure all instance have the cell mappings set
16:54:29 andrewbogott yeah, I think that's likely the issue. I need to get out of my deployment window and then I'll circle back and debug :)
16:55:01 andrewbogott hm, I thought that the cell mapping thing was handled by an upgrade script many versions ago
16:55:09 andrewbogott but since I only have one cell it should be easy to fix by hand in the db
16:56:14 sean-k-mooney it could be deleted but unachived/purged isntances

Earlier   Later