Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-05-20
11:53:04 brinzhang_ sean-k-mooney: IMO, you should move https://review.opendev.org/#/c/729563/1/nova/compute/api.py@4041 this common block code to the cyborg as a common function,
11:53:39 sean-k-mooney yep that is more or less what i was referint too when i commented on your follow up
11:53:41 brinzhang_ sean-mooney: I was rebased on your patch, if I move this that you cannot call it in your patch
11:54:02 sean-k-mooney i noticed you wer basically doint the same thing and extracting it to a common function we can both use makes sense
11:56:06 sean-k-mooney brinzhang_: i was also being lazy and suggesting you could refactor my code :P but ill see if i can update it later today
11:56:07 brinzhang after you update, I will update again, to call the common function ^
11:56:51 brinzhang sean-k-mooney: ok, if I have time tomorrow, I will do
11:57:25 brinzhang I am meeting now, after meetting , I will go home today.
11:57:44 sean-k-mooney brinzhang: cool no worries i have a meeting in an hour. im not sure if i will get it done before then but ill try to update it in the afternoon
11:58:03 brinzhang sean-k-mooney: thanks ^^
11:58:59 sean-k-mooney brinzhang: while you are hear you said i should handel the excepiotn in rebuild like is done at X but the link you provded did not have any expcetion handeling
11:59:12 sean-k-mooney can you let me know what you wanted me to do
11:59:36 sean-k-mooney brinzhang: im refering to this comment https://review.opendev.org/#/c/715326/11/nova/conductor/manager.py@1232
12:00:01 brinzhang create_and_bind_arqs will raise an exception, when bind failed, so I think that should catch the exception
12:00:49 sean-k-mooney ah right
12:01:47 sean-k-mooney i taught i was handeling that elsewhere but perhaps i missed updating this code path
12:01:55 sean-k-mooney ill take a look
12:02:07 brinzhang ah, you are busy in works
12:03:03 sean-k-mooney not too busy but im trying to start wraping my head around the numa in plamcnet code changes so im trying to context switch away form cyborg for a day or two
12:03:31 sean-k-mooney that said i have not really started on the numa in placment stuff yet so i still have the cyborg context loaded in my brain for the moment
12:04:33 sean-k-mooney im still happy to review your cyborg patches but that why im slitly distracted as im preparing a second dev env to work on the numa stuff in parallel
12:04:50 brinzhang I can understand you sean-k-mooney, I will try my best to do these thing, to reduce your work in cybrog
12:06:06 brinzhang sean-k-money: hope you can keep cyborg service in your environment, maybe need your test in your environment for some actions
12:06:17 sean-k-mooney brinzhang: well i actully dont mind working on the cycborg stuff its just on on our internal roadmap for the next major release e.g. 17.0 to have it productised so i have to blance it with stuff that is on our road map
12:06:37 sean-k-mooney brinzhang: :) that i why im setting up a second multi node dev env for numa
12:06:57 sean-k-mooney brinzhang: i have openstack deploy at home so im just created 2 more vms for working on numa
12:07:08 sean-k-mooney im keeping my cyborg vms running too
12:07:25 brinzhang thanks ^^
12:08:04 brinzhang I have to continue my meetting sean-k-mooney
12:13:55 nightmare_unreal hello, sometimes when i run openstack server migrate , it does not throw any error but doesn't actually migrate. But when I run cmd again it works ( if I run it 3rd time then it throws error which is normal)
12:14:28 nightmare_unreal is it possible that sometimes the cmd are ignored or something ? I am not sure ( I will try to reproduce it)
12:14:43 sean-k-mooney nightmare_unreal: no the command should not be ignored
12:14:54 sean-k-mooney we auto rollback on some internal errors
12:15:15 sean-k-mooney did you do resize --confrim on the second migration before the third
12:15:39 sean-k-mooney if you did not confim the cold migration then i would expect the third to error
12:15:42 nightmare_unreal if I don't do resize confirm then it throws error : which is expected and intended behavior
12:15:58 nightmare_unreal but once i do confirm and run cmd again it works ( which it should IMO)
12:16:04 sean-k-mooney yep
12:16:37 sean-k-mooney so where you confriming after the second sucessfful migration and then seeing it error on the third migration
12:17:56 sean-k-mooney if you look in the compute node logs for the first migration there likely is an error somewhere that caused a rollback. similarly i would expect there to be an error somewhere for the thrid failid migration
12:17:57 nightmare_unreal I got it
12:18:02 nightmare_unreal i can share the screenshot
12:18:06 nightmare_unreal this happened again
12:18:31 nightmare_unreal cold migration strange behavior https://usercontent.irccloud-cdn.com/file/NvweRHRz/weird_cold.png
12:18:31 sean-k-mooney sure, link?
12:18:41 nightmare_unreal is it available ? above
12:18:48 sean-k-mooney yep
12:19:04 sean-k-mooney so migration is an async operation
12:19:26 sean-k-mooney so i dont think we expect the client to return anything do we?
12:19:27 nightmare_unreal i ran the 1st cmd check the status it was still in SHUTOFF , I ran 2nd time now it's in RESIZE mode
12:19:52 nightmare_unreal now I ran 3rd time it throws error like it should Cannot 'migrate' instance aef212b9-6de7-4869-877c-a21c0b08de69 while it is in vm_state resized
12:19:55 sean-k-mooney did you run them back to back
12:20:05 sean-k-mooney well iguess you waited
12:20:08 sean-k-mooney to check it was off
12:20:11 nightmare_unreal I waited for like 5 sec
12:20:21 lyarwood yeah it takes longer to shutoff the instance
12:20:32 lyarwood we try to do a soft power off iirc
12:20:32 sean-k-mooney ya i was wondering if you were racing with the instance state change
12:20:35 nightmare_unreal the VM was already in shutoff mode though
12:20:52 lyarwood if it's already shutoff then that's different
12:21:21 sean-k-mooney you still could be racing with the change to move ti to resizing
12:21:26 nightmare_unreal thing is I am making script to do some automation for migration . and I rely on status changes for e.g. if state changes from SHUTOFF to resize then cmd worked
12:21:41 nightmare_unreal probably
12:22:02 sean-k-mooney nightmare_unreal: the first command should still work without the second
12:22:26 nightmare_unreal true , maybe it should throw error then.
12:22:38 sean-k-mooney maybe just add a retry loop to your state change check
12:22:51 sean-k-mooney e.g. try 3 times with a 5 second sleep or something
12:22:56 nightmare_unreal yeah i was thinking same. maybe 3 tries with some timeout
12:23:06 nightmare_unreal cool thanks sean-k-mooney
12:55:31 openstackgerrit Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301
12:56:21 jsuchome lyarwood: new PS ^. It's not pretty, but I think it's more readable than before.
13:23:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596
13:23:51 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595
13:24:28 lyarwood kashyap: random question, what's the preferred way of polling blockjobs today? A comment from Matt I've removed from the comments above suggests there was an events framework we could use?
13:24:38 lyarwood jsuchome: ack will look shortly
13:24:57 kashyap lyarwood: Hiya
13:25:17 kashyap lyarwood: Yes, the idea we discussed in the past was to listen on:
13:25:36 kashyap _BLOCK_JOB_READY events that QEMU (which in turn libvirt also relays) emits once a copy job has synced
13:26:04 kashyap lyarwood: I recall writin up that design somewhere, need to dig; but buried in a few downstream tasks before I go offline tomm & Fri
13:26:27 kashyap Give me a few, I'll find the relevant pointers :)
13:26:29 lyarwood ack np
13:26:33 lyarwood thanks
13:29:06 kashyap lyarwood: Context-1: the thorny problem CI in Nova identified was this write-up: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html ("Debugging blockRebase() - "active block copy not ready for pivot")
13:29:38 openstack bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa
13:29:38 kashyap lyarwood: Context-2: from the above, I filed this libvirt bug in 2016, and is fixed (in Jan-2019): https://bugzilla.redhat.com/show_bug.cgi?id=1382165
13:31:45 kashyap lyarwood: Context-3: https://libvirt.org/git/?p=libvirt.git;a=commit;h=73ce3911a
13:31:49 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596
13:31:49 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595
13:31:53 kashyap (There "some clients" is referring to OpenStack :-))
13:32:25 kashyap So yeah, in short: what's needed is to listen on VIR_DOMAIN_BLOCK_JOB_READY event.
13:32:49 kashyap Instead of polling (to avoid the race that we debugged in the Context-1 link)
13:34:23 lyarwood https://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainEventBlockJobCallback using this I assume?
13:34:32 lyarwood we don't want to listen over QMP
13:35:01 kashyap lyarwood: No, we're not doing over QMP, of course
13:35:17 kashyap libvirt itself emits events; there's even an XML flag for the 'ready' bit
13:35:43 lyarwood kk
13:35:52 kashyap (The actual thing is being transmitted over QMP anyway; but we don't manually do it; that's what I meant)
13:36:52 lyarwood yup understood thanks
13:37:16 kashyap And check out this bit (read it in full in the URL): "The attribute 'ready', if present, tracks progress of the job [...]": https://libvirt.org/formatdomain.html
13:37:26 kashyap [Under the 'mirror' documentation]

Earlier   Later