| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-20 | |||
| 09:48:35 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: [WIP] Use annotated ddt for test_cpu_policy_constraint https://review.opendev.org/692205 | |
| 09:48:38 | jsuchome | lyarwood: Hi, I've addressed your comments, in some parts we still disagree (same with some dansmith's ones btw) | |
| 09:50:48 | lyarwood | jsuchome: ack, I'll take a look now | |
| 10:13:30 | openstackgerrit | Huaqiang Wang proposed openstack/nova master: test_hardware: show test parameter details for causing a test failure https://review.opendev.org/692205 | |
| 10:47:16 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Cyborg shelve/unshelve support https://review.opendev.org/729563 | |
| 11:08:11 | openstackgerrit | Merged openstack/nova master: trivial: Remove remaining '_LE' instances https://review.opendev.org/728963 | |
| 11:20:10 | lyarwood | jsuchome: sorry was side tracked, just updated. | |
| 11:20:59 | lyarwood | jsuchome: apologies for the mistakes before, I really think we need to break this up a little to make the logic more clear to reviewers | |
| 11:26:09 | jsuchome | lyarwood: the main part is somewhat confusing, I agree ... I'm not sure though how to make it better | |
| 11:26:58 | jsuchome | maybe by moving the verification bits into different method... | |
| 11:52:28 | lyarwood | jsuchome: yeah I think that would be useful | |
| 11:53:04 | brinzhang_ | sean-k-mooney: IMO, you should move https://review.opendev.org/#/c/729563/1/nova/compute/api.py@4041 this common block code to the cyborg as a common function, | |
| 11:53:39 | sean-k-mooney | yep that is more or less what i was referint too when i commented on your follow up | |
| 11:53:41 | brinzhang_ | sean-mooney: I was rebased on your patch, if I move this that you cannot call it in your patch | |
| 11:54:02 | sean-k-mooney | i noticed you wer basically doint the same thing and extracting it to a common function we can both use makes sense | |
| 11:56:06 | sean-k-mooney | brinzhang_: i was also being lazy and suggesting you could refactor my code :P but ill see if i can update it later today | |
| 11:56:07 | brinzhang | after you update, I will update again, to call the common function ^ | |
| 11:56:51 | brinzhang | sean-k-mooney: ok, if I have time tomorrow, I will do | |
| 11:57:25 | brinzhang | I am meeting now, after meetting , I will go home today. | |
| 11:57:44 | sean-k-mooney | brinzhang: cool no worries i have a meeting in an hour. im not sure if i will get it done before then but ill try to update it in the afternoon | |
| 11:58:03 | brinzhang | sean-k-mooney: thanks ^^ | |
| 11:58:59 | sean-k-mooney | brinzhang: while you are hear you said i should handel the excepiotn in rebuild like is done at X but the link you provded did not have any expcetion handeling | |
| 11:59:12 | sean-k-mooney | can you let me know what you wanted me to do | |
| 11:59:36 | sean-k-mooney | brinzhang: im refering to this comment https://review.opendev.org/#/c/715326/11/nova/conductor/manager.py@1232 | |
| 12:00:01 | brinzhang | create_and_bind_arqs will raise an exception, when bind failed, so I think that should catch the exception | |
| 12:00:49 | sean-k-mooney | ah right | |
| 12:01:47 | sean-k-mooney | i taught i was handeling that elsewhere but perhaps i missed updating this code path | |
| 12:01:55 | sean-k-mooney | ill take a look | |
| 12:02:07 | brinzhang | ah, you are busy in works | |
| 12:03:03 | sean-k-mooney | not too busy but im trying to start wraping my head around the numa in plamcnet code changes so im trying to context switch away form cyborg for a day or two | |
| 12:03:31 | sean-k-mooney | that said i have not really started on the numa in placment stuff yet so i still have the cyborg context loaded in my brain for the moment | |
| 12:04:33 | sean-k-mooney | im still happy to review your cyborg patches but that why im slitly distracted as im preparing a second dev env to work on the numa stuff in parallel | |
| 12:04:50 | brinzhang | I can understand you sean-k-mooney, I will try my best to do these thing, to reduce your work in cybrog | |
| 12:06:06 | brinzhang | sean-k-money: hope you can keep cyborg service in your environment, maybe need your test in your environment for some actions | |
| 12:06:17 | sean-k-mooney | brinzhang: well i actully dont mind working on the cycborg stuff its just on on our internal roadmap for the next major release e.g. 17.0 to have it productised so i have to blance it with stuff that is on our road map | |
| 12:06:37 | sean-k-mooney | brinzhang: :) that i why im setting up a second multi node dev env for numa | |
| 12:06:57 | sean-k-mooney | brinzhang: i have openstack deploy at home so im just created 2 more vms for working on numa | |
| 12:07:08 | sean-k-mooney | im keeping my cyborg vms running too | |
| 12:07:25 | brinzhang | thanks ^^ | |
| 12:08:04 | brinzhang | I have to continue my meetting sean-k-mooney | |
| 12:13:55 | nightmare_unreal | hello, sometimes when i run openstack server migrate , it does not throw any error but doesn't actually migrate. But when I run cmd again it works ( if I run it 3rd time then it throws error which is normal) | |
| 12:14:28 | nightmare_unreal | is it possible that sometimes the cmd are ignored or something ? I am not sure ( I will try to reproduce it) | |
| 12:14:43 | sean-k-mooney | nightmare_unreal: no the command should not be ignored | |
| 12:14:54 | sean-k-mooney | we auto rollback on some internal errors | |
| 12:15:15 | sean-k-mooney | did you do resize --confrim on the second migration before the third | |
| 12:15:39 | sean-k-mooney | if you did not confim the cold migration then i would expect the third to error | |
| 12:15:42 | nightmare_unreal | if I don't do resize confirm then it throws error : which is expected and intended behavior | |
| 12:15:58 | nightmare_unreal | but once i do confirm and run cmd again it works ( which it should IMO) | |
| 12:16:04 | sean-k-mooney | yep | |
| 12:16:37 | sean-k-mooney | so where you confriming after the second sucessfful migration and then seeing it error on the third migration | |
| 12:17:56 | sean-k-mooney | if you look in the compute node logs for the first migration there likely is an error somewhere that caused a rollback. similarly i would expect there to be an error somewhere for the thrid failid migration | |
| 12:17:57 | nightmare_unreal | I got it | |
| 12:18:02 | nightmare_unreal | i can share the screenshot | |
| 12:18:06 | nightmare_unreal | this happened again | |
| 12:18:31 | sean-k-mooney | sure, link? | |
| 12:18:31 | nightmare_unreal | cold migration strange behavior https://usercontent.irccloud-cdn.com/file/NvweRHRz/weird_cold.png | |
| 12:18:41 | nightmare_unreal | is it available ? above | |
| 12:18:48 | sean-k-mooney | yep | |
| 12:19:04 | sean-k-mooney | so migration is an async operation | |
| 12:19:26 | sean-k-mooney | so i dont think we expect the client to return anything do we? | |
| 12:19:27 | nightmare_unreal | i ran the 1st cmd check the status it was still in SHUTOFF , I ran 2nd time now it's in RESIZE mode | |
| 12:19:52 | nightmare_unreal | now I ran 3rd time it throws error like it should Cannot 'migrate' instance aef212b9-6de7-4869-877c-a21c0b08de69 while it is in vm_state resized | |
| 12:19:55 | sean-k-mooney | did you run them back to back | |
| 12:20:05 | sean-k-mooney | well iguess you waited | |
| 12:20:08 | sean-k-mooney | to check it was off | |
| 12:20:11 | nightmare_unreal | I waited for like 5 sec | |
| 12:20:21 | lyarwood | yeah it takes longer to shutoff the instance | |
| 12:20:32 | sean-k-mooney | ya i was wondering if you were racing with the instance state change | |
| 12:20:32 | lyarwood | we try to do a soft power off iirc | |
| 12:20:35 | nightmare_unreal | the VM was already in shutoff mode though | |
| 12:20:52 | lyarwood | if it's already shutoff then that's different | |
| 12:21:21 | sean-k-mooney | you still could be racing with the change to move ti to resizing | |
| 12:21:26 | nightmare_unreal | thing is I am making script to do some automation for migration . and I rely on status changes for e.g. if state changes from SHUTOFF to resize then cmd worked | |
| 12:21:41 | nightmare_unreal | probably | |
| 12:22:02 | sean-k-mooney | nightmare_unreal: the first command should still work without the second | |
| 12:22:26 | nightmare_unreal | true , maybe it should throw error then. | |
| 12:22:38 | sean-k-mooney | maybe just add a retry loop to your state change check | |
| 12:22:51 | sean-k-mooney | e.g. try 3 times with a 5 second sleep or something | |
| 12:22:56 | nightmare_unreal | yeah i was thinking same. maybe 3 tries with some timeout | |
| 12:23:06 | nightmare_unreal | cool thanks sean-k-mooney | |
| 12:55:31 | openstackgerrit | Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301 | |
| 12:56:21 | jsuchome | lyarwood: new PS ^. It's not pretty, but I think it's more readable than before. | |
| 13:23:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595 | |
| 13:23:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596 | |
| 13:24:28 | lyarwood | kashyap: random question, what's the preferred way of polling blockjobs today? A comment from Matt I've removed from the comments above suggests there was an events framework we could use? | |
| 13:24:38 | lyarwood | jsuchome: ack will look shortly | |
| 13:24:57 | kashyap | lyarwood: Hiya | |
| 13:25:17 | kashyap | lyarwood: Yes, the idea we discussed in the past was to listen on: | |
| 13:25:36 | kashyap | _BLOCK_JOB_READY events that QEMU (which in turn libvirt also relays) emits once a copy job has synced | |
| 13:26:04 | kashyap | lyarwood: I recall writin up that design somewhere, need to dig; but buried in a few downstream tasks before I go offline tomm & Fri | |
| 13:26:27 | kashyap | Give me a few, I'll find the relevant pointers :) | |
| 13:26:29 | lyarwood | ack np | |
| 13:26:33 | lyarwood | thanks | |
| 13:29:06 | kashyap | lyarwood: Context-1: the thorny problem CI in Nova identified was this write-up: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html ("Debugging blockRebase() - "active block copy not ready for pivot") | |
| 13:29:38 | kashyap | lyarwood: Context-2: from the above, I filed this libvirt bug in 2016, and is fixed (in Jan-2019): https://bugzilla.redhat.com/show_bug.cgi?id=1382165 | |
| 13:29:38 | openstack | bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa | |
| 13:31:45 | kashyap | lyarwood: Context-3: https://libvirt.org/git/?p=libvirt.git;a=commit;h=73ce3911a | |
| 13:31:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595 | |
| 13:31:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596 | |
| 13:31:53 | kashyap | (There "some clients" is referring to OpenStack :-)) | |