| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-05-20 | |||
| 12:07:08 | sean-k-mooney | im keeping my cyborg vms running too | |
| 12:07:25 | brinzhang | thanks ^^ | |
| 12:08:04 | brinzhang | I have to continue my meetting sean-k-mooney | |
| 12:13:55 | nightmare_unreal | hello, sometimes when i run openstack server migrate , it does not throw any error but doesn't actually migrate. But when I run cmd again it works ( if I run it 3rd time then it throws error which is normal) | |
| 12:14:28 | nightmare_unreal | is it possible that sometimes the cmd are ignored or something ? I am not sure ( I will try to reproduce it) | |
| 12:14:43 | sean-k-mooney | nightmare_unreal: no the command should not be ignored | |
| 12:14:54 | sean-k-mooney | we auto rollback on some internal errors | |
| 12:15:15 | sean-k-mooney | did you do resize --confrim on the second migration before the third | |
| 12:15:39 | sean-k-mooney | if you did not confim the cold migration then i would expect the third to error | |
| 12:15:42 | nightmare_unreal | if I don't do resize confirm then it throws error : which is expected and intended behavior | |
| 12:15:58 | nightmare_unreal | but once i do confirm and run cmd again it works ( which it should IMO) | |
| 12:16:04 | sean-k-mooney | yep | |
| 12:16:37 | sean-k-mooney | so where you confriming after the second sucessfful migration and then seeing it error on the third migration | |
| 12:17:56 | sean-k-mooney | if you look in the compute node logs for the first migration there likely is an error somewhere that caused a rollback. similarly i would expect there to be an error somewhere for the thrid failid migration | |
| 12:17:57 | nightmare_unreal | I got it | |
| 12:18:02 | nightmare_unreal | i can share the screenshot | |
| 12:18:06 | nightmare_unreal | this happened again | |
| 12:18:31 | sean-k-mooney | sure, link? | |
| 12:18:31 | nightmare_unreal | cold migration strange behavior https://usercontent.irccloud-cdn.com/file/NvweRHRz/weird_cold.png | |
| 12:18:41 | nightmare_unreal | is it available ? above | |
| 12:18:48 | sean-k-mooney | yep | |
| 12:19:04 | sean-k-mooney | so migration is an async operation | |
| 12:19:26 | sean-k-mooney | so i dont think we expect the client to return anything do we? | |
| 12:19:27 | nightmare_unreal | i ran the 1st cmd check the status it was still in SHUTOFF , I ran 2nd time now it's in RESIZE mode | |
| 12:19:52 | nightmare_unreal | now I ran 3rd time it throws error like it should Cannot 'migrate' instance aef212b9-6de7-4869-877c-a21c0b08de69 while it is in vm_state resized | |
| 12:19:55 | sean-k-mooney | did you run them back to back | |
| 12:20:05 | sean-k-mooney | well iguess you waited | |
| 12:20:08 | sean-k-mooney | to check it was off | |
| 12:20:11 | nightmare_unreal | I waited for like 5 sec | |
| 12:20:21 | lyarwood | yeah it takes longer to shutoff the instance | |
| 12:20:32 | sean-k-mooney | ya i was wondering if you were racing with the instance state change | |
| 12:20:32 | lyarwood | we try to do a soft power off iirc | |
| 12:20:35 | nightmare_unreal | the VM was already in shutoff mode though | |
| 12:20:52 | lyarwood | if it's already shutoff then that's different | |
| 12:21:21 | sean-k-mooney | you still could be racing with the change to move ti to resizing | |
| 12:21:26 | nightmare_unreal | thing is I am making script to do some automation for migration . and I rely on status changes for e.g. if state changes from SHUTOFF to resize then cmd worked | |
| 12:21:41 | nightmare_unreal | probably | |
| 12:22:02 | sean-k-mooney | nightmare_unreal: the first command should still work without the second | |
| 12:22:26 | nightmare_unreal | true , maybe it should throw error then. | |
| 12:22:38 | sean-k-mooney | maybe just add a retry loop to your state change check | |
| 12:22:51 | sean-k-mooney | e.g. try 3 times with a 5 second sleep or something | |
| 12:22:56 | nightmare_unreal | yeah i was thinking same. maybe 3 tries with some timeout | |
| 12:23:06 | nightmare_unreal | cool thanks sean-k-mooney | |
| 12:55:31 | openstackgerrit | Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301 | |
| 12:56:21 | jsuchome | lyarwood: new PS ^. It's not pretty, but I think it's more readable than before. | |
| 13:23:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595 | |
| 13:23:51 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596 | |
| 13:24:28 | lyarwood | kashyap: random question, what's the preferred way of polling blockjobs today? A comment from Matt I've removed from the comments above suggests there was an events framework we could use? | |
| 13:24:38 | lyarwood | jsuchome: ack will look shortly | |
| 13:24:57 | kashyap | lyarwood: Hiya | |
| 13:25:17 | kashyap | lyarwood: Yes, the idea we discussed in the past was to listen on: | |
| 13:25:36 | kashyap | _BLOCK_JOB_READY events that QEMU (which in turn libvirt also relays) emits once a copy job has synced | |
| 13:26:04 | kashyap | lyarwood: I recall writin up that design somewhere, need to dig; but buried in a few downstream tasks before I go offline tomm & Fri | |
| 13:26:27 | kashyap | Give me a few, I'll find the relevant pointers :) | |
| 13:26:29 | lyarwood | ack np | |
| 13:26:33 | lyarwood | thanks | |
| 13:29:06 | kashyap | lyarwood: Context-1: the thorny problem CI in Nova identified was this write-up: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105158.html ("Debugging blockRebase() - "active block copy not ready for pivot") | |
| 13:29:38 | kashyap | lyarwood: Context-2: from the above, I filed this libvirt bug in 2016, and is fixed (in Jan-2019): https://bugzilla.redhat.com/show_bug.cgi?id=1382165 | |
| 13:29:38 | openstack | bugzilla.redhat.com bug 1382165 in libvirt "virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs`" [Unspecified,Closed: nextrelease] - Assigned to pkrempa | |
| 13:31:45 | kashyap | lyarwood: Context-3: https://libvirt.org/git/?p=libvirt.git;a=commit;h=73ce3911a | |
| 13:31:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595 | |
| 13:31:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596 | |
| 13:31:53 | kashyap | (There "some clients" is referring to OpenStack :-)) | |
| 13:32:25 | kashyap | So yeah, in short: what's needed is to listen on VIR_DOMAIN_BLOCK_JOB_READY event. | |
| 13:32:49 | kashyap | Instead of polling (to avoid the race that we debugged in the Context-1 link) | |
| 13:34:23 | lyarwood | https://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainEventBlockJobCallback using this I assume? | |
| 13:34:32 | lyarwood | we don't want to listen over QMP | |
| 13:35:01 | kashyap | lyarwood: No, we're not doing over QMP, of course | |
| 13:35:17 | kashyap | libvirt itself emits events; there's even an XML flag for the 'ready' bit | |
| 13:35:43 | lyarwood | kk | |
| 13:35:52 | kashyap | (The actual thing is being transmitted over QMP anyway; but we don't manually do it; that's what I meant) | |
| 13:36:52 | lyarwood | yup understood thanks | |
| 13:37:16 | kashyap | And check out this bit (read it in full in the URL): "The attribute 'ready', if present, tracks progress of the job [...]": https://libvirt.org/formatdomain.html | |
| 13:37:26 | kashyap | [Under the 'mirror' documentation] | |
| 13:39:29 | lyarwood | kashyap: yeah we already poll on that | |
| 13:40:21 | kashyap | Ah, nod. /me bbiab; need to finish this other bit early. | |
| 13:40:26 | lyarwood | kashyap: I added that a while ago https://review.opendev.org/#/c/399735/ | |
| 13:40:29 | kashyap | (But I reviewed your clean-up; looks good to me) | |
| 13:40:54 | lyarwood | ack thanks | |
| 13:41:06 | kashyap | lyarwood: Oh, yea; now I recall it. Thx | |
| 14:29:43 | lyarwood | artom / melwitt / stephenfin ; would you all mind taking a swing at https://review.opendev.org/#/c/631294/ sometime this week? | |
| 14:30:46 | artom | lyarwood, ack | |
| 14:31:55 | lyarwood | thanks, it's a total pig thanks to the backport requirement but all I can do at the moment | |
| 14:32:28 | openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: template: consider openstack client besides novaclient https://review.opendev.org/717722 | |
| 14:47:53 | stephenfin | lyarwood: ack, I was waiting on the WIP patches theren ow | |
| 14:47:54 | stephenfin | *now | |
| 15:47:39 | jsuchome | lyarwood: fyi, I missed broken tests, will have to rewrite it once more... | |
| 15:50:15 | lyarwood | jsuchome: ack np I was waiting on CI for that very reason :) | |
| 15:54:49 | jsuchome | I accidentally ran wrong tests localy and was surprised it was passing :-) | |
| 16:05:29 | openstackgerrit | Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301 | |
| 16:11:52 | bauzas | gibi: fwiw, I'll be off tomorrow and Friday | |
| 16:12:19 | bauzas | May the Force be with French folks | |
| 16:15:20 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'Migration' https://review.opendev.org/723572 | |
| 16:15:21 | openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Remove 'NovaObjectDictCompat' from 'InstancePCIRequest' https://review.opendev.org/723573 | |
| 16:16:59 | openstackgerrit | Merged openstack/nova stable/stein: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/726862 | |
| 16:17:18 | openstackgerrit | Merged openstack/nova master: Add admin doc information about image cache resource accounting https://review.opendev.org/729350 | |
| 16:17:33 | openstackgerrit | Merged openstack/nova master: Increase code reuse in test_numa_servers https://review.opendev.org/714125 | |
| 16:30:21 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Track blockjob status in DEBUG when polling for completion https://review.opendev.org/729595 | |
| 16:30:22 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Remove blockjob.end == 0 workaround resolved in libvirt v2.3.0 https://review.opendev.org/729596 | |
| 16:33:16 | openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Remove log translations https://review.opendev.org/728966 | |