Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-24
22:14:57 sean-k-mooney ya can you enable debug logs on both nodes and try again
22:15:13 spatel Let me go home and collect all nova debug and file BUG
22:15:40 spatel it seems source just deleting instance before nova migration complete...
22:16:18 sean-k-mooney spatel: sure its late here too so i was going to be droping ofline soon once my pizzia is readdy
22:16:49 sean-k-mooney if you can file a bug and attach some debug logs for the source and destination we can see why it is not copying the disk
22:17:45 spatel i will
22:17:48 spatel thanks!
22:18:06 sean-k-mooney spatel: by the way what release are you running
22:18:36 spatel queens
22:18:50 sean-k-mooney there was a error we caught just as Rocky was shipping where we treated migration compution as a success without checking if there was an error
22:18:59 sean-k-mooney ah so ya you might be hitting that bug
22:19:09 spatel damn it!! !
22:19:13 spatel really!!
22:19:29 spatel send me that BUG and i will check code
22:19:29 sean-k-mooney basically if libvirt through an error after migrtion started we did not handel it properly
22:20:40 spatel I am leaving and need to shutdown my pc but you can send me email on satish.txt@gmail.com
22:35:35 openstackgerrit Merged openstack/nova stable/pike: Revert "Make host_aggregate_map dictionary case-insensitive" https://review.openstack.org/605268
22:35:41 openstackgerrit Merged openstack/nova stable/pike: Enforce case-sensitive hostnames in aggregate host add https://review.openstack.org/605269
#openstack-nova - 2018-10-25
00:18:29 openstackgerrit melanie witt proposed openstack/nova stable/rocky: Add rolling upgrade info to enable_consoleauth workaround reno https://review.openstack.org/613174
00:54:18 openstackgerrit Tao Li proposed openstack/python-novaclient master: Deprecate the unused instance-name https://review.openstack.org/602520
01:11:46 openstackgerrit melanie witt proposed openstack/nova master: Add nova-status upgrade check for consoles https://review.openstack.org/611214
02:51:04 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add a hacking rule for deprecated assertion methods https://review.openstack.org/613184
03:03:37 openstackgerrit Fan Zhang proposed openstack/nova master: Retry after hitting libvirt error VIR_ERR_OPERATION_INVALID in live migration. https://review.openstack.org/612272
04:29:29 openstackgerrit Merged openstack/python-novaclient master: Deprecate the unused instance-name https://review.openstack.org/602520
05:38:09 openstackgerrit yenai proposed openstack/nova master: Delete redundant spaces https://review.openstack.org/613199
07:40:22 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Remove unnecessary minimum microversion https://review.openstack.org/613226
07:46:31 openstackgerrit Takashi NATSUME proposed openstack/nova master: api-ref: Add descriptions of error cases https://review.openstack.org/610429
07:46:49 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add API ref guideline for examples https://review.openstack.org/604060
07:47:03 openstackgerrit Takashi NATSUME proposed openstack/nova master: Add API ref guideline for body text https://review.openstack.org/605628
08:36:55 openstackgerrit Tetsuro Nakamura proposed openstack/nova-specs master: Spec: Support filtering by forbidden aggregate https://review.openstack.org/603352
08:38:27 openstackgerrit Seyeong Kim proposed openstack/nova master: Enable connection_info refresh for new-style attachments https://review.openstack.org/579004
09:37:34 pooja_jadhav hi team, I facing issue on nova-compute : (1054, u"Unknown column 'block_device_mapping.volume_type' in 'field list'"). For that I have did nova-manage db sync and api_db sync as well. still facing same issue. Can anyone help me to resolve this issue?
10:09:15 openstackgerrit Jan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type https://review.openstack.org/572081
10:48:08 openstackgerrit Radoslav Gerganov proposed openstack/nova stable/pike: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/613263
10:50:46 openstackgerrit Radoslav Gerganov proposed openstack/nova stable/pike: Update resources once in update_available_resource https://review.openstack.org/612295
11:02:03 openstackgerrit Radoslav Gerganov proposed openstack/nova stable/queens: Don't persist zero allocation ratios in ResourceTracker https://review.openstack.org/613271
11:04:32 openstackgerrit Radoslav Gerganov proposed openstack/nova stable/queens: Update resources once in update_available_resource https://review.openstack.org/612294
12:21:52 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Add recreate test for bug 1799892 https://review.openstack.org/613304
12:21:52 openstack bug 1799892 in OpenStack Compute (nova) "Placement API crashes with 500s in Rocky upgrade with downed compute nodes" [Undecided,New] https://launchpad.net/bugs/1799892
12:21:52 openstackgerrit Tetsuro Nakamura proposed openstack/nova master: Consider root id is None in the database case https://review.openstack.org/613305
12:34:54 cdent mnaser gets the gold star award for finding good bugs
12:48:18 openstackgerrit Jan Gutter proposed openstack/os-vif master: Extend port profiles with datapath offload type https://review.openstack.org/572081
13:15:08 openstackgerrit Adam Spiers proposed openstack/nova-specs master: Add spec for libvirt driver launching AMD SEV-encrypted instances https://review.openstack.org/609779
13:22:54 openstackgerrit Merged openstack/nova master: Remove the CachingScheduler https://review.openstack.org/611723
13:24:51 spatel sean-k-mooney: morning
13:25:13 spatel I did debug and here is the source machine log http://paste.openstack.org/show/733042/
13:25:13 sean-k-mooney spatel: hi o/
13:26:45 spatel There are some ERROR
13:27:09 spatel something like this - MessagingTimeout: Timed out waiting for a reply to message ID 80fcdfa3d90d4c128284822024982706
13:27:19 sean-k-mooney the messaging timeout is strange but may just meean the compute node was not back up after your config change
13:27:49 spatel Let me tell you what happened...
13:27:53 spatel 1. build foo instance
13:28:06 spatel 2. did block migration - and it failed with error
13:28:28 spatel 3. tried again and succeeded but vm failed to boot on destination compute node
13:29:23 spatel if you see my log i have break them in two section first one is for first attempt and second block is retried
13:29:55 sean-k-mooney right
13:30:20 sean-k-mooney the second migration looks ok bu i take it you had the same truncated disk on the dest
13:30:44 spatel yes it had only 1MB disk on destination, it didn't copy full disk
13:31:24 spatel are you seeing anything interesting in logs?
13:32:01 sean-k-mooney no it looks relitivly normal
13:32:08 spatel hmm!
13:32:43 spatel Do you need this line in - live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE
13:33:10 spatel I have meeting i will be back in few mi n
13:33:11 spatel min
13:34:17 sean-k-mooney spatel: i think we removed those flags a few releases go from the config
13:35:06 sean-k-mooney mdbooth: are you aware of any bugs related to block_migration with raw images
13:35:39 mdbooth sean-k-mooney: Can you be more specific?
13:36:00 sean-k-mooney mdbooth: the migration is completing but the disk image is truncated on the destination
13:36:11 mdbooth Oh...
13:36:26 sean-k-mooney mdbooth: then on reboot the vm is borked because the disk is curropt
13:39:15 mdbooth block migration == cold migration, right?
13:39:35 sean-k-mooney mdbooth: no live migration with block migrations of disk
13:39:46 mdbooth ... and the guest is fine until reboot?
13:39:52 sean-k-mooney e.g. no shared sotrage and not boot form volume
13:40:15 sean-k-mooney mdbooth: yes the guest is alive after live migration and then if you reboot it it erros out
13:40:19 mdbooth That's improbable.
13:40:30 mdbooth There's something going on there.
13:40:36 mdbooth something else...
13:40:38 sean-k-mooney mdbooth: fine is proably an over statement the kernel is running from ram at that point
13:40:56 sean-k-mooney if you try to use the disk its proably going to break
13:41:07 mdbooth ENOTENOUGHINFO
13:48:34 openstackgerrit Balazs Gibizer proposed openstack/nova master: Reject forced move with nested source allocation https://review.openstack.org/605785
13:50:13 openstackgerrit Balazs Gibizer proposed openstack/nova master: Run negative server moving tests with nested RPs https://review.openstack.org/604125
13:50:45 sean-k-mooney mdbooth: by the way looking at http://paste.openstack.org/show/733042/ is it valid to use imagecahing with raw images?
13:53:13 mdbooth sean-k-mooney: I don't know what I'm looking at there, seems truncated. However yes, imagecache is always used. I rewrote all that stuff to make it a whole lot more obvious a couple of years ago, btw, but it didn't land.
13:53:15 openstackgerrit Balazs Gibizer proposed openstack/nova master: Handle allocations consuming only from the child RPs https://review.openstack.org/608298
13:55:00 sean-k-mooney mdbooth: ok you looking at part of a systemd log for the n-cpus i think. its an openstack ansible deployment so not really sure how they do logging
13:55:21 openstackgerrit Merged openstack/nova master: Drop legacy cold migrate allocation compat code https://review.openstack.org/611970
13:55:41 sean-k-mooney mdbooth: anywya what it feels like is only the delta disk was copied insteda of the full disk
13:56:04 mdbooth sean-k-mooney: There is no delta disk for raw.
13:56:15 sean-k-mooney when spatel requires i might suggest moving to qcow2 for not raw
13:56:25 mdbooth sean-k-mooney: Is it possible that they've got qcow2 on the source and flat on the dest?
13:56:36 mdbooth Because if so then I'm pretty sure that would break.
13:57:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add request_spec.RequestGroup versioned object https://review.openstack.org/568840
13:57:04 sean-k-mooney am maybe i would be very surpried but that could explain things
13:57:24 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add requested_resources field to RequestSpec https://review.openstack.org/567267
13:57:29 mdbooth The reason being we explicitly only copy the delta, as we copy the backing file explicitly oob.
13:57:53 mdbooth But if the dest was raw/flat there would be no backing file.
13:57:58 openstackgerrit Balazs Gibizer proposed openstack/nova master: Add bandwidth related standard resource classes https://review.openstack.org/570847
13:57:59 sean-k-mooney mdbooth: ill ask for more info when spatel is back on line. they were going to open a bug for this anyway

Earlier   Later