| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-24 | |||
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126 | |
| 19:28:20 | openstack | bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | |
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115 | |
| 20:14:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836 | |
| 20:25:16 | openstackgerrit | Merged openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 21:22:56 | openstackgerrit | Merged openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999 | |
| 21:23:13 | cfriesen | so are we recommending setting send_service_user_token to True now? would we ever change the default to be True? | |
| 21:35:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 21:41:30 | openstackgerrit | Merged openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773 | |
| 21:41:36 | openstackgerrit | Merged openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774 | |
| 21:44:51 | spatel | sean-k-mooney: hey | |
| 21:45:07 | spatel | I am having issue with block migration | |
| 21:45:32 | spatel | it migrate full instance but didn't copy full disk.img file and my VM is failed to boot | |
| 21:45:33 | sean-k-mooney | are you using config drive | |
| 21:45:38 | spatel | no | |
| 21:46:00 | sean-k-mooney | ok config drive breaks that i think | |
| 21:46:25 | spatel | http://paste.openstack.org/show/732988/ This is what i have in nova.conf | |
| 21:46:25 | sean-k-mooney | hum that is strange | |
| 21:46:55 | sean-k-mooney | so the migration suceeded form a nova api point of view but actully failed? | |
| 21:47:20 | spatel | Horizon migrating vm but when i reboot machine it put me in emergency mode of linux ( when i check disk.img file its just few KB in size) | |
| 21:48:16 | spatel | Migration succeeded and i can see my full VM migration to new compute node and its running also but as soon as i reboot it put me in emergency mode let me show you logs | |
| 21:48:30 | sean-k-mooney | so img images are usally raw images. for qcow images i know we have a disk layering/caching thing we use | |
| 21:49:07 | spatel | This is what instance look like after reboot http://paste.openstack.org/show/732989/ | |
| 21:49:17 | spatel | raw image | |
| 21:49:22 | sean-k-mooney | im not sure if we do the same disk caching thing with raw images | |
| 21:49:44 | spatel | You think it could be image issue ? | |
| 21:50:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 21:50:54 | sean-k-mooney | am no not nessicarlly but if it was using the same disk offest stuff we use for qcow the vm image would be small unless you wrote a lot of data to it after it booted | |
| 21:51:37 | sean-k-mooney | that said line 11 XFS (sda1): last sector read failed | |
| 21:51:51 | spatel | on source node size was 101MB | |
| 21:51:57 | sean-k-mooney | that looks like file system curruption | |
| 21:52:01 | spatel | and destination node its few KB | |
| 21:52:26 | sean-k-mooney | right am you not on shared storage by the way | |
| 21:52:52 | spatel | No i don't have shared storage | |
| 21:52:59 | sean-k-mooney | e.g. you dont have /var/libvirt/... on nfs | |
| 21:53:05 | spatel | no | |
| 21:53:06 | sean-k-mooney | ok cool just chekcing | |
| 21:53:11 | spatel | sure | |
| 21:53:21 | sean-k-mooney | do you have logs form nova for the migration | |
| 21:53:27 | spatel | let me pull | |
| 21:53:34 | sean-k-mooney | e.g. the n-cpu logs for sorce and dest | |
| 21:53:50 | sean-k-mooney | it seam like the image just got truncated but i dont know why that would happen | |
| 21:55:13 | spatel | http://paste.openstack.org/show/732990/ | |
| 21:55:19 | spatel | this is destination compute node logs | |
| 21:57:00 | spatel | I am using swap disk it shouldn't be an issue | |
| 21:57:33 | sean-k-mooney | swap will be copied also so ya it shoudl be fine | |
| 21:57:48 | openstackgerrit | Vladyslav Drok proposed openstack/nova stable/pike: Fix resize_instance rpcapi call https://review.openstack.org/603439 | |
| 21:58:07 | spatel | let me try again and see | |
| 21:59:01 | sean-k-mooney | this is a little suspicious Unknown base file: /var/lib/nova/instances/_base/68e4d13dacff5cffeaacecf533afab659ec3e170 | |
| 22:00:11 | spatel | hmm! | |
| 22:00:21 | spatel | let me try again can capture fresh log | |
| 22:00:49 | spatel | i have spun up new vm and it has 98Mdisk file | |
| 22:00:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Migrate old style volume attachments on nova-compute startup https://review.openstack.org/549130 | |
| 22:01:52 | spatel | This is what i am doing for migration Horizon > live migration > select block migration | |
| 22:03:35 | sean-k-mooney | any luck reporducing? | |
| 22:03:53 | sean-k-mooney | by the way im assuming you are not using sriov interface on that vm right | |
| 22:04:00 | spatel | no SR-IOV | |
| 22:04:20 | sean-k-mooney | ok good because that does not wrok :) | |
| 22:04:25 | sean-k-mooney | at least not yet | |
| 22:04:31 | spatel | with in 30 second nova said migration completed and i can see host also updated in horizon | |
| 22:05:04 | spatel | here is the fresh log http://paste.openstack.org/show/732991/ | |
| 22:05:12 | spatel | now going to reboot my instance | |
| 22:06:23 | spatel | failed to boot | |
| 22:06:29 | spatel | Entering emergency mode. Exit the shell to continue. | |
| 22:06:54 | sean-k-mooney | hum if you log in to host ostack-compute-27.v1v0x.net does its n-cpu log have any errors | |
| 22:06:56 | spatel | disk size is 3.3M | |
| 22:07:07 | spatel | n-cpu log? | |
| 22:07:12 | spatel | is that a log file? | |
| 22:07:19 | sean-k-mooney | nova compute agent | |
| 22:07:56 | sean-k-mooney | in the developement installer devstack its refered to as n-cpu for short | |
| 22:08:16 | sean-k-mooney | so same log but for source node | |
| 22:08:41 | spatel | This is the log of compute-27 http://paste.openstack.org/show/732992/ | |
| 22:09:04 | spatel | no error there | |
| 22:09:27 | spatel | what is this ? [instance: 2ad8cdf5-4db7-4024-a35f-343340ad27ee] Instance not resizing, skipping migration. | |
| 22:11:06 | sean-k-mooney | we use the same code path for migration and resizing vms since it basically the same thing | |
| 22:11:31 | spatel | hmm | |
| 22:12:45 | sean-k-mooney | its being logged from here https://github.com/openstack/nova/blob/12fcfc5e2b51b563529a5bc0b2990816bbbda80b/nova/compute/resource_tracker.py#L1127 | |
| 22:12:45 | spatel | Do you think my nova config is wrong? | |
| 22:13:16 | sean-k-mooney | no i think your nova config is fine that said we might need to enable dbug loggin to get to the bottom of what is going on | |
| 22:13:47 | spatel | So i have manually copy disk file from source to destination and VM is up | |
| 22:14:33 | spatel | look like compute node not not copying full disk file or source just deleting instance before copy finish | |
| 22:14:36 | openstackgerrit | Merged openstack/nova master: Add debug logs for when provider inventory changes https://review.openstack.org/597560 | |
| 22:14:44 | sean-k-mooney | ya that should not be required | |
| 22:14:57 | sean-k-mooney | ya can you enable debug logs on both nodes and try again | |
| 22:15:13 | spatel | Let me go home and collect all nova debug and file BUG | |
| 22:15:40 | spatel | it seems source just deleting instance before nova migration complete... | |
| 22:16:18 | sean-k-mooney | spatel: sure its late here too so i was going to be droping ofline soon once my pizzia is readdy | |
| 22:16:49 | sean-k-mooney | if you can file a bug and attach some debug logs for the source and destination we can see why it is not copying the disk | |
| 22:17:45 | spatel | i will | |
| 22:17:48 | spatel | thanks! | |
| 22:18:06 | sean-k-mooney | spatel: by the way what release are you running | |
| 22:18:36 | spatel | queens | |
| 22:18:50 | sean-k-mooney | there was a error we caught just as Rocky was shipping where we treated migration compution as a success without checking if there was an error | |
| 22:18:59 | sean-k-mooney | ah so ya you might be hitting that bug | |