Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-24
18:47:02 sean-k-mooney in the instance request spec im assuming
18:47:24 sean-k-mooney or somewhare in the instance extra stuff in the db
18:47:38 cfriesen no, InstanceNUMACell.cpu_topology
18:48:07 sean-k-mooney does that actully give you the cpu_topology or the vcpu to pcpu mappings
18:48:36 sean-k-mooney i have learned that we are terrible at naming anything related to numa in the code
18:50:27 cfriesen topology...threads/cores/sockets
18:51:10 cfriesen there's also InstanceNUMACell.siblings to show guest HT siblings
18:54:04 sean-k-mooney cfriesen: cool. i still think you need to convice dansmith and jaypipes there is a need for it. the periodic task i think would have value and be a easir sell as it will activly detect there is an issue you should investagate
19:23:38 openstackgerrit Merged openstack/nova stable/rocky: Move live_migration.pre.start to the start of the method https://review.openstack.org/612714
19:23:44 openstackgerrit Merged openstack/nova stable/rocky: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612715
19:28:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115
19:28:20 openstack bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem)
19:28:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126
20:14:34 openstackgerrit Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836
20:25:16 openstackgerrit Merged openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562
20:58:28 openstackgerrit Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665
20:58:28 openstackgerrit Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663
20:58:29 openstackgerrit Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785
20:58:29 openstackgerrit Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934
20:58:30 openstackgerrit Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658
20:58:30 openstackgerrit Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829
21:22:56 openstackgerrit Merged openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999
21:23:13 cfriesen so are we recommending setting send_service_user_token to True now? would we ever change the default to be True?
21:35:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420
21:41:30 openstackgerrit Merged openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773
21:41:36 openstackgerrit Merged openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774
21:44:51 spatel sean-k-mooney: hey
21:45:07 spatel I am having issue with block migration
21:45:32 spatel it migrate full instance but didn't copy full disk.img file and my VM is failed to boot
21:45:33 sean-k-mooney are you using config drive
21:45:38 spatel no
21:46:00 sean-k-mooney ok config drive breaks that i think
21:46:25 sean-k-mooney hum that is strange
21:46:25 spatel http://paste.openstack.org/show/732988/ This is what i have in nova.conf
21:46:55 sean-k-mooney so the migration suceeded form a nova api point of view but actully failed?
21:47:20 spatel Horizon migrating vm but when i reboot machine it put me in emergency mode of linux ( when i check disk.img file its just few KB in size)
21:48:16 spatel Migration succeeded and i can see my full VM migration to new compute node and its running also but as soon as i reboot it put me in emergency mode let me show you logs
21:48:30 sean-k-mooney so img images are usally raw images. for qcow images i know we have a disk layering/caching thing we use
21:49:07 spatel This is what instance look like after reboot http://paste.openstack.org/show/732989/
21:49:17 spatel raw image
21:49:22 sean-k-mooney im not sure if we do the same disk caching thing with raw images
21:49:44 spatel You think it could be image issue ?
21:50:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265
21:50:54 sean-k-mooney am no not nessicarlly but if it was using the same disk offest stuff we use for qcow the vm image would be small unless you wrote a lot of data to it after it booted
21:51:37 sean-k-mooney that said line 11 XFS (sda1): last sector read failed
21:51:51 spatel on source node size was 101MB
21:51:57 sean-k-mooney that looks like file system curruption
21:52:01 spatel and destination node its few KB
21:52:26 sean-k-mooney right am you not on shared storage by the way
21:52:52 spatel No i don't have shared storage
21:52:59 sean-k-mooney e.g. you dont have /var/libvirt/... on nfs
21:53:05 spatel no
21:53:06 sean-k-mooney ok cool just chekcing
21:53:11 spatel sure
21:53:21 sean-k-mooney do you have logs form nova for the migration
21:53:27 spatel let me pull
21:53:34 sean-k-mooney e.g. the n-cpu logs for sorce and dest
21:53:50 sean-k-mooney it seam like the image just got truncated but i dont know why that would happen
21:55:13 spatel http://paste.openstack.org/show/732990/
21:55:19 spatel this is destination compute node logs
21:57:00 spatel I am using swap disk it shouldn't be an issue
21:57:33 sean-k-mooney swap will be copied also so ya it shoudl be fine
21:57:48 openstackgerrit Vladyslav Drok proposed openstack/nova stable/pike: Fix resize_instance rpcapi call https://review.openstack.org/603439
21:58:07 spatel let me try again and see
21:59:01 sean-k-mooney this is a little suspicious Unknown base file: /var/lib/nova/instances/_base/68e4d13dacff5cffeaacecf533afab659ec3e170
22:00:11 spatel hmm!
22:00:21 spatel let me try again can capture fresh log
22:00:49 spatel i have spun up new vm and it has 98Mdisk file
22:00:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Migrate old style volume attachments on nova-compute startup https://review.openstack.org/549130
22:01:52 spatel This is what i am doing for migration Horizon > live migration > select block migration
22:03:35 sean-k-mooney any luck reporducing?
22:03:53 sean-k-mooney by the way im assuming you are not using sriov interface on that vm right
22:04:00 spatel no SR-IOV
22:04:20 sean-k-mooney ok good because that does not wrok :)
22:04:25 sean-k-mooney at least not yet
22:04:31 spatel with in 30 second nova said migration completed and i can see host also updated in horizon
22:05:04 spatel here is the fresh log http://paste.openstack.org/show/732991/
22:05:12 spatel now going to reboot my instance
22:06:23 spatel failed to boot
22:06:29 spatel Entering emergency mode. Exit the shell to continue.
22:06:54 sean-k-mooney hum if you log in to host ostack-compute-27.v1v0x.net does its n-cpu log have any errors
22:06:56 spatel disk size is 3.3M
22:07:07 spatel n-cpu log?
22:07:12 spatel is that a log file?
22:07:19 sean-k-mooney nova compute agent
22:07:56 sean-k-mooney in the developement installer devstack its refered to as n-cpu for short
22:08:16 sean-k-mooney so same log but for source node
22:08:41 spatel This is the log of compute-27 http://paste.openstack.org/show/732992/
22:09:04 spatel no error there
22:09:27 spatel what is this ? [instance: 2ad8cdf5-4db7-4024-a35f-343340ad27ee] Instance not resizing, skipping migration.
22:11:06 sean-k-mooney we use the same code path for migration and resizing vms since it basically the same thing
22:11:31 spatel hmm
22:12:45 spatel Do you think my nova config is wrong?
22:12:45 sean-k-mooney its being logged from here https://github.com/openstack/nova/blob/12fcfc5e2b51b563529a5bc0b2990816bbbda80b/nova/compute/resource_tracker.py#L1127
22:13:16 sean-k-mooney no i think your nova config is fine that said we might need to enable dbug loggin to get to the bottom of what is going on
22:13:47 spatel So i have manually copy disk file from source to destination and VM is up
22:14:33 spatel look like compute node not not copying full disk file or source just deleting instance before copy finish
22:14:36 openstackgerrit Merged openstack/nova master: Add debug logs for when provider inventory changes https://review.openstack.org/597560
22:14:44 sean-k-mooney ya that should not be required
22:14:57 sean-k-mooney ya can you enable debug logs on both nodes and try again

Earlier   Later