| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-24 | |||
| 18:38:05 | sean-k-mooney | im kindof assuming any resonable size cloud that is going to have this problem is liekly exporting there logs to elastic serach and or similar and could set up an alert for it | |
| 18:39:32 | melwitt | sean-k-mooney: commented | |
| 18:41:10 | sean-k-mooney | melwitt: thanks | |
| 18:46:33 | cfriesen | sean-k-mooney: actually, I was wrong. we do have the actual guest CPU topology in the InstanceNUMACell, so we could display it too. | |
| 18:47:02 | sean-k-mooney | in the instance request spec im assuming | |
| 18:47:24 | sean-k-mooney | or somewhare in the instance extra stuff in the db | |
| 18:47:38 | cfriesen | no, InstanceNUMACell.cpu_topology | |
| 18:48:07 | sean-k-mooney | does that actully give you the cpu_topology or the vcpu to pcpu mappings | |
| 18:48:36 | sean-k-mooney | i have learned that we are terrible at naming anything related to numa in the code | |
| 18:50:27 | cfriesen | topology...threads/cores/sockets | |
| 18:51:10 | cfriesen | there's also InstanceNUMACell.siblings to show guest HT siblings | |
| 18:54:04 | sean-k-mooney | cfriesen: cool. i still think you need to convice dansmith and jaypipes there is a need for it. the periodic task i think would have value and be a easir sell as it will activly detect there is an issue you should investagate | |
| 19:23:38 | openstackgerrit | Merged openstack/nova stable/rocky: Move live_migration.pre.start to the start of the method https://review.openstack.org/612714 | |
| 19:23:44 | openstackgerrit | Merged openstack/nova stable/rocky: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612715 | |
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126 | |
| 19:28:20 | openstack | bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | |
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115 | |
| 20:14:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836 | |
| 20:25:16 | openstackgerrit | Merged openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 21:22:56 | openstackgerrit | Merged openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999 | |
| 21:23:13 | cfriesen | so are we recommending setting send_service_user_token to True now? would we ever change the default to be True? | |
| 21:35:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 21:41:30 | openstackgerrit | Merged openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773 | |
| 21:41:36 | openstackgerrit | Merged openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774 | |
| 21:44:51 | spatel | sean-k-mooney: hey | |
| 21:45:07 | spatel | I am having issue with block migration | |
| 21:45:32 | spatel | it migrate full instance but didn't copy full disk.img file and my VM is failed to boot | |
| 21:45:33 | sean-k-mooney | are you using config drive | |
| 21:45:38 | spatel | no | |
| 21:46:00 | sean-k-mooney | ok config drive breaks that i think | |
| 21:46:25 | spatel | http://paste.openstack.org/show/732988/ This is what i have in nova.conf | |
| 21:46:25 | sean-k-mooney | hum that is strange | |
| 21:46:55 | sean-k-mooney | so the migration suceeded form a nova api point of view but actully failed? | |
| 21:47:20 | spatel | Horizon migrating vm but when i reboot machine it put me in emergency mode of linux ( when i check disk.img file its just few KB in size) | |
| 21:48:16 | spatel | Migration succeeded and i can see my full VM migration to new compute node and its running also but as soon as i reboot it put me in emergency mode let me show you logs | |
| 21:48:30 | sean-k-mooney | so img images are usally raw images. for qcow images i know we have a disk layering/caching thing we use | |
| 21:49:07 | spatel | This is what instance look like after reboot http://paste.openstack.org/show/732989/ | |
| 21:49:17 | spatel | raw image | |
| 21:49:22 | sean-k-mooney | im not sure if we do the same disk caching thing with raw images | |
| 21:49:44 | spatel | You think it could be image issue ? | |
| 21:50:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 21:50:54 | sean-k-mooney | am no not nessicarlly but if it was using the same disk offest stuff we use for qcow the vm image would be small unless you wrote a lot of data to it after it booted | |
| 21:51:37 | sean-k-mooney | that said line 11 XFS (sda1): last sector read failed | |
| 21:51:51 | spatel | on source node size was 101MB | |
| 21:51:57 | sean-k-mooney | that looks like file system curruption | |
| 21:52:01 | spatel | and destination node its few KB | |
| 21:52:26 | sean-k-mooney | right am you not on shared storage by the way | |
| 21:52:52 | spatel | No i don't have shared storage | |
| 21:52:59 | sean-k-mooney | e.g. you dont have /var/libvirt/... on nfs | |
| 21:53:05 | spatel | no | |
| 21:53:06 | sean-k-mooney | ok cool just chekcing | |
| 21:53:11 | spatel | sure | |
| 21:53:21 | sean-k-mooney | do you have logs form nova for the migration | |
| 21:53:27 | spatel | let me pull | |
| 21:53:34 | sean-k-mooney | e.g. the n-cpu logs for sorce and dest | |
| 21:53:50 | sean-k-mooney | it seam like the image just got truncated but i dont know why that would happen | |
| 21:55:13 | spatel | http://paste.openstack.org/show/732990/ | |
| 21:55:19 | spatel | this is destination compute node logs | |
| 21:57:00 | spatel | I am using swap disk it shouldn't be an issue | |
| 21:57:33 | sean-k-mooney | swap will be copied also so ya it shoudl be fine | |
| 21:57:48 | openstackgerrit | Vladyslav Drok proposed openstack/nova stable/pike: Fix resize_instance rpcapi call https://review.openstack.org/603439 | |
| 21:58:07 | spatel | let me try again and see | |
| 21:59:01 | sean-k-mooney | this is a little suspicious Unknown base file: /var/lib/nova/instances/_base/68e4d13dacff5cffeaacecf533afab659ec3e170 | |
| 22:00:11 | spatel | hmm! | |
| 22:00:21 | spatel | let me try again can capture fresh log | |
| 22:00:49 | spatel | i have spun up new vm and it has 98Mdisk file | |
| 22:00:57 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Migrate old style volume attachments on nova-compute startup https://review.openstack.org/549130 | |
| 22:01:52 | spatel | This is what i am doing for migration Horizon > live migration > select block migration | |
| 22:03:35 | sean-k-mooney | any luck reporducing? | |
| 22:03:53 | sean-k-mooney | by the way im assuming you are not using sriov interface on that vm right | |
| 22:04:00 | spatel | no SR-IOV | |
| 22:04:20 | sean-k-mooney | ok good because that does not wrok :) | |
| 22:04:25 | sean-k-mooney | at least not yet | |
| 22:04:31 | spatel | with in 30 second nova said migration completed and i can see host also updated in horizon | |
| 22:05:04 | spatel | here is the fresh log http://paste.openstack.org/show/732991/ | |
| 22:05:12 | spatel | now going to reboot my instance | |
| 22:06:23 | spatel | failed to boot | |
| 22:06:29 | spatel | Entering emergency mode. Exit the shell to continue. | |
| 22:06:54 | sean-k-mooney | hum if you log in to host ostack-compute-27.v1v0x.net does its n-cpu log have any errors | |
| 22:06:56 | spatel | disk size is 3.3M | |
| 22:07:07 | spatel | n-cpu log? | |
| 22:07:12 | spatel | is that a log file? | |
| 22:07:19 | sean-k-mooney | nova compute agent | |
| 22:07:56 | sean-k-mooney | in the developement installer devstack its refered to as n-cpu for short | |
| 22:08:16 | sean-k-mooney | so same log but for source node | |
| 22:08:41 | spatel | This is the log of compute-27 http://paste.openstack.org/show/732992/ | |
| 22:09:04 | spatel | no error there | |
| 22:09:27 | spatel | what is this ? [instance: 2ad8cdf5-4db7-4024-a35f-343340ad27ee] Instance not resizing, skipping migration. | |
| 22:11:06 | sean-k-mooney | we use the same code path for migration and resizing vms since it basically the same thing | |
| 22:11:31 | spatel | hmm | |
| 22:12:45 | sean-k-mooney | its being logged from here https://github.com/openstack/nova/blob/12fcfc5e2b51b563529a5bc0b2990816bbbda80b/nova/compute/resource_tracker.py#L1127 | |
| 22:12:45 | spatel | Do you think my nova config is wrong? | |
| 22:13:16 | sean-k-mooney | no i think your nova config is fine that said we might need to enable dbug loggin to get to the bottom of what is going on | |
| 22:13:47 | spatel | So i have manually copy disk file from source to destination and VM is up | |