Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-24
18:31:09 sean-k-mooney cfriesen: would a error log message generate by one of the periodic task on the compute agent not be more useful?
18:31:40 sean-k-mooney e.g. dicoverd instance x with pinning y expect z
18:32:57 openstackgerrit sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034
18:34:24 sean-k-mooney melwitt: i updated the placement wsgi patch again based on more talks with mdbooth and cdent earilier
18:34:41 sean-k-mooney melwitt: do you still want me to drop the second unit test https://review.openstack.org/#/c/610034/5/nova/tests/unit/api/openstack/placement/test_db_api.py
18:34:52 sean-k-mooney if so i can respin it quickly
18:37:07 cfriesen sean-k-mooney: an error log like that is not a bad idea, actually.
18:38:05 sean-k-mooney im kindof assuming any resonable size cloud that is going to have this problem is liekly exporting there logs to elastic serach and or similar and could set up an alert for it
18:39:32 melwitt sean-k-mooney: commented
18:41:10 sean-k-mooney melwitt: thanks
18:46:33 cfriesen sean-k-mooney: actually, I was wrong. we do have the actual guest CPU topology in the InstanceNUMACell, so we could display it too.
18:47:02 sean-k-mooney in the instance request spec im assuming
18:47:24 sean-k-mooney or somewhare in the instance extra stuff in the db
18:47:38 cfriesen no, InstanceNUMACell.cpu_topology
18:48:07 sean-k-mooney does that actully give you the cpu_topology or the vcpu to pcpu mappings
18:48:36 sean-k-mooney i have learned that we are terrible at naming anything related to numa in the code
18:50:27 cfriesen topology...threads/cores/sockets
18:51:10 cfriesen there's also InstanceNUMACell.siblings to show guest HT siblings
18:54:04 sean-k-mooney cfriesen: cool. i still think you need to convice dansmith and jaypipes there is a need for it. the periodic task i think would have value and be a easir sell as it will activly detect there is an issue you should investagate
19:23:38 openstackgerrit Merged openstack/nova stable/rocky: Move live_migration.pre.start to the start of the method https://review.openstack.org/612714
19:23:44 openstackgerrit Merged openstack/nova stable/rocky: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612715
19:28:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126
19:28:20 openstack bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem)
19:28:20 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115
20:14:34 openstackgerrit Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836
20:25:16 openstackgerrit Merged openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562
20:58:28 openstackgerrit Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663
20:58:28 openstackgerrit Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665
20:58:29 openstackgerrit Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934
20:58:29 openstackgerrit Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785
20:58:30 openstackgerrit Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829
20:58:30 openstackgerrit Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658
21:22:56 openstackgerrit Merged openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999
21:23:13 cfriesen so are we recommending setting send_service_user_token to True now? would we ever change the default to be True?
21:35:21 openstackgerrit Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420
21:41:30 openstackgerrit Merged openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773
21:41:36 openstackgerrit Merged openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774
21:44:51 spatel sean-k-mooney: hey
21:45:07 spatel I am having issue with block migration
21:45:32 spatel it migrate full instance but didn't copy full disk.img file and my VM is failed to boot
21:45:33 sean-k-mooney are you using config drive
21:45:38 spatel no
21:46:00 sean-k-mooney ok config drive breaks that i think
21:46:25 spatel http://paste.openstack.org/show/732988/ This is what i have in nova.conf
21:46:25 sean-k-mooney hum that is strange
21:46:55 sean-k-mooney so the migration suceeded form a nova api point of view but actully failed?
21:47:20 spatel Horizon migrating vm but when i reboot machine it put me in emergency mode of linux ( when i check disk.img file its just few KB in size)
21:48:16 spatel Migration succeeded and i can see my full VM migration to new compute node and its running also but as soon as i reboot it put me in emergency mode let me show you logs
21:48:30 sean-k-mooney so img images are usally raw images. for qcow images i know we have a disk layering/caching thing we use
21:49:07 spatel This is what instance look like after reboot http://paste.openstack.org/show/732989/
21:49:17 spatel raw image
21:49:22 sean-k-mooney im not sure if we do the same disk caching thing with raw images
21:49:44 spatel You think it could be image issue ?
21:50:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265
21:50:54 sean-k-mooney am no not nessicarlly but if it was using the same disk offest stuff we use for qcow the vm image would be small unless you wrote a lot of data to it after it booted
21:51:37 sean-k-mooney that said line 11 XFS (sda1): last sector read failed
21:51:51 spatel on source node size was 101MB
21:51:57 sean-k-mooney that looks like file system curruption
21:52:01 spatel and destination node its few KB
21:52:26 sean-k-mooney right am you not on shared storage by the way
21:52:52 spatel No i don't have shared storage
21:52:59 sean-k-mooney e.g. you dont have /var/libvirt/... on nfs
21:53:05 spatel no
21:53:06 sean-k-mooney ok cool just chekcing
21:53:11 spatel sure
21:53:21 sean-k-mooney do you have logs form nova for the migration
21:53:27 spatel let me pull
21:53:34 sean-k-mooney e.g. the n-cpu logs for sorce and dest
21:53:50 sean-k-mooney it seam like the image just got truncated but i dont know why that would happen
21:55:13 spatel http://paste.openstack.org/show/732990/
21:55:19 spatel this is destination compute node logs
21:57:00 spatel I am using swap disk it shouldn't be an issue
21:57:33 sean-k-mooney swap will be copied also so ya it shoudl be fine
21:57:48 openstackgerrit Vladyslav Drok proposed openstack/nova stable/pike: Fix resize_instance rpcapi call https://review.openstack.org/603439
21:58:07 spatel let me try again and see
21:59:01 sean-k-mooney this is a little suspicious Unknown base file: /var/lib/nova/instances/_base/68e4d13dacff5cffeaacecf533afab659ec3e170
22:00:11 spatel hmm!
22:00:21 spatel let me try again can capture fresh log
22:00:49 spatel i have spun up new vm and it has 98Mdisk file
22:00:57 openstackgerrit Matt Riedemann proposed openstack/nova master: Migrate old style volume attachments on nova-compute startup https://review.openstack.org/549130
22:01:52 spatel This is what i am doing for migration Horizon > live migration > select block migration
22:03:35 sean-k-mooney any luck reporducing?
22:03:53 sean-k-mooney by the way im assuming you are not using sriov interface on that vm right
22:04:00 spatel no SR-IOV
22:04:20 sean-k-mooney ok good because that does not wrok :)
22:04:25 sean-k-mooney at least not yet
22:04:31 spatel with in 30 second nova said migration completed and i can see host also updated in horizon
22:05:04 spatel here is the fresh log http://paste.openstack.org/show/732991/
22:05:12 spatel now going to reboot my instance
22:06:23 spatel failed to boot
22:06:29 spatel Entering emergency mode. Exit the shell to continue.
22:06:54 sean-k-mooney hum if you log in to host ostack-compute-27.v1v0x.net does its n-cpu log have any errors
22:06:56 spatel disk size is 3.3M
22:07:07 spatel n-cpu log?
22:07:12 spatel is that a log file?
22:07:19 sean-k-mooney nova compute agent
22:07:56 sean-k-mooney in the developement installer devstack its refered to as n-cpu for short
22:08:16 sean-k-mooney so same log but for source node
22:08:41 spatel This is the log of compute-27 http://paste.openstack.org/show/732992/
22:09:04 spatel no error there

Earlier   Later