| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-24 | |||
| 18:13:42 | sean-k-mooney | cfriesen: yes but we can also say 2 cpus per socket instead of max | |
| 18:13:50 | sean-k-mooney | anyway that is just a taught | |
| 18:14:37 | sean-k-mooney | it sound like dansmith and jaypipes would prefer this not to be in the api anyway so maybe you could do it as an osc or nova client feature | |
| 18:21:41 | cfriesen | dansmith: jaypipes: currently there's no way for an admin to look at the expected virt/phys mapping without going into the database. do we expect that nova admins will always have raw DB access? | |
| 18:22:11 | sean-k-mooney | they dont need db acess | |
| 18:22:29 | sean-k-mooney | they just need to be able to do a flaovr show and image show + look at teh libvirt xml | |
| 18:22:47 | cfriesen | sean-k-mooney: no, I'm talking about which specific guest vcpu maps to which specific host CPU | |
| 18:22:59 | sean-k-mooney | that in the libvirt xml | |
| 18:23:20 | cfriesen | sean-k-mooney: that's the hypervisor view, not nova's view (which in buggy cases can be different) | |
| 18:23:48 | sean-k-mooney | the database is not going to help you in does cases to debug what went wrong | |
| 18:24:13 | sean-k-mooney | we have no way to get the numa_toplogy blob from the moment when nova was caluating the pinning | |
| 18:24:22 | cfriesen | sure it can...if I can see that the entry in the database matched the previous mappings from before I did a migration... | |
| 18:25:25 | sean-k-mooney | wait your talking about migrtiaon with cpu pinning wich today is not supported | |
| 18:25:31 | cfriesen | cold migration is | |
| 18:25:48 | sean-k-mooney | cold migration yes but that wont break in this case | |
| 18:26:07 | sean-k-mooney | the xml will be regenerated on the new host | |
| 18:26:38 | cfriesen | throw in power outages and downed compute nodes and lost messages and migration reverts | |
| 18:27:31 | sean-k-mooney | so when the db is in an undefiend state it may not agree with the hypervior | |
| 18:27:50 | sean-k-mooney | yes that is true. not sure this will help with htat | |
| 18:28:13 | cfriesen | sean-k-mooney: it'll at least tell us what the problem is | |
| 18:28:37 | sean-k-mooney | the problem being the db is borked | |
| 18:29:22 | sean-k-mooney | if the vm is running it means the hypervior pinned it correctly based on the info it had at the time. | |
| 18:30:02 | sean-k-mooney | it should never be the case that the db is correct and vm is wrong in the cold migrate case | |
| 18:30:15 | sean-k-mooney | live migrate this can invert | |
| 18:31:09 | sean-k-mooney | cfriesen: would a error log message generate by one of the periodic task on the compute agent not be more useful? | |
| 18:31:40 | sean-k-mooney | e.g. dicoverd instance x with pinning y expect z | |
| 18:32:57 | openstackgerrit | sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034 | |
| 18:34:24 | sean-k-mooney | melwitt: i updated the placement wsgi patch again based on more talks with mdbooth and cdent earilier | |
| 18:34:41 | sean-k-mooney | melwitt: do you still want me to drop the second unit test https://review.openstack.org/#/c/610034/5/nova/tests/unit/api/openstack/placement/test_db_api.py | |
| 18:34:52 | sean-k-mooney | if so i can respin it quickly | |
| 18:37:07 | cfriesen | sean-k-mooney: an error log like that is not a bad idea, actually. | |
| 18:38:05 | sean-k-mooney | im kindof assuming any resonable size cloud that is going to have this problem is liekly exporting there logs to elastic serach and or similar and could set up an alert for it | |
| 18:39:32 | melwitt | sean-k-mooney: commented | |
| 18:41:10 | sean-k-mooney | melwitt: thanks | |
| 18:46:33 | cfriesen | sean-k-mooney: actually, I was wrong. we do have the actual guest CPU topology in the InstanceNUMACell, so we could display it too. | |
| 18:47:02 | sean-k-mooney | in the instance request spec im assuming | |
| 18:47:24 | sean-k-mooney | or somewhare in the instance extra stuff in the db | |
| 18:47:38 | cfriesen | no, InstanceNUMACell.cpu_topology | |
| 18:48:07 | sean-k-mooney | does that actully give you the cpu_topology or the vcpu to pcpu mappings | |
| 18:48:36 | sean-k-mooney | i have learned that we are terrible at naming anything related to numa in the code | |
| 18:50:27 | cfriesen | topology...threads/cores/sockets | |
| 18:51:10 | cfriesen | there's also InstanceNUMACell.siblings to show guest HT siblings | |
| 18:54:04 | sean-k-mooney | cfriesen: cool. i still think you need to convice dansmith and jaypipes there is a need for it. the periodic task i think would have value and be a easir sell as it will activly detect there is an issue you should investagate | |
| 19:23:38 | openstackgerrit | Merged openstack/nova stable/rocky: Move live_migration.pre.start to the start of the method https://review.openstack.org/612714 | |
| 19:23:44 | openstackgerrit | Merged openstack/nova stable/rocky: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612715 | |
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115 | |
| 19:28:20 | openstack | bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | |
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126 | |
| 20:14:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836 | |
| 20:25:16 | openstackgerrit | Merged openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 21:22:56 | openstackgerrit | Merged openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999 | |
| 21:23:13 | cfriesen | so are we recommending setting send_service_user_token to True now? would we ever change the default to be True? | |
| 21:35:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 21:41:30 | openstackgerrit | Merged openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773 | |
| 21:41:36 | openstackgerrit | Merged openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774 | |
| 21:44:51 | spatel | sean-k-mooney: hey | |
| 21:45:07 | spatel | I am having issue with block migration | |
| 21:45:32 | spatel | it migrate full instance but didn't copy full disk.img file and my VM is failed to boot | |
| 21:45:33 | sean-k-mooney | are you using config drive | |
| 21:45:38 | spatel | no | |
| 21:46:00 | sean-k-mooney | ok config drive breaks that i think | |
| 21:46:25 | sean-k-mooney | hum that is strange | |
| 21:46:25 | spatel | http://paste.openstack.org/show/732988/ This is what i have in nova.conf | |
| 21:46:55 | sean-k-mooney | so the migration suceeded form a nova api point of view but actully failed? | |
| 21:47:20 | spatel | Horizon migrating vm but when i reboot machine it put me in emergency mode of linux ( when i check disk.img file its just few KB in size) | |
| 21:48:16 | spatel | Migration succeeded and i can see my full VM migration to new compute node and its running also but as soon as i reboot it put me in emergency mode let me show you logs | |
| 21:48:30 | sean-k-mooney | so img images are usally raw images. for qcow images i know we have a disk layering/caching thing we use | |
| 21:49:07 | spatel | This is what instance look like after reboot http://paste.openstack.org/show/732989/ | |
| 21:49:17 | spatel | raw image | |
| 21:49:22 | sean-k-mooney | im not sure if we do the same disk caching thing with raw images | |
| 21:49:44 | spatel | You think it could be image issue ? | |
| 21:50:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |
| 21:50:54 | sean-k-mooney | am no not nessicarlly but if it was using the same disk offest stuff we use for qcow the vm image would be small unless you wrote a lot of data to it after it booted | |
| 21:51:37 | sean-k-mooney | that said line 11 XFS (sda1): last sector read failed | |
| 21:51:51 | spatel | on source node size was 101MB | |
| 21:51:57 | sean-k-mooney | that looks like file system curruption | |
| 21:52:01 | spatel | and destination node its few KB | |
| 21:52:26 | sean-k-mooney | right am you not on shared storage by the way | |
| 21:52:52 | spatel | No i don't have shared storage | |
| 21:52:59 | sean-k-mooney | e.g. you dont have /var/libvirt/... on nfs | |
| 21:53:05 | spatel | no | |
| 21:53:06 | sean-k-mooney | ok cool just chekcing | |
| 21:53:11 | spatel | sure | |
| 21:53:21 | sean-k-mooney | do you have logs form nova for the migration | |
| 21:53:27 | spatel | let me pull | |
| 21:53:34 | sean-k-mooney | e.g. the n-cpu logs for sorce and dest | |
| 21:53:50 | sean-k-mooney | it seam like the image just got truncated but i dont know why that would happen | |
| 21:55:13 | spatel | http://paste.openstack.org/show/732990/ | |
| 21:55:19 | spatel | this is destination compute node logs | |
| 21:57:00 | spatel | I am using swap disk it shouldn't be an issue | |
| 21:57:33 | sean-k-mooney | swap will be copied also so ya it shoudl be fine | |
| 21:57:48 | openstackgerrit | Vladyslav Drok proposed openstack/nova stable/pike: Fix resize_instance rpcapi call https://review.openstack.org/603439 | |
| 21:58:07 | spatel | let me try again and see | |
| 21:59:01 | sean-k-mooney | this is a little suspicious Unknown base file: /var/lib/nova/instances/_base/68e4d13dacff5cffeaacecf533afab659ec3e170 | |