| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-24 | |||
| 17:46:36 | cfriesen | or at least might not be able to | |
| 17:46:40 | dansmith | cfriesen: right, it's that case I don't get | |
| 17:47:07 | dansmith | cfriesen: like, if I can log into the guest, what does it matter what the topology is? | |
| 17:47:39 | dansmith | I mean I can come up with completely synthetic reasons, but they're exceedingly weak, which is what I said above | |
| 17:49:19 | dansmith | sorry.. "can't log into the guest" | |
| 17:54:18 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Update allocation_ratios in placement inventory if config changes https://review.openstack.org/613126 | |
| 17:55:33 | cfriesen | most of the uses that we added it for were admin-level, admittedly. Like figuring out why something is having a hard time scheduling on a migration or showing the expected virt/phys mapping. For an end user it's really just about showing all the available information about the instance and making it so they don't have to jump through hoops to get it. | |
| 17:56:30 | cdent | "making it so they don't have to jump through hoops to get it" <- that ought to be compelling enough? | |
| 18:00:27 | openstackgerrit | Gaudenz Steinlin proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | |
| 18:01:34 | sean-k-mooney | cdent: its 3 command to 1 but if some clouds hide flavor extra spec then i guess maybe they cant run the 3 commands | |
| 18:01:55 | sean-k-mooney | cfriesen: had you also planned to show cpu topplogy or just numa topology | |
| 18:02:51 | sean-k-mooney | cfriesen: as we both know they can and usually are very different by defualt on openstack | |
| 18:03:16 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove unused code https://review.openstack.org/613127 | |
| 18:03:17 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove unused Quota driver methods https://review.openstack.org/613128 | |
| 18:03:17 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove QuotaDriver.destroy_all_by_project() https://review.openstack.org/613129 | |
| 18:03:18 | openstackgerrit | Jay Pipes proposed openstack/nova master: quota: remove default kwarg on get_class_quotas() https://review.openstack.org/613130 | |
| 18:04:29 | jaypipes | melwitt, dansmith: some cleanups of the quota system ^^ | |
| 18:05:28 | cfriesen | sean-k-mooney: we currently show memory size, page size, and which guest CPUs are associated with each guest numa node. showing virtual CPU topology (sockets/cores/threads) would be a lot trickier since that's not in the nova DB. | |
| 18:06:12 | sean-k-mooney | well it is if set in the flavor or image else it up to the virt driver | |
| 18:06:15 | jaypipes | melwitt, dansmith: more patches on the way but those are a good first chunk of removing cruft | |
| 18:06:50 | sean-k-mooney | cfriesen: so if we were going this route it would be nice to include it if set in image or flavor | |
| 18:07:36 | cfriesen | sean-k-mooney: don't we have scenarios where we say "max cpus per socket"? in that case only the virt driver knows the actual number | |
| 18:13:42 | sean-k-mooney | cfriesen: yes but we can also say 2 cpus per socket instead of max | |
| 18:13:50 | sean-k-mooney | anyway that is just a taught | |
| 18:14:37 | sean-k-mooney | it sound like dansmith and jaypipes would prefer this not to be in the api anyway so maybe you could do it as an osc or nova client feature | |
| 18:21:41 | cfriesen | dansmith: jaypipes: currently there's no way for an admin to look at the expected virt/phys mapping without going into the database. do we expect that nova admins will always have raw DB access? | |
| 18:22:11 | sean-k-mooney | they dont need db acess | |
| 18:22:29 | sean-k-mooney | they just need to be able to do a flaovr show and image show + look at teh libvirt xml | |
| 18:22:47 | cfriesen | sean-k-mooney: no, I'm talking about which specific guest vcpu maps to which specific host CPU | |
| 18:22:59 | sean-k-mooney | that in the libvirt xml | |
| 18:23:20 | cfriesen | sean-k-mooney: that's the hypervisor view, not nova's view (which in buggy cases can be different) | |
| 18:23:48 | sean-k-mooney | the database is not going to help you in does cases to debug what went wrong | |
| 18:24:13 | sean-k-mooney | we have no way to get the numa_toplogy blob from the moment when nova was caluating the pinning | |
| 18:24:22 | cfriesen | sure it can...if I can see that the entry in the database matched the previous mappings from before I did a migration... | |
| 18:25:25 | sean-k-mooney | wait your talking about migrtiaon with cpu pinning wich today is not supported | |
| 18:25:31 | cfriesen | cold migration is | |
| 18:25:48 | sean-k-mooney | cold migration yes but that wont break in this case | |
| 18:26:07 | sean-k-mooney | the xml will be regenerated on the new host | |
| 18:26:38 | cfriesen | throw in power outages and downed compute nodes and lost messages and migration reverts | |
| 18:27:31 | sean-k-mooney | so when the db is in an undefiend state it may not agree with the hypervior | |
| 18:27:50 | sean-k-mooney | yes that is true. not sure this will help with htat | |
| 18:28:13 | cfriesen | sean-k-mooney: it'll at least tell us what the problem is | |
| 18:28:37 | sean-k-mooney | the problem being the db is borked | |
| 18:29:22 | sean-k-mooney | if the vm is running it means the hypervior pinned it correctly based on the info it had at the time. | |
| 18:30:02 | sean-k-mooney | it should never be the case that the db is correct and vm is wrong in the cold migrate case | |
| 18:30:15 | sean-k-mooney | live migrate this can invert | |
| 18:31:09 | sean-k-mooney | cfriesen: would a error log message generate by one of the periodic task on the compute agent not be more useful? | |
| 18:31:40 | sean-k-mooney | e.g. dicoverd instance x with pinning y expect z | |
| 18:32:57 | openstackgerrit | sean mooney proposed openstack/nova master: harden placement init under wsgi https://review.openstack.org/610034 | |
| 18:34:24 | sean-k-mooney | melwitt: i updated the placement wsgi patch again based on more talks with mdbooth and cdent earilier | |
| 18:34:41 | sean-k-mooney | melwitt: do you still want me to drop the second unit test https://review.openstack.org/#/c/610034/5/nova/tests/unit/api/openstack/placement/test_db_api.py | |
| 18:34:52 | sean-k-mooney | if so i can respin it quickly | |
| 18:37:07 | cfriesen | sean-k-mooney: an error log like that is not a bad idea, actually. | |
| 18:38:05 | sean-k-mooney | im kindof assuming any resonable size cloud that is going to have this problem is liekly exporting there logs to elastic serach and or similar and could set up an alert for it | |
| 18:39:32 | melwitt | sean-k-mooney: commented | |
| 18:41:10 | sean-k-mooney | melwitt: thanks | |
| 18:46:33 | cfriesen | sean-k-mooney: actually, I was wrong. we do have the actual guest CPU topology in the InstanceNUMACell, so we could display it too. | |
| 18:47:02 | sean-k-mooney | in the instance request spec im assuming | |
| 18:47:24 | sean-k-mooney | or somewhare in the instance extra stuff in the db | |
| 18:47:38 | cfriesen | no, InstanceNUMACell.cpu_topology | |
| 18:48:07 | sean-k-mooney | does that actully give you the cpu_topology or the vcpu to pcpu mappings | |
| 18:48:36 | sean-k-mooney | i have learned that we are terrible at naming anything related to numa in the code | |
| 18:50:27 | cfriesen | topology...threads/cores/sockets | |
| 18:51:10 | cfriesen | there's also InstanceNUMACell.siblings to show guest HT siblings | |
| 18:54:04 | sean-k-mooney | cfriesen: cool. i still think you need to convice dansmith and jaypipes there is a need for it. the periodic task i think would have value and be a easir sell as it will activly detect there is an issue you should investagate | |
| 19:23:38 | openstackgerrit | Merged openstack/nova stable/rocky: Move live_migration.pre.start to the start of the method https://review.openstack.org/612714 | |
| 19:23:44 | openstackgerrit | Merged openstack/nova stable/rocky: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612715 | |
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1799727 https://review.openstack.org/613115 | |
| 19:28:20 | openstack | bug 1799727 in OpenStack Compute (nova) "CPU_Allocation_Ratio from nova.conf doesn't update exisiting providers" [High,In progress] https://launchpad.net/bugs/1799727 - Assigned to Matt Riedemann (mriedem) | |
| 19:28:20 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update reserved/allocation_ratio in placement inventory if config changes https://review.openstack.org/613126 | |
| 20:14:34 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add more documentation for online_data_migrations CLI https://review.openstack.org/605836 | |
| 20:25:16 | openstackgerrit | Merged openstack/nova stable/queens: Fix up compute rpcapi version for pike release https://review.openstack.org/612562 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Make CellDatabases fixture reentrant https://review.openstack.org/611665 | |
| 20:58:28 | openstackgerrit | Dan Smith proposed openstack/nova master: Modify get_by_cell_and_project() to get_not_deleted_by_cell_and_projects() https://review.openstack.org/607663 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Minimal construct plumbing for nova list when a cell is down https://review.openstack.org/567785 | |
| 20:58:29 | openstackgerrit | Dan Smith proposed openstack/nova master: Refactor scatter-gather utility to return exception objects https://review.openstack.org/607934 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | |
| 20:58:30 | openstackgerrit | Dan Smith proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | |
| 21:22:56 | openstackgerrit | Merged openstack/nova master: libvirt: fix disk_bus handling for root disk https://review.openstack.org/584999 | |
| 21:23:13 | cfriesen | so are we recommending setting send_service_user_token to True now? would we ever change the default to be True? | |
| 21:35:21 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.openstack.org/541420 | |
| 21:41:30 | openstackgerrit | Merged openstack/nova stable/queens: Move live_migration.pre.start to the start of the method https://review.openstack.org/612773 | |
| 21:41:36 | openstackgerrit | Merged openstack/nova stable/queens: Ensure attachment cleanup on failure in driver.pre_live_migration https://review.openstack.org/612774 | |
| 21:44:51 | spatel | sean-k-mooney: hey | |
| 21:45:07 | spatel | I am having issue with block migration | |
| 21:45:32 | spatel | it migrate full instance but didn't copy full disk.img file and my VM is failed to boot | |
| 21:45:33 | sean-k-mooney | are you using config drive | |
| 21:45:38 | spatel | no | |
| 21:46:00 | sean-k-mooney | ok config drive breaks that i think | |
| 21:46:25 | sean-k-mooney | hum that is strange | |
| 21:46:25 | spatel | http://paste.openstack.org/show/732988/ This is what i have in nova.conf | |
| 21:46:55 | sean-k-mooney | so the migration suceeded form a nova api point of view but actully failed? | |
| 21:47:20 | spatel | Horizon migrating vm but when i reboot machine it put me in emergency mode of linux ( when i check disk.img file its just few KB in size) | |
| 21:48:16 | spatel | Migration succeeded and i can see my full VM migration to new compute node and its running also but as soon as i reboot it put me in emergency mode let me show you logs | |
| 21:48:30 | sean-k-mooney | so img images are usally raw images. for qcow images i know we have a disk layering/caching thing we use | |
| 21:49:07 | spatel | This is what instance look like after reboot http://paste.openstack.org/show/732989/ | |
| 21:49:17 | spatel | raw image | |
| 21:49:22 | sean-k-mooney | im not sure if we do the same disk caching thing with raw images | |
| 21:49:44 | spatel | You think it could be image issue ? | |
| 21:50:39 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional test for AggregateMultiTenancyIsolation + migrate https://review.openstack.org/571265 | |